Moción de censura ~ Palabra de Abascal vs la de Casado ~ Data shy ends

La moción de censura nos da la oportunidad de re-usar

Dos políticos han sido protagonistas de la moción de censura que se ha desarrollado recientemente en el Congreso de los Diputados: Santiago Abascal y Pablo Casado. Ambos han destacado por sus discursos en sede parlamentaria, por motivos que no son el objeto de este post.

El objetivo de este post no es otro que utilizar varios conceptos introducidos en post anteriores para analizar sus discursos desde el punto de vista de técnicas básicas dentro del Procesamiento del Lenguaje Natural.

Ficheros de los discursos

Tanto VOX como el PP han publicado los ficheros PDF con el contenido de los discursos de moción de censura de sus líderes. El de Abascal lo podéis encontrar aquí, y el de Casado aquí.

Hemos utilizado una forma simple y que todo el mundo puede hacer: almacenar en un fichero de texto diferente (.TXT) cada discurso. Hay herramientas online que fácilmente convierten un fichero PDF o una página HTML en un fichero de texto. Con ese fichero PDF por discurso, el único trabajo «manual» que hemos hecho ha sido convertirlo a un fichero TXT. Y se acabó el trabajo manual: no hay que volver a tocar los ficheros nunca más, ni para limpiarlos de caracteres extraños. De eso ya se encargarán nuestros métodos.

Para todos aquellos que quieran hacer sus propios análisis con los ficheros de los discursos de cada político, aquí tenéis el link, una vez hemos transformado el PDF en un fichero de texto plano.

download

Pre-procesado y análisis

Tal y como hemos detallado en anteriores posts, lo primero es realizar un pre-procesado para poder disponer de nuestros datos (palabras). Hay muchas formas de pre-procesar la información para que se puedan utilizar las técnicas NLP. En nuestro caso, hemos leído los ficheros de los discursos, limpiado de caracteres extraños, divididos en palabras y eliminado las stop words. En este post lo explicamos en detalle.

Después, estos son los análisis que hemos realizado:

Cuenta de palabras totales vs palabras útiles, así como su distribución, tal y como hicimos en este post.
Bag of Words, tal y hicimos en este post. En este caso hemos identificado las 15 palabras más utilizadas y los 10 binomios de palabas más usados en cada discurso.
TF-IDF, tal y como hicimos en este post, identificando las 15 palabras con mayor score y los 10 binomios de palabras con mayor score de cada discurso.

Palabras totales vs. palabras útiles de los discursos

Haciendo un simple conteo de las palabras totales vs. palabras útiles del discurso de cada líder político, vemos que Abascal se extendió bastante más en su discurso con mucha diferencia (12563 frente a 4883 palabras). Recordemos que las palabras útiles son las que aportan significado y para quedarnos con ellas simplemente hemos eliminado las stop words de nuestro corpus que contiene los discursos. En nuestro análisis, hemos incluido como stops words, además de las típicas del castellano, «señor», «abascal», «casado», «partido», «popular», «vox», «señorías», «moción» y «censura».

En el gráfico de abajo podemos ver cuántas palabras contiene cada discurso, y cuántas de ellas son útiles.

Moción de censura: conteo de palabras por discurso

Distribución de palabras totales vs. palabras útiles de los discursos

Aunque en el anterior gráfico podíamos intuir cómo se repartía el porcentaje de palabras útiles respecto al total de palabras en el discurso de cada político, si lo visualizamos en forma de «donut», el resultado nos dice que el porcentaje es del 41,51% de palabras útiles de Abascal y el 37,25% de palabras útiles de Casado.

Moción de censura: distribución de palabras útiles por político

Bag of Words sobre los discursos

Recordando el concepto de Bag of Words, hemos asignado un identificador único a cada palabra distinta de cada discurso y contar las veces que se repite esa palabra a lo largo del discurso (corpus). Hay una característica importante de este método: no importa el orden en el que están dispuestas las palabras, lo único que se tiene en cuenta es el número de ocurrencias de cada palabra en cada documento del corpus, en este caso, cada discurso.

Las palabras más repetidas por Abascal fueron «España», «españoles» y «gobierno». Para Casado fueron «España», «Sánchez» y «gobierno».

Lo mismo, pero con binomios de palabras

Otro análisis interesante es, en vez de contabilizar el número de veces que se usan las palabras de forma individual, considerar un análisis de los binomios de palabras y ver qué sale.

En los siguientes gráficos vemos el resultado: Para Abascal, los binomios más utilizados son «millones personas» «Pedro Sánchez», «dieciocho sistemas» y «consenso progre». Para Casado, los binomios más utilizados son «bloque ruptura» y «izquierda derecha», «unida diversa» y «Unión europea».

TF-IDF de los discursos

Hacemos un pequeño repaso de lo que es TF-IDF: significa Term Frequency times Inverse Document Frequency (es decir, TF x IDF) y es una métrica o score que establece un valor sobre la importancia de una palabra («p») en un documento («p»). Aquí term es análogo a palabra.

Utilizando esta técnica, el discurso de moción de censura de Abascal tuvo como palabras más frecuentes «España», «españoles» y «gobierno». Para Casado, las palabras más frecuentes fueron «España», «Sánchez» y «gobierno».

Esta técnica funciona especialmente bien cuando hay varios documentos a comparar. Es decir, si hubiéramos tenido a nuestra disposición todos los discursos de cada político, nos habría dado mejores resultados.

Finalmente, nos binomios de palabras

El método TD-IDF también se puede aplicar sobre binomios de palabras, es decir, sobre todos los pares de palabras que aparecen de forma consecutiva en los discursos de Abascal y Casado.

Según la técnica TF-IDF, los binomios más utilizados por Abascal fueron «millones personas», «soberanía nacional» y «dieciocho sistemas». Para Casado fueron «bloque ruptura», «España izquierda», «izquierda derecha» y «unida izquierda».

Si te interesa el código, aquí lo tienes

import pandas as pd
import string

import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

import os
from os import listdir
from os.path import isfile, join

import spacy
from spacy.tokens import Doc

import re

import plotly
import plotly.graph_objects as go

import scattertext as st
from scattertext import CorpusFromPandas, produce_scattertext_explorer

from IPython.display import IFrame
from IPython.core.display import display, HTML

import nltk
from nltk.tokenize import RegexpTokenizer

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer

from nltk.corpus import stopwords
nltk.download('stopwords')
nltk.download('wordnet')

# Function to return a list of stop words to consider
def create_stop_words():
    # We create a stop word list
    stops = set(stopwords.words("spanish"))

    # We define individual numbers and letters as stop words
    all_letters_numbers = string.digits + string.ascii_letters

    stops = stops.union(["señor", "abascal", "casado", 
                         "partido", "popular", "vox", "señorías", "moción",
                         "censura"
                        ])  # add some stopwords
    
    stops = stops.union(list(all_letters_numbers))
    
    return stops

g_parties_dict = {
                  "Casado": "./Mocion/Casado/",
                  "Abascal": "./Mocion/Abascal/"
}

g_parties_color_dict = {
                        "Casado": ["#b3d9ff", "#0080ff", "interpolateBlues"],
                        "Abascal": ["#c2f0c2", "#33cc33", "interpolateGreens"]}

g_nlp = spacy.load('es_core_news_sm')
g_stop_words = create_stop_words() | g_nlp.Defaults.stop_words
g_not_category_name='Resto de partidos'
g_charts_root_dir = './Charts'

def list_dirs(directory):
    """Returns all directories in a given directory
    """
    return [f for f in pathlib.Path(directory).iterdir() if f.is_dir()]


def list_files(directory, ext):
    """Returns all files in a given directory
    """
    onlyfiles = [join(directory, f) for f in listdir(directory) if isfile(join(directory, f)) and f.endswith(ext)]
    return sorted(onlyfiles)

# Function to remove numbers and small words (1 or 2 letters) from a document
def num_and_short_word_preprocessor(tokens):
    # Regular expression for numbers
    no_numbers = re.sub('(\d)+', '', tokens.lower())
    
    # Regular expression for 1-letter and 2-letter words
    no_short_words = re.sub(r'\b\w{1,2}\b', '', no_numbers)
    
    return no_short_words

# Function to tokenize a document wihthout using stopwords
def custom_tokenizer(doc):
    word_tokenizer = RegexpTokenizer(r'\w+')
    tokens = word_tokenizer.tokenize(doc)

    return tokens

# Function to tokenize a document using stopwords
def custom_tokenizer_filtering_stopwords(doc):
    word_tokenizer = RegexpTokenizer(r'\w+')
    tokens = word_tokenizer.tokenize(doc)

    filtered_tokens = [token for token in tokens if token not in g_stop_words]
    
    return filtered_tokens

def create_Doc(row):
    doc = Doc(g_nlp.vocab, words=row)
    doc.is_parsed = True
    
    return doc

def create_party_programs_corpus_with_sections():

    all_party_program_section_name_l = []
    all_party_corpus_lc = []
    all_party_name = list()

    for party_name, party_program_location in g_parties_dict.items():
        # List of files and their corresponding names that make up the party program
        party_program_section_file_l = list_files(party_program_location, ".txt")
        party_program_section_name_l = list(map(lambda x: os.path.split(x)[1], party_program_section_file_l))

        # List that contains the program document corpus
        party_program_corpus = list()

        # Fill out the corpus
        for section_file_name in party_program_section_file_l:

            with open(section_file_name, 'r', encoding="utf8") as section_file:
                section_file_data = section_file.read().replace('\n', '. ')

            party_program_corpus.append(section_file_data)
            all_party_name.append(party_name)

        # Clean up sections
        party_corpus_lc = [section.lower() for section in party_program_corpus] # to lowercase
        party_corpus_lc = [re.sub("(\d)+", '', section) for section in party_corpus_lc] # no numbers
        party_corpus_lc = [re.sub("[€º”—“«»&gt;•‘’!""#$%&amp;'()*+,-./:;?@[\]^_`{|}~]+", ' ', 
                                  section) for section in party_corpus_lc] # no strange characters
        party_corpus_lc = [re.sub('\s\s+', ' ', section) for section in party_corpus_lc] # no multiple spaces
        party_corpus_lc = [re.sub(r'\b\w{1}\b', '', section) for section in party_corpus_lc]

        all_party_corpus_lc += party_corpus_lc

        # Cleanup section names
        all_party_program_section_name_l += party_program_section_name_l
        all_party_program_section_name_l = [re.sub(".txt", '', 
                                                section_name) for section_name in all_party_program_section_name_l]
        all_party_program_section_name_l = [re.sub("(\d)+[-]", '',
                                                section_name) for section_name in all_party_program_section_name_l]  
        
        
    token_raw_party_section = [custom_tokenizer(section) for section in all_party_corpus_lc]
    token_filtered_party_section = [custom_tokenizer_filtering_stopwords(section) for section in all_party_corpus_lc]
    
    # Create DataFrame
    party_corpus_df = pd.DataFrame(data = { 
                                    "Party": all_party_name,
                                    "Section_name": all_party_program_section_name_l,
                                    "Section_text": all_party_corpus_lc,
                                    "Tokenized_raw_section_text": token_raw_party_section, 
                                    "Tokenized_filtered_section_text": token_filtered_party_section, 
                                    })

    parsed = party_corpus_df.Tokenized_filtered_section_text.apply(create_Doc)

    party_corpus_df["Parsed_section_text"] = parsed

    return party_corpus_df

def get_word_breakdown(party_corpus_df):
    
    def get_num_tokens(values):
        flat_list = [item for sublist in values for item in sublist]
        return len(flat_list)


    total_words = party_corpus_df.groupby("Party").Tokenized_raw_section_text.agg(get_num_tokens)  
    useful_words = party_corpus_df.groupby("Party").Tokenized_filtered_section_text.agg(get_num_tokens)
    
    words_per_party = pd.DataFrame(total_words).assign(x=useful_words.values)
    words_per_party.reset_index(level=0, inplace=True)
    words_per_party.columns = ['Party', 'Total words', 'Useful words']
    
    return words_per_party

def create_wordcount_pie(words_per_party):

    import locale
    locale.setlocale(locale.LC_NUMERIC, 'es_ES')
    
    words_per_party = words_per_party.sort_values(by=['Total words'], ascending=False).reset_index(drop=True)

    legends_l = []
    for party in words_per_party['Party']:
        legends_l.append("{} - totales".format(party))
        legends_l.append("{} - útiles".format(party))

    sns.set(color_codes=True)
    sns.set_style("whitegrid")

    figure = plt.figure(figsize=(7, 7))
    ax_pie = figure.add_subplot(1, 1, 1)   

    startingRadius = 1.7 + (0.3* (len(words_per_party)-1))
    for index, row in words_per_party.iterrows():

        party_name = row["Party"]
        total_words = row["Total words"]
        useful_words = row["Useful words"]

        useless_words = row["Total words"] - row["Useful words"]

        party_colors = g_parties_color_dict[party_name]

        useful_words_pct = (useful_words / total_words) * 100
        
        localized_useful_words_pct = locale.format_string('%.2f', useful_words_pct, grouping=True)

        label_text = "{}: {}% útiles".format(party_name, localized_useful_words_pct)
        label_text_2 = "{}".format(total_words)

        remaining_pct = 100 - useful_words_pct

        donut_sizes = [remaining_pct, useful_words_pct]

        ax_pie.text(-0.48, startingRadius - 0.26, label_text, 
                 horizontalalignment='left', verticalalignment='center',
                    fontsize=12, fontweight='bold',color='black')

        ax_pie.text(0.1 - startingRadius, 0, useful_words, 
                 horizontalalignment='left', verticalalignment='center',
                    fontsize=12, fontweight='normal',color='black', rotation = 45)

        ax_pie.text(-0.3 + startingRadius, 0, total_words, 
                 horizontalalignment='left', verticalalignment='center',
                    fontsize=12, fontweight='normal',color='black', rotation = 45)

        donut = ax_pie.pie(donut_sizes, 
                radius = startingRadius, 
                startangle = 270, 
                colors=party_colors,
                labels = [total_words, useful_words],
                rotatelabels = False,
                counterclock = True,
                labeldistance = 0.95 - 0.025*index,
                wedgeprops = {"edgecolor": "white", 'linewidth': 1},
                textprops = dict(rotation_mode = 'default', 
                                 va='center', 
                                 ha='center', 
                                 #rotation=45,
                                 wrap=True,
                                 #position=(2,0),
                                 visible=False
                )
        )

        startingRadius-=0.80

    # create circle and place onto pie chart
    circle = ax_pie.add_patch(plt.Circle(xy=(0, 0), radius=0.35, facecolor='white'))

    horiz_offset = 0.8
    vert_offset = -0.30

    _ = ax_pie.legend(legends_l,
                 bbox_to_anchor=(horiz_offset, vert_offset),
                 ncol=2)

    _ = ax_pie.set_title('Distribución de palabras (útiles/totales) en el discurso', 
                         fontsize=18, x=0.5, y=1.35)

def create_wordcount_stacked_bar(words_per_party):
    words_per_party = words_per_party.sort_values(by=['Total words'], ascending=False).reset_index(drop=True)

    sns.set(color_codes=True)
    sns.set_style("whitegrid")

    figure = plt.figure(figsize=(15, 5))
    ax_1 = figure.add_subplot(1, 1, 1)  
    ax_1.set_ymargin(0.15)


    parties = words_per_party["Party"]

    total_words_colors = [g_parties_color_dict[key][0] for key in parties.values]
    useful_words_colors = [g_parties_color_dict[key][1] for key in parties.values]

    total_words_legends = ["{} - no útiles".format(key) for key in parties.values]
    useful_words_legends = ["{} - útiles".format(key) for key in parties.values]

    useless_words = words_per_party["Total words"] - words_per_party["Useful words"]


    pie_1 = ax_1.barh(parties, words_per_party["Total words"], 
             color=total_words_colors)

    pie_2 = ax_1.barh(parties, words_per_party["Useful words"], 
             color=useful_words_colors)  

    for i, v in enumerate(words_per_party["Total words"]):
        plt.text(v/1.5, i + .45, str(useless_words[i]), color=total_words_colors[i], fontweight='bold', fontsize=16)
        plt.text(v/1.0, i + .05, str(v), color='black', fontweight='normal', fontsize=13)

    for i, v in enumerate(words_per_party["Useful words"]):    
        plt.text(v/4, i + .45, str(v), color=useful_words_colors[i], fontweight='bold', fontsize=16)


    # we also need to switch the labels
    ax_1.set_xlabel('Número de palabras en el discurso', fontsize=18)  

    horiz_offset = 0.98
    vert_offset = 1


    _ = ax_1.legend(pie_1 + pie_2, total_words_legends + useful_words_legends, 
                    bbox_to_anchor=(horiz_offset, vert_offset),
                   ncol=2)



def merge_party_sections(party_corpus_df):
    corpus_groupby_party = party_corpus_df.groupby("Party", as_index=False)

    party_aggregated_sections = corpus_groupby_party["Section_text"].apply(lambda x: ' '.join(x))

    party_sections_df = pd.DataFrame(data = {"Section_text": party_aggregated_sections.values, 
                                             "Party": list(corpus_groupby_party["Party"].groups.keys())})   
    
    return party_sections_df

def create_corpora_metric(corpora, doc_names, metric, stop, ngram = 1, num_freq_words=25, num_freq_words_section = 5, 
                     debug = False): 
    
    if metric == "bow":
        vectorizer = CountVectorizer(stop_words = stop, ngram_range = (ngram, ngram), 
                                 preprocessor = num_and_short_word_preprocessor, tokenizer=custom_tokenizer)
    elif metric =="tf-idf":
        vectorizer = TfidfVectorizer(stop_words = stop, ngram_range = (ngram, ngram), 
                                 preprocessor = num_and_short_word_preprocessor, tokenizer=custom_tokenizer)
    else:
        raise Exception('Metric not supported')
    
    vec = vectorizer.fit(corpora)
    doc_term_mat = vec.transform(corpora)
    
    # START: Calculate the most frequent words per section (using word ID)
    freq_words_set = set()
    for section in range(len(corpora)):
        section_info = doc_term_mat[section,]
        
        freq_words_section = [(idx, section_info[0, idx]) for word, idx in vec.vocabulary_.items()]
        sorted_freq_words_section = sorted(freq_words_section, key = lambda x: x[1], reverse=True)
        
        top_freq_words_section = [word[0] for word in sorted_freq_words_section]
        
        freq_words_set.update(top_freq_words_section[0:num_freq_words_section])
        
    # END
     
    # START: Calculate the most frequest words overall (using word ID)
    all_sections_info = doc_term_mat.sum(axis=0)
    all_freq_words_section = [(idx, all_sections_info[0, idx]) for word, idx in vec.vocabulary_.items()]
    all_sorted_freq_words_section = sorted(all_freq_words_section, key = lambda x: x[1], reverse=True)
    
    all_top_freq_words_section = [word[0] for word in all_sorted_freq_words_section]
    freq_words_set.update(all_top_freq_words_section[0:num_freq_words])
    # END
        
    
    freq_words_key_l = list(freq_words_set) # list of most frequent word IDs
    reverse_dictionary = {v: k for k, v in vec.vocabulary_.items()} # dictionary ID: word
    freq_words_value_l = [reverse_dictionary[word_key] for word_key in freq_words_key_l] # list of most frequent words
    
    #DataFrame including the most frequent words per section and overall
    freq_words_df = pd.DataFrame((doc_term_mat[:,freq_words_key_l]).todense(), 
                                  columns=freq_words_value_l, 
                                  index=doc_names)
    
    return freq_words_df

def create_metric_dict_by_party(party_corpus_df, metric = "bow", 
                                ngram = 1, num_freq_words=10, num_freq_words_section = 5):
    
    parties = party_corpus_df.Party.unique()
    metric_by_party = dict()

    # METRIC (BOW or TF-IDF) for individual parties
    for party in parties:
        party_corpora = party_corpus_df[party_corpus_df.Party == party]

        party_section_texts = party_corpora.Section_text.tolist()
        party_section_names = party_corpora.Section_name.tolist()

        freq_words_df = create_corpora_metric(party_section_texts, 
                                                        party_section_names, 
                                                        metric,
                                                        g_stop_words, 
                                                        ngram = ngram, 
                                                        num_freq_words=num_freq_words, 
                                                        num_freq_words_section = num_freq_words_section)
        
        metric_by_party[party] = freq_words_df
    
    # METRIC (BOW or TF-IDF) for the overall corpora including all parties    
    party_sections_df = merge_party_sections(party_corpus_df)
    
    overall_section_texts = party_sections_df.Section_text.tolist()
    overall_section_names = party_sections_df.Party.tolist()
    
    overall_freq_words_df = create_corpora_metric(overall_section_texts, 
                                                        overall_section_names, 
                                                        metric,
                                                        g_stop_words, 
                                                        ngram = ngram, 
                                                        num_freq_words=num_freq_words, 
                                                        num_freq_words_section = num_freq_words_section)
    
    metric_by_party["All"] = overall_freq_words_df
        
    return metric_by_party

def normalize_word_df(word_df):
    norm_word_df = word_df.copy()
    total_per_party = norm_word_df.sum(axis=1)
    grand_total = sum(total_per_party)

    for party in norm_word_df.index:
        norm_word_df.loc[party] = (norm_word_df.loc[party] / total_per_party[party]) * 100
        
    return norm_word_df

def show_df_as_bubble(word_df,
                      num_words = 2,
                      normalize = False,
                      standard_sizeref = True,
                      title = 'Palabras con mayor score por partido', 
                      xtitle = 'Palabras', 
                      ytitle = 'Partido',
                      sub_dir = 'bow', 
                      file_name = 'bow_bubble',
                      to_file = False):
   
    if normalize:
        word_df = normalize_word_df(word_df)
        title = title + " (normalizado)"

    # Keep the top 'num_words' words
    word_df.reindex(word_df.sum().sort_values(ascending=False).index, axis=1)
    
    num_words = min(num_words, word_df.shape[1])
    word_df = word_df.iloc[:, :num_words]
    
    # Sort in alphabetical order
    word_df = word_df.reindex(sorted(word_df.columns), axis=1)

    # Get the word labels
    word_l = word_df.columns.values
    num_words = len(word_l)
    # List that contains every bubble row. Each element is a party.
    bubble_data = []

    xs = list(range(word_df.shape[1]))

    for num_row in range(len(word_df.values)):
        index_name = word_df.index[num_row]
        index_color = g_parties_color_dict[index_name][1]
        index_values = word_df.values[num_row]
        
        if normalize:
            sizeref = 0.01 * max(index_values)
            hover_text = ["{:.2f}%".format(index_value) for index_value in index_values]
        else:
            if standard_sizeref:
                sizeref = 1
            else:
                sizeref = 0.01 * max(index_values)
            hover_text = index_values
            
        
        section_bubble = go.Scatter(
            x = word_l, 
            y = [num_row + 1] * num_words,
            name = index_name,
            showlegend = True,
            mode = 'markers',
            hovertext = hover_text,
            hoverinfo = "x+text",
            marker_size = index_values,
            marker = dict(color = index_color, sizeref = sizeref),
        )
        bubble_data.append(section_bubble)

    fig = go.Figure(data=bubble_data)
    
    fig.update_layout(
        title = dict(text = title, xanchor = 'center', x = 0.5, font = dict(size=20)),
        yaxis = dict(
            tickmode = 'array',
            tickvals = list(range(1, num_words + 1)),
            ticktext = word_df.index
        ),
        legend = dict(itemsizing = 'constant')
    )
    
    fig.update_xaxes(tickangle=315, title_text=xtitle, title_font=dict(size=18))
    fig.update_yaxes(title_text=ytitle, title_font=dict(size=18))
    
    if to_file:
        complete_file_name = '{}/{}/{}.html'.format(g_charts_root_dir, sub_dir, file_name)
        plotly.offline.plot(fig, filename = complete_file_name, auto_open=False)
        
    fig.show()

# Read files
party_corpus_df = create_party_programs_corpus_with_sections()

# Word count analysis
words_per_party = get_word_breakdown(party_corpus_df)

create_wordcount_pie(words_per_party)
create_wordcount_stacked_bar(words_per_party)

# BOW analysis
bow_by_party = create_metric_dict_by_party(party_corpus_df, metric = "bow", ngram = 1, 
                                           num_freq_words=15, num_freq_words_section = 5)

two_gram_bow_by_party = create_metric_dict_by_party(party_corpus_df, metric = "bow", ngram = 2, 
                                                 num_freq_words=10, 
                                                 num_freq_words_section = 10)

all_party_words = bow_by_party['All']
all_party_2_words = two_gram_bow_by_party['All']

show_df_as_bubble(all_party_words, num_words = 15, normalize = False,
                  title = 'Palabras más usadas por político',ytitle = 'Político',
                  to_file = True, sub_dir = 'bow', file_name = 'bow_partidos')


show_df_as_bubble(all_party_2_words, num_words = 15, normalize = True,
                  title = 'Binomios de palabras por político', 
                  xtitle = 'Binomios de palabras', 
                  ytitle = 'Político',
                  to_file = True, sub_dir = 'bow', file_name = 'bow_2_partidos')

# TF-IDF analysis

td_idf_by_party = create_metric_dict_by_party(party_corpus_df, metric = "tf-idf", ngram = 1, 
                                           num_freq_words=15, num_freq_words_section = 15)

two_gram_tf_idf_by_party = create_metric_dict_by_party(party_corpus_df, metric = "tf-idf", ngram = 2, 
                                                 num_freq_words=10, 
                                                 num_freq_words_section = 10)

tdidf_all_party_words = td_idf_by_party['All']
tdidf_all_party_2_words = two_gram_tf_idf_by_party['All']

show_df_as_bubble(tdidf_all_party_words, num_words = 250, normalize = False, standard_sizeref = False,
                  title = 'Palabras con mayor score por político', ytitle = 'Político',
                  to_file = True, sub_dir = 'tfidf', file_name = 'tfidf_partidos')

show_df_as_bubble(tdidf_all_party_2_words, num_words = 250, normalize = False, standard_sizeref = False,
                  title = 'Binomios de palabras con mayor score por político', ytitle = 'Político',
                  to_file = True, sub_dir = 'tfidf', file_name = 'tfidf_2_partidos')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

import pandas as pd

import string

import matplotlib.pyplot as plt

%matplotlib inline

import seaborn as sns

import os

from os import listdir

from os.path import isfile, join

import spacy

from spacy.tokens import Doc

import re

import plotly

import plotly.graph_objects as go

import scattertext as st

from scattertext import CorpusFromPandas, produce_scattertext_explorer

from IPython.display import IFrame

from IPython.core.display import display, HTML

import nltk

from nltk.tokenize import RegexpTokenizer

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

from nltk.corpus import stopwords

nltk.download('stopwords')

nltk.download('wordnet')

# Function to return a list of stop words to consider

def create_stop_words():

# We create a stop word list

stops = set(stopwords.words("spanish"))

# We define individual numbers and letters as stop words

all_letters_numbers = string.digits + string.ascii_letters

stops = stops.union(["señor", "abascal", "casado",

"partido", "popular", "vox", "señorías", "moción",

"censura"

]) # add some stopwords

stops = stops.union(list(all_letters_numbers))

return stops

g_parties_dict = {

"Casado": "./Mocion/Casado/",

"Abascal": "./Mocion/Abascal/"

}

g_parties_color_dict = {

"Casado": ["#b3d9ff", "#0080ff", "interpolateBlues"],

"Abascal": ["#c2f0c2", "#33cc33", "interpolateGreens"]}

g_nlp = spacy.load('es_core_news_sm')

g_stop_words = create_stop_words() | g_nlp.Defaults.stop_words

g_not_category_name='Resto de partidos'

g_charts_root_dir = './Charts'

def list_dirs(directory):

"""Returns all directories in a given directory

"""

return [f for f in pathlib.Path(directory).iterdir() if f.is_dir()]

def list_files(directory, ext):

"""Returns all files in a given directory

"""

onlyfiles = [join(directory, f) for f in listdir(directory) if isfile(join(directory, f)) and f.endswith(ext)]

return sorted(onlyfiles)

# Function to remove numbers and small words (1 or 2 letters) from a document

def num_and_short_word_preprocessor(tokens):

# Regular expression for numbers

no_numbers = re.sub('(\d)+', '', tokens.lower())

# Regular expression for 1-letter and 2-letter words

no_short_words = re.sub(r'\b\w{1,2}\b', '', no_numbers)

return no_short_words

# Function to tokenize a document wihthout using stopwords

def custom_tokenizer(doc):

word_tokenizer = RegexpTokenizer(r'\w+')

tokens = word_tokenizer.tokenize(doc)

return tokens

# Function to tokenize a document using stopwords

def custom_tokenizer_filtering_stopwords(doc):

word_tokenizer = RegexpTokenizer(r'\w+')

tokens = word_tokenizer.tokenize(doc)

filtered_tokens = [token for token in tokens if token not in g_stop_words]

return filtered_tokens

def create_Doc(row):

doc = Doc(g_nlp.vocab, words=row)

doc.is_parsed = True

return doc

def create_party_programs_corpus_with_sections():

all_party_program_section_name_l = []

all_party_corpus_lc = []

all_party_name = list()

for party_name, party_program_location in g_parties_dict.items():

# List of files and their corresponding names that make up the party program

party_program_section_file_l = list_files(party_program_location, ".txt")

party_program_section_name_l = list(map(lambda x: os.path.split(x)[1], party_program_section_file_l))

# List that contains the program document corpus

party_program_corpus = list()

# Fill out the corpus

for section_file_name in party_program_section_file_l:

with open(section_file_name, 'r', encoding="utf8") as section_file:

section_file_data = section_file.read().replace('\n', '. ')

party_program_corpus.append(section_file_data)

all_party_name.append(party_name)

# Clean up sections

party_corpus_lc = [section.lower() for section in party_program_corpus] # to lowercase

party_corpus_lc = [re.sub("(\d)+", '', section) for section in party_corpus_lc] # no numbers

party_corpus_lc = [re.sub("[€º”—“«»>•‘’!""#$%&'()*+,-./:;?@[\]^_`{|}~]+", ' ',

section) for section in party_corpus_lc] # no strange characters

party_corpus_lc = [re.sub('\s\s+', ' ', section) for section in party_corpus_lc] # no multiple spaces

party_corpus_lc = [re.sub(r'\b\w{1}\b', '', section) for section in party_corpus_lc]

all_party_corpus_lc += party_corpus_lc

# Cleanup section names

all_party_program_section_name_l += party_program_section_name_l

all_party_program_section_name_l = [re.sub(".txt", '',

section_name) for section_name in all_party_program_section_name_l]

all_party_program_section_name_l = [re.sub("(\d)+[-]", '',

section_name) for section_name in all_party_program_section_name_l]

token_raw_party_section = [custom_tokenizer(section) for section in all_party_corpus_lc]

token_filtered_party_section = [custom_tokenizer_filtering_stopwords(section) for section in all_party_corpus_lc]

# Create DataFrame

party_corpus_df = pd.DataFrame(data = {

"Party": all_party_name,

"Section_name": all_party_program_section_name_l,

"Section_text": all_party_corpus_lc,

"Tokenized_raw_section_text": token_raw_party_section,

"Tokenized_filtered_section_text": token_filtered_party_section,

})

parsed = party_corpus_df.Tokenized_filtered_section_text.apply(create_Doc)

party_corpus_df["Parsed_section_text"] = parsed

return party_corpus_df

def get_word_breakdown(party_corpus_df):

def get_num_tokens(values):

flat_list = [item for sublist in values for item in sublist]

return len(flat_list)

total_words = party_corpus_df.groupby("Party").Tokenized_raw_section_text.agg(get_num_tokens)

useful_words = party_corpus_df.groupby("Party").Tokenized_filtered_section_text.agg(get_num_tokens)

words_per_party = pd.DataFrame(total_words).assign(x=useful_words.values)

words_per_party.reset_index(level=0, inplace=True)

words_per_party.columns = ['Party', 'Total words', 'Useful words']

return words_per_party

def create_wordcount_pie(words_per_party):

import locale

locale.setlocale(locale.LC_NUMERIC, 'es_ES')

words_per_party = words_per_party.sort_values(by=['Total words'], ascending=False).reset_index(drop=True)

legends_l = []

for party in words_per_party['Party']:

legends_l.append("{} - totales".format(party))

legends_l.append("{} - útiles".format(party))

sns.set(color_codes=True)

sns.set_style("whitegrid")

figure = plt.figure(figsize=(7, 7))

ax_pie = figure.add_subplot(1, 1, 1)

startingRadius = 1.7 + (0.3* (len(words_per_party)-1))

for index, row in words_per_party.iterrows():

party_name = row["Party"]

total_words = row["Total words"]

useful_words = row["Useful words"]

useless_words = row["Total words"] - row["Useful words"]

party_colors = g_parties_color_dict[party_name]

useful_words_pct = (useful_words / total_words) * 100

localized_useful_words_pct = locale.format_string('%.2f', useful_words_pct, grouping=True)

label_text = "{}: {}% útiles".format(party_name, localized_useful_words_pct)

label_text_2 = "{}".format(total_words)

remaining_pct = 100 - useful_words_pct

donut_sizes = [remaining_pct, useful_words_pct]

ax_pie.text(-0.48, startingRadius - 0.26, label_text,

horizontalalignment='left', verticalalignment='center',

fontsize=12, fontweight='bold',color='black')

ax_pie.text(0.1 - startingRadius, 0, useful_words,

horizontalalignment='left', verticalalignment='center',

fontsize=12, fontweight='normal',color='black', rotation = 45)

ax_pie.text(-0.3 + startingRadius, 0, total_words,

horizontalalignment='left', verticalalignment='center',

fontsize=12, fontweight='normal',color='black', rotation = 45)

donut = ax_pie.pie(donut_sizes,

radius = startingRadius,

startangle = 270,

colors=party_colors,

labels = [total_words, useful_words],

rotatelabels = False,

counterclock = True,

labeldistance = 0.95 - 0.025*index,

wedgeprops = {"edgecolor": "white", 'linewidth': 1},

textprops = dict(rotation_mode = 'default',

va='center',

ha='center',

#rotation=45,

wrap=True,

#position=(2,0),

visible=False

)

startingRadius-=0.80

# create circle and place onto pie chart

circle = ax_pie.add_patch(plt.Circle(xy=(0, 0), radius=0.35, facecolor='white'))

horiz_offset = 0.8

vert_offset = -0.30

_ = ax_pie.legend(legends_l,

bbox_to_anchor=(horiz_offset, vert_offset),

ncol=2)

_ = ax_pie.set_title('Distribución de palabras (útiles/totales) en el discurso',

fontsize=18, x=0.5, y=1.35)

def create_wordcount_stacked_bar(words_per_party):

words_per_party = words_per_party.sort_values(by=['Total words'], ascending=False).reset_index(drop=True)

sns.set(color_codes=True)

sns.set_style("whitegrid")

figure = plt.figure(figsize=(15, 5))

ax_1 = figure.add_subplot(1, 1, 1)

ax_1.set_ymargin(0.15)

parties = words_per_party["Party"]

total_words_colors = [g_parties_color_dict[key][0] for key in parties.values]

useful_words_colors = [g_parties_color_dict[key][1] for key in parties.values]

total_words_legends = ["{} - no útiles".format(key) for key in parties.values]

useful_words_legends = ["{} - útiles".format(key) for key in parties.values]

useless_words = words_per_party["Total words"] - words_per_party["Useful words"]

pie_1 = ax_1.barh(parties, words_per_party["Total words"],

color=total_words_colors)

pie_2 = ax_1.barh(parties, words_per_party["Useful words"],

color=useful_words_colors)

for i, v in enumerate(words_per_party["Total words"]):

plt.text(v/1.5, i + .45, str(useless_words[i]), color=total_words_colors[i], fontweight='bold', fontsize=16)

plt.text(v/1.0, i + .05, str(v), color='black', fontweight='normal', fontsize=13)

for i, v in enumerate(words_per_party["Useful words"]):

plt.text(v/4, i + .45, str(v), color=useful_words_colors[i], fontweight='bold', fontsize=16)

# we also need to switch the labels

ax_1.set_xlabel('Número de palabras en el discurso', fontsize=18)

horiz_offset = 0.98

vert_offset = 1

_ = ax_1.legend(pie_1 + pie_2, total_words_legends + useful_words_legends,

bbox_to_anchor=(horiz_offset, vert_offset),

ncol=2)

def merge_party_sections(party_corpus_df):

corpus_groupby_party = party_corpus_df.groupby("Party", as_index=False)

party_aggregated_sections = corpus_groupby_party["Section_text"].apply(lambda x: ' '.join(x))

party_sections_df = pd.DataFrame(data = {"Section_text": party_aggregated_sections.values,

"Party": list(corpus_groupby_party["Party"].groups.keys())})

return party_sections_df

def create_corpora_metric(corpora, doc_names, metric, stop, ngram = 1, num_freq_words=25, num_freq_words_section = 5,

debug = False):

if metric == "bow":

vectorizer = CountVectorizer(stop_words = stop, ngram_range = (ngram, ngram),

preprocessor = num_and_short_word_preprocessor, tokenizer=custom_tokenizer)

elif metric =="tf-idf":

vectorizer = TfidfVectorizer(stop_words = stop, ngram_range = (ngram, ngram),

preprocessor = num_and_short_word_preprocessor, tokenizer=custom_tokenizer)

else:

raise Exception('Metric not supported')

vec = vectorizer.fit(corpora)

doc_term_mat = vec.transform(corpora)

# START: Calculate the most frequent words per section (using word ID)

freq_words_set = set()

for section in range(len(corpora)):

section_info = doc_term_mat[section,]

freq_words_section = [(idx, section_info[0, idx]) for word, idx in vec.vocabulary_.items()]

sorted_freq_words_section = sorted(freq_words_section, key = lambda x: x[1], reverse=True)

top_freq_words_section = [word[0] for word in sorted_freq_words_section]

freq_words_set.update(top_freq_words_section[0:num_freq_words_section])

# END

# START: Calculate the most frequest words overall (using word ID)

all_sections_info = doc_term_mat.sum(axis=0)

all_freq_words_section = [(idx, all_sections_info[0, idx]) for word, idx in vec.vocabulary_.items()]

all_sorted_freq_words_section = sorted(all_freq_words_section, key = lambda x: x[1], reverse=True)

all_top_freq_words_section = [word[0] for word in all_sorted_freq_words_section]

freq_words_set.update(all_top_freq_words_section[0:num_freq_words])

# END

freq_words_key_l = list(freq_words_set) # list of most frequent word IDs

reverse_dictionary = {v: k for k, v in vec.vocabulary_.items()} # dictionary ID: word

freq_words_value_l = [reverse_dictionary[word_key] for word_key in freq_words_key_l] # list of most frequent words

#DataFrame including the most frequent words per section and overall

freq_words_df = pd.DataFrame((doc_term_mat[:,freq_words_key_l]).todense(),

columns=freq_words_value_l,

index=doc_names)

return freq_words_df

def create_metric_dict_by_party(party_corpus_df, metric = "bow",

ngram = 1, num_freq_words=10, num_freq_words_section = 5):

parties = party_corpus_df.Party.unique()

metric_by_party = dict()

# METRIC (BOW or TF-IDF) for individual parties

for party in parties:

party_corpora = party_corpus_df[party_corpus_df.Party == party]

party_section_texts = party_corpora.Section_text.tolist()

party_section_names = party_corpora.Section_name.tolist()

freq_words_df = create_corpora_metric(party_section_texts,

party_section_names,

metric,

g_stop_words,

ngram = ngram,

num_freq_words=num_freq_words,

num_freq_words_section = num_freq_words_section)

metric_by_party[party] = freq_words_df

# METRIC (BOW or TF-IDF) for the overall corpora including all parties

party_sections_df = merge_party_sections(party_corpus_df)

overall_section_texts = party_sections_df.Section_text.tolist()

overall_section_names = party_sections_df.Party.tolist()

overall_freq_words_df = create_corpora_metric(overall_section_texts,

overall_section_names,

metric,

g_stop_words,

ngram = ngram,

num_freq_words=num_freq_words,

num_freq_words_section = num_freq_words_section)

metric_by_party["All"] = overall_freq_words_df

return metric_by_party

def normalize_word_df(word_df):

norm_word_df = word_df.copy()

total_per_party = norm_word_df.sum(axis=1)

grand_total = sum(total_per_party)

for party in norm_word_df.index:

norm_word_df.loc[party] = (norm_word_df.loc[party] / total_per_party[party]) * 100

return norm_word_df

def show_df_as_bubble(word_df,

num_words = 2,

normalize = False,

standard_sizeref = True,

title = 'Palabras con mayor score por partido',

xtitle = 'Palabras',

ytitle = 'Partido',

sub_dir = 'bow',

file_name = 'bow_bubble',

to_file = False):

if normalize:

word_df = normalize_word_df(word_df)

title = title + " (normalizado)"

# Keep the top 'num_words' words

word_df.reindex(word_df.sum().sort_values(ascending=False).index, axis=1)

num_words = min(num_words, word_df.shape[1])

word_df = word_df.iloc[:, :num_words]

# Sort in alphabetical order

word_df = word_df.reindex(sorted(word_df.columns), axis=1)

# Get the word labels

word_l = word_df.columns.values

num_words = len(word_l)

# List that contains every bubble row. Each element is a party.

bubble_data = []

xs = list(range(word_df.shape[1]))

for num_row in range(len(word_df.values)):

index_name = word_df.index[num_row]

index_color = g_parties_color_dict[index_name][1]

index_values = word_df.values[num_row]

if normalize:

sizeref = 0.01 * max(index_values)

hover_text = ["{:.2f}%".format(index_value) for index_value in index_values]

else:

if standard_sizeref:

sizeref = 1

else:

sizeref = 0.01 * max(index_values)

hover_text = index_values

section_bubble = go.Scatter(

x = word_l,

y = [num_row + 1] * num_words,

name = index_name,

showlegend = True,

mode = 'markers',

hovertext = hover_text,

hoverinfo = "x+text",

marker_size = index_values,

marker = dict(color = index_color, sizeref = sizeref),

)

bubble_data.append(section_bubble)

fig = go.Figure(data=bubble_data)

fig.update_layout(

title = dict(text = title, xanchor = 'center', x = 0.5, font = dict(size=20)),

yaxis = dict(

tickmode = 'array',

tickvals = list(range(1, num_words + 1)),

ticktext = word_df.index

legend = dict(itemsizing = 'constant')

)

fig.update_xaxes(tickangle=315, title_text=xtitle, title_font=dict(size=18))

fig.update_yaxes(title_text=ytitle, title_font=dict(size=18))

if to_file:

complete_file_name = '{}/{}/{}.html'.format(g_charts_root_dir, sub_dir, file_name)

plotly.offline.plot(fig, filename = complete_file_name, auto_open=False)

fig.show()

# Read files

party_corpus_df = create_party_programs_corpus_with_sections()

# Word count analysis

words_per_party = get_word_breakdown(party_corpus_df)

create_wordcount_pie(words_per_party)

create_wordcount_stacked_bar(words_per_party)

# BOW analysis

bow_by_party = create_metric_dict_by_party(party_corpus_df, metric = "bow", ngram = 1,

num_freq_words=15, num_freq_words_section = 5)

two_gram_bow_by_party = create_metric_dict_by_party(party_corpus_df, metric = "bow", ngram = 2,

num_freq_words=10,

num_freq_words_section = 10)

all_party_words = bow_by_party['All']

all_party_2_words = two_gram_bow_by_party['All']

show_df_as_bubble(all_party_words, num_words = 15, normalize = False,

title = 'Palabras más usadas por político',ytitle = 'Político',

to_file = True, sub_dir = 'bow', file_name = 'bow_partidos')

show_df_as_bubble(all_party_2_words, num_words = 15, normalize = True,

title = 'Binomios de palabras por político',

xtitle = 'Binomios de palabras',

ytitle = 'Político',

to_file = True, sub_dir = 'bow', file_name = 'bow_2_partidos')

# TF-IDF analysis

td_idf_by_party = create_metric_dict_by_party(party_corpus_df, metric = "tf-idf", ngram = 1,

num_freq_words=15, num_freq_words_section = 15)

two_gram_tf_idf_by_party = create_metric_dict_by_party(party_corpus_df, metric = "tf-idf", ngram = 2,

num_freq_words=10,

num_freq_words_section = 10)

tdidf_all_party_words = td_idf_by_party['All']

tdidf_all_party_2_words = two_gram_tf_idf_by_party['All']

show_df_as_bubble(tdidf_all_party_words, num_words = 250, normalize = False, standard_sizeref = False,

title = 'Palabras con mayor score por político', ytitle = 'Político',

to_file = True, sub_dir = 'tfidf', file_name = 'tfidf_partidos')

show_df_as_bubble(tdidf_all_party_2_words, num_words = 250, normalize = False, standard_sizeref = False,

title = 'Binomios de palabras con mayor score por político', ytitle = 'Político',

to_file = True, sub_dir = 'tfidf', file_name = 'tfidf_2_partidos')

Takeaway

Los discursos de moción de censura de Abascal y Casado nos han dado la oportunidad para re-usar masivamente el trabajo realizado en posts anteriores para limpiar, pre-procesar y analizar documentos.

En primer lugar hemos realizado el ejercicio sencillo de contar palabras, además de conocer qué porcentaje de palabras aporta poco significado a un discurso.

Después hemos contado las palabras y nos hemos quedado con las que más se repiten y hemos realizado un análisis adicional sobre los binomios de palabras más usados.

Finalmente hemos buscado las palabras que mejor caracterizan al discurso, utilizando la técnica TF-IDF, aunque habríamos sacado más rendimiento a la téncica si hubiésemos tenido varios discursos por cada polícito.

Ver en Kaggle

Moción de censura: la palabra de Abascal contra la de Casado

La moción de censura nos da la oportunidad de re-usar

Ficheros de los discursos

Pre-procesado y análisis

Palabras totales vs. palabras útiles de los discursos

Distribución de palabras totales vs. palabras útiles de los discursos

Bag of Words sobre los discursos

Lo mismo, pero con binomios de palabras

TF-IDF de los discursos

Finalmente, nos binomios de palabras

Si te interesa el código, aquí lo tienes

Takeaway

Enviar comentario Cancelar la respuesta

Entradas recientes

Archivos

Moción de censura: la palabra de Abascal contra la de Casado

La moción de censura nos da la oportunidad de re-usar

Ficheros de los discursos

Pre-procesado y análisis

Palabras totales vs. palabras útiles de los discursos

Distribución de palabras totales vs. palabras útiles de los discursos

Bag of Words sobre los discursos

Lo mismo, pero con binomios de palabras

TF-IDF de los discursos

Finalmente, nos binomios de palabras

Si te interesa el código, aquí lo tienes

Takeaway

Comparte esto:

Enviar comentario Cancelar la respuesta

Entradas recientes

Archivos