Automatic grouping of news from online newspapers using Machine Learning techniques for clustering texts in Portuguese

Abstract

Clustering is a technique of organizing data into groups whose members have some similarity. Thus, this research aimed to use the techniques of Natural Language Processing, Machine Learning and Clustering to create clusters of news from a sample collected from the main online newspapers. It was found that the pre-processing step requires an effort to guarantee the quality of the results. The complexity of the Portuguese language, the need to update the list of stopwords, the difficulties related to the detection of the most important characteristics and the high dimensionality of the data were evidenced during all stages of this study. The k-means clustering algorithm obtained the best results for this type of information and Hierarchical Clustering had difficulties, since similar news were allocated to different groups. Affinity Propagation, on the other hand, disagreed as to the ideal number of clusters, but achieved a good performance when grouping by similarity.