Effectiveness of Preprocessing Algorithms for Natural Language Processing Applications
Kirill Smelyakov, Danil Karachevtsev, Denis Kulemza, Yehor Samoilenko, Oleh Patlan, Anastasiya Chupryna
Abstract
this article is devoted to identifying features and analyzing the effectiveness of stop word removal, stemming and text lemmatization algorithms that are used in the widely used NLTK and Spacy libraries when preparing data in Natural Language Processing applications in determining the publication topic; work is based on the analysis of the results of numerous experiments on the application of the considered text processing algorithms for such common topics of web publications as sports, economics and news.
Topics & Concepts
Computer scienceLemmatisationPreprocessorNatural language processingArtificial intelligenceNatural languageAlgorithmText processingWord (group theory)Data pre-processingStop wordsInformation retrievalLinguisticsPhilosophyCybersecurity and Information SystemsInformation Systems and Technology ApplicationsScientific Research and Philosophical Inquiry