News keywords extraction algorithm based on TextRank and classified TF-IDF

Ao Xiong, Xin Yu, Derong Liu, Hongkang Tian

202019 citationsDOI

Abstract

TextRank algorithm tends to extract words with frequent occurrence as keywords, while TF-IDF only considers the word frequency relation in the text library to extract keywords. In order to combine the advantages of the two algorithms, this paper proposes a Weighted TF-IDF of the Same Category Historical News Library and TextRank (TFSL-TR). This method first uses the classification model based on LSTM to classify the news, and calculates the TF-IDF value as the first weight of the word by using the news library of the target news category. Then use the TextRank algorithm to calculate the second weight of words. Finally, sum the two weights by weight and take the TopK words as keywords in order of size. The experiment was carried out in the Chinese news library, and the results showed that, compared with the traditional method, TFSL-TR could effectively improve the accuracy of keyword extraction.

Topics & Concepts

tf–idfWord (group theory)Computer scienceArtificial intelligenceKeyword extractionOrder (exchange)Value (mathematics)AlgorithmNatural language processingPattern recognition (psychology)MathematicsMachine learningTerm (time)PhysicsGeometryEconomicsFinanceQuantum mechanicsAdvanced Text Analysis TechniquesWeb Data Mining and AnalysisInformation Retrieval and Search Behavior