A Survey of Modern Crawler Methods
Zixiang Chang
Abstract
Web crawler is a kind of computer program to browse the World Wide Web (WWW) automatically and efficiently. In the information age, due to the explosive growth of Internet pages, it has become exceedingly difficult and time-consuming for people to search information, therefore people need the help of crawler to get the information they need. In this paper, the existing crawler technology is briefly summarized, the purpose is for beginners or people interested in the field of crawler can have a preliminary understanding and cognition of crawler. This paper first introduces the background of crawler technology and its classification and use. Then it summarizes some common libraries and frameworks of modern crawlers through tables, which is convenient for readers to compare and understand. Secondly, this paper reviews the literature of modern crawler technology, and describes how people realize crawler application through library or framework. The practical application of crawler technology is also briefly explained.