Evaluation of Focused Web Crawlers in Real-World Scenarios
Dharmesh Dhabliya, Ahmed Hussein Alawadi, Vijay Kumar Rai, N.T. Velusudha, Kousik Sarathy
Abstract
This study evaluates the practical performance of focused web crawlers, tools essential for extracting targeted data from the ever-expanding web. Moving beyond theoretical analysis, we configure crawlers for specific real-world scenarios such as academic research and social media monitoring, optimizing them for precise data retrieval across a dataset of 1,000 domains and 10 million URLs. Performance is measured through meticulous logging of resource use and data freshness, while quality is ensured via cross-verification with a benchmark dataset. Ethical compliance is rigorously maintained throughout the process. The comparative analysis of different crawlers provides a framework for their selection and deployment, culminating in actionable recommendations for their use in various industries, reflective of the complex demands faced by users in a dynamic digital landscape.