A REVIEW ON WEB CRAWLER

ABSTRACT

With the advancements in the internet, searching web is much important. To revoke the web pages automatically, web crawler is used. Web crawler feeds on a seed URL and visits all the following URLs to gather information. Crawlers act together with thousands of web servers over periods expanding from weeks to several years. These crawlers visits several thousands of pages every second, includes a high-performance fault manager, are platform independent or dependent and are able to get used to a wide range of configurations without including additional hardware.This paper focused on introduction and fundamentals of web crawler, architecture of web crawler and advantages and disadvantages of web crawler.

FULL TEXT