# What is Web Crawlers?

They are software bots that automatically browse pages on the internet and record data.

## Overview
They form the foundation of search engines. By following links on a page, they jump to other pages and index everything they see, creating a massive database. They can be thought of as digital travelers mapping the internet.

*Analogy: They are like an attendant in a library who opens every book one by one, takes notes on the information inside, and then directs you to other books.*

## How it works
They start with a specific initial list. They read the text, images, and links on the page and process them into their own database, then continue to move toward the new links they find.

## Where it is used
They are used in search engine indexing, data collection projects, and price comparison sites.

## Commonly confused with
They can be confused with web scraping; a crawler traverses the entire network, while a scraper extracts data from a specific page.

## Frequently asked questions
**Can they visit every site?**
Site owners can determine which parts bots can or cannot visit using a 'robots.txt' file.


## Related terms
- [Web Scraping](/en/dictionary/web-scraping/)
- [Data Pipeline](/en/dictionary/data-pipeline/)

---
Source: TreScout Dictionary · https://trescout.com/en/dictionary/web-crawlers/