← Discover
Discover · GitHub · Crawl4AI↑ +154 today

Prepare Web Data for AI

Crawl4AI; is an open source web crawler and scraper optimized for large language models. It converts web pages into clean and structured formats (Markdown, etc.) that artificial intelligence models can easily process.

Updates

  • August 31, 2026: Stars 75,853 → 80,563, latest release v0.9.3 (August 31, 2026).
  • August 2, 2026: Stars 67,194 → 75,853, latest release v0.9.2 (July 15, 2026).

What you get

  • It converts web content into a clean and AI-friendly format.
  • Specially optimized for LLM processes.
  • It is fast and open source.

Installation

Using pip (PyPI)
pip install crawl4ai

Running it

Crawl Web Page for AI
crawl4ai-download

Installation (single command)

🤖 Paste this into your AI agent (Claude Code · Codex · Antigravity)

Help me install open source web crawler called Crawl4AI; Let's try it by installing it with 'pip install -U crawl4ai', then running 'crawl4ai-setup' and converting a web page to LLM-friendly Markdown with the command 'crwl https://www.nbcnews.com/business -o markdown'.

Related dictionary terms

MarkdownLLMOpen SourceArtificial Intelligence
Who it is forWeb data collectors for AI/data
DifficultyMedium · Python
What offersLLM-friendly web crawl + scrape
PreconditionPython
FeeFree · open source (Apache-2.0)
License: Apache-2.0 · you can use it freely, modify it, make commercial use (also includes patent protection).

Links

TreScout did not build this tool · we found it in GitHub trends and wrote it up. This page describes the repository as of 2026-05-29: The star count and our text belong to that day, the repository may have changed since. Check the repository link for the current state. This page was machine-translated from the Turkish original · the Turkish version prevails.