CommonCrawl: the free web archive most LLMs are trained on
Hundreds of terabytes of the open web, free, no account required.
Most LLMs were trained on this. It's free, public, and sitting in an S3 bucket anyone can pull from.
Ask how most large language models got their reading list and the honest answer routes through one free archive. CommonCrawl has been crawling the open web since 2008 and handing the result out for nothing: no account, no key, no invoice. Here is what is actually in it, how to pull it without downloading the whole internet, and why we keep it on our list for AI and search feature work.
What it actually is
CommonCrawl is a nonprofit web crawl, released as a new snapshot roughly every month (named like CC-MAIN-2026-30), archived continuously since 2008. Each monthly snapshot runs into the hundreds of terabytes. Nobody gates access to it: hosting is sponsored by AWS, the data sits in a public S3 bucket, and the license is effectively open, use it for research, training, or a product feature.
Three file formats, one crawl
- WARC: the raw record, full HTTP request and response, headers, TLS and protocol version, exactly what a browser would have seen.
- WAT: the same crawl reduced to computed metadata as JSON: links found on the page, header fields, structure, no page body.
- WET: the same pages again, stripped down to extracted plain text, nothing else.
Pick the format that matches the job. Training a text model wants WET. Mapping link structure wants WAT. Anything that needs the actual bytes as they were served, headers included, wants WARC.
Getting a slice without downloading the internet
The naive move is grabbing a full monthly segment and regretting it. CommonCrawl publishes a CDXJ URL index alongside every crawl: look up a domain or URL in the index first, get back the exact WARC file and byte offset, then pull just that record over HTTP or S3. From AWS, `aws s3 cp --no-sign-request s3://commoncrawl/...` in us-east-1 avoids egress cost entirely. From anywhere else, the same files sit behind plain HTTPS at data.commoncrawl.org, wget and curl both work, still no account.
Why it matters for AI and search work
CommonCrawl is not just one dataset among many, it is the base ingredient behind most LLM pretraining corpora: C4, RefinedWeb, RedPajama and others are CommonCrawl put through a cleaning pipeline, not a separate crawl. If a client project needs real web scale text (evaluating a classifier, sampling realistic queries, prototyping a search or embedding feature) this is the place to pull from instead of writing a crawler and inheriting its politeness rules, robots.txt handling, and legal grey areas yourself. When we picked eval queries for ternlight, our on-device semantic search, the point of reference was real-world text distribution, and CommonCrawl is the honest place to sample that from.
The honest catch
None of this is curated. A raw crawl is full of duplicate pages, boilerplate navigation, spam, mixed languages and the occasional broken encoding, which is exactly why C4 and RefinedWeb exist as separate papers: cleaning a raw crawl into something trainable is most of the work, not the download. And WARC is a real format with real edge cases, reach for a library like `warcio` in Python instead of parsing it by hand.
Next step: start at commoncrawl.org/get-started for the index docs and a working example before pulling anything. More of what we reach for is on the tools page. If your team needs a hand turning a raw web-scale dataset into something a model or a search feature can actually use, write to us at hello@gattyworks.com.