Data Quality Beats Token Count

A web filter reads every document in the raw pool and keeps a small fraction of them. Each read must be almost free, or the filter spends the compute it was built to save.

That constraint shapes the whole data pipeline. You crawl, you parse HTML into text, you filter for language and quality and toxicity, and you drop the duplicates. Four jobs, one cheap pattern.

The filter must cost orders of magnitude less than the training

Most filters run one recipe. You hold a small target set T that you like, and a huge raw set R that you have, Common Crawl for example. You want the part of R that looks like T. Train a cheap model on T, sometimes on R as well, and score every document in R.

You spend the score in one of two ways. Keep the documents above a threshold, or resample with a probability tied to the score. The filter must also generalize past T, because T is small and the pool it judges is the web.

Now price it. If you keep 1 percent of the web, the filter reads a hundred documents for every one that reaches training. Multiply the per-document cost by that hundred before you compare it against the run. Heavy scorers lose on that arithmetic before anyone argues about their accuracy.

Three cheap scorers, one pattern

An n-gram model estimates the probability of a word from the n-1 words before it. Count the n-grams in a corpus, then read the next-word probability off the counts for that context. Sparsity breaks the naive version. Many reasonable n-grams never appear, the problem grows with n, and raw counts return a zero for text that is fine.

Smoothing fixes that, usually Kneser-Ney, the version KenLM implements. When the counts for a long context are weak, the model falls back to shorter contexts and interpolates across the lengths. Train KenLM on clean text, usually Wikipedia, then score raw pages by probability or normalized perplexity. Low perplexity means the page reads like the training corpus, and high perplexity means junk.

CCNet and the early LLaMA pipelines ran exactly this. They trained KenLM on Wikipedia, scored Common Crawl paragraphs by perplexity, sorted, and kept the best portion. The model catches obvious garbage and non-language text. Repetition fools it, because the window is a few words wide.

fastText is the discriminative option and it is built for speed. A plain bag-of-words classifier over word counts works, and the weight matrix grows enormous. fastText maps each word to a low-dimensional embedding, averages the embeddings into one document vector, and runs a linear classifier on that vector.

Word order comes back through hashed n-grams. Build word n-grams, bigrams for example. Hash them into a fixed number of bins and treat the bins as extra tokens. Different n-grams collide in one bin, and the averaged weights stay usable anyway.

The filter itself is binary. The target set T supplies the positives and random samples from R supply the negatives. Train it, run it over R, and keep the documents that score high on target.

Importance sampling replaces the boundary with a ratio. You want samples from a target distribution p, and you can only draw from q. So draw x from q, weight it by p(x) divided by q(x), and resample by weight. T is your sample from p and R is your sample from q.

T is small, so fitting a rich p is hard. The cheap version uses hashing again. Hash tokens or n-grams into buckets, count the bucket frequencies in T and in R, and smooth both into probabilities. Score a document by multiplying its bucket probabilities under each distribution, then take the ratio.

That keeps more diversity than a sharp in-or-out classifier, because it matches the full shape of T rather than a boundary. The three scorers differ in what they model and agree on the move. Each one learns how much a document looks like T, and the score decides what survives.

Use an expensive model once, then distill the signal

Quality has no single definition. Grammar, coherence, low spam, and educational value all get called quality, and a cheap model approximates any of them at best. So teams define quality by example. The positive set is the definition.

The GPT-3 pipeline used curated sources as positives, books and Wikipedia and WebText, with random Common Crawl as the negatives. Train a linear classifier and filter Common Crawl by its score. Smooth sampling sometimes takes the place of a hard cut.

The first LLaMA pipeline moved the positives one hop out. Pages linked from Wikipedia became the target and Wikipedia itself stayed out of the set, with random crawl pages as negatives. The classifier kept what it labeled positive.

Phi-1 paid a strong model for the definition. Start with the Python subset of The Stack. Ask GPT-4 one question of each file: "How educational is this file for a student learning basic coding concepts?" About 100k labeled files define the target.

GPT-4 over the whole subset is out of the question, so distill it. Compute embeddings with a pre-trained model. Train a random forest on those embeddings to imitate the GPT-4 labels. Run the forest over the full Python subset.

The unfiltered subset reached about 12 percent on HumanEval after 96k steps. The filtered subset reached about 17 percent after 36k steps. The filter bought the score and the steps together.

The pattern holds beyond Python. Pay a strong model once on a small sample to define the signal. Distill the signal into a classifier you can afford. Run the cheap one over everything.

Cheap filters see local patterns only

These scorers read local word patterns. Long-range coherence and factual correctness sit outside what they measure. Shuffle the sentences inside a document and the n-gram statistics barely move, so the shuffled version scores about the same. Text written to beat the filter passes it.

So they belong early, where the job is to remove the worst of the web for almost nothing. The deeper quality signals arrive later, once the pool is small enough to afford them.

Language identification runs the same machinery. fastText ships a pre-trained language identification model trained on multilingual sources, and it returns a probability for each language. Pipelines keep the pages where the probability of English clears a threshold, 0.5 for example. The identification model handles ordinary sentences well and struggles with short snippets, code, formulas, dialect, and code-switching.

The reason to bother is arithmetic. Compute is fixed, so tokens spent on other languages come out of the language you care about. BLOOM trained on about 30 percent English and shows the trade against an English-only model of the same size.

Toxicity filtering is the same shape once more. The Jigsaw Toxic Comments set labels Wikipedia talk-page comments as toxic, severe toxic, obscene, threat, insult, and identity hate. Dolma-style pipelines train one fastText classifier to separate hate from safe text and a second to separate NSFW from safe. Both run over the raw text, and the pipeline drops or down-weights the high scorers. Both miss where the context carries the meaning, and both hold up as bulk filters.

Filtering changes the training distribution. Every threshold is a claim about what the model must learn. The data note made that point about the domain mix, and here it narrows to one number in a scoring pass.

Deduplication is a different job from quality filtering

Quality filtering says this document is bad, remove it. Deduplication says this document is fine, keep fewer copies. Both run in the same pipeline, and the questions they answer do not overlap.

Duplicates waste compute, because the run trains again on text it already read. They also raise memorization. Repetition increases the chance of verbatim recall, and verbatim recall is where the copyright and privacy problems start.

The web supplies both kinds. Exact duplicates are identical text across mirrors and reposts. Near-duplicates are the same page after a small edit: boilerplate, templates, localized versions, licenses, and mass-copied artifacts.

Three choices define a deduplication run before any algorithm shows up.

• The unit can be a sentence, a paragraph, a fixed-length span, or the whole document.
• The match can be exact, high overlap, or semantic similarity.
• The action can remove every copy but one, or cap the frequency.

The hard part is the count. Pairwise similarity over billions of documents is impossible, so the working methods turn comparison into hashing and bucketing.

Exact deduplication is hashing with a Bloom filter

Exact deduplication is three steps. Hash each unit, usually a paragraph or a span. Group by hash. Keep one member of each group.

C4 runs this on three-sentence spans, and a repeated span survives in one place only. That produces strange edits inside otherwise fine documents, and the cost stays near-linear in the number of spans.

Holding every hash in memory gets expensive, so pipelines reach for a Bloom filter. The filter is a bit array of length m with k hash functions, and every bit starts at zero. Insert sets the k hashed positions to 1.

The query is asymmetric.

• If any of the k bits is 0, the item is new, and that answer is never wrong.
• If all k bits are 1, the item is probably a duplicate, and sometimes that answer is wrong.

False positives come from collisions, and the rate depends on m, n, and k. One k is optimal for a given m and n. Dolma runs paragraph-level exact deduplication this way, with the false positive rate tuned to about 10^-15. A false positive drops a paragraph the corpus never saw before, so the rate goes very low. Standard Bloom filters only insert, which fits a one-pass pipeline.

MinHash turns similarity into collision probability

Near-duplicates need a similarity measure and a way to find the similar pairs without comparing every pair. The measure is Jaccard similarity over sets. Turn each document into a set of word shingles or n-grams, then divide the intersection size by the union size. Pipelines call two documents near-duplicates above a high threshold, often about 0.99.

Computing that over all pairs is out, so MinHash converts it into hashing. Pick a random hash over every possible set element. Define h(S) as the element of S with the smallest hash value. The probability that two documents collide is their similarity.

Repeat with independent hashes, and the fraction of matches estimates the similarity. Banding turns that estimate into a lookup. Compute n MinHash values per document as a signature. Split the signature into b bands of r rows, with n equal to b times r. Two documents become candidates when every hash inside one band matches.

One band matches with probability s raised to r, where s is the Jaccard similarity. The chance of becoming a candidate follows from that.

The pair r and b set the shape of the cutoff. A larger r sharpens the transition and raises the effective threshold. A larger b creates more collisions at lower similarity. Buckets come out the other end, and any exact comparison you still want happens inside a bucket.

The evaluation note named decontamination as a duty, and this is the machinery that does it. A benchmark item sitting in the training corpus makes the score on that benchmark meaningless. Run MinHash with the benchmark on one side, and drop the crawl documents that land in the same bucket. The operation is identical. Only the target set changes.

Paraphrases escape all of it. A paraphrase carries the same meaning in different words, so the n-gram sets barely overlap. Embedding the documents and searching that space finds them, at a cost above hashing. Loose thresholds there erase useful diversity, so this stage needs a stricter threshold than the exact ones.

The Builder Test

Price the filter against the run before you write it. Measure the cost of one forward pass. Multiply by the size of the raw pool. Compare that number against the training run it protects. If the two numbers sit within one order of magnitude, the filter has no benefit left to give.

Then read what it does. Sample the documents it kept and the documents it dropped, and read both samples yourself. If the sample surprises you, the model learns that surprise across the whole corpus.

What Carries

Cheap scoring buys expensive compute back. Every gate you can afford to run over the raw pool is compute the training run never spends on text you did not want.

Write the threshold down next to the run. Anyone reading the result later needs to know what the corpus was allowed to contain. The corpus is clean at that point. The next question is what the model does with an instruction after it reads all of it.