Data Quality Filtering v1 — Provenance, Dedup, and Toxicity Screens

Abstract

We describe Data Quality Filtering v1, the pipeline used to assemble the first Origin pretraining mixture. The design goal was to maximize useful signal per token while reducing three concrete risks: PII leakage, toxic content generation, and near-duplicate mass inflating token counts without adding information. We document each pipeline stage, the tooling choices, the impact on downstream training probes, and the failure modes we identified for subsequent versions.

1. Background

Early scaling observations demonstrated a consistent delta between filtered and unfiltered pretraining corpora on instruction-follow probes: at matched token budgets, filtered data scored ~12 points higher on IdonAI's internal rubric (1–5 scale). This result made a production-grade filtering pipeline a prerequisite for Origin, not an optimization.

The Founding Research Charter committed IdonAI to "data quality over data volume" and to maintaining provenance documentation for audits. v1 was the first concrete implementation of both commitments.

2. Pipeline Architecture

The v1 pipeline runs as a DAG of sequential stages. Each stage can emit documents, hold documents for review, or hard-block them from the final corpus.

2.1 Stage overview

Total effective reduction from raw candidate pool to final corpus: approximately 38–42% of raw candidate tokens. The remaining tokens are of substantially higher average quality.

2.2 License gate

We ingest only sources with clear licensing or established web crawl permissions. Proprietary data with unclear licensing is quarantined pending legal review. This is both an ethical and a legal requirement. The gate is conservative: when in doubt, exclude.

2.3 Quality classifier

A lightweight n-gram and feature-based classifier trained on human-labeled quality signals. Signals include:

• Presence of boilerplate patterns (cookie consent pages, navigation menus, footer text scraped as body)
• SEO keyword stuffing signatures (high density of keyword repetition without coherent prose)
• Low information-per-sentence score (short, repetitive, or auto-generated snippets)
• "Link farm" structure (mostly URLs with minimal contextual text)

Documents scoring below threshold are downweighted (not hard-blocked) unless score is very low. This preserves borderline technical content while reducing low-value noise.

2.4 Toxicity and CSAM screening

CSAM (child sexual abuse material) is an absolute block—any document with a positive signal from PhotoDNA-equivalent image hash screening or text classifiers is removed and reported.

Toxicity uses a classifier to score hate speech, graphic violence, and sexual content. Scoring:

• High toxicity: hard block
• Medium toxicity: domain-dependent; some domains (e.g., medical, harm-reduction) retain medium-toxicity content with context
• Low toxicity: pass through

We do not aim to remove all adult or difficult content—legal fiction, medical literature, and harm-reduction material are valuable. The classifier's job is to block content that has no plausible legitimate use in a general-purpose pretraining corpus.

2.5 PII heuristics

Regular-expression and model-based heuristics reduce:

• Email addresses (substituted with [EMAIL])
• Phone numbers (substituted with [PHONE])
• Social security numbers / national ID patterns
• API keys and credential patterns (high-confidence regex)

We do not claim comprehensive PII removal—heuristics have recall limitations. Documents with high PII density are downweighted further after heuristic passes.

2.6 Deduplication

Exact dedup uses document-level hashing. Documents with identical hashes are collapsed to a single copy with the highest-quality source attribution retained.

MinHash near-dedup uses locality-sensitive hashing with shingling to identify documents with high n-gram overlap (Jaccard similarity > 0.8 threshold). Near-duplicates arise from web mirrors, content syndication, and republication. Removing them reduces training noise and memorization risk.

Combined, deduplication removes roughly 20% of candidate documents that were otherwise passing quality filters.

2.7 Domain balancing

Unmanaged web scrapes over-represent news, social media, and legal documents. We apply per-domain token caps and upweighting multipliers to:

• Upweight: Technical documentation, scientific literature, educational text, code repositories
• Downweight: Overrepresented news domains, social media
• Cap: Any single domain at ≤X% of final token count to prevent single-source dominance

Exact balancing weights are versioned as data pipeline artifacts. The Dataset Mixture Ablations note evaluates how these weights affect downstream performance.

3. Impact on Training

We ran a controlled comparison of identical Origin-scale training runs at matched token budgets: one on unfiltered corpus, one on v1-filtered corpus.

Filtering improved every measured downstream probe. The toxic continuation rate improvement is the most dramatic: a 59% reduction from the unfiltered baseline with zero capability regression.

4. Failure Modes Identified

4.1 Over-filtering of technical forums

The quality classifier disproportionately downweighted technical Q&A and forum content due to structural features (short answers, code fragments, colloquial writing) that superficially resembled low-quality text. Post-hoc analysis found that some highly valuable Stack Overflow-style content was removed. v2 added domain-specific classifier adjustments for technical forums.

4.2 Near-dedup false positives

MinHash near-dedup removed legitimate document variants that shared boilerplate sections (e.g., scientific papers with common Methods boilerplate, textbook chapters from different editions). False positive rate was approximately 3% of removed near-duplicates—acceptable but worth monitoring.

4.3 Multilingual coverage gaps

The language ID and quality classifier were English-primary. Non-English content received inconsistent filtering quality, with some languages over-filtered (high false-positive toxicity rates due to classifier cultural bias) and others under-filtered. Multilingual improvements were targeted in subsequent versions.

4.4 PII heuristic recall

Estimated PII recall for emails/phones is ~85%; structured national IDs ~70%. This is not comprehensive. Users should not assume pretraining data is PII-free—they should treat it as meaningfully reduced.

5. Code Corpora

Code repositories require separate handling:

• License filtering is more aggressive (GPL, AGPL, and similar copyleft licenses excluded from certain mix components based on legal guidance)
• PII heuristics are extended with API key and credential regex
• Comment-stripping experiments showed mixed results; retained for v1
• Dedup rates are higher than prose due to code forking and reuse patterns

Code fraction and its effect on downstream performance are studied in Dataset Mixture Ablations.

6. Later Evolution

v1 enabled Origin's pretraining run. Subsequent pipeline versions added:

• v1.5: Domain-specific quality classifier adjustments for technical forums and scientific literature
• v2: Code-specific filtering (stricter credential redaction, license gate improvements, comment-quality classifiers)
• v2.5: Multimodal pair cleaning for captioned images and audio transcripts (in support of NextGen pretraining)
• v3: Citation-heavy scientific pack processing, supporting improved GPQA Diamond performance in NextGen

7. Limitations

v1 is English-primary. Quality gains for non-English languages are smaller and less validated. The toxicity classifier reflects annotator demographics and may under-block content from under-represented cultural contexts. PII recall is not comprehensive. The license gate reflects our legal team's 2023 interpretation—later guidance may require re-evaluation of specific sources. Filtering cannot address all memorization risk—dedup reduces but does not eliminate memorizable sequences.

8. Conclusion

Data Quality Filtering v1 delivered a substantial improvement in training signal quality at a modest reduction in corpus size. The key insights—filtering multiplies per-token value, deduplication is essential for honest token counts, and domain balancing should be versioned and evaluated like architecture choices—became permanent practices at IdonAI. Every subsequent Origin and NextGen training run starts from a v1+ pipeline.

See Early Scaling Observations for pre-pipeline experiments and Dataset Mixture Ablations for downstream effects of mixture weights.