Origin — Technical Report on IdonAI's Fast Production Model

Abstract

We present Origin, IdonAI's latency-optimized production language model for interactive chat, coding assistance, and cost-efficient tool use. Origin is the default fast path inside Quark and the workhorse model for high-QPS workloads. This report describes architecture (decoder-only transformer with grouped-query attention), data and training recipe, evaluation on MMLU, HumanEval, MATH, GSM8K, and GPQA Diamond, serving characteristics, distillation from larger teachers, safety interfaces, ablations, and limitations.

Origin targets Flash-class quality: strong everyday reasoning and coding at interactive latency, not maximum scores on the hardest scientific suites—those are reserved for NextGen. The design philosophy is that a fast model used on 90% of turns is more valuable to users than a brilliant model that stalls.

Headline results

1. Introduction

1.1 Motivation

IdonAI's product thesis required a model that feels immediate. Empirically, users abandon assistants that stall on simple turns before the model demonstrates capability. The interaction quality of a fast, good-enough answer beats a perfect answer delivered after a long wait for the majority of everyday tasks—coding autocomplete, document summarization, Q&A, and single-file debugging.

This asymmetry motivates a two-tier model family: Origin (fast, efficient) and NextGen (deep, frontier). Quark routes between them. The routing architecture is described in the Quark routing prototype note; this report focuses solely on Origin.

1.2 Design goals

1. Interactive time-to-first-token (TTFT) under realistic continuous batching loads
2. Strong coding and short-form reasoning competitive with Flash-class contemporaries
3. High-reliability structured output for tool-calling pipelines
4. Shared tokenizer with NextGen enabling distillation and routing transparency
5. Safety refusal behavior compatible with Quark's product-layer filters
6. Efficient KV-cache profile enabling high concurrency on modern accelerators

1.3 Non-goals

Origin deliberately excludes several capabilities: native multimodal understanding (image, audio, video input—NextGen), context windows beyond 128K tokens (NextGen targets 1M), and maximum performance on competition mathematics or graduate-level scientific benchmarks. Scoping these out kept the training budget tractable and the serving path fast.

2. Architecture

2.1 Backbone

Origin is a decoder-only autoregressive transformer with the following design choices:

• Pre-normalization (RMSNorm) before attention and FFN sub-layers for stable training at scale
• SwiGLU feed-forward networks replacing standard ReLU or GELU variants; empirically stronger per-parameter on reasoning benchmarks
• Rotary positional embeddings (RoPE) for improved length generalization; extended during long-context mid-training
• No bias terms in linear projections for parameter efficiency

2.2 Grouped-query attention

Full multi-head attention (MHA) scales KV-cache memory linearly with both heads and sequence length, severely limiting concurrency at serving time. Origin adopts grouped-query attention (GQA) with a head grouping factor chosen to recover ≥98% of MHA quality on internal probe suites while reducing KV memory proportionally—see the GQA study for full ablations. This single architectural choice is among the most consequential for Origin's cost profile.

2.3 Context window

The production context limit is 128K tokens, enabled by two mechanisms: (1) long-context mid-training on sequences up to 128K with RoPE frequency scaling, and (2) needle-in-a-haystack validation at 32K and 128K depth. The 128K window is not uniformly reliable—see Section 5.3 and Limitations.

2.4 Structured decoding

Origin ships with constrained decoding support for JSON and XML schemas. Tool-calling pipelines in Quark route through a schema-conditioned decoding path, reducing exact-valid JSON failures from free-form generation baselines. Full methodology and metrics are in the structured output constraints note.

2.5 Parameter scale

We do not publish exact parameter counts at this time. Origin is in the multi-billion dense parameter class, smaller than NextGen's sparse-MoE total-parameter count but comparable in active-per-token compute.

3. Data and training

3.1 Pretraining corpus

The pretraining mixture was assembled via the Data Quality Filtering v1 pipeline: filtered web text, code repositories (multiple languages), books, academic text, and technical documentation. Key choices:

• Code density elevated relative to general web models—justified by early scaling observations showing improved HumanEval without proportional MMLU regression at moderate fractions
• Near-deduplication (MinHash) reduces repetition and memorization risk
• Domain balancing caps overrepresented sources (e.g., a single large forum) and upweights high-quality long-form reasoning documents
• Toxicity and CSAM screens applied before finalization; PII heuristics reduce exposure of personal information

Dataset mixture ablations (code in 10%, 25%, 40%) are reported in the dataset mixture ablations note; 25% code became Origin's production prior.

3.2 Training stages

Origin was trained in five sequential stages:

The long-context mid-training stage and its needle evaluation protocol are documented in the long-context mid-training note. Preference optimization methodology is in Preference Optimization v1.

3.3 Optimizer

AdamW with cosine learning-rate schedule and gradient clipping. We experimented with linear cooldown phases; cosine remains the default. Batch sizes scale over training to improve sample efficiency.

3.4 Data we do not train on

Private Quark conversations are not used in training without explicit opt-in programs, legal review, and user notification. This policy is stated in the founding research charter.

4. Evaluation protocol

All published numbers use IdonAI's standardized evaluation harness (described in Eval Harness Standardization): fixed prompt templates, temperature 0 for greedy academic suites, logged few-shot counts, and pinned harness commit hashes. Third-party leaderboards may differ by prompt phrasing, few-shot framing, and post-processing—treat external comparisons as band-matching rather than exact.

Contamination risk: best-effort test-set isolation during SFT mixing; we do not claim perfect contamination control across the full pretraining corpus.

5. Results

5.1 Core academic benchmarks

Origin performs at the top of the Flash-class band across all suites. GPQA Diamond at 48.1% is the weakest relative position—graduate science reasoning is where NextGen's deeper compute budget matters most.

5.2 Product-oriented probes

5.3 Long-context (≤128K)

128K context is production-ready for single-fact retrieval. Multi-hop reasoning at maximum depth is weaker; users should prefer structured prompts with explicit indexing for long documents.

5.4 Latency characteristics

Under controlled Origin serving canaries (see Origin Serving Canary):

Absolute millisecond values are omitted—they depend on hardware generation and cluster load. The key point is that Origin consistently meets interactive latency SLOs that NextGen cannot match at scale.

6. Serving architecture

Origin is deployed under:

• Continuous batching with paged KV cache — eliminates KV fragmentation, maximizes GPU utilization
• Chunked prefill — prevents long-context prefill from starving short interactive turns; policy details in Continuous Batching Notes
• Prefix caching — system prompts shared across Quark sessions are cached; reduces redundant prefill computation by ~60% on standard Quark traffic
• Speculative decoding (experimental) — draft token generation on small auxiliary heads; not yet production-enabled; benchmarked in internal canaries
• Canary promotion gates — every checkpoint passes origin-serving canaries before Quark traffic is shifted (shadow → 1% → 10% → 100%)

Origin is the preferred model for high-QPS Quark consumer traffic. NextGen serves deep-reasoning and multimodal turns with separate capacity planning.

7. Distillation from NextGen

Selective distillation from NextGen improved Origin on coding and short-form reasoning without meaningful latency regression. Methodology from Distilling NextGen into Origin:

What we distilled: Coding patch patterns, structured-output formats, short-form chain-of-thought reasoning on math word problems.

What we did not distill: Long-context synthesis (hurts latency/memory), multimodal trace patterns (not applicable to text-only Origin), and NextGen's full reasoning chains (too slow to decode).

Distillation binds the model family together economically: NextGen continues to push the capability frontier; Origin industrializes what NextGen has already proven.

8. Safety interfaces

Origin's safety posture combines model-level and system-level mechanisms:

1. SFT refusal demonstrations — narrow policy v0 refusals introduced during instruction tuning to establish refusal fluency without over-refusal
2. Preference optimization — honesty and harmlessness axes; anti-sycophancy preferences to prevent agreement collapse
3. Quark product-layer filters — output filters for disallowed categories, prompt injection detection, and enterprise policy overlays; these sit outside Origin and are not part of this report
4. Red-team gates — every Origin release candidate passes Red-Team Pack v1 (later v2/v3) before promotion; any full-fail blocks release

Model-only safety is insufficient. The 2025 safety evaluation card shows Quark + filters achieving 97.4% safe completion on jailbreak pack v3; Origin model-only achieves lower rates. System-level safety design is documented in Responsible AI & Safety Research.

Over-refusal on benign hard prompts was reduced from ~14% to 8.9% aggregate through the program described in Over-Refusal Reduction. Dual dashboards (jailbreak + over-refusal) are required to avoid safety-helpfulness tradeoff confusion.

9. Ablations

The following ablations were run on Origin-scale models during architecture selection:

The GQA and long-context mid-training decisions are non-negotiable for Origin's production target. The code-heavy pretraining fraction is the most impactful single data decision.

10. Limitations

Hallucinations. Origin generates plausible but incorrect facts, citations, and code identifiers. Citation fabrication under pressure is not solved; see calibration and abstention work.

Hard science gap. GPQA Diamond at 48.1% situates Origin below frontier reasoning models. Graduate-level biology, chemistry, and physics questions should route to NextGen.

Language coverage. English dominates pretraining and fine-tuning. Non-English quality degrades, particularly for low-resource languages. We do not publish per-language breakdowns at this release.

128K ≠ uniform quality. Multi-hop reasoning at full 128K context is meaningfully weaker than retrieval at the same length. Treating the context window as a reliable reasoning surface for complex tasks at maximum depth is incorrect.

No native multimodal input. Image, audio, and video inputs are not supported. Multimodal Quark turns are handled by NextGen.

Structured output edge cases. JSON exact-valid rate is 92.1%; the remaining 7.9% are real failures concentrated on deeply nested schemas and long payloads. Downstream tools must validate model output independently.

Adversarial robustness. Origin is not robust to sophisticated multi-turn jailbreaks without the Quark product layer. Model-only deployment for safety-sensitive applications is unsupported.

11. Conclusion

Origin is IdonAI's production fast model: Flash-class quality, interactive serving characteristics, and the default path for high-QPS Quark traffic. It achieves 81.4% MMLU, 78.9% HumanEval, 74.2% MATH, 91.8% GSM8K, and 48.1% GPQA Diamond with a 128K context window.

The architectural bets—GQA, long-context mid-training, shared tokenizer with NextGen, and constrained decoding—were each validated by ablation. Selective distillation from NextGen provides a path to incrementally improve Origin without full retraining cycles.

NextGen extends the family for maximum reasoning depth and multimodal capability. Together they instantiate the two-tier vision in the Founding Research Charter. Quark routes between them based on task complexity, latency requirements, and cost.

Product: Quark · Model page: /models/origin · NextGen report: /research/introducing-nextgen

References

Internal technical reports (unpublished except where linked):

• IdonAI Research (2022). Founding Research Charter. /research/founding-research-charter
• IdonAI Research (2022). Early Scaling Observations on Small Dense Transformers. /research/early-scaling-observations
• IdonAI Research (2023). Data Quality Filtering v1. /research/data-quality-filtering-v1
• IdonAI Research (2023). Tokenizer Design Notes. /research/tokenizer-design-notes
• IdonAI Research (2023). Grouped-Query Attention for Interactive Serving. /research/gqa-kv-cache-efficiency
• IdonAI Research (2023). Instruction Tuning Pilot. /research/instruction-tuning-pilot
• IdonAI Infrastructure (2023). Origin Serving Canaries. /research/origin-serving-canary
• IdonAI Research (2024). Preference Optimization v1. /research/preference-optimization-v1
• IdonAI Research (2024). Long-Context Mid-Training. /research/long-context-midtraining
• IdonAI Research (2024). Dataset Mixture Ablations. /research/dataset-mixture-ablations-2024
• IdonAI Research (2025). Distilling NextGen into Origin. /research/distillation-origin-from-nextgen
• IdonAI Research (2025). Structured Output Constraints. /research/structured-output-constraints
• IdonAI Safety (2024). Red-Team Pack v1. /research/red-team-pack-v1
• IdonAI Safety (2025). Over-Refusal Reduction. /research/over-refusal-reduction
• Shazeer (2019). Fast Transformer Decoding: One Write-Head is All You Need.
• Ainslie et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.