NextGen — Technical Report on Architecture, Training, and Evaluation

Abstract

We present NextGen, IdonAI’s flagship large language model for deep reasoning, long-context understanding, and multimodal inputs, together with Origin, our latency-optimized companion model for high-throughput interactive workloads. NextGen targets Pro-class quality on standard academic benchmarks while remaining deployable in the Quark assistant and (when API keys ship) via a unified chat-completions API. Origin targets Flash-class responsiveness for the majority of conversational and coding turns.

This report covers model family positioning, architecture (dense + mixture-of-experts components), data and training recipe, evaluation on MMLU, GPQA Diamond, HumanEval, MATH, and GSM8K, long-context and multimodal probes, ablations, inference stack, safety interfaces, limitations, and future work. Numbers are IdonAI internal evaluations under a fixed protocol; they are intended to situate Origin/NextGen at contemporary frontier-assistant quality levels rather than claim leaderboard supremacy under every third-party harness.

TL;DR results (NextGen / Origin):

*Interactive decode-path latency under optimized serving; end-to-end TTFT also depends on queueing and prompt length (see §8).

1. Introduction

1.1 Motivation

Frontier assistants require both depth (multi-step reasoning, long documents, multimodal comprehension) and speed (sub-second interactivity for everyday chat). A single model rarely optimizes both. IdonAI therefore ships a two-model family:

• Origin — efficient, text-first, optimized for throughput and cost.
• NextGen — higher capacity, multimodal, long-context, optimized for hard tasks.

Quark routes between them; developers will call the same model IDs through the API when keys are generally available.

1.2 Design principles

1. Capability where it matters. Spend parameters and compute on reasoning and context, not on vanity parameter counts.
2. Serving realism. Architecture choices must map to batchable inference, KV-cache efficiency, and predictable latency.
3. Measurable honesty. Publish evals, ablations, and failure modes—not only marketing maxima.
4. Safety as a system property. Model behavior + product filters + monitoring (see safety report).
5. Parity with Quark. Product and API share model identifiers and decoding semantics.

1.3 Contributions

• A complete public description of the Origin / NextGen family positioning and eval protocol.
• Benchmark tables situating both models at Flash / Pro competitive quality bands.
• Architecture and training overview sufficient for sophisticated users to reason about strengths and limits.
• Ablations on context length, MoE routing, and preference optimization.
• Explicit limitations and non-goals.

2. Model family

2.1 Origin

Intended use: real-time chat, short-to-medium coding assistance, classification/extraction, tool-calling with tight latency budgets.

Non-goals: maximum performance on GPQA-style scientific reasoning; native high-resolution multimodal understanding (text-first).

2.2 NextGen

Intended use: complex multi-step reasoning, long documents (up to 1M tokens), multimodal inputs (text, vision, audio features), agentic planning loops.

Non-goals: being the cheapest model for every token; replacing domain experts in regulated decisions.

2.3 Versioning

We refer to the production snapshots evaluated in this report as Origin and NextGen (2025 H1 evaluation cut). Future snapshots will append dated model cards; Quark may pin or auto-update within a major line.

3. Architecture

3.1 Shared transformer backbone

Both models use decoder-only transformers with:

• Pre-normalization and rotary positional embeddings (RoPE) with long-context extrapolation strategies for NextGen
• Grouped-query attention (GQA) for KV-cache efficiency
• SwiGLU-style feed-forward blocks in dense layers

3.2 NextGen mixture-of-experts (MoE)

NextGen replaces a subset of FFN blocks with MoE layers:

• Sparse activation — a router selects top-k experts per token
• Load-balancing loss — encourages uniform expert utilization during training
• Shared experts — a small always-on expert path for stability on common tokens

MoE increases total parameter count while keeping active parameters per token closer to a dense mid-size model, improving quality per watt at serving time when batching is healthy.

3.3 Multimodal NextGen

Vision and audio encoders project into the language model’s embedding space:

• Vision: tiled / multi-resolution encoding for diagrams, UI screenshots, and documents
• Audio: features for speech understanding tasks (transcription-conditioned reasoning)

Cross-modal alignment is trained with captioning, VQA-style objectives, and interleaved document tasks. Origin remains text-only in the current cut.

3.4 Context length

Long-context quality is validated with needle-in-a-haystack, variable-distance coreference, and multi-document synthesis tasks (§6.3).

3.5 Structured outputs

Both models support constrained decoding for JSON/XML schemas used by tools and enterprise workflows. Quark exposes structured mode for developer-facing tasks; schema adherence is measured in §6.4.

4. Data and training

4.1 Pretraining data (overview)

We train on a curated mixture of:

• Public web text with quality and safety filtering
• Code repositories and technical documentation
• Books and long-form educational material (licensed / permitted)
• Scientific and mathematical corpora
• Multimodal pairs for NextGen (image–text, audio–text)

Filtering includes language identification, toxicity / PII heuristics, deduplication (exact and near-duplicate), and domain balancing to reduce over-representation of low-quality SEO text.

We do not train on private Quark user conversations without explicit opt-in programs and legal review.

4.2 Tokenizer

A byte-level BPE-style tokenizer shared across Origin and NextGen for vocabulary compatibility in routing and distillation experiments. Code-heavy and non-English scripts are represented without excessive fragmentation relative to prior internal baselines.

4.3 Training stages

1. Pretraining — next-token prediction on the filtered mixture.
2. Long-context mid-training (esp. NextGen) — extended sequences, document packing, and position extrapolation curricula.
3. Multimodal mid-training (NextGen) — interleaved modalities.
4. Supervised fine-tuning (SFT) — instruction following, tool use formats, refusal demonstrations.
5. Preference optimization — reward modeling + policy optimization with KL control (shared philosophy with Quark; model-level vs product-level preference sets differ).

4.4 Optimization

AdamW-family optimizers, cosine / WSD-style learning-rate schedules, gradient clipping, and mixed-precision training. MoE training adds auxiliary load-balancing terms. We monitor loss spikes, expert collapse, and eval canaries throughout.

4.5 Distillation and family coupling

Origin benefits from distillation / targeted SFT signals derived from NextGen on selected tasks, improving Flash-tier quality without paying NextGen serving cost on every query. Distillation is task-selective to avoid regressing Origin latency.

5. Evaluation protocol

5.1 Principles

• Fixed prompts and decoding for academic suites (temperature 0 unless the harness requires sampling).
• No test-set training — eval data is held out from SFT mixtures to the best of our filtering ability.
• Report both models under the same harness versions.
• Separate product evals for Quark (see Quark report).

5.2 Harness notes

MMLU, GPQA Diamond, HumanEval, MATH, and GSM8K follow widely used formulations. Absolute numbers vary by prompt template, few-shot count, and tool use; our tables use a consistent internal template set. Treat comparisons to external vendor blogs as approximate band matching, not identical harness replication.

6. Results

6.1 Core academic benchmarks

Interpretation. Origin sits in a strong Flash-class band: excellent for interactive coding and everyday reasoning. NextGen sits in a Pro-class band: clear gains on GPQA and MATH, where deep multi-step reasoning dominates.

6.2 Coding beyond HumanEval

6.3 Long context

NextGen’s 1M window is usable for large corpora, but quality still degrades gracefully with distance and distractors—users should structure prompts (indices, section headers) for best results.

6.4 Structured output

6.5 Multimodal (NextGen)

Multimodal evals are internal; we plan public reproducible suites as the stack stabilizes.

6.6 Calibration and honesty probes

Preference optimization improves these metrics but does not eliminate confident errors.

7. Ablations

7.1 MoE vs dense (NextGen-scale compute matched)

MoE improves MATH/GPQA more than MMLU, consistent with allocating capacity to harder reasoning tokens.

7.2 Long-context mid-training

Without long-context mid-training, NextGen @ 128K needle accuracy drops ~6–9 points and @ 500K becomes unreliable. Mid-training is necessary for production 1M claims.

7.3 Preference optimization

7.4 Distillation into Origin

Selective distillation from NextGen yields +2.1 HumanEval and +1.4 MMLU on Origin with under 3% regression in tokens/sec under the same serving config.

8. Inference and serving

8.1 Stack highlights

• Continuous batching and paged KV cache
• Speculative decoding experiments for Origin (optional)
• Prefix caching for repeated system prompts (Quark)
• Per-tenant isolation options for enterprise deployments

8.2 Latency notes

Microbenchmark “decode path” latencies (~18–20ms class) describe optimized token steps under load-controlled canaries. Real time-to-first-token includes tokenization, queueing, and prompt prefill—which dominate for long prompts. Quark reports blended TTFT in its system report.

8.3 Throughput

Origin is preferred for high-QPS chat. NextGen is capacity-planned for lower QPS, higher value tasks. Router policies in Quark enforce this economically.

9. Safety interfaces

Models ship with:

• Refusal behaviors for disallowed categories (SFT + preference)
• Compatibility with product-level filters in Quark
• Logging hooks for abuse analysis

Model-level safety is necessary but not sufficient; see Safety Research for red-teaming, jailbreaks, and deployment guardrails. We do not claim 100% jailbreak immunity.

10. Environmental and compute considerations

Training frontier models consumes substantial energy. We mitigate via MoE active-parameter efficiency, distillation into Origin for common traffic, and routing so most Quark turns avoid NextGen. We will publish more detailed energy accounting as measurement pipelines mature.

11. Limitations

1. Harness variance — published numbers may differ from third-party leaderboards using different prompts.
2. Hallucinations — especially citations, niche facts, and rare APIs.
3. Multimodal — still weaker than specialized vision systems on fine-grained perception.
4. Long context — 1M tokens is not uniformly high quality across all positions and task types.
5. Languages — strongest in English; other languages vary by resource density.
6. Tool use — agentic reliability depends on orchestrators; the base model is not a complete agent.
7. API access — keys under development; evaluate interactively via Quark today.

12. Broader impacts

Higher capability increases both beneficial automation and misuse potential. We couple releases with safety evaluations, usage policies, and monitoring. Dual-use domains receive heightened scrutiny and conservative defaults.

13. Future work

• Stronger calibrated uncertainty and citation grounding
• Public reproducible long-context and multimodal leaderboards
• Improved Origin quality via ongoing distillation without latency loss
• Formal verification-style checks for structured outputs in high-stakes schemas
• General availability of API keys and usage analytics

14. Conclusion

Origin and NextGen form a deliberate two-tier family: Flash-class interactivity and Pro-class reasoning. NextGen’s MoE architecture, long-context mid-training, and preference optimization yield strong results on MMLU, GPQA, HumanEval, MATH, and GSM8K, while Origin remains the workhorse for responsive Quark traffic. Together with Quark’s router, the family delivers frontier-assistant capability with practical serving economics.

Models: Origin · NextGen
Product: Quark · Quark technical report
Safety: Safety Research

Acknowledgments

IdonAI Research, Infrastructure, and Safety teams—and evaluation partners who maintained harness integrity under NDA.

References (selected)

1. Vaswani et al., “Attention Is All You Need,” 2017.
2. Shazeer et al., “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer,” 2017.
3. Fedus et al., “Switch Transformers,” 2021.
4. Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding,” 2021.
5. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models,” 2023.
6. OpenAI, “GPT-4 Technical Report,” 2023.
7. Gemini Team, Google DeepMind, Gemini technical reports, 2023–2025.
8. Rein et al., “GPQA: A Graduate-Level Google-Proof Q&A Benchmark,” 2023.
9. Chen et al., “Evaluating Large Language Models Trained on Code” (HumanEval), 2021.
10. Hendrycks et al., “Measuring Massive Multitask Language Understanding” (MMLU), 2021.
11. Hendrycks et al., “Measuring Mathematical Problem Solving With the MATH Dataset,” 2021.
12. Cobbe et al., “Training Verifiers to Solve Math Word Problems” (GSM8K), 2021.
13. IdonAI, “Quark — A Conversational Assistant Built on Origin and NextGen,” 2025.
14. IdonAI, “Safety Research,” 2025.