Early Scaling Observations on Small Dense Transformers

Abstract

We report early scaling experiments on dense decoder-only transformers trained on a filtered English-plus-code mixture. The experiments were not intended to produce a production model—they were intended to calibrate intuitions about loss-vs-compute, data mixture sensitivity, and serving constraints before committing to Origin-scale training. We cover model sizes from 120M to 1.3B parameters, evaluate on cross-entropy loss, HumanEval-lite, GSM8K-lite, and instruction-following probes, and identify three decisions that carried directly into Origin: data filtering, grouped-query attention, and a Flash-tier production target rather than a single maximal model.

1. Background and Motivation

By mid-2022 the public scaling law literature (Kaplan et al., 2020; Hoffmann et al., 2022) had established that dense transformer loss improved smoothly with compute. What the literature did not address well was how downstream task performance—particularly code generation and instruction following—translated from pretraining loss, or how serving constraints should interact with architecture search.

IdonAI's founding charter required that architecture proposals answer concrete serving questions: Can this be batched? What is the KV-cache cost at interactive load? We designed these early experiments to answer those questions at small scale before they became expensive.

2. Experimental Setup

2.1 Model configurations

All models share architecture choices: pre-RMSNorm, SwiGLU FFNs, rotary positional embeddings, no bias terms in linear projections. Context window: 2K initially, extended to 4K in mid-run ablations.

2.2 Training setup

• Optimizer: AdamW, cosine schedule, gradient clipping
• Tokenizer: Byte-level BPE precursor (later finalized for Origin/NextGen—see Tokenizer Design Notes)
• Data: English web + code, partially filtered (pre-v1 pipeline)
• Compute: Internal GPU cluster; experiments run at varying FLOPs budgets to trace scaling curves

2.3 Evaluation suite

3. Results

3.1 Smooth loss scaling

Cross-entropy improved predictably with compute across all three sizes following a power-law trend consistent with published results. No evidence of discontinuous "emergence" on our small probes—gains were gradual. This result is unsurprising but important to verify on our specific data mixture before committing to larger runs.

Loss reduction tracks roughly as expected from compute-optimal theory. Code BPB improves faster than English BPB with scale, consistent with code having more structured, learnable patterns.

3.2 Code vs. prose mixture sensitivity

We ran paired experiments varying code fraction (10%, 25%, 40%) at fixed 350M scale and fixed compute:

Higher code fraction consistently improved HumanEval-lite but degraded long-form writing rubrics. The 25% working point became the starting prior for Origin's production mixture (validated at full scale in Dataset Mixture Ablations).

3.3 Serving reality check

We profiled S-1.3B under a prototype continuous batching implementation across batch sizes 1–32:

At S-1.3B scale with full multi-head attention, GPU memory constrained concurrency significantly. GQA prototypes recovered concurrency at equivalent quality—see Section 4.2.

3.4 Instruction following and unfiltered data

Unfiltered web pretraining produced models that scored 2.4/5 on instruction-follow probes; filtered versions scored 3.1/5 at matched compute. The delta motivated full investment in the Data Quality Filtering v1 pipeline before any Origin-scale run.

4. Key Findings

4.1 No discontinuous emergence at this scale

Instruction-following, code generation, and math reasoning all improved gradually with scale on our suite. We treat emergence claims with appropriate skepticism and attribute apparent discontinuities in published work largely to measurement choices.

4.2 GQA necessary for serving viability

Full MHA made S-1.3B difficult to batch efficiently. We ran prototype GQA experiments with group factors of 2, 4, and 8 and found group factor 4–8 recovered over 97% of MHA quality on all probes while cutting KV memory by 4–8×. This was a non-negotiable finding for Origin's serving target; the GQA study contains the full production evaluation.

4.3 Width vs. depth at fixed FLOPs

At fixed compute budget, depth-balanced models (equal increases in both depth and width) outperformed width-only or depth-only scaling:

The depth-balanced result held consistently; we adopted this as the default architectural scaling principle.

5. Negative Results

Several hypotheses did not pan out:

• Naive width-only scaling consistently underperformed depth-balanced schedules at equivalent FLOPs—reported above
• Unfiltered web at 40% code introduced instruction-follow regressions from SEO and auto-generated content that offset the coding gains
• 2K context at all pretraining made downstream long-context fine-tuning significantly more expensive, motivating the mid-training curriculum later used in Origin (see Long-Context Mid-Training)
• Very long cosine warmup (>5% of training) did not improve final loss and increased schedule sensitivity

6. Implications for Origin and Beyond

These 2022 experiments directly shaped three Origin design decisions:

1. Data filtering investment — the gap between filtered and unfiltered instruction-follow performance justified the engineering cost of the full Data Quality Filtering v1 pipeline
2. GQA adoption — concurrency results made GQA non-optional for Flash-tier serving; confirmed in production by the GQA study
3. Flash-tier target — the compute requirements for a single maximal model at production QPS are prohibitive; a two-tier architecture (Origin + NextGen) with routing was validated as the economically sensible path, as described in the Founding Research Charter

The mixture sensitivity results also shaped NextGen's different data balance (heavier on long-form reasoning and multimodal pairs).

7. Limitations

These experiments operated at toy scale. Results from 120M–1.3B models do not transfer reliably to all phenomena at 10B+ scale. The evaluation suite was internal and not comparable to 2025–2026 standardized harnesses. English-heavy data creates English-heavy conclusions; multilingual scaling was not studied here.

8. Conclusion

Early scaling experiments are cheap and high-leverage. The key findings—smooth loss scaling, code mixture sensitivity, GQA necessity for serving, and depth-balanced architecture preference—each paid dividends in the Origin training run. The most valuable results were the negative ones: they prevented expensive mistakes at scale.

See Origin Technical Report for how these lessons were applied in production.