Safety Research — Red-Teaming, Alignment, and Deployment Guardrails

Abstract

This report describes IdonAI’s safety research and deployment practice for the Origin and NextGen models and the Quark assistant. We outline threat models, data and training mitigations, automated and human red-teaming, quantitative safety evaluations, product-layer guardrails, incident response, and known limitations. Safety is treated as a system property: model behavior, product policy, monitoring, and organizational process must work together.

We do not claim perfect safety. We claim a repeatable engineering process with published metrics, clear ownership, and conservative defaults in high-risk domains.

1. Introduction

Capability improvements expand both beneficial use and misuse. IdonAI’s safety program aims to:

1. Prevent severe harm categories by default (policy + model + filters).
2. Detect novel attacks through continuous red-teaming.
3. Respond to incidents with defined severity levels and SLAs.
4. Measure regressions so capability launches cannot silently weaken safety.
5. Communicate residual risk to users and enterprise customers.

This document is the public counterpart to internal safety runbooks. Product-specific routing and honesty metrics also appear in the Quark and NextGen technical reports.

2. Threat model

2.1 Assets

• Model weights and proprietary eval sets
• User conversation content and account integrity
• IdonAI infrastructure and API credentials (when generally available)
• Reputation and downstream user safety

2.2 Adversaries

2.3 Harm categories (policy summary)

We maintain a tiered policy. Disallowed (deny actionable assistance): child sexual exploitation, violent crimes and terrorism facilitation, biological / chemical weapons assistance, high-risk cyber offense, scams/fraud playbooks, and other categories aligned with widely adopted industry policies. Restricted: medical/legal/financial advice (informational only; escalate to professionals), self-harm (encourage help resources), and dual-use technical topics (high-level public knowledge only).

Exact policy text evolves; enterprise contracts may add domain-specific constraints.

3. Safety in the training stack

3.1 Data filtering

Pretraining and SFT data pass through:

• CSAM and severe violence classifiers / hash matching where applicable
• PII detection and reduction heuristics
• Deduplication to reduce memorization of rare personal strings
• Domain balancing to avoid over-weighting toxic niches

3.2 Refusal and redirection SFT

Supervised demonstrations teach:

• Clear refusals with brief rationale
• Safe partial answers when only high-level information is appropriate
• Escalation language for self-harm and crisis situations

3.3 Preference optimization for harmlessness

Preference pairs explicitly include safety axes (harmlessness vs over-refusal). We track over-refusal as a regression metric so models do not become useless on benign edge cases (e.g., security education, fiction, medical information).

3.4 What training does not solve

Training-time mitigations fail open under novel jailbreaks, distribution shift, and powerful adaptive attackers. Hence product filters, monitoring, and iterative red-teaming remain mandatory.

4. Red-teaming program

4.1 Cadence

4.2 Attack classes

1. Jailbreaks — roleplay, obfuscation, encoding, multi-turn priming
2. Policy ambiguity — borderline dual-use requests
3. Prompt injection — malicious instructions in retrieved or pasted content
4. Tool abuse — if/when tools can fetch or execute (defense in depth)
5. Data exfiltration — coaxing system prompts or training snippets
6. Sycophancy / unsafe compliance — user pressure to violate policy

4.3 Scoring

Attacks are scored on a rubric: blocked, partial leak, full fail. Full fails block release or trigger mitigations before promotion to Quark production.

5. Quantitative evaluations

Figures below are from internal suites on the Quark-production stack (models + safety layer) unless noted as model-only.

5.1 Disallowed-category suites

5.2 Jailbreak packs

5.3 Prompt injection

5.4 Over-refusal

We actively tune to reduce over-refusal without sacrificing disallowed-category performance.

5.5 Honesty-related safety

Fabricated authoritative citations and false medical certainty are tracked as safety-adjacent harms. See Quark report honesty tables; mitigations include refusal to invent sources and uncertainty language.

6. Deployment guardrails (Quark and platform)

6.1 Layered controls

Request → Input filters → Model (Origin/NextGen) → Output filters → Client
                │                                      │
                └────────── Monitoring / rate limits ──┘

1. Input screening — regex/classifier hits for severe categories; anomaly scores.
2. Model policy — trained refusals.
3. Output filtering — secondary classifiers on completions; re-roll or refuse on hit.
4. Rate limits — per account / IP; escalating challenges on abuse.
5. Audit logs — security-relevant events with retention controls.

6.2 Enterprise

Enterprise deployments may add:

• VPC / dedicated capacity
• Custom retention and DLP integrations
• SSO and role-based admin
• Domain-specific policy packs

Contact enterprise@idonai.com.

6.3 Status and transparency

Platform status is published at /status. Safety incidents meeting public thresholds will be disclosed with timelines and mitigations.

7. Organizational process

• Launch reviews — capability increases require safety sign-off.
• Kill switches — ability to disable routes, tighten filters, or pin model versions.
• Vendor / contractor access — least privilege, NDAs, audited data access.
• Employee access to production conversations — restricted, justified, logged.

8. Limitations and residual risk

1. Adaptive attackers will find new jailbreaks; metrics lag novel attacks.
2. Classifier errors cause both misses and over-refusals.
3. Multimodal inputs create new steganographic and indirect injection paths.
4. Long context can bury malicious instructions far from the user turn.
5. Evaluation coverage is incomplete for all languages and dialects.
6. Open-ended dual-use knowledge cannot be fully removed from pretrained models.

Users must maintain human oversight for high-stakes domains.

9. Related IdonAI publications

• Quark technical report — assistant routing, honesty, product safety metrics
• NextGen technical report — model capabilities and training-time safety interfaces
• Trust Center: /trust

10. Conclusion

IdonAI safety work combines training-time mitigations, continuous red-teaming, layered product guardrails, and explicit measurement of both under-refusal and over-refusal. Origin, NextGen, and Quark ship under this program, with conservative defaults for high-risk categories and an engineering process designed to improve under attack.

Contact: security@idonai.com for vulnerability reports · support@idonai.com for product safety questions

References (selected)

1. Weidinger et al., “Ethical and social risks of harm from Language Models,” 2021.
2. Ganguli et al., “Red Teaming Language Models to Reduce Harms,” 2022.
3. Bai et al., “Constitutional AI,” 2022.
4. OpenAI, “GPT-4 System Card,” 2023.
5. Perez et al., “Red Teaming Language Models with Language Models,” 2022.
6. Greshake et al., “Not What You’ve Signed Up For” (indirect prompt injection), 2023.
7. IdonAI, Quark and NextGen technical reports, 2025.