OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support

"OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support"

thumbnail: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/oncoagent-thumbnail.png authors:- user: oncoagent-research tags:

• oncology
• multi-agent
• LangGraph
• RAG
• QLoRA
• AMD
• open-source
• clinical-ai
• healthcare

OncoAgent: A Dual-Tier Multi-Agent Framework for Privacy-Preserving Oncology Clinical Decision Support

Technical preprint · May 2026 · OncoAgent Research Group

We present OncoAgent, an open-source, privacy-preserving clinical decision support system for oncology. OncoAgent combines a dual-tier fine-tuned LLM architecture with a state-of-the-art multi-agent LangGraph topology, a four-stage Corrective RAG pipeline over 70+ physician-grade NCCN and ESMO guidelines, and a three-layer reflexion safety validator enforcing a strict Zero-PHI policy.

The system routes clinical queries through an additive complexity scorer to either a 9B parameter speed-optimised model (Tier 1) or a 27B deep-reasoning model (Tier 2), both fine-tuned via QLoRA on a corpus of 266,854 real and synthetically generated oncological cases using the Unsloth framework on AMD Instinct MI300X hardware (192 GB HBM3).

Sequence packing on MI300X enabled full-dataset fine-tuning in approximately 50 minutes — a 56× throughput acceleration over API-based generation. Post-fix, CRAG document grading achieved a 100% success rate with a mean RAG confidence score of 2.3+. The complete system is 100% open source and deployable on-premises, eliminating proprietary cloud API dependency and preserving patient data sovereignty.

Keywords: clinical decision support, oncology AI, multi-agent systems, retrieval-augmented generation, QLoRA, AMD ROCm, open-source healthcare AI, HITL safety, LangGraph, Corrective RAG

Oncology is one of the most information-dense and cognitively demanding domains in clinical medicine. The volume, heterogeneity, and rapid evolution of evidence-based guidelines — from the National Comprehensive Cancer Network (NCCN) to the European Society for Medical Oncology (ESMO) — create a persistent knowledge gap between published evidence and bedside practice.

AI-assisted clinical decision support systems hold transformative potential for closing this gap, yet most commercially available systems fail in three critical ways:

Hallucinated recommendationsnot grounded in validated guidelinesCloud API dependencythat precludes on-premises deployment in privacy-sensitive hospital environmentsMonolithic LLM architecturesprone to context saturation under complex multi-comorbidity presentations

OncoAgent is designed around three core principles:

**Architectural decomposition:**Clinical reasoning is decomposed across eight specialised LangGraph nodes, each with a bounded, auditable function.**Grounded generation:**All model outputs are anchored to a curated vector knowledge base through a four-stage retrieval pipeline with explicit relevance gating.**Hardware sovereignty:**The full inference and training stack runs natively on AMD Instinct MI300X using ROCm and open-source frameworks — enabling hospital deployment without data exfiltration.

Large language models have demonstrated significant promise in clinical NLP tasks including diagnostic coding, literature summarisation, and patient communication. Domain-specific fine-tuning approaches — exemplified by BioMedLM, Med-PaLM 2, and ClinicalBERT — consistently improve performance on medical benchmarks over general-purpose models. OncoAgent extends this line of work by targeting the specific subdomain of oncological triage and treatment pathway recommendation, where hallucination consequences are most severe.

Decomposed multi-agent systems have emerged as a principled approach to complex reasoning tasks. OncoAgent synthesises four canonical SOTA patterns:

Claude Code pattern— deterministic safety harnesses separated from LLM reasoningHermes Agent pattern— structured tool-calling with per-session memory isolationCorrective RAG(Shi et al., 2024) — document relevance grading and query reformulationReflexion(Shinn et al., 2023) — self-correcting generation via feedback-augmented retry loops

Standard bi-encoder retrieval is ill-suited for clinical domains where terminological precision is critical (e.g., "tyrosine kinase inhibitor" vs. "TKI"). OncoAgent implements a multi-stage pipeline with cross-encoder re-ranking, and integrates Hypothetical Document Embeddings (HyDE; Gao et al., 2022) to resolve medical synonym mismatches by projecting natural language queries into the guideline embedding space.

OncoAgent is implemented as a stateful directed graph using LangGraph. The system state is represented as an immutable AgentState

TypedDict containing 11 logical sections and approximately 30 typed keys. Each node appends to specific keys without mutating upstream data, preserving a complete audit trail.

The 8-node topology is:

Router → Ingestion → Corrective RAG → Specialist ↔ Critic → HITL Gate → Formatter → END
↓
Fallback → END

Key properties:

• 5 conditional edges
• 1 reflexion retry loop (max 2 iterations)
• 1 mandatory HITL interrupt for high-complexity or low-confidence outputs

Case complexity is quantified using a weighted additive model prior to specialist invocation:

S = w_cancer + w_stage + w_mutations + w_treatment

Where:

| Factor | Condition | Weight | |---|---|---| | Cancer type | Rare | +0.40 | | Cancer type | Unknown primary | +0.30 | | Stage | Stage IV | +0.25 | | Stage | Stage III | +0.15 | | Mutations | ≥2 identified | +0.30 | | Mutations | Single | +0.15 | | Prior treatment | Any keyword match | +0.10 |

Decision boundary: S ≥ 0.5

→ Tier 2 (Qwen 3.6-27B deep reasoning) · S < 0.5

→ Tier 1 (Qwen 3.5-9B speed triage)

Validation: A Stage IV pancreatic carcinoma case with KRAS + BRCA2 mutations correctly produced S = 0.80

, routing to Tier 2. ✅

Clinicians may also manually override the tier selection through the UI.

The CRAG node grades each retrieved document for clinical relevance before forwarding to the Specialist. Documents that fail binary relevance classification trigger automatic query reformulation (max 1 retry). This eliminates the primary hallucination source in RAG pipelines — retrieval of plausibly titled but semantically irrelevant content.

After migrating from Qwen 3.5 to Qwen 2.5 Instruct for the grading step, success rate improved from 0% → 100%, with RAG confidence score reaching 2.3+ on uterine cancer triage tests.

The Critic node runs a three-layer validation cascade before any output reaches the HITL gate:

Formatting check— validates structural compliance with the OncoCoT output schemaSafety check— deterministic rule-based scan for prohibited output patterns (absolute dosing without guideline citation, drug interaction omissions, etc.)LLM entailment check— verifies that the Specialist's recommendation is fully supported by the retrieved RAG context

On FAIL, the Critic's specific feedback is injected back into the Specialist context for a retry (max 2 iterations). Crucially, the Critic runs as deterministic code, not LLM-controlled logic — ensuring safety enforcement cannot be bypassed by adversarial prompting.

The HITL gate provides a mandatory clinician interrupt for all Tier 2 cases and any output where rag_confidence < 0.3

. A dedicated Fallback node catches unrecoverable failures and returns a clinically safe refusal — "Información no concluyente en las guías provistas"

— avoiding hallucinated alternatives under any failure mode.

The PatientMemoryStore

module assigns each patient session a unique thread_id

(format PT-XXXX

), passed as a configurable

parameter to LangGraph's native checkpointing system. This enforces strict per-patient memory isolation while enabling iterative multi-turn consultations within a session.

The knowledge base was constructed from 77 direct physician guideline PDFs identified by a concurrent web scraper that processed 138 NCCN detail pages in under 60 seconds. Text extraction used PyMuPDF

(fitz) for block-level structural parsing, preserving the semantic reading order of multi-column clinical layouts.

A regex-based sanitisation step strips institutional branding prior to ingestion. Patient-facing materials are excluded via heuristic filtering. The resulting corpus covers 70+ professional oncological guidelines across all major cancer types including HCC, NSCLC, breast, colorectal, and neuroendocrine tumours.

Standard general-purpose embedding models (e.g., all-MiniLM-L6-v2

) were rejected due to poor clinical terminology semantics. OncoAgent uses:

Embeddings:pritamdeka/S-PubMedBert-MS-MARCO

— fine-tuned on PubMed and MS-MARCO for asymmetric medical semantic searchVector store:LocalChromaDBpersistent index — zero-cloud, Zero-PHI compliant

| Stage | Component | Function | Configuration | |---|---|---|---|

1. Recall | PubMedBERT Bi-Encoder | Wide-net retrieval | top-15 candidates |
2. Distance Gate | Cosine Distance Filter | Anti-hallucination floor | threshold = 0.10 |
3. Re-Ranking | Cross-Encoder (MS-MARCO MiniLM) | Joint query-document relevance | top-5 returned |
4. Context Trimming | Character-Budget Limiter | Fit within LLM context window | max 6,000 chars |

Anti-Hallucination Policy: Any query failing Stage 2 returns "Información no concluyente en las guías provistas"

without invoking the Specialist. This guarantees zero hallucinated recommendations for out-of-domain clinical inputs.

Distance threshold calibration against the NCCN corpus established:

• Medical-query distances: ~0.06–0.09
• Out-of-domain distances: ~0.11–0.15
• Hard threshold: 0.10

An optional HyDE module generates a hypothetical guideline paragraph and uses it as the embedding anchor for Stage 1 retrieval, resolving synonym mismatches (e.g., "neoplasia pulmonar"

vs. "lung carcinoma"

).

| Source | Type | Samples | Notes | |---|---|---|---| | PMC-Patients | Real clinical cases | ~85,000 | PubMed Central patient reports | | Asclepius | Real clinical data | ~85,000 | Curated medical QA corpus | | OncoCoT Synthetic | Synthetic (Qwen 3.6-27B) | 96,941 | Generated on MI300X at ~6,800 cases/hr · rejection rate 0.65% | Total | — | 266,854 | 90/10 train/eval split · SHA-256 hashed · deduplicated |

All cases use ChatML template for Qwen compatibility. Thinking tokens were disabled (chat_template_kwargs: {enable_thinking: False}

) to prevent JSON parse corruption.

Both tiers use 4-bit NormalFloat4 (NF4) quantisation via BitsAndBytes, with LoRA adapters targeting all major projection modules: q_proj

, k_proj

, v_proj

, o_proj

, gate_proj

, up_proj

, down_proj

.

| Parameter | Tier 1 (Qwen 3.5-9B) | Tier 2 (Qwen 3.6-27B) | |---|---|---| | Per-device batch size | 4 | 2 | | Gradient accumulation | 4 | 8 | | Effective batch size | 16 | 16 | | Learning rate | 2×10⁻⁴ | 1×10⁻⁴ | LoRA rank (r) | 16 | 32 | | Sequence packing | True, 2048 tokens | True, 2048 tokens | | Early stopping | Patience = 3 | Patience = 3 | | Quantisation | NF4 4-bit | NF4 4-bit |

The original HuggingFace transformers

• PEFT pipeline failed on the MI300X due to two independent issues:

• Tokenisation conflicts between trl

v0.24.0 strict EOS validation and theQwen3VLProcessor

wrapper - Insufficient VRAM headroom for target effective batch sizes under standard precision

Migration to Unsloth's FastLanguageModel resolved both simultaneously:

VRAM reduction:~60% drop in peak usage (from OOM to stable ~64 GB on 192 GB device)Training speed:~2× improvement to ~16 s/step at effective batch 16

AMD ROCm-specific adaptations required:

# 1. Pass inner tokenizer, not the Qwen3VLProcessor wrapper
trainer = SFTTrainer(tokenizer=model.get_tokenizer(), ...)
# 2. Prevent incompatible EOS injection
training_args = SFTConfig(eos_token=None, ...)
# 3. AMD-specific bitsandbytes for ROCm 6.2/gfx942
# pip install bitsandbytes --find-links <amd-continuous-release-wheel>
# 4. BF16 workaround (is_bf16_supported() returns False on ROCm despite hardware support)
training_args = TrainingArguments(fp16=True, ...)
# Final deployment uses native BF16:
model = AutoModelForCausalLM.from_pretrained(..., torch_dtype=torch.bfloat16)

Sequence packing via packing=True

in SFTConfig

concatenates multiple short clinical records into single 2048-token sequences, eliminating padding overhead and drastically reducing forward pass count.

The combined effect of Unsloth kernels and sequence packing on the MI300X enabled full-dataset fine-tuning of the 266,854-sample corpus in approximately 50 minutes — against an initial 5-hour estimate — representing roughly a 6× training time compression. GPU utilisation peaked at ~70%, with consistent throughput at ~11.3 s/iteration.

Checkpoint-1000 results:Tier 1 adapter trained for 1,339 steps · training loss ≈ 0.05 · adapter size 187 MB · verified against 11-file manifest includingadapter_model.safetensors

,adapter_config.json

, andtokenizer.json

.

The system supports adaptive inference routing: local BF16 inference via the LocalModelManager

singleton when ROCm is available, with graceful fallback to the Featherless.ai API for high availability.

A dedicated Zero-PHI redaction node runs as the first processing step in the Ingestion node, before any text reaches an LLM. It identifies and replaces Protected Health Information (patient names, dates of birth, MRN numbers, addresses, facility identifiers) with clinically neutral placeholders. The redacted representation is stored in AgentState

; the original text is discarded.

This ensures that no PHI reaches any downstream LLM call — local or remote — and satisfies HIPAA de-identification requirements by design rather than policy.

The system's safety guarantees are enforced at four independent layers. A failure at any single layer does not compromise the overall posture.

| Layer | Mechanism | Addresses | |---|---|---| L1: Retrieval Gate | Distance Gate (cosine threshold 0.10) | Out-of-domain hallucinations | L2: Confidence Gate | RAG confidence score < 0.3 → block | Low-quality retrieval grounding | L3: Reflexion Critic | Formatting + safety + LLM entailment (max 2 retries) | Unsupported or unsafe Specialist outputs | L4: HITL Gate | Mandatory clinician interrupt for Tier 2 / flagged cases | High-complexity cases requiring expert judgment |

Layers 1 and 2 operate at the retrieval layer. Layer 3 at the generation layer. Layer 4 at the deployment layer. All Layer 3 checks run as deterministic code — not LLM-controlled logic — preventing safety bypass via adversarial prompting.

The OncoAgent UI is implemented as a real-time streaming Gradio application in a ChatGPT-style conversational layout. It features:

**Left sidebar:Session controls, KPI tiles, evidence source tabsMain chat area:**Live agentic reasoning updates as each node completes

Real-time transparency is achieved via LangGraph's .stream(stream_mode="updates")

API, which emits {node_name: node_output}

dictionaries as each node completes. The UI maps each node to a human-readable clinical label (e.g., corrective_rag

→ "Retrieving NCCN/ESMO guidelines"), providing clinicians with full pipeline visibility.

The rag_confidence

score and retrieved source count are prominently surfaced, giving clinicians immediate visibility into the quality of guideline grounding behind each recommendation.

The interface was designed to WCAG 2.1 AA standards — Lucide-style inline SVG icons, slate-900/sky-500 dark theme, Figtree/Inter typography, prefers-reduced-motion

media query, all transitions capped at 200 ms.

| Component | Metric | Value | |---|---|---| Knowledge Base | Guidelines ingested | 70+ | | PDFs parsed | 138 in < 60 s | | | Index parsing errors | 0 | | CRAG Pipeline | Document grading success rate (post-fix) | 100% | | RAG confidence score (uterine cancer test) | 2.3+ (was 0.0 pre-fix) | | | Parallel grading latency (3–5 docs) | < 5 s | | Complexity Router | Stage IV pancreatic + KRAS + BRCA2 | Score = 0.80 → Tier 2 ✅ | Training (Tier 1, 9B) | Full 266k-sample training time | ~50 min (vs. 5 hr estimate) | | Steady-state throughput | ~11.3–16 s/step | | | GPU utilisation (MI300X) | ~70% peak | | | VRAM utilisation (Unsloth) | ~64 GB / 192 GB | | | Training loss at checkpoint-1000 | ~0.05 | | | Synthetic data throughput (MI300X vs. API) | 6,800 vs. 120 cases/hr (56× ↑) | | | Synthetic corpus rejection rate | 0.65% | | Graph Topology | Compiled nodes verified | 8 / 8 | | Module test suites passed | 6 / 6 | | UI | Browser timeouts during triage | 0 | | UI rendering latency | < 200 ms |

The ability to run the complete OncoAgent stack — training, inference, RAG, and UI — on a single AMD MI300X instance without cloud API dependencies is not merely an engineering convenience. In hospital environments governed by HIPAA (US), GDPR (EU), and equivalent national frameworks, the legal and ethical obligation to maintain data within controlled infrastructure is absolute. OncoAgent demonstrates that SOTA multi-agent clinical AI is achievable within this constraint.

The 56× synthetic data generation acceleration (from ~120 to ~6,800 cases/hr) and the ~6× training time compression together represent a significant practical contribution to the feasibility of domain-specific fine-tuning in time-constrained settings. These results suggest that AMD's CDNA3 architecture, when paired with Unsloth's Triton kernel optimisations and SFT sequence packing, may be substantially underutilised by standard HuggingFace training pipelines — and that the performance gap can be closed without changes to the underlying model architecture.

Several limitations warrant acknowledgement:

• The training corpus relies on approximately 36% synthetically generated cases. Clinical accuracy validation against board-certified oncologist judgments has not yet been performed at scale.
• The current knowledge base covers NCCN guidelines primarily in English; ESMO and non-English clinical corpora remain for future work.
• The Tier 1 adapter represents checkpoint-1000 of a potentially longer trajectory; full convergence and downstream clinical benchmark evaluation (MedQA, USMLE-style oncology subsets) are planned for subsequent releases.

OncoAgent establishes a complete, open-source, privacy-preserving clinical decision support architecture for oncology that integrates SOTA multi-agent design patterns, domain-specific fine-tuning, and a four-stage grounded retrieval pipeline.

The system demonstrates that production-grade clinical AI does not require proprietary infrastructure: the full stack — including 266k-sample QLoRA fine-tuning, 70+ guideline RAG, eight-node LangGraph orchestration, three-layer reflexion safety validation, and real-time clinical streaming UI — runs on a single AMD Instinct MI300X instance under ROCm.

The architectural contributions — particularly the synthesis of Corrective RAG, Reflexion, and HITL gating into a single coherent safety stack — represent a replicable blueprint for domain-specific clinical AI deployments where hallucination consequences are life-critical.

All code, adapter weights, and the OncoCoT synthetic corpus will be released publicly on Hugging Face Spaces and GitHub.

• Singhal, K. et al. (2023). Large language models encode clinical knowledge. Nature, 620, 172–180. - Nori, H. et al. (2023). Can generalist foundation models outcompete special-purpose tuning? Case study in medicine. arXiv:2311.16452. - Wang, L. et al. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 186345. - Shi, W. et al. (2024). Corrective Retrieval Augmented Generation. arXiv:2401.15884. - Shinn, N. et al. (2023). Reflexion: Language agents with verbal reinforcement learning. NeurIPS 2023. - Nogueira, R. and Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. - Gao, L. et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels. arXiv:2212.10496. - Hu, E.J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. - Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. - Han, S. et al. (2024). LangGraph: Building stateful multi-actor applications with LLMs. LangChain Technical Report.

OncoAgent is intended as a clinical decision support tool. All outputs require review by licensed medical professionals prior to any clinical application.