How it works

Overview

Two levels, one memory

OmiCairn works at two nested levels. Both use the same model backbone, the same containerized execution and the same experience store. Inside a task, repair, review and verification loops correct the work. Across tasks, experience replay carries lessons forward.

Node level

An analysis tree

One analysis request becomes a tree of small, executable steps. Each step is planned, run in a container, repaired if it fails and reviewed before anything is built on it.

Used for the case studies and the benchmark.

Run level

A verification-gated research run

A hypothesis-driven run moves through eight phases. Four produce work and four check it. A run continues only when the current phase passes its gate.

Used for the neurodegeneration campaign.

Across tasks

Experience replay

Fixes, review findings, pitfalls and expert directives are stored as experience items. Confirmed items are shown to later agents as advice. The model itself never changes.

How items are promoted

Node level

An analysis tree that repairs and reviews itself

Each analysis is stored as a tree. The root registers the input data without changing it. Every other node is one analytical step that reads the outputs of its parent. The tree records the code, parameters, logs, inputs, outputs and review decisions of every step, including the attempts that failed.

review
superseded, kept for provenance
repair or re-attempt

Running code is not the same as a correct analysis. A step first has to run without error; the BugFixer agent repairs failures a limited number of times. A step that ran is then reviewed. If the review fails in a way that means the outputs do not reflect the intended analysis, the step is attempted again as a new sibling node that carries the reviewer’s feedback. The failed attempt stays in the tree, and nothing is built on it.

What a step is

Each step is a function block: Python, R or shell code with a manifest of its inputs, outputs and parameters and a list of software requirements. All blocks share one entry point, so any block can be run, rerun or swapped the same way. The FunctionCreator agent first looks for a validated block in the library and adapts it; only if none fits does it write a new one. A separate check reads the code before it runs.

What review looks at

Code first collects evidence from the step: how many output files and figures it produced, how large they are, how many warnings the logs contain, whether the logs show real computation on the input, and whether outputs contain placeholder markers such as “mock”, “dummy” or “example data”. The Reviewer agent then judges the step against its task. It names concerns by type:

Cheating: results not derived from the input data, such as hard-coded or simulated values reported as findings.
Placeholder: stub outputs.
ML-practice violation: improper data splitting, information leakage or missing validation.
Implementation mismatch: the code does not do what the step says.

A request to redo the step, or a critical concern of one of these types, blocks the step. Smaller concerns are passed to the Orchestrator without stopping the analysis.

Changing a step

If a user edits a step, changes its parameters or asks for a rerun, only that step and the steps that depend on it are recomputed. All other outputs and all earlier execution records are kept.

The agents

MainAgent: A deterministic controller. It runs the session, creates re-attempts and calls the other agents. It uses the model only to interpret the request.
Orchestrator: Decides whether the request is satisfied and, if not, plans the next steps from the tree, the failures so far and curated domain knowledge.
FunctionCreator: Reuses a validated function block or writes a new one for each planned step.
BugFixer: Repairs a failing step in a bounded session: it reads logs and files, edits code, installs packages and runs diagnostics.
Reviewer: Judges whether a step that ran is an adequate analysis, using the evidence code collected.
WebAgent: Retrieves software documentation, code repositories, biological databases and, when enabled, scientific literature.

Every agent must return output in a fixed schema. Attempts, repairs, branches and the size of the tree are all bounded, so a session always ends.

Overview of OmiCairn’s agents. User inputs (a biomedical question, research objective and uploaded data) go to the Orchestrator agent, which plans function blocks. A loop runs from Planning to Execution to Review and back to Planning for rework. The FunctionCreator generates and registers function blocks in a library (normalization, spatial mapping, marker detection, survival analysis). The Web agent retrieves documentation, literature and annotations. The Reviewer checks results against the user’s intent. The BugFixer repairs execution errors and reruns steps. A human expert inspects the workflow and guides re-analysis; final outputs are a reproducible, evidence-linked report. — Figure 1 of the OmiCairn manuscript: the Orchestrator plans, the FunctionCreator builds missing function blocks, the Web agent retrieves source-tracked information, the Reviewer checks outputs and the BugFixer repairs errors. A human expert can inspect the workflow and guide re-analysis at any point.Figure from the OmiCairn manuscript (in preparation).

Run level

Eight phases, four of which check

A research run starts from a written question. It moves through eight phases in order. Every phase except CROSS-CHECK is carried out by a model-based agent; CROSS-CHECK is plain code. When a checking phase finds a problem, it sends the run back to the phase that can fix it.

What each phase does and where it may send the run
Phase	What it does	May send the run to
SCOPE agent	Turns the brief into a question, collects the literature and finds candidate datasets. Checks whether the data can answer the question.	PLAN; SCOPE again if data are missing
PLAN agent	States the hypotheses, a decision rule for each, and the analysis plan. Freezes them as a pre-registration.	EXECUTE; PLAN again if underpowered; SCOPE if data are missing
EXECUTE agent, sandbox	Writes and runs the analysis code. Records estimates, confidence intervals, P values and comparisons in a structured results file, with figures and notes.	CROSS-CHECK
CROSS-CHECK code only	Deterministic gates on the results. Layer 1.	REPORT; EXECUTE on any error
REPORT agent	Writes a structured research report in which each statement points to its evidence.	CLAIM-AUDIT
CLAIM-AUDIT claim-review agent	Checks the report sentence by sentence. Layer 2.	REVIEW; REPORT
REVIEW reviewer agent	Judges the work as a whole. Layer 3.	PANEL-REVIEW; REPORT, EXECUTE, PLAN or SCOPE; halt
PANEL-REVIEW three reviewer agents	Three independent reviewers must all accept. Layer 3.	done; REPORT, EXECUTE or PLAN

Going back is part of the design

Each phase may only make the decisions allowed for it; any other decision is rejected and the phase runs again. Every route back is counted. Each phase can run only a limited number of times, and a run can only go back a limited number of times. A run that would exceed either limit halts instead of looping.

Each pass through a phase writes to a new directory that is never overwritten. Later phases can read earlier work but cannot change it; an edited report is saved as a new file. This is what makes a run auditable after the fact.

Runs can also stop for reasons outside this logic, such as an infrastructure failure or an exhausted usage quota. In the campaign, 17 of 41 genuine runs reached multi-agent review. Halted runs and their hypotheses are published too.

A run can also be started in ideation mode, in which it only proposes research directions and ends before SCOPE.

produces work
checks work
route back

Every checking phase can send the run back. Each vertical line is a route back to one phase; a dot marks a phase that can use it. REVIEW can reach every earlier phase that produces work, or halt the run.

Accountability

Three layers of checking

The layers look at different things. Code checks that the numbers are internally valid. A claim-review agent checks that every sentence of the report is backed by those numbers. Reviewer agents judge whether the analysis answers the question. A failure at any layer sends the run back.

Layer 1 · CROSS-CHECK

Deterministic gates

Run by code alone, without a language model.

Every confidence interval is finite and ordered, and every estimate lies inside its interval.
Bounded metrics, such as AUC, correlations and P values, lie in their allowed ranges.
A comparison with a baseline on the same samples reports the paired difference and its interval.
Intervals quoted in the notes are identical to those in the results file.
The headline comparison was produced by the code that ran in this stage, not recovered from a failed stage or typed in by hand, and is not degenerate.
No hypothesis is called supported unless its decision rule was met.
The pre-registration has not changed since PLAN.

Further gates only warn: whether the minimum detectable effect is reported, whether declared figures exist, whether software lock files are present, and whether multiple comparisons are corrected.

Any error sends the run back to EXECUTE.

Layer 2 · CLAIM-AUDIT

Claim-review agent

Reads the report one sentence at a time.

Each sentence is classified (quantitative, qualitative or methodological claim, narrative, definition, prediction or filler) and labelled supported, partially supported, unsupported, over-claimed, under-claimed, filler or style. Claims need evidence of the right kind:

A number must cite an entry in the results file, and the stored value, rounded as quoted, must match.
A qualitative claim must cite a publication from the run’s literature set.
A methodological claim must cite a file or a field of the pre-registration.

The agent then edits the report using existing results only: filler is removed, over-claims are softened, under-claims strengthened. A sentence it cannot fix stays in the report and is flagged for human revision. Code then verifies the audit ledger, and every reference must resolve in Crossref or PubMed.

Unsupported or excess over-claimed sentences send the report back to REPORT.

Layer 3 · REVIEW and PANEL-REVIEW

Reviewer agents

Separate from the agents that did the analysis.

An internal reviewer agent receives the audited report, the results, the audit ledger and the cross-check warnings. It scores eleven dimensions: novelty, methodological rigor, statistical validity, domain depth, figure quality, clarity, references, honest limitations, reproducibility, anti-fabrication and comprehensiveness. Each over-claim left after the audit costs anti-fabrication points.

Its decision names the cause: weak analysis goes back to EXECUTE, weak presentation to REPORT, a flawed design to PLAN, a question that contradicts the data to SCOPE. A run that cannot be saved halts.

A panel of three reviewer agents follows: a domain expert, a methodologist or statistician, and a reproducibility advocate. They do not see each other’s assessments.

The run ends as done only if all three accept. These are agents inside the system, not human peer reviewers. On the campaign pages, “completed the pipeline” means reached this review, not accepted by it.

What the checks did in the campaign

Across the 57-run neurodegeneration campaign. Source: campaign release and the OmiCairn manuscript.

2,766

report sentences audited by the claim-review agent

in the 21 runs that reached sentence-level auditing

240

claim-integrity issues detected

149 corrected automatically, 91 flagged for human revision

68

of the 91 flagged issues were over-claims

15 were filler sentences and 8 unsupported statements

19 of 32

executing runs re-ran analyses

in response to cross-check errors or review findings

34

review-gate rejections followed by targeted rework

routes back from REVIEW to an earlier phase

Flagged sentences remain in the delivered reports; no human edits were made during the runs. Reaching review shows that a run passed through the pipeline, not that its conclusions are valid.

Pre-registration

The rule is fixed before the analysis runs

In PLAN, every hypothesis gets a decision rule. The rule is stored with a cryptographic hash. EXECUTE and CROSS-CHECK recompute the hash; any change is reported as pre-registration drift and blocks the run. A hypothesis can be called supported only if its rule is met.

Anatomy of a decision rule

Metric

What is estimated, for example a donor-level AUC or an effect size.

Direction

At least, more than, at most or less than.

Threshold

The value the estimate must reach, written down in PLAN.

Null value (optional)

A value, such as chance level, that the 95% confidence interval must exclude.

The rule is met only if the estimate passes the threshold in the stated direction and, when a null value is given, the interval excludes it.

Which comparisons matter

The pre-registration also names one headline comparison, the load-bearing comparisons on which a verdict depends, and the discriminating comparisons that separate competing explanations. CROSS-CHECK fails the run if any of them is missing.

Verdicts need an evidence path

Each run keeps a knowledge graph of hypotheses, findings, datasets and publications. A hypothesis can be marked supported, partially supported or refuted only through a finding whose evidence path points to a non-empty file that a pipeline stage produced.

Can the data answer the question?

At the end of SCOPE, and again at the end of PLAN, the agent rates the match between question and data as feasible, underpowered or needing data. Missing data send the run back to SCOPE to revise the question or the sources; an underpowered design sends PLAN back to change the design, for example to pooled cohorts.

How verdicts appear on this site

Every hypothesis of the campaign carries one of these outcomes. The first five count as evaluated.

Supported: The pre-registered decision rule was met.
Partially supported: Some parts of the decision rule were met, but not all.
Not supported: The test ran and the decision rule was not met.
Inconclusive: The test ran but was underpowered, ambiguous or confounded, so the data cannot settle it.
Not testable (data): Not tested, or tested only through a proxy, because suitable data were unavailable or unsuitable.
Not tested (run did not execute): Never tested: the run halted, failed technically or ran out of time or compute, or the hypothesis was proposed but never run.

Browse all hypotheses and their evidence

Expert in the loop

Humans stay in the loop

OmiCairn is built to be steered by computational biologists, not to replace them. Experts act at four points:

Notes. An expert can attach a note to a phase. The agents must treat it as authoritative.
Steering. An expert can reopen a specific phase. A reopened phase counts toward the same limits as any other route back.
Flagged sentences. Sentences the claim-review agent cannot fix are left in the report and flagged for human revision.
Experience. An expert confirms, edits or rejects experience items that are not promoted automatically, and expert directives are stored as experience for later runs.

Across tasks

Experience replay: learning without changing the model

Every task leaves feedback behind: errors and the repairs that fixed them, review concerns and the revisions that cleared them, audit failures, reviewer comments and expert notes. OmiCairn keeps this feedback as structured experience and shows it to later agents. What an agent is shown changes; the model, its instructions, the review criteria and the gate thresholds do not.

Collection

After each task or run, a reflection agent reads the record and proposes candidate items. Each item stores the situation that triggered it, the reusable content, the agents and phases it applies to, links to its source, and how it was verified. A candidate that repeats an existing item is merged into it, and the item’s count of independent observations goes up.

Confidence-gated promotion

Confidence is highest for items confirmed by an observed outcome and seen repeatedly, for example a fix after which the code ran, or a revision after which a step passed review. It is lowest for items that rest only on an agent’s interpretation.

Expert directives become active directly.
Outcome-verified items with enough confidence that do not conflict with an active item become active automatically.
Everything else waits for a human computational biologist, who accepts, edits or rejects it. Until then it is never shown to an agent.

Rejected items are kept with the reason, so the same proposal is not raised again. When an expert activates an item that conflicts with an older one, the older item is retired.

Replay

When an agent acts, it is shown the active items that apply to its role and phase, ranked by similarity to its current situation: debugging fixes to the BugFixer, function blocks to the FunctionCreator, pitfalls and practices to the planning, coding and phase agents. Each item is shown with its class and confidence, as advice rather than a rule. Every replay is logged, so a later output can be traced to the experience that shaped it.

Starting from an empty store, OmiCairn accumulated about 439 items over the work reported in the manuscript, spanning reusable analysis skills, experiences and policies. How much the full store, and expert confirmation, improve later work has not yet been measured.

human decision
replayed to agents
next task

Experience is promoted only when an outcome or an expert backs it. Items an agent merely believes are held for a human computational biologist. Only active items are ever replayed.

Limits

What these checks do not guarantee

Consistency is not truth. The checks verify that numbers, sentences and verdicts trace back to code that ran on the stated data. They cannot show that the biology is right.
Reviewer agents are part of the system. Their review is not human peer review. In the manuscript, two external domain reviewers scored seven campaign reports lower than the internal reviewer did.
Completing the pipeline is not validation. A run that reached review passed through every phase; it was not thereby shown to be correct.
Many tests were weak. Many executed analyses were underpowered, used fallback tissues or surrogate methods, or could not reach the intended data. The adjudication note of each hypothesis says so.
Flagged problems were left in place. Sentences flagged for human revision remain in the delivered reports.

See it in practice

How OmiCairn works