Fittest MCP Server

Fittest

The fittest survive, not the smartest. An evolving agent council that decides what gets compute and lab time.

Intelligence is abundant; ground truth is not. ML models screen candidate molecules and materials by the million, but only a few get a high-fidelity calculation (DFT) or a lab experiment. Fittest is a council of three independent foundation machine-learned interatomic potentials (MLIPs: MACE-MPA-0, AIMNet2, Orb-v3) and four role agents (Skeptic, Calibrator, Oracle, Curator) that share one log. For every structure it decides which answer to serve, how far to trust it, and which structures get ground truth. In this repo ground truth is DFT read from Transition1x (lookups, not live calculations); the decision is the same for lab time.

The project asks two questions, each tested against baselines or a pre-registered rule:

1. What if the models decide as a council? Their disagreement tells the council when to be skeptical (§2.3, §3.3).
2. Can the council evolve to give better predictions? Yes, once the test for new members was fixed: fine-tuned children replace their parents (§2.5).

Built at ScienceClaw 2026 (MIT Media Lab) for the NVIDIA ALCHEMI challenge, on NVIDIA ALCHEMI Toolkit 0.3.0-rc. Formerly TrustSwarm; the old GitHub URL redirects here.

• Code: this repo (fittest/ package, MIT licence)
• Hosted MCP server: https://fittest-mcp.westus2.cloudapp.azure.com/mcp
• Authoritative numbers, all generated from the result files: results/RESULTS.md (Idea 1), results/evolution/SUMMARY.md (Idea 2) and results/story_numbers.json (scripts/story_numbers.py: the critic values, run F's lineages and its absolute errors)

---

TL;DR

• Problem: the best model fails silently where it matters. On 287 held-out Transition1x reactions every model's force error peaks at the transition state, the top of the energy hill that sets the reaction rate. AIMNet2 leads at the reactant (median 0.22 vs 0.35 eV/Å for Orb) and loses its lead at the transition state (0.46 vs 0.47), with no warning. Its own seed ensemble does not warn either: its spread is 2.4× smaller than its error.
• The council: each decision has an owner, no central planner. Three models answer independently and never see each other's answers. The Calibrator keeps the track record and chooses what to serve, the Skeptic audits trusted answers at random and vetoes, the Oracle spends the ground-truth budget, and the Curator breeds and retires members. They coordinate only through an append-only log, so every decision replays. The agents are deterministic programs, not LLMs; an LLM agent uses the whole council as one tool (MCP).
• Idea 1: decide as a council (leak-free split, 200 DFT label lookups per arm).
  • Disagreement between independent models flags when the best model is wrong, with no labels: AUROC 0.917 against "AIMNet2 wrong" (pool split, 1,672 frames). Copies of one model are overconfident: their spread is 2.4–36× smaller than their error.
  • Served force RMSE 0.206 eV/Å vs 0.286 for the best baseline (−28%). The council discovered from 200 labels that averaging AIMNet2 and Orb is the best answer; that average, picked in hindsight with eval labels, is as good (0.202).
  • Adding one role, the Skeptic's random audits of trusted answers, cuts calibration error (ECE) from 0.370 to 0.079 (4.6×).
  • Negative: the council's own risk score ranks its remaining errors worse than the simple baselines (AUROC 0.73 vs 0.80–0.87).
• The worst model is the best critic. As an answerer MACE-MPA-0 is wrong on 99.9% of eval frames (Orb 91.3%, AIMNet2 57.2%). As a critic it is the only foundation model whose removal makes the council worse at spotting mistakes: mean ΔAUROC +0.040 over 491 fitness checks (AIMNet2 −0.067, Orb −0.037).
• Idea 2: let the council evolve (pre-registered, failures included). The Curator breeds fine-tuned children on the labels the council already paid for; a child must beat its parent on labels it never trained on, and members that add nothing are retired. The first four designs (runs A–D) failed their rule or were not confirmed. A diagnosis showed why: the test's energy check on a few labels was noise and threw out the best children. With a force-only test (run F, 12 new seeds, 800 lookups for both arms) the evolving council beats the static council in 12 of 12 seeds: −0.026 eV/Å [−0.030, −0.021] (−12%), and −14% on 2,475 structures from new chemical formulas that no run had read. In total that is 0.181 vs 0.286 eV/Å against the best Idea 1 baseline on the same eval frames (−37%), and below the hindsight average. 108 children were bred, 43 sit in the final councils, and 32 of the 36 foundation models were retired. Caveat: growing the council without selection (run E) already gives −0.018 of the −0.026; selection's own share is not isolated yet.
• Use it now. Add the hosted MCP server to Claude Code: claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp, then ask "Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0." Structures are never stored (logging policy).

---

Contents

1. Use it now — MCP server, local install, Python API, ALCHEMI hook
2. How it works — the problem, the council, Idea 1, the worst model as critic, Idea 2 (evolution), ALCHEMI usage
3. Validation — data, split, metrics, the Idea 1 ablation, self-repair
4. What failed and limitations — and next steps
5. Reproduce
6. Repo map, privacy, licence, acknowledgements

---

1. Use it now

1.1 Hosted MCP server (no install)

fittest-mcp exposes the council to any MCP client over streamable HTTP. The three model agents (MACE-MPA-0, AIMNet2, Orb-v3) run as warm CPU processes on an Azure VM, and their disagreement is turned into a verdict with the Calibrator state learned in the final run (fittest/snapshots/trust_snapshot.json, identical to results/trust_snapshot.json). Landing page: https://fittest-mcp.westus2.cloudapp.azure.com/, public counts at /stats, health at /health. Full guide: docs/MCP.md.

Limits: ≤ 20 structures per call, ≤ 100 atoms each, non-periodic, coordinates within ±1000 Å, no atoms closer than 0.3 Å; element symbols must be exact (Cl or cl; PDB-style C1 is fine, ambiguous CA or HE1 is an error); per-client rate limit (burst 20, then 1 call per 2 s) plus a per-IP limit; 120 s per request; model evaluations are serialized.

The verdict rule. One rule feeds check_structures, explain and the usage counts. A structure gets trust only if all four hold:

1. every model agent supports its elements (AIMNet2 sets the limit; the full committee covers H B C N O F Si P S Cl As Se Br I), otherwise send_to_dft with reason unsupported_elements and no trust score;
2. the forces do not all vanish (max |F| < 0.05 eV/Å for every model means symmetry or exact equilibrium, where disagreement carries no information);
3. σ_F < τ;
4. trust_score ≥ 0.5.

Otherwise the verdict is send_to_dft, with every failing condition listed in verdict_reasons. τ = 0.620 eV/Å is fixed: it is the 70th percentile of σ_F on the round-0 pool, which the Calibrator's threshold rule never moved. trust_score is the Calibrator's Beta posterior mean of P(served answer correct) for the served candidate in the structure's (σ_F, mean |F|) bucket, built from DFT oracle labels on Transition1x with prior Beta(2, 1), so it is never exactly 1. It comes with trust_score_90ci (the 90% credible interval) and labels_behind_score (the DFT labels in that bucket); a bucket with few labels gives a wide interval, so read the score with it. "Correct" means force RMSE ≤ 0.25 eV/Å and relative energy error ≤ 0.25 eV (relative to the reaction's reactant). The ECE 0.080 in §3.3 is for arm D's risk score (max of the isotonic P(served wrong) and the Skeptic's p_S); the calibration of this per-bucket trust_score has not been measured beyond the 12-frame spot check below.

Claude Code

claude mcp add --transport http fittest https://fittest-mcp.westus2.cloudapp.azure.com/mcp

Cursor (~/.cursor/mcp.json)

{ "mcpServers": { "fittest": { "url": "https://fittest-mcp.westus2.cloudapp.azure.com/mcp" } } }

Claude Desktop / claude.ai: Settings → Connectors → Add custom connector → https://fittest-mcp.westus2.cloudapp.azure.com/mcp.

Example prompt: "Use fittest to check whether I can trust an ML potential on this water molecule: O 0 0 0, H 0.96 0 0, H -0.24 0.93 0." Atoms may be separated by newlines, commas or semicolons. In the Round-2 verification run this prompt, used verbatim, gave trust, trust score 0.857 (90% CI 0.61–0.99, 4 labels behind it), recommended AIMNet2, σ_F = 0.378 eV/Å below τ = 0.620 eV/Å (docs/MCP.md, verification log).

Hosted CPU vs the GPU run. On the 12 held-out Transition1x frames of results/service_demo.xyz the hosted server's σ_F ≥ τ flags agree 12/12 with the earlier GPU run (results/service_demo.json), with max |Δσ_F| = 0.0005 eV/Å. Under the verdict rule 7 of the 12 are trusted and none of the trusted frames is DFT-wrong; the earlier τ-only rule trusted 2 DFT-wrong frames.

Domain. The Calibrator learned from gas-phase C/H/N/O Transition1x reactions with at most 7 heavy atoms, including transition states and NEB paths. A domain_warning is added for elements outside C/H/N/O that the committee still supports, larger molecules, isolated atoms or fragments and unusually short bonds; there the trust score is an extrapolation. Absolute energies are on each model's own reference (PBE for MACE and Orb, ωB97M for AIMNet2), so served_energy_eV values from different models are not comparable and a batch whose served model differs carries an energy_warning. Compare reference_energy_eV (always AIMNet2) across structures instead.

1.2 Local MCP server

# demo only: a plain uvx install has no model agents
claude mcp add fittest-demo -- uvx --from git+https://github.com/Pavel-Tk/fittest fittest-mcp --fake-models

# real models: install into an environment that has them (documented, not yet tested end to end)
uv venv fittest-env --python 3.12
uv pip install --python fittest-env/bin/python "fittest[models] @ git+https://github.com/Pavel-Tk/fittest" \
    --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match   # CPU torch
# optional: Orb-v3 in its own env (orb-models conflicts with the Toolkit), as in scripts/setup_cpu.sh
uv venv orb-env --python 3.12
uv pip install --python orb-env/bin/python torch --index-url https://download.pytorch.org/whl/cpu
uv pip install --python orb-env/bin/python "dm-tree>=0.1.9" orb-models ase numpy \
    --extra-index-url https://download.pytorch.org/whl/cpu --index-strategy unsafe-best-match
# then register ONE of these two (same name, alternatives):
claude mcp add fittest-local -- $PWD/fittest-env/bin/fittest-mcp          # (a) MACE + AIMNet2 only
claude mcp add fittest-local -e FITTEST_ORB_PYTHON=$PWD/orb-env/bin/python -- $PWD/fittest-env/bin/fittest-mcp   # (b) full committee

A plain uvx --from git+… install has no model agents. It serves only --fake-models, CPU stand-ins whose verdicts mean nothing (useful to test the wiring); without that flag the tools return an error pointing to --fake-models, the [models] extra and the hosted server. For real verdicts, install into an environment that has the models and point your client at that environment's fittest-mcp. This real-models local install is documented but has not been tested end to end (fresh environment through to a verdict); the hosted server is the tested path.

The [models] extra installs MACE-MPA-0 and AIMNet2 through the ALCHEMI Toolkit. Orb-v3 needs its own Python environment because orb-models pins dependencies that conflict with the Toolkit; set FITTEST_ORB_PYTHON to that environment's Python. With only MACE and AIMNet2 the server serves a partial committee and every response carries a warning: τ and the Calibrator's buckets and ledger were built on the full three-model committee, so verdicts are approximate. Local mode runs over stdio on CPU: the commands above install CPU-only torch. --device cuda needs CUDA builds of torch in both environments instead (install without the CPU index, as scripts/setup_gpu.sh does). It spawns one process per installed model agent, each with its own random socket key, logs only to ~/.fittest-mcp/usage.sqlite and never reports anywhere.

1.3 Python API: the trust service

git clone https://github.com/Pavel-Tk/fittest && cd fittest
uv venv .venv --python 3.11 && source .venv/bin/activate   # Windows: .venv\Scripts\activate
uv pip install -e ".[dev,service]"

from fittest.service import TrustSnapshot, score, fake_predictors, connect_models

snap = TrustSnapshot.load("results/trust_snapshot.json")      # Calibrator state from the final run
# real model agents (one process each; GPU or CPU):
#   clients = connect_models(snap.models, base_port=6300, orb_python="/path/to/orb/env/python", device="cuda")
#   predictors = {m: c.predict for m, c in clients.items()}
predictors = fake_predictors(snap.models)                      # CPU stand-ins: wiring only, verdicts meaningless

water = {"Z": [8, 1, 1], "R": [[0, 0, 0], [0.96, 0, 0], [-0.24, 0.93, 0]]}
for r in score([water], predictors, snap):
    print(r["sigma_F"], r["flagged"], r["trust"], r["recommended_model"], r["action"])
# keys: sigma_F, sigma_F_max, mean_abs_fbar, flagged, trust, recommended_model, bucket,
#       action ("send to oracle / DFT" | "use <candidate>"), served_forces, served_members, per_model

HTTP and CLI wrappers of the same function:

python -m fittest.service serve --snapshot results/trust_snapshot.json --port 8000 [--fake-models]
curl -X POST localhost:8000/trust -H 'content-type: application/json' \
     -d '{"systems":[{"Z":[8,1,1],"R":[[0,0,0],[0.96,0,0],[-0.24,0.93,0]]}]}'
python -m fittest.service score my_molecules.xyz --snapshot results/trust_snapshot.json

score raises if a model agent the snapshot was learned with is missing: σ_F and its bins only mean something for the committee that produced them.

1.4 The ALCHEMI hook: DisagreementHook

A drop-in Toolkit hook for any FusedStage, FIRE2 or NEB run, with any committee of two or more models. At admission and every every steps it asks every committee member for forces (one round trip per model, concurrently), computes σ_F per system, and flags systems with σ_F ≥ τ. With graduate=True a flagged system gets status = exit_status, leaves the batch at the next refill_check, and the SizeAwareSampler refills its slot; the flagged system goes to your DFT queue.

from nvalchemi.dynamics import FIRE2, ConvergenceHook, FusedStage, SizeAwareSampler
from nvalchemi.dynamics.base import DynamicsStage
from fittest import paths
from fittest.hook import DisagreementHook
from fittest.model_server import spawn
from fittest.models.mace_batched import BatchedMACE

committee = {m: spawn(m, 6100 + k, python=paths.model_python(m)).connect()   # one process per model agent
             for k, m in enumerate(["mace_mpa0", "aimnet2", "orb"])}
flagged = []
hook = DisagreementHook(committee, tau=0.57,          # eV/Å; float, or a zero-arg callable read at every check
                        every=10, stage="both",       # pristine geometry at admission + every 10 steps
                        graduate=True,                # flagged -> leaves the batch, sampler refills the slot
                        id_map=lambda sid: ds.load_order[sid],
                        on_flag=lambda rec: flagged.append(rec["key"]))

driver = BatchedMACE().model                          # any Toolkit BaseModelMixin drives the dynamics
relax = FIRE2(model=driver, dt=0.05, n_steps=50, convergence_hook=ConvergenceHook.from_fmax(0.05))
for h in driver.make_neighbor_hooks():
    relax.register_hook(h, stage=DynamicsStage.BEFORE_COMPUTE)
sampler = SizeAwareSampler(ds, max_atoms=1024, max_edges=None, max_batch_size=48)
fused = FusedStage(sub_stages=[(0, relax)], sampler=sampler, sinks=[], refill_frequency=1, hooks=[hook])
with fused:
    fused.run(batch=None, n_steps=20000)

hook.records[flagged[0]]   # sigma_admission, checks[{step, sigma, sigma_max, tau, energy}], at_flag{Z, R, forces}
hook.summary()             # n_flagged, round trips, gather time, dead models, failures

ds is any dataset that follows the Toolkit sampler protocol; a complete example is in docs/REPRODUCE.md §2. For NEB use NEB(..., extra_hooks=[DisagreementHook(..., graduate=False)]); without dynamics use fittest.hook.DisagreementMonitor(committee, tau).check(keys, Z, R, batch, step=0, phase="x"). Committee members can be ModelClients, objects with .predict(Z, R, batch[, charge]), or plain callables returning (E, F). A member that raises or times out is logged and retired after max_failures consecutive failures; checks continue while two members answer. To add your own model as an agent, see docs/REPRODUCE.md §3.

---

2. How it works

2.1 The problem: ground truth is scarce, and models fail silently where it matters

Discovery is a funnel. ML models on GPUs screen candidates by the hundred million, and agents now set up and check the runs. What stays scarce is ground truth: a high-fidelity simulation or a lab experiment for each of the few candidates that reach the end. Two questions follow: which candidates should get ground truth, and which cheap answers can be trusted without it? Both need to know when a model is wrong, and the models do not say.

Labs spend months deciding whether a foundation MLIP is good enough for their chemistry. The models return a number for every structure, with no warning when the structure is outside what they learned. On the 3,157 held-out Transition1x test frames (287 reactions), median force RMSE against DFT (ωB97x/6-31G(d)) rises from reactant to transition state for every model (results/fig/numbers.json, fig1). The best model, AIMNet2, leads clearly at the reactant and ties Orb at the transition state, where the reaction is decided:

(eV/Å, median per-frame force RMSE.)

2.2 The council: agents and the blackboard

Each agent holds private information and talks to the others only through an append-only blackboard (fittest.blackboard.Blackboard; the full arm-D log is results/blackboard_armD.jsonl, 5,921 events). The round driver advances the clock and makes no decisions; there is no central planner. The agents are deterministic programs, not LLMs: the same inputs give the same decisions, and an LLM agent can use the whole council as a tool (§1).

flowchart LR
    S["Stream<br/>label-free structures"] --> P["ModelAgents x3<br/>MACE · AIMNet2 · Orb<br/>(one process each)"]
    P -->|"prediction"| M{"Monitor / DisagreementHook<br/>σ_F ≥ τ ?"}
    M -->|"pass"| C["Calibrator<br/>ledger · verdict · trust"]
    M -->|"flag"| B["Bids<br/>ModelAgents + Skeptic"]
    K["Skeptic<br/>private references,<br/>planted failures"] -->|"veto, audits"| B
    C -->|"verdict"| K
    B --> O["Oracle<br/>k DFT labels / round"]
    O -->|"oracle_result: true error<br/>of every candidate"| C
    C -->|"reputation"| P
    C -->|"threshold τ"| M
    C -->|"worst model + labels"| U["Curator<br/>LoRA fine-tune + gate"]
    U -->|"finetune_report"| C

What the Calibrator serves. Candidates are every non-empty subset mean of the models (3 singletons, 3 pairs, the 3-way consensus). The context bucket is label-free: σ_F quantile bin (4) × mean |F̄| bin (3). Per bucket it serves the candidate with the highest posterior probability of being correct. Its risk score is an isotonic P(served answer wrong | feature); the feature (σ_F or the spread among the served candidate's members) is chosen by 5-fold reaction-grouped CV on its own labels (it chose σ_F: CV AUROC 0.756 vs 0.752). τ = smallest σ with ledger P(wrong | σ ≥ τ) ≥ 0.5, floored at the round-0 70th percentile of σ_F. In the final run τ stayed at that floor (0.620; see §4).

What happened in the final run (arm D). 200 oracle calls: 50 exploration audits and 150 won by bids (9,745 bids placed). The hook flagged 601 systems; the Skeptic vetoed 1,788 trusted verdicts (1,202 on pool systems; vetoes on eval are advisory). On eval the Calibrator served mean(aimnet2+orb) on 1,373 of 1,485 frames and aimnet2 on 112. Final reputations were nearly equal (aimnet2 0.89, mace_mpa0 0.89, orb 0.89, skeptic 0.86).

2.3 Idea 1: decide as a council

The idea comes from LLMs: Mixture-of-Agents asks several models and lets an aggregator decide. Here the members are independent foundation MLIPs, trained by different groups on different data. Unlike multi-agent debate, they never see each other's answers and are never pushed to agree; their disagreement is the signal.

The signal. For M models predicting forces F_m on a structure with n atoms:

F̄_i  = mean_m F_m,i
s_i  = sqrt( mean_m ||F_m,i − F̄_i||² )      per-atom spread
σ_F  = mean_i s_i                            per-structure disagreement (eV/Å)

(fittest.signals.force_spread; the hook uses an identical vectorised twin, tested to agree.)

• Committee of independent models (MACE-MPA-0, AIMNet2, Orb): Spearman(σ_F, AIMNet2 force error) 0.73; the spread is about 1.8× the error.
• Seed ensemble (the four shipped AIMNet2 members): Spearman 0.68, but the spread is about 0.4× the error — overconfident.
• On the 1,485 eval frames: the AIMNet2 ensemble's median error is 2.4× its spread; four MACE LoRA seeds (arm B′) are 36× under-dispersed (spread 0.008 eV/Å, error 0.278 eV/Å).

Seed ensembles can rank errors well while being badly under-dispersed: ranking and calibration are different properties. This is why the council uses models trained on different data by different groups.

What the council does with it (full ablation in §3.3): disagreement flags when the best model is wrong (AUROC 0.917 against "AIMNet2 wrong", pool split); the Calibrator serves the best subset per context bucket, 0.206 eV/Å vs 0.286 for the best baseline (−28%); the Skeptic's random audits of trusted answers cut ECE from 0.370 to 0.079. The council's own risk score ranks its remaining errors worse than the baselines' scores (AUROC 0.73 vs 0.80–0.87).

2.4 The worst model is the best critic

The evolutionary loop (§2.5) scores every member in two roles: as an answerer (how much the served error rises without it, Δ_ans) and as a critic (how much the AUROC of the disagreement signal drops without it, Δ_crit). Pooled over every fitness evaluation of the three foundation models in all evolution runs (results/story_numbers.json):

"Wrong" here is force RMSE > 0.25 eV/Å on the 1,485 eval frames (results/fig/numbers.json). The council never serves MACE alone, yet MACE is the one foundation model whose removal consistently costs the council error detection. Our reading, not tested: MACE was trained on different data (PBE materials) and disagrees with the others where they share a blind spot. Caveat: in run F on GPU the loop still retired base MACE in 12 of 12 seeds; in the CPU runs, which breed only AIMNet2 children, base MACE survived in 18 of 24 seeds (E-cpu, F-cpu).

2.5 Idea 2: let the council evolve

Status: confirmed in run F (pre-registered, 12 new seeds, and again on frames no run had read). Four earlier designs failed or were not confirmed; all are reported. Full write-up, controls and exploratory runs: results/evolution/SUMMARY.md.

The static council has fixed members. The evolutionary loop (fittest/evolution.py, fittest/evolution_c.py, scripts/evolve_c.py) asks whether a council that breeds, tests and retires its own members serves better answers at the same DFT budget as a static control. One generation, as in run F (results/evolution_c/PREREG.md, results/evolution_f/PREREG.md):

flowchart LR
    T["1 Triage<br/>members predict the pool;<br/>Calibrator chooses what to serve"] --> O["2 Oracle<br/>200 labels per generation:<br/>75% highest σ_F, 25% audits of trusted frames"]
    O --> F["3 Fitness<br/>marginal contribution<br/>answerer Δ_ans · critic Δ_crit"]
    F --> S["4 Selection<br/>retire members whose<br/>contribution CI includes 0"]
    S --> B["5 Breeding<br/>one child per family,<br/>fine-tuned where the council is weak"]
    B --> G["6 Gate<br/>child ≤ parent force RMSE<br/>on select labels; clones rejected"]
    G --> H["7 Hot-swap<br/>survivors + gated children;<br/>ledger re-fitted"]
    H --> T

• Budget and controls. 4 generations × 200 Transition1x lookups = 800, the same for both arms. The static control runs the identical triage and oracle procedure with the members fixed at MACE-MPA-0 + AIMNet2 + Orb. Every generation of every run is scored on the same 1,485 eval frames with 1,000 shared reaction-bootstrap resamples, so comparisons are paired. Eval never feeds back: only the final scorer (EvalScorer) reads eval labels.
• Breed / select split. The pool's 16 endpoint-geometry components are split once into select (6 components, 561 frames, 51 reactions) and breed (10 components, 1,111 frames, 101 reactions); 40% of each generation's labels go to select and 60% to breed. Children train only on breed labels; the gate and fitness use select labels, which no member trained on (asserted).
• Niches, label-free: flagged = σ_F ≥ τ₀ (the 70th percentile of pool σ_F, the high-disagreement, transition-state-like region), trusted = the rest. Only flagged breeds specialists, because that is where the council is weak and where triage spends its labels.
• Breeding. After generations 0, 1 and 2, the Curator breeds one child per family (AIMNet2, Orb-v3, MACE-MPA-0) from the fittest living member of that family, trained on the breed labels of the flagged niche. Head-only fine-tune (AIMNet2 mlps.2 + outputs, Orb heads, MACE model.readouts), Adam, 200 steps, force + reaction-relative energy loss. Later children mutate their parent's learning rate and energy weight (×½ or ×2) and start from its head weights.
• Gate. On the select labels of the niche, the child's force RMSE must be ≤ its parent's. Clone rejection: Pearson r > 0.95 between the child's and a member's deviations from the population-mean force, or median per-frame force RMS difference < 0.02 eV/Å. Failing children are stillborn and stay in the family tree. Runs A–D also required reactant-relative ΔE MAE ≤ parent, and run A the ALCHEMI build_acceptance_report from-scratch ratio ≤ 1.01; run F dropped the energy check (below).
• Fitness = marginal contribution to the council, on clean labels:
  • answerer: Δ_ans(m) = err(P \ m) − err(P), where err is the mean served error in wrongness units (F_RMSE / 0.25 + |ΔE_rel| / 0.25) and the served answer comes from a Calibrator fitted out-of-fold (5 reaction-grouped folds), so a member cannot look useful by overfitting the labels it is scored on;
  • critic: Δ_crit(m) = AUROC(σ_F(P) → P wrong) − AUROC(σ_F(P \ m) → P\m wrong);
  • 400 paired reaction-bootstrap resamples give each Δ an SE and 95% CI; fitness = Δ_ans/SE_ans + Δ_crit/SE_crit; the role (answerer or critic) is the larger z-score.
• Selection. Recomputed after every death. A member is eligible when neither Δ has a CI lower bound above 0 and fitness ≤ 0.5; the least fit eligible member dies while the council exceeds 3, then the least fit dies while it exceeds 6. Foundation models can be retired too.
• Decision rule (run F, fixed before it ran): the seed-level 95% t-interval of mean Δ served force RMSE (evolved − static, eval, final generation) lies entirely below 0, at most 1 of 12 seeds is significantly worse, and ECE does not regress at the seed level. Runs A–C used a stricter rule: a significant win in every seed, also against the run's own generation 0.

Every pre-registered design (Δ = final generation − static at the same budget, served force RMSE on eval, eV/Å; * = paired CI excludes 0; E and F: seed-level mean with 95% t-interval):

Run F also passes the original every-seed rule, with lower relative-energy error (−0.039 eV) and lower ECE (−0.021) than static.

Frames no run had read. The run F fix came from a diagnosis that read earlier eval results, so a check on never-read frames was pre-registered before scoring (PREREG_fresh_frames.md, FRESH_FRAMES.md): 2,475 structures, 225 reactions, Transition1x val formulas. Run F: −0.026 [−0.034, −0.019], 12 of 12 seeds better (served force RMSE 0.191 → 0.164 eV/Å, −14%). Run E: −0.018 [−0.026, −0.011], 12 of 12.

In absolute terms (results/story_numbers.json): on eval the evolved council serves 0.181 eV/Å (mean of 12 seeds), against 0.207 for the static council, 0.202 for the hindsight AIMNet2+Orb average and 0.286 for the best Idea 1 baseline (−37%). The baselines' answers use no labels, so their error does not depend on the budget.

Lineages (run F, GPU, 12 seeds, results/evolution_f/gpu*/b800*/family_tree.json): 108 children bred (9 per seed), 9 stillborn, 6 rejected as clones, 50 retired later, 43 in the final councils. 32 of the 36 foundation models were retired (MACE 12, Orb 12, AIMNet2 8), and the final councils have 3 to 6 members. AIMNet2 and Orb build real lineages (children of children); 33 of the 36 MACE children were bred from the base model again.

What we learned on the way

• The energy check was noise. Most children of runs A–C failed the gate on reaction-relative energy (run A: 15 of 18 stillborn; GPU run C: 36 of 54, 35 of them on rel-E). On CPU, that check's sign agreed with eval only 38% of the time (r = −0.15), while the select-set force change predicted eval (r = 0.87). It passed 39 of 114 AIMNet2 children against 91 of 114 Orb children, and the runs that ended worse than static were those with no AIMNet2 specialist left (DIAGNOSIS.md). Run F removed only that check.
• Growth vs selection. Growing the council without selection (run E) gives −0.018, the full loop of run F −0.026, on different seeds. A sham control (children with learning rate 0; CPU, exploratory) shows that run E's force gain comes from learning. What selection adds on its own needs both arms on the same seeds (§4).
• Budget. The run F design is not significant at 400 labels (−0.010 [−0.023, +0.004]) and wins at 1,600 (−0.030); exploratory, seeds 24–29. The loop needs enough labels.
• Noise. Run-to-run noise is about 0.02 eV/Å (the static council alone moves 0.202 → 0.220 between seeds), and paired CIs cover eval resampling only. That is why runs D–F use 6 to 12 new seeds and seed-level intervals.

Family trees of runs A and B: results/fig/fig8_family_tree_run{A,B}_s{0,1}.png; per-generation curves: results/fig/fig7_evolution_run{A,B}_s{0,1}.png; budget axis: results/evolution_c/fig9_budget.png.

2.6 How the ALCHEMI Toolkit is used

In-flight screen (600 pool frames, MACE-MPA-0 driver, τ = 0.571 = 70th percentile of cached σ_F, check every 10 steps, ≤ 50 FIRE2 steps per system, L4 GPU):

Batched NEB (8 reactions, AIMNet2 driver, climbing image, all 8 converged in 21 s): along the DFT reference bands, σ_F follows force error (median per-reaction Spearman 0.96, pooled 0.85; two reactions at 0.12 and 0.16). σ_F does not track relative-energy error (median Spearman 0.14, pooled −0.20). The maximum of σ_F is at the DFT transition state in 3/8 reactions and within one image of it in 6/8.

Documentation truncated — see the full README on GitHub.