Fixture mode uses committed JSON (placeholder commits — not for measure_repo). HuggingFace mode pulls Verified, Multilingual, and Multi-SWE-bench (ByteDance-Seed/Multi-SWE-bench, ~1632 instances). To swap offline fixtures for real rows: choose huggingface, check Replace pilot fixtures, then enqueue.
measure_repo
ByteDance-Seed/Multi-SWE-bench
Requires sourced instances. Sets fixed depth + repo metrics + partitions.
Parent repository must already have fixed_module_depth.
fixed_module_depth
By default, batches skip instances that already have metrics and take the next unmeasured ids (up to the batch limit).
P1–P3 / P1b are offline. P4 needs model_outcomes JSON. P5 checks OpenRouter catalog.
Runs measure_repo then measure_instance for every pilot roster id. Skips already-measured repos/instances unless Force is checked.
measure_instance
Live completion for each seeded litellm_route. Requires OPENROUTER_API_KEY on the engine (unless mock).
litellm_route
OPENROUTER_API_KEY
Requires every instance to have metrics or a permanent setup_failure.
Requires locked prereg with JSON regression_spec and completed full-run coverage (unless allow_incomplete).
regression_spec
Requires study prereg lock (or scope=pilot + exploratory=true). Prefer batch enqueue from Runs or Pilot. Single-cell form is for OPS smoke only.
scope=pilot
exploratory=true
Fail-closed gold check: one verified cell per pilot repo (~12 Docker pulls). Catches build-after-patch, title selectors, and monorepo Jest misses before a full 24-cell pilot_gold_run. Any unresolved repo fails the job.
pilot_gold_run
Fail-closed plumbing check: one verified cell per language × one included model × 1 rep (exploratory). Fails on empty_response / test_crash / patch-apply errors; soft unresolved (model miss) still passes the smoke.
empty_response
test_crash
Apply each pilot instance’s reference patch through the same Docker eval path (no LLM). Default roster: all 24 in config/pilot_instances.yaml. Job finishes with a resolve report — <100% is operator GO/NO-GO, not an automatic job failure. Default test timeout is 1800s (large P2P suites like matplotlib). Prefer pilot_harness_smoke first when changing the test runner.
config/pilot_instances.yaml
pilot_harness_smoke