nirs4all-core → dag-ml migration — war room¶
Status: groundwork / prep phase (no code migration started). Production nirs4all is untouched and stays untouched. This folder is the consolidated study + plan so it never gets lost again. Assembled 2026-06-23 from a full multi-agent recon of every migration doc across the ecosystem (15 clusters, 83 docs classified).
Goal of the chantier: make dag-ml (Rust + C-ABI reproducible DAG coordinator) the real execution core of the production Python library nirs4all, without breaking the frozen 0.9.x/0.10.x public surface and without disturbing the production
main.
This is an index + synthesis, not a copy dump. Every source doc is referenced at its
canonical home (cross-repo docs use repo:path notation, dag-ml docs use relative links) so
nothing drifts. Companion proposals live next to this file:
WORKING_STRATEGY.md— how to develop the migration while keeping prod alive (repo / branch / worktree / backend-selector). Answers ask #3.PARITY_AND_PERF_HARNESS.md— how to guarantee non-regression (parity) and measure performance, automated. Answers ask #4.
TL;DR readiness verdict¶
dag-ml 0.2.0 is ready to back the nirs4all control plane, not yet to run production numerics. It is a closed, CI-gated, contract-frozen Rust coordinator built for exactly this problem (reproducible, OOF/leakage-safe, identity-keyed DAG coordination). It already implements every coordination capability the migration needs. By deliberate design it never touches feature matrices — all NIRS numerics must be supplied by nirs4all as host controllers, and today the only production host adapter is sklearn.
The migration is therefore mostly a nirs4all-side decoupling + bridging job, not a “finish dag-ml” job.
The map — where the prepared study actually lives¶
Read these three tiers in order. Tier 3 is a trap: it is parallel design, not the plan.
Tier 2 — nirs4all side: what we migrate from + the executable gate¶
Doc ( |
Rel. |
Why it matters |
|---|---|---|
|
★★★ |
The single most migration-relevant nirs4all-side doc: reviewed cross-repo co-design with shared contracts, ownership split, phased N0–N8 / D0–D6 roadmaps. |
|
★★★ |
Decomposition backlog for the 12 >2k-line god classes (orchestrator / merge / branch / base_model / workspace_store / predictions / dataset) that own OOF/refit/branch-merge/storage — prerequisite to crossing the ABI. |
|
★★★ |
Existing replay / chain / expanded_config / trace / bundle round-trip + its gaps — maps onto dag-ml’s replay/lineage mandate. |
|
★★★ |
Canonical description of the production core being migrated FROM: Orchestrator→Executor→StepRunner→controllers over a mutable |
|
★★ |
File-by-file generic-vs-NIRS boundary map; names the |
|
★★★ |
The executable gate — see |
Tier 3 — Parallel designs — donor/parity input only, NOT the plan¶
These never reference dag-ml; they independently reinvent large parts of it. Treat as a requirements / parity inventory (“what must be preserved”), never as the architecture to build.
Doc ( |
What to harvest |
Why it’s not the plan |
|---|---|---|
|
Feature-Preservation Matrix, identity schema, node taxonomy, leakage edge cases, numerical-equivalence harness (r²±0.001), public-surface checklist |
A Dec-2025 pure-Python ground-up rewrite. Keys folds by row position (dag-ml rejects this). Internally self-contradictory on backward-compat. |
|
Generic-core stays/moves boundary map |
A third parallel design (“pipeforge”, generic Python core). |
Cross-repo integration contracts (adjacent, not core)¶
dag-ml-data:docs/ADR-0001-nirs4all-connector-ownership.md · nirs4all-methods:docs/nirs4all_integration_map.md (per-class PLS/model → libn4m swap; largely blocked today) · nirs4all-core:docs/PARITY.md (full-Python nirs4all = oracle of record) · nirs4all-formats:docs/INTEGRATION_NIRS4ALL.md (leaf readers, no dag-ml content).
Suggested reading order (first day on the chantier)¶
../CAPABILITY_MATRIX.md— what dag-ml takes over, feature by feature.../COORDINATOR_SPEC.md(the nirs4all confrontation section) — how it maps.../design/DSL_NIRS4ALL_PARITY.md— exactly which DSL constructs are covered / gapped.../adr/ADR-17-cutover-rollback.md+../adr/ADR-01-compatibility-ledger.md— the safety strategy.../STATUS.md+../SUPPORTED.md+../HOST_ADAPTER_BACKLOG.md— what’s real today + the wire protocol.nirs4all:tests/integration/parity/README.md+_registry.py— the gate.nirs4all:docs/_internal/god_classes_modularization.md— the decoupling work.../design/source/_archive/dag_ml_externalization_from_code.md§(9-step) +dag_ml_specification_v1.md§19 — the detailed extraction order.
dag-ml readiness (the honest ledger)¶
Ready (control plane): full COMPILE→PLAN→FIT_CV→SELECT→REFIT→PREDICT→EXPLAIN with
deterministic scheduling; nirs4all-compatible JSON DSL importer (list/dict pipelines,
_or_/_cartesian_/_chain_/_grid_/_range_/_log_range_/_zip_/_sample_, branch/merge/split/sources);
identity-keyed FoldSet + OOF-join-by-sample-id + leakage refusal by default (UC6/UC11 as
conformance fixtures); deterministic variant generation/selection with fingerprints; replay
bundles + provenance (RO-Crate / PROV / OpenLineage); C-ABI + process-adapter JSONL path with
a shipped sklearn adapter; heterogeneous multisource conformance pack (D8/D9).
Gaps blocking core migration:
No Python-object/YAML frontend — the Rust importer eats serialized JSON; nirs4all must serialize live pipeline objects / splitters / sklearn instances to portable descriptors (today
operator_classis a short name, lossy for live instances).No production host controllers beyond sklearn — all 42 nirs4all model estimators currently route through process adapters; the nirs4all-methods/libn4m swap is a separate, largely-blocked effort.
Production data-provider path incomplete — multi-node graphs need per-node dag-ml-data provider wiring; host-filtered
branch_viewmodes (by_metadata/by_tag/by_filter) not yet production.Performance unmeasured — only two ignored ~1.5 s sanity probes; no throughput/memory/end-to-end campaign benchmarks; per-task JSONL process-adapter overhead is the biggest unknown at nirs4all scale.
nirs4all-side decoupling not done —
SpectroDataset.x()+TransformerMixinControllerwavelength injection hotspots + the 12 god classes must be split along the dag-ml ownership seam.Parity oracle is a scaffold — no captured gold baseline, tolerances recorded but unenforced, no dag-ml backend hook wired in yet.
Cross-repo reducer-vocabulary drift —
robust_mean/exclude_outliersexist data-side only; must reconcile on the ADR-10 release train.
Open decisions — only the maintainer can make these (see proposals for recommendations)¶
Destination — dag-ml directly as the new core (recommended; per CAPABILITY_MATRIX/COORDINATOR_SPEC) vs the intermediate generic “pipeforge” Python lib vs the v2_design pure-Python rewrite. Pick one; retire the other two (or mark parity-input-only) before any decomposition.
Mechanism — process-adapter JSONL (only stable cross-language path today, per-task overhead) vs in-process C-ABI / a future PyO3 binding. Gates the GIL/async (ADR-15) + perf work.
Backward-compat posture — sanction ADR-14 “managed-debt” shims for the migration window (conflicts with nirs4all/CLAUDE.md’s absolute no-shims rule)?
Sequencing — god-class decomposition + gold-baseline capture before the bridge, or in parallel? (Externalization notes recommend extract-protocols-first.)
v1 scope — basic surface (baseline + linear/branch/merge/stacking/generators) only, or include heterogeneous multi-source repetitions (N0–N8/D0–D6) from day one?
Numerics ownership — run nirs4all operators as-is via the sklearn process adapter, or pursue the nirs4all-methods/libn4m swap concurrently?
Tracked vs local docs — several dag-ml core docs are gitignored/local-only (
.gitignore23–32). Commit them (so the migration plan is versioned), or keep local?Perf gate — acceptable end-to-end overhead vs legacy (ADR-17 dual-run) before flipping the default backend?
Recon provenance: 5f20f507…/tasks/w3tix7niq.output (full structured synthesis), 2026-06-23.
