# DAG-ML Coordinator Short Spec

Status: working normative spec.

This document is the short alignment contract for the product. It resolves the
ambiguity left by the long design documents: DAG-ML is a Rust pipeline
coordinator, not an ML algorithm library and not a data framework.

## Product Goal

DAG-ML must virtually execute any data pipeline that can be expressed as:

1. a compiled DAG of data/model/control dependencies;
2. an experimental campaign plan: variants, splits, repetitions, refit, predict,
   explain;
3. external operators reachable through language bindings/controllers;
4. typed data contracts and identity tables supplied by `dag-ml-data` or an
   equivalent data provider.

The product promise is:

- reproducible execution;
- traceable execution;
- leakage-safe OOF campaigns;
- real compiled DAG semantics;
- high-performance Rust orchestration and scheduling;
- external operators, data buffers and fitted objects kept outside the core.

## Non-Negotiable Principles

1. The Rust core owns logic, not domain algorithms.
2. Operators are external payloads: sklearn estimators, torch modules, C++
   methods, bioinformatics tools, filters, splitters, augmenters.
3. Controllers live in bindings or native plugin layers. They adapt external
   operators to the Rust core contract.
4. The Rust core owns compilation, planning, validation, scheduling, RNG,
   fingerprints, lineage, cache keys, OOF joins, and campaign state.
5. The Rust core never inspects feature buffers, images, spectra, tensors,
   sequences, graphs, or fitted model internals.
6. The Rust core may inspect only identity, fold membership, prediction tables,
   target/scoring tables, descriptors, fingerprints and controller manifests.
7. Every join, split, merge and prediction use is keyed by stable identities, not
   by row position.
8. Unsafe leakage paths are refused by default and must be explicit, traceable and
   searchable when allowed.

## Ownership Boundary

| Area | Rust core owns | Binding/controller owns | Data layer owns |
|---|---|---|---|
| DSL | compiled IR target, not syntax sugar | frontend syntax helpers | none |
| DAG | graph topology, ports, branches, merges | external node behavior | source contracts only |
| Campaign | variants, split plan, phases, refit/replay | splitter/model execution | sample relation facts |
| Controllers | manifest validation, invocation protocol | actual adapter code | none |
| Data | views by identity, DataPlan refs | host handles | schema, sources, adapters, collation |
| Execution | scheduler, tasks, phase gates, reducers | fit/predict/transform/split calls | materialization |
| Reproducibility | RNG contexts, fingerprints, canonical plans | local RNG consumption from seed | schema/data fingerprints |
| Traceability | lineage records, cache keys, artifact refs | artifact serialization hooks | fitted data-adapter refs |
| OOF safety | fold validation, prediction store, OOF join | emits predictions with identity | group/origin/repetition facts |

## Vocabulary

### Operator

External business payload. Examples: sklearn estimator, torch module, C++ PLS,
BLAST wrapper, aligner, filter predicate, augmenter, splitter.

An operator is never owned by the Rust core.

Frontend syntax should therefore prefer minimal opaque references when the
operator is unambiguous. A bare `SNV` is compiled as a transform node whose
operator payload remains `"SNV"`; a binding-side registry may instantiate the
class and route it to a generic transformer controller. Verbose descriptors
(`class`, `function`, explicit params, controller hints) are only required when
the payload needs parameters or when registry routing would otherwise be
ambiguous.

If controller manifests are present while compiling a DSL graph or execution
plan, their `operator_selectors` may also classify a minimal alias before the
graph ports are frozen. This lets a custom bare alias stay compact even when
Rust cannot infer its category by name; if multiple operator kinds claim it,
the compiler rejects the alias and asks for explicit syntax.

### Controller

Binding-side or native adapter that knows how to call an operator. A Python
controller may call sklearn. A native controller may call C++. An R controller may
call mlr3. A bioinformatics controller may call a local library or process.

Controllers are registered through a manifest and invoked through ABI/binding
contracts.

### Rust Core

The coordinator. It compiles, plans, validates, schedules and records execution.
It treats controllers as callable endpoints with declared capabilities.

### Data Provider

The owner of source storage and heavy buffers. For this workspace, `dag-ml-data`
is the reference contract. Other data providers can exist if they satisfy the
same identity/data-plan boundary.

## Controller Contract

Each controller must expose a manifest equivalent to:

| Field | Purpose |
|---|---|
| `controller_id` | stable unique id |
| `controller_version` | semantic version used in fingerprints |
| `operator_kind` | model, transform, splitter, augmenter, filter, metric, generator, tuner |
| `supported_phases` | compile, plan, fit_cv, select, refit, predict, explain |
| `input_ports` / `output_ports` | typed port contracts |
| `data_requirements` | `ModelInputSpec`, aux inputs, required sources or target |
| `capabilities` | deterministic, thread_safe, process_safe, needs_python_gil, emits_predictions, consumes_oof_predictions, emits_artifacts, stateful, emits_relation, accepts_task_batch, accepts_static_subgraph |
| `operator_selectors` | optional alias/class/function/ref/type selectors used by the registry to route opaque operator payloads before falling back to generic kind routing |
| `batch_patterns` | optional static patterns for controller-side multitask execution, including allowed node kinds, axes, limits and compatibility constraints |
| `fit_scope` | stateless, fold_train, full_train, inference_only |
| `rng_policy` | uses core seed, ignores seed, externally deterministic, nondeterministic |
| `artifact_policy` | serializable, host_only, content_addressed, replay_required |

Minimum callable operations:

| Operation | Used for |
|---|---|
| `describe` | manifest and static contracts |
| `plan` | controller-specific plan details, no heavy compute |
| `invoke` | phase-specific execution over handles and identity views |
| `invoke_batch` | optional execution of a validated group of logical tasks or a closed static subgraph |
| `release` | lifecycle cleanup for host handles |

Specialized operations such as `fit`, `predict`, `transform`, `split`,
`augment`, `score` may exist behind `invoke`, but the Rust core should schedule
typed tasks rather than know library-specific APIs.

`invoke_batch` is never a license for controller-owned topology. It is a
transport and scheduling optimisation for logical tasks that the Rust core has
already planned. A controller may use it to launch many preprocessings on one
GPU call or to execute a closed, static subgraph, but it must return one
logical response per member task so cache, OOF validation, lineage and replay
remain coordinator-owned.

Controller inputs:

- opaque data handles;
- selected identity view;
- phase;
- fold id, branch path, variant id, trial id;
- deterministic `SeedContext`;
- data-plan/fingerprint refs;
- controller params.

Data-plan references are `DataBinding` contracts owned by the coordinator:

- node id and input name;
- `dag-ml-data` request id;
- schema fingerprint;
- data-plan fingerprint;
- optional relation fingerprint;
- output representation;
- feature set id used for `feature_arrow` requests, defaulting to input name
  when omitted;
- feature fusion selectors may be passed through the same `feature_arrow`
  bytes-view when a data provider supports `dag-ml-data` multi-source fusion;
- source ids;
- view policy for fold train, fold validation, refit and predict.

The actual data plan and relation table remain external. `dag-ml-data` can emit
a coordinator envelope containing these fingerprints plus coordinator relation
records; `dag-ml` validates that an execution campaign binds to the exact
envelope before a controller receives any handle.

The shared serialization contract is versioned as
`CoordinatorDataPlanEnvelope` v1. `dag-ml` consumes the subset represented by
`ExternalDataPlanEnvelope`, rejects unsupported future versions at runtime, and
publishes the current JSON Schema at
`docs/contracts/coordinator_data_plan_envelope.schema.json`. The schema is the
wire-contract artifact; Rust validation remains responsible for semantic
checks such as fingerprint equality and relation membership in the active
campaign fold set.

At execution time, the scheduler does not give the controller the raw
materialized data handle directly. It asks the data provider for a scoped view
derived from the active phase and `DataViewPolicy`: `FIT_CV` tasks receive a
fold-train view for fitting and a separate fold-validation view for OOF
prediction, refit/full-train views carry the full training sample ids, and
replay predict views are marked as predict partitions. The handles visible in
`TaskRequest` are scoped data-view handles, and the same request carries a
`data_views` map keyed like `input_handles` so bindings can inspect the selected
partition without guessing from the handle. The parent handle remains
traceability state owned by the provider. Unsafe data views are rejected unless
declared explicitly: `FIT_CV` fitting cannot use full-train or validation
partitions, validation/predict views cannot include augmented rows, and excluded
rows cannot be included unless the corresponding `DataViewPolicy.unsafe_flags`
entry is present.

Controller outputs:

- opaque data/model/artifact handles;
- prediction blocks;
- sample relation deltas;
- metrics;
- structured errors;
- artifact refs or serialization bytes.

## Public Method Shape

The data/model piloting contract must be visible in public Rust and binding
APIs. It must not be hidden inside controller-specific side effects.

Minimum public coordinator flow:

| Method | Inputs | Output |
|---|---|---|
| `compile` | frontend pipeline IR, frontend registry | `GraphSpec`, `CampaignSpec` |
| `plan` | graph, campaign, controller manifests, dataset schema, data planner | immutable `ExecutionPlan` |
| `fit_cv` | execution plan, data provider handle, controller registry, stores | `CVResult` with prediction and lineage refs |
| `select` | CV result, ranking policy | `SelectedGraph` / selected variant refs |
| `refit` | selected graph, execution plan, data provider handle, stores | `RefitResult` / bundle inputs |
| `export_bundle` | selected graph, refit result, artifacts, fingerprints | `ExecutionBundle` |
| `predict` | bundle, new data provider handle | prediction blocks |
| `explain` | bundle, new data provider handle, target node/method | explanation payload refs |

Minimum controller-facing request/response shapes:

| Type | Required fields |
|---|---|
| `ControllerPlanRequest` | node id, operator params, phase set, data requirements, input/output ports, data schema fingerprint |
| `SplitRequest` | identity table, sample relation table, split policy, seed context |
| `TaskRequest` | phase, node id, fold id, branch path, variant id plus generated choices/fingerprint, data view, data-plan refs, input handles, prediction input metadata, artifact input metadata, seed context |
| `TaskResponse` | output handles, prediction blocks, sample relation deltas, metrics, artifacts, lineage payload |
| `TaskBatchRequest` | batch id, batch pattern id, canonical member task ids, ordered `TaskRequest` members, parent seed, resource limits |
| `TaskBatchResponse` | batch id, one `TaskResponse` per member task id, optional parent batch metrics/lineage |

Every shape-changing operation must declare the affected domain:

| Domain | Examples | Contract |
|---|---|---|
| row domain | sample filtering, sample augmentation, repetitions, group splits | changes identity/sample relation or view membership |
| feature domain | preprocessing, feature augmentation, feature selection, source fusion | changes representation/schema/feature names |
| target domain | y-transform, target aggregation, multi-target mapping | changes target space and inverse-transform requirements |
| prediction domain | model predict, OOF join, aggregation of repetitions | changes prediction block shape and aggregation level |

## DAG, Campaign Plan And Splits

This is the key correction.

There are two related but distinct plans:

### GraphPlan

The compiled DAG of dependency semantics:

- transforms;
- y-transforms;
- models;
- feature joins;
- prediction joins;
- source joins;
- fork/map/branch;
- aggregators;
- filters;
- augmentation nodes;
- generator nodes;
- tuners;
- explain nodes.

Graph nodes describe dependency and phase behavior. External behavior is executed
by controllers.

### CampaignPlan

The experimental execution plan around the graph:

- root seed;
- variants/search space;
- split strategy;
- repeated campaigns;
- nested CV policy;
- selection/ranking policy;
- refit policy;
- predict/explain replay policy;
- scheduler/resource policy.

### Generation Ownership

Generation has two different meanings and the ownership must stay explicit:

- compile-time generation of variants/search spaces belongs to `dag-ml`, because
  it changes campaign fingerprints, variant ids, seeds, lineage and selection;
- runtime generation of data, features, models or synthetic samples is an
  external operator capability. The payload lives in controllers or
  `dag-ml-data`, while `dag-ml` validates seeds, origin/sample relations,
  shape deltas, fold boundaries and lineage.

The current core represents compile-time generation through `GenerationSpec`
and `VariantPlan`. Choices may carry typed node-parameter overrides; the
scheduler lowers those into the effective `NodePlan.params` and
`params_fingerprint` sent to controllers for that variant. When a graph declares
`search_space_fingerprint`, plan compilation verifies it against the canonical
campaign `GenerationSpec` fingerprint before variants are enumerated. Runtime
generators are graph nodes/controllers with explicit capabilities such as
`generates_data`, `generates_model` or `expands_variants`; they are not allowed
to mutate identity or training boundaries without emitting relation and shape
deltas.

### Splitters

Splitters are not ordinary data-transform operators.

A splitter is a controller capability invoked by the coordinator during campaign
planning or early execution. It produces a `FoldSet`. The Rust core validates the
`FoldSet` against identity, group, target, repetition and origin constraints.

Native identity splitters such as KFold and GroupKFold may be shipped by the Rust
core for deterministic cross-language behavior. Feature-dependent splitters such
as KS/SPXY/sklearn splitters are external controller calls.

Implementation consequence:

- `NodeKind::Split` must not mean "data transform node".
- If a split appears in frontend syntax, the compiler must lower it into
  `CampaignPlan.split_invocation`, not a normal feature-flow node.
- A legacy or compatibility graph node may exist only as a control node that
  emits no feature data and is excluded from model/data transforms.

## Identity And Leakage Units

The coordinator must distinguish physical rows from logical samples. This is
critical for products with repeated measurements: several `X` observations can
share one `Y`.

| Identity | Meaning | Owned by |
|---|---|---|
| `ObservationId` | physical row/acquisition in one source | data provider |
| `SampleId` | logical sample requested by the user | data provider, validated by core |
| `TargetId` | target/label unit; may cover one or more samples | data provider, validated by core |
| `GroupId` | leakage unit such as plant, product, patient, plot, batch | data provider, validated by core |
| `OriginId` | original sample/observation from which an augmented row was derived | data provider/controller, validated by core |
| `RepetitionId` | repeated acquisition id inside a sample | data provider |

Split policies must declare the leakage unit:

| `split_unit` | Rule |
|---|---|
| `observation` | only the same observation is atomic; unsafe for repeated X / one Y unless explicitly allowed |
| `sample` | all observations of one sample stay on the same fold side |
| `target` | all samples sharing one target stay on the same fold side |
| `group` | all samples/observations sharing one group stay on the same fold side |

Defaults:

- repeated measurements default to `split_unit="sample"`;
- group ids, when present and requested, dominate sample ids for leakage;
- augmentation origin constraints are always checked unless explicitly disabled by
  a traceable unsafe policy;
- fold membership is stored at sample/target/group level and broadcast to
  observations during materialization.

The core refuses a split when:

- a requested split unit is absent from the sample relation table;
- one leakage unit appears in both train and validation of a fold;
- an augmented row derived from a validation origin appears in train;
- repeated observations of the same sample are split across train/validation
  under `split_unit="sample"`;
- a feature-dependent splitter returns folds not expressible in stable ids.

### Heterogeneous Multi-Source Units

For pipelines where one physical sample carries several observations per source
with asymmetric cardinalities, unit domains are named by an `EntityUnitLevel`:
`physical_sample` (the `SampleId`-level leakage/target/output unit),
`source_sample` (one source's observations of a physical sample), `observation`
(one acquisition, `ObservationId`) and `combo` (a derived observation built from
one observation per source for the same physical sample). In the mainline a
`combo` is a relation-backed derived observation/unit carrying
`component_observation_ids` and `origin_sample_id`; it is not a public
`PredictionLevel`. The public prediction levels stay
`observation`/`sample`/`target`/`group`, and cartesian/combo work reduces to
sample-level output by identity, never by row position. Promoting `combo` or
`source_sample` to a first-class public prediction level is a deferred,
explicitly-gated decision. The frozen vocabulary (`EntityUnitLevel`,
`PredictionUnitId`, `ReductionPlan`, `RepresentationPlan`, `FitInfluencePolicy`),
the mainline decision and the ADR-02 migration checklist are in
`docs/adr/ADR-19-multisource-unit-vocabulary.md`.

## Repetitions, Aggregation And Refit

Repeated observations are first-class. The core must support models trained on
individual observations while also evaluating and selecting on aggregated
sample-level predictions.

Example: several spectra (`ObservationId`) for the same product (`SampleId`) with
one chemical value (`TargetId`).

Required prediction levels:

| Level | Meaning |
|---|---|
| `observation` | one prediction per physical acquisition |
| `sample` | aggregation of observations for the same sample |
| `target` | aggregation of samples for the same target unit |
| `group` | aggregation at group level, only if explicitly requested |

Required aggregation policies:

| Policy field | Purpose |
|---|---|
| `aggregation_level` | observation, sample, target, group |
| `method` | mean, weighted_mean, median, vote, custom_controller |
| `weights` | none, quality, repetition_count, controller_emitted |
| `emit_parallel_metrics` | whether raw and aggregated metrics are both computed |
| `selection_metric_level` | which level drives variant/model selection |
| `store_raw_predictions` | keep observation-level predictions for audit |
| `store_aggregated_predictions` | keep aggregated predictions for ranking/replay |

The Rust core implements deterministic observation-to-sample and
sample-to-target/group aggregation for `mean`, `median`, `vote` and
`weighted_mean`. Weighted means require an explicit weight policy;
`controller_emitted` and `quality` weights are read from observation prediction
blocks, while `repetition_count` weights sample-to-target/group aggregation by
the number of observations attached to each sample. `custom_controller` remains
an explicit controller responsibility.

FIT_CV requirements:

1. Models may fit on observation-level rows if the data plan says so.
2. Validation predictions are first captured at the controller-emitted level.
3. The Rust core aggregates predictions by identity according to policy.
4. Metrics are computed in parallel when requested. The core provides
   identity-aligned regression scoring for validated sample/target/group
   prediction blocks (`mse`, `rmse`, `mae`, `r2`) and refuses positional-only
   or mismatched unit sets. Metric reports preserve the scored prediction
   origin: producer node, partition, optional fold and optional prediction id.
5. Selection must declare which metric level is authoritative. Selection
   policies can require a `metric_level`, and candidates whose metric metadata
   is missing or comes from another level are rejected before ranking.
6. OOF joins use the declared aggregation level and must not mix raw and
   aggregated predictions implicitly.
7. Replay-facing prediction contracts carry `prediction_level` explicitly.
   Sample-level replay caches carry `sample_ids`; target/group replay caches
   carry `unit_ids` typed as `PredictionUnitId`, validated in bundle records,
   payloads and file/in-memory stores. Aggregated caches are not preloaded into
   the sample OOF store.

REFIT requirements:

1. The final fit boundary is the selected training universe, excluding held-out
   test samples.
2. Refit may train on all repeated observations of selected samples if the
   `DataModelShapePlan` declares observation-level fitting.
3. Refit prediction outputs must preserve the same aggregation policy used for
   selection unless the user explicitly chooses a different predict policy.
4. Final prediction blocks may store both raw repetition predictions and
   aggregated sample predictions.
5. A bundle must record whether the selected model was chosen by observation,
   sample, target or group metrics.
6. Prediction requirements, prediction-cache records, prediction-cache payloads
   and `NodeTask.prediction_inputs` must preserve the prediction level, so a
   target/group metric decision cannot be replayed through an implicit
   sample-level cache.

## Data/Model Shape Plan

The coordinator must make shape control explicit because augmentations,
selection, fusion and aggregation change what a controller sees.

Each model or transform node receives a `DataModelShapePlan`:

| Field | Meaning |
|---|---|
| `input_granularity` | observation, sample, target, group |
| `target_granularity` | observation, sample, target, group |
| `fit_rows` | train observations/samples allowed for fit |
| `predict_rows` | validation/test/final rows expected for predict |
| `feature_namespace` | source/branch/augmentation prefixes |
| `feature_schema_fingerprint` | stable identity of column/feature layout |
| `target_space` | raw, transformed, scaled, encoded |
| `aggregation_policy` | how controller predictions are reduced |
| `augmentation_policy` | sample/feature augmentation rules |
| `selection_policy` | feature selection and supervised selection fit scope |

Shape-changing controllers must return a shape delta:

| Delta | Examples | Required validation |
|---|---|---|
| `row_delta` | sample filter, sample augmentation, separation branch | identity and fold boundaries remain valid |
| `feature_delta` | feature augmentation, selection, source fusion | feature schema fingerprint changes deterministically |
| `target_delta` | y scaling, target encoding | inverse-transform and target space are recorded |
| `prediction_delta` | probability output, repetition aggregation | prediction columns and aggregation level are recorded |

The core validates deltas before downstream tasks can consume them.
Feature deltas are checked against any declared
`feature_schema_fingerprint`; lineage may also echo the data/model shape and
aggregation-policy fingerprints, and when present those fingerprints must match
the compiled `NodePlan`.

## Augmentation, Selection, Filtering And Fusion

These operations are leakage-sensitive because they can change rows, features or
target spaces.

### Sample Augmentation

Sample augmentation creates new observations or samples. Default policy:

- FIT_CV: train partition only;
- validation/test rows are never augmented for training metrics;
- each augmented row carries `OriginId`;
- an augmented row inherits target, group and fold boundary from its origin;
- OOF and scoring are reported on original identities unless policy opts into
  augmented reporting.
The default Rust contract rejects sample augmentation across all partitions, or
sample augmentation without origin/target/group inheritance, unless the
corresponding `AugmentationPolicy.unsafe_flags` entry is present.

Forbidden by default:

- validation-origin augmentation appearing in train;
- augmented rows counted as independent samples for group/sample metrics;
- augmented validation/test rows entering view requests without an explicit
  unsafe flag;
- sample augmentation after a prediction join unless explicitly declared safe.

### Feature Augmentation

Feature augmentation changes columns/features but not identity.

Rules:

- stateless feature augmentation may run on train/validation/test if declared
  deterministic and fitted nowhere;
- stateful feature augmentation must fit on fold train and apply to validation;
- supervised feature augmentation is treated like supervised feature selection and
  must use fold-train targets only;
- feature namespaces and schema fingerprints must change deterministically;
- source/branch/augmentation provenance is preserved for merge and explain.

### Feature Selection

Feature selection is a transform that may be supervised or unsupervised.

Rules:

- supervised feature selection fits only inside the current train boundary;
- selected feature masks are artifacts with fold/refit lineage;
- supervised feature selection must store masks so CV/refit replay can audit
  which features were fitted inside each train boundary;
- validation/test/final data only receive `apply`, never `fit`;
- downstream feature joins must verify compatible selected schemas or use an
  explicit missing-feature policy;
- selection masks used at REFIT are recorded separately from CV-fold masks.

### Sample Filtering / Exclusion

Filtering changes row membership.

Rules:

- filters must declare whether they affect train only, predict only, all
  partitions, or branch-local views;
- train-only exclusion cannot silently remove validation samples from scoring;
- separation branches must produce disjoint or explicitly overlapping identity
  sets according to branch policy;
- selector-driven separation branches are declared as campaign
  `branch_view_plans` over source ids, metadata, tags or provider-native filter
  expressions before host data providers materialize branch-local views;
- all filters emit a `row_delta` with before/after identity fingerprints.

### Source Fusion And Merge

Feature/source fusion changes feature shape and possibly missingness.

Rules:

- sample alignment remains data-layer work, but the core records and validates
  the chosen alignment plan fingerprint;
- feature joins are namespace-stable and branch-aware;
- prediction joins are OOF-checked by the core;
- mixed joins must declare which inputs are raw features and which are
  predictions, with separate leakage checks.

## Phase Model

The coordinator executes:

```text
COMPILE -> PLAN -> FIT_CV -> SELECT -> REFIT -> PREDICT -> EXPLAIN
```

| Phase | Rust core responsibility | Controller responsibility |
|---|---|---|
| `COMPILE` | parse frontend IR into graph/campaign specs, freeze topology | provide descriptors if needed |
| `PLAN` | resolve controllers, ports, data plans, variants, split invocations | describe/plan only |
| `FIT_CV` | schedule fold/variant/branch tasks, enforce fold views, record predictions | fit/transform/predict/split/augment |
| `SELECT` | rank variants, reject unsafe variants by policy | optional metric helpers |
| `REFIT` | replay selected graph on full train boundary | fit final artifacts |
| `PREDICT` | replay bundle on new data, validate schema/fingerprints | predict/transform only |
| `EXPLAIN` | replay to explanation target | controller-specific explanation |

## ExecutionPlan

An `ExecutionPlan` is the Rust-owned, immutable plan after compile and plan:

- graph fingerprint;
- campaign fingerprint;
- data-plan fingerprints;
- controller manifests and versions;
- variant list or lazy enumerator;
- split invocation specs and resulting fold fingerprints;
- leakage unit policy;
- data/model shape plans per node;
- aggregation policies for prediction/evaluation/refit;
- topological order and deterministic parallel node levels;
- phase execution schedules expanded by variant and fold;
- controller multitask group templates for eligible known task groups or closed
  static subgraphs;
- phase gates per node;
- expected input/output contracts;
- cache key templates;
- lineage templates;
- scheduler policy.

No controller may mutate the `ExecutionPlan`.

## Runtime Execution

The Rust core runs task batches in deterministic dependency order.

Parallelism dimensions:

- variants;
- folds;
- branches;
- independent DAG nodes;
- controller multitask groups;
- controller-declared internal parallelism.

Scheduler rules:

1. Ready tasks may run in parallel only if dependencies and resource policy allow.
2. Reduction order is canonical: sort by variant id, fold id, branch path, node id.
3. Nested parallelism is controlled by controller capabilities and resource policy.
4. Python/GIL-bound controllers may be process-scheduled; native thread-safe
   controllers may be thread-scheduled.
5. Compatible ready tasks may be coalesced into a controller multitask batch
   only when a manifest-declared pattern admits them. Default compatibility is
   same phase, same fold, same data view and no hidden dependency between
   members.
6. Batches can be split by resource limits, and may fall back to scalar
   `TaskRequest`s unless the manifest explicitly marks the group as required.
7. A task result is accepted only after runtime validation.

## Runtime Validation

Before dispatch, the Rust core checks:

- required ports exist;
- phase is allowed by controller manifest;
- data plan fingerprint matches;
- requested data views match the active phase, fold id and partition;
- view sample ids are in the active fold/partition;
- validation prediction sample ids are contained in the fold-validation view;
- leakage unit membership is compatible with the active fold;
- task data/model shape plan matches the phase and controller manifest;
- seed is derived from the canonical path;
- unsafe policy is explicit if required.

For `TaskBatchRequest`, every member `TaskRequest` is checked first. The batch
itself is then checked against the selected `batch_patterns` entry: allowed
node kinds, coalescing axes, canonical member order, phase/fold/view/branch
compatibility, `max_tasks` and resource hints.

After dispatch, the Rust core checks:

- output ports match declared contracts;
- prediction blocks carry producer, fold, partition, sample ids and target names;
- no prediction row is positional-only;
- train/validation/test/final partitions are legal for the phase;
- augmentation origins do not cross fold boundaries;
- group/repetition/target leakage units remain on one side of a fold;
- shape deltas are declared before downstream consumption;
- aggregation level and prediction columns match the node policy;
- artifact refs, portable backend/URI/content metadata and handle lifetimes are
  registered;
- lineage was recorded.

For `TaskBatchResponse`, every member response is validated as if it had been
returned by a scalar invocation. Parent batch metrics and lineage are retained
only as extra provenance; they cannot replace member-level records.

## OOF And Leakage Rules

OOF safety is a Rust core invariant.

Rules:

1. Training a downstream model on upstream predictions requires validation OOF
   predictions by default.
2. `partition="train"` predictions cannot feed downstream training unless
   `allow_train_predictions_as_features=true` and a second lineage flag records
   `leakage_acknowledged=true`.
3. Every producer must provide exactly one validation prediction per requested
   sample unless an explicit aggregation policy says otherwise.
4. Producers merged by a prediction join must share compatible fold structure.
5. Augmented observations inherit origin leakage boundaries.
6. Group, target and repetition leakage units are validated from sample relation
   facts supplied by the data layer.
7. OOF joins are by sample identity, never by row position.
8. Repetition aggregation is explicit; raw observation predictions and aggregated
   sample/target predictions are never silently mixed.
9. Refit uses the selected aggregation and shape policies unless the predict/refit
   policy explicitly overrides them and records the override.

### Stacking Refit Coverage Policy

Prediction-stacking meta-nodes default to
`stacking_oof_refit_contract.policy="require_full_coverage"`. Under that
default, REFIT may train a meta-model only when each upstream OOF producer covers
the complete refit sample universe from validation OOF blocks. Partial OOF
coverage is rejected with the stable cause `partial_oof_without_policy`.

A stacking node may explicitly opt out of meta-model REFIT by carrying
`metadata.stacking_oof_refit_contract.policy="cv_only"`, or may run the CV path
and skip REFIT only when coverage is incomplete with
`"skip_refit_on_incomplete_oof"`. Both policies keep malformed OOF invalid:
non-validation blocks, missing fold ids, unknown folds, fold/sample mismatches
and duplicate validation samples are refused before any skip decision is
accepted.

## Traceability

Every accepted task emits or updates a `LineageRecord` containing:

- run id;
- graph/campaign/variant fingerprints;
- node id;
- phase;
- fold id;
- branch path;
- controller id/version;
- params fingerprint;
- data-plan fingerprint;
- data/model shape fingerprint;
- aggregation level and policy fingerprint;
- input lineage refs;
- output handle/artifact refs;
- seed;
- unsafe flags;
- metrics;
- timing and resource hints.

The lineage graph must be enough to answer:

- why was this sample in this fold?
- which controller produced this artifact?
- which seed and params were used?
- which data schema and data plan were used?
- did any unsafe leakage path occur?
- can this predict/explain run replay the training bundle?

Bundle and prediction-cache payload schemas are versioned artifacts. The core
publishes a `SchemaMigrationPolicy` for each artifact with current/min readable
and writable versions plus explicit automatic migration edges. No implicit
migration is allowed: old versions are accepted only if the policy declares a
migration edge, future versions are refused, and version `0` is always invalid.

## Performance Requirements

The Rust core must be designed as the high-performance coordination layer:

- compact immutable specs;
- canonical fingerprints without reparsing large payloads;
- parallel scheduler over variants/folds/branches/nodes;
- deterministic reducers;
- no feature-buffer copies through the core;
- opaque handle arenas with explicit release;
- prediction store optimized for identity joins;
- controller capability-aware scheduling;
- zero-copy Arrow/DLPack boundaries where identity, predictions or tensors cross.

The core must not chase performance by moving domain algorithms inside Rust. The
performance target is orchestration, validation, scheduling, lineage, and OOF
joins at scale.

## Confrontation With Current nirs4all Pipeline

Current nirs4all is the prototype. It proves the product need, but not the final
architecture.

| nirs4all today | DAG-ML target |
|---|---|
| Python-only runtime | Rust core with Python/R/JS/native bindings |
| Conceptual DAG reconstructed from sequential steps | Real compiled immutable DAG |
| Controllers do work and also carry some orchestration side effects | Controllers only execute declared external behavior |
| Split step mutates pipeline/dataset state | Splitter controller invocation produces `FoldSet` in `CampaignPlan` |
| Branching stored in mutable contexts/snapshots | `Fork/Map` graph semantics with explicit branch path |
| Merge reconstructs features/predictions in Python controllers | Rust-owned feature/prediction join contracts and OOF validation |
| Repetition averages such as `avg` / `w_avg` are controller/runtime conventions | Aggregation policy is explicit, fingerprinted and evaluated in parallel with raw predictions |
| Several spectra can share one product target through dataset conventions | `ObservationId`, `SampleId`, `TargetId`, `GroupId`, `OriginId` are first-class leakage identities |
| Feature/sample augmentation and selection are controller-specific effects | Shape deltas, fit scopes and augmentation/selection policies are validated by the core |
| NIRS-shaped data assumptions leak through dataset/model logic | Generic data contracts through `dag-ml-data` |
| Dynamic routing through Python registry | Binding registries export controller manifests to Rust |
| Trace, artifacts and prediction store are pipeline-specific | Rust-owned lineage/cache/artifact/prediction contracts |
| Reproducibility depends on Python conventions and local discipline | Core-derived seeds, fingerprints and deterministic reducers |
| Leakage safety exists but is embedded in controllers | Leakage safety is a core invariant checked around every task |

The migration path is not "port nirs4all as-is". It is:

1. extract the implicit DAG and campaign semantics;
2. encode them as `GraphPlan` and `CampaignPlan`;
3. move orchestration, validation and scheduling to Rust;
4. keep operators external through controllers;
5. keep data generic through data contracts;
6. replay nirs4all use cases as conformance tests, not as product constraints.

## What Must Be Implemented Next

The next implementation layer must make the coordinator visible:

1. `ControllerManifest`
2. `ControllerRegistry`
3. `CampaignSpec`
4. `GraphPlan`
5. `ExecutionPlan`
6. `NodePlan`
7. `NodeTask`
8. `NodeResult`
9. `RunContext`
10. sequential scheduler
11. in-memory prediction store
12. in-memory lineage recorder
13. mock controller conformance tests
14. split invocation model that produces and validates `FoldSet`
15. `LeakageUnitPolicy`
16. `AggregationPolicy`
17. `DataModelShapePlan`
18. `ShapeDelta`
19. augmentation, feature-selection and filtering policies

The existing OOF/fold/data-plan code remains useful, but it is not the product
shape by itself. The product shape starts when a compiled `ExecutionPlan` drives
controllers through tasks and validates their outputs.

## Acceptance Checks For This Spec

An implementation is aligned only if all answers are "yes":

1. Can the core compile a frozen graph and campaign plan without executing
   external code?
2. Can splitters be external controllers while the core owns the validated
   `FoldSet`?
3. Can model/process/augmentation/filter operators remain external payloads?
4. Can the scheduler parallelize variants, folds and branches without changing
   results?
5. Can the core reject OOF leakage without inspecting feature buffers?
6. Can a run be replayed from fingerprints, controller versions, data-plan refs,
   artifacts and lineage?
7. Can the same core coordinate Python, native C++, R, JS or bioinformatics
   controllers with the same safety rules?
8. Can repeated observations with one target be trained, scored, aggregated,
   selected and refit without leakage or hidden metric-level changes?
9. Can feature/sample augmentation, feature selection, filtering and fusion
   declare shape deltas that the core validates before downstream use?

If any answer is "no", the implementation has drifted from the product goal.