# Shared Contracts

This directory contains wire-contract artifacts shared with `dag-ml-data`, plus
DAG-ML-specific publication schemas. `dag-ml` remains the consumer and semantic
validator: it checks fingerprints, campaign fold membership, OOF boundaries and
leakage policies before any controller receives a handle.

## Heterogeneous Multi-Source Vocabulary and Evolution

The heterogeneous multi-source repetitions roadmap
(`docs/HETEROGENEOUS_MULTISOURCE_REPETITIONS_ROADMAP.md`) extends several of the
schemas below with optional unit-level fields.
`docs/adr/ADR-19-multisource-unit-vocabulary.md` freezes the vocabulary
(`physical_sample`, `source_sample`, `observation`, `combo`, `EntityUnitLevel`,
`PredictionUnitId`, `ReductionPlan`, `RepresentationPlan`, `FitInfluencePolicy`)
and records the mainline decision that combos are relation-backed derived
observations rather than a public `PredictionLevel`. Each phase that touches a
contract here follows the ADR-19 / ADR-02 checklist: optional fields first,
defaults/dual-read, fixture and conformance-pack update, CHANGELOG entry, and an
explicit C ABI decision. A first-class public `combo` / `source_sample`
prediction level remains deferred and gated.

## Coordinator Data Plan Envelope v1

Schema: `coordinator_data_plan_envelope.schema.json`

Canonical fixture: `examples/fixtures/data/coordinator_data_plan_envelope_nir.json`

Conformance pack: `conformance_pack.v1.json`

Parity oracle handoff: `parity_oracle.v1.json`

Public C ABI snapshot: `abi_snapshot.v1.json`

Runtime type consumed here: `ExternalDataPlanEnvelope`

Producer type in `dag-ml-data`: `CoordinatorDataPlanEnvelope`

The envelope binds a data plan to stable schema, plan and relation
fingerprints. It may carry coordinator relation records for sample, target,
group, origin, source and augmentation identity. The JSON Schema documents the
portable shape of that envelope; Rust validation enforces the stronger semantic
rules that depend on the active campaign.

Short-term policy: both repositories keep a JSON-identical conformance fixture
for this envelope plus a copy of the v1 schema, and test that the published
artifact declares the Rust-supported version. `scripts/validate_contracts.py`
compares the fixture and schema copies when `DAG_ML_DATA_REPO` points to a
sibling checkout, validates the shared conformance-pack digests, and CI checks
out that peer explicitly. When development moves into a monorepo, this file
should become a single generated or shared contract artifact used by both
crates.

The D8 multisource audit extends the shared conformance pack with seven named
scenarios: `multisource_a2_b3_c2.v1`, `sample_level_late_fusion.v1`,
`cartesian_combo_to_sample_reducer.v1`, `missing_source_with_fallback.v1`,
`stacking_oof_contract.v1`, `invalid_unit_join.v1` and
`row_vs_sample_selection_mismatch.v1`. These scenarios bind the D1-D7 public
surface changes to schema digests, canonical fixtures and concrete Rust/contract
test references. They intentionally remain metadata: the core still validates
relations, fingerprints, OOF safety and representation replay without owning
feature buffers or host model objects.

## Parity Oracle v1

Manifest: `parity_oracle.v1.json`

This is the producer-side handoff for the future `nirs4all` compatibility
ledger. It does not wire `nirs4all`; instead it names the parity cases,
fixtures, Python/WASM gates and invariants that the consumer ledger must bind
to public API rows before bridge work starts. `scripts/validate_contracts.py`
checks the manifest shape, verifies referenced `dag-ml`/`dag-ml-data` fixtures
when the sibling checkout is present, pins its digest in
`conformance_pack.v1.json`, and requires the manifest to stay byte-identical
across both repositories.

## Public C ABI Snapshot v1

Snapshot: `abi_snapshot.v1.json`

Header: `crates/dag-ml-capi/include/dag_ml.h`

`scripts/validate_abi_snapshot.py` checks the header SHA-256 against the
snapshot and runs in CI. Any C ABI header change must update this manifest in
the same review so downstream hosts can see the ABI movement explicitly.
The shared `dag-ml-data` conformance pack also requires the producer-side
multi-target Arrow helper
`dagmldata_coordinator_multi_target_arrow_json`, which is consumed as a data
provider capability rather than as a `dag-ml` header symbol.

## Coordinator Branch View v1

Schema: `coordinator_branch_view.schema.json`

Mirrors `dag-ml-data`'s `coordinator_branch_view.v1` byte-for-byte except for
the `$id` (each repo declares its own domain). The normalized SHA-256 (with
`$id` stripped) is pinned identically in both repos' `conformance_pack.v1.json`
so the wire contract cannot drift. `BranchViewPlan` records in
`dag-ml`/`crates/dag-ml-core/src/data.rs` accept the same shape and the
in-memory `dag-ml-data` provider executes `by_source` natively.

## Fitted Adapter Ref v1

Schema: `fitted_adapter_ref.schema.json`

Mirrors `dag-ml-data`'s `fitted_adapter_ref.v1` byte-for-byte except for the
`$id`. Same normalized-SHA-256 enforcement in both `conformance_pack.v1.json`
files. The producer type is `FittedAdapterRef` in `dag-ml-data`; `dag-ml`
relies on this schema to validate fitted-adapter records that flow through
the data layer at refit time.

## Feature Fusion Selector v1

Schema: `feature_fusion_selector.schema.json`

Canonical fixture: `examples/fixtures/data/feature_fusion_selector_nir_chem.json`

Runtime shape passed through data-provider `feature_arrow` when the provider
supports `dag-ml-data` multi-source fusion:
`{ schema_version, feature_set_id, sources, alignment, combination_plan?,
representation_plan?, policy? }`, where each source maps a `source_id` to a
provider-owned `feature_set_id` and optional column subset. The optional D6
plans describe host-owned representation work such as cartesian, sampled
cartesian, fixed stack and padded/masked stack materialization; the core
validates identity, unit, replay and provenance contracts without materializing
feature buffers. This keeps `DagMlDataVTable` ABI-compatible while making
feature fusion explicit.

## GraphSpec v1

Schema: `graph_spec.schema.json`

Canonical fixture: `examples/branch_merge_oof_graph.json`

Runtime type: `GraphSpec`

C ABI: `DAG_ML_GRAPH_SPEC_SCHEMA_VERSION`,
`dagml_graph_spec_contract_json`, `dagml_graph_validate_json`

This is the portable graph object produced by the DSL compiler and consumed by
the execution-plan builder. The schema documents node kinds, ports, edge
contracts, OOF prediction edges and lineage propagation flags so host bindings
can reject malformed graph JSON before controller resolution or scheduling.
Rust validation remains the semantic authority for uniqueness, endpoint checks,
port-kind alignment and cycle refusal.

Prediction-stacking meta-nodes reserve the optional node metadata key
`stacking_oof_refit_contract`. Its current shape is
`{"policy": "require_full_coverage" | "cv_only" | "skip_refit_on_incomplete_oof"}`.
The default is `require_full_coverage`: REFIT consumes validation OOF only when
the producer covers the complete refit sample universe. `cv_only` always skips
the stacking node during REFIT, while `skip_refit_on_incomplete_oof` skips only
when otherwise well-formed validation OOF is incomplete. Invalid OOF still fails
with a stable cause such as `partial_oof_without_policy` or
`non_validation_partition`; Rust validates the metadata object and the OOF
coverage semantics.

## Pipeline DSL v1

Schema: `pipeline_dsl.schema.json`

Canonical compatibility fixture: `examples/pipeline_dsl_nirs4all_compat.json`

Runtime parser: `parse_pipeline_dsl_json`

C ABI: `DAG_ML_PIPELINE_DSL_SCHEMA_VERSION`,
`dagml_pipeline_dsl_contract_json`, `dagml_pipeline_dsl_validate_json`,
`dagml_pipeline_dsl_compile_json`,
`dagml_pipeline_dsl_compile_artifact_json`,
`dagml_pipeline_dsl_execution_plan_build_json`

This is the public input contract for both canonical `PipelineDslSpec` JSON and
serialized nirs4all-style list/dict JSON. The schema documents the accepted
portable surface: canonical step kinds plus compatibility keys such as
`pipeline`, `preprocessing`, `model`, `branch`, `merge`, `split`, `sources`,
`_or_`, `_cartesian_`, `_chain_`, `_grid_`, `_range_`, `_log_range_`, `_zip_`
and `_sample_`. The compatibility profile also accepts minimal nirs4all-style
operator aliases: short strings such as `SNV`/`PLSRegression`, plain
`{"class": ...}` / `{"function": ...}` / `{"ref": ...}` objects, and
`{"name": ..., "step": ...}` wrappers. Rust classifies those aliases only far
enough to build the safe plan: splitters become campaign `SplitInvocation`
entries, obvious estimators become model nodes, obvious tuners such as
`OptunaTuner` become tuner nodes, chart aliases become chart nodes, and
everything else remains an external transform for host controller resolution.
When the compiler is given controller manifests (`compile-pipeline-dsl
--controllers` or `build-pipeline-dsl-plan`), selector-only aliases can refine
this default before graph ports are frozen: a custom bare alias such as
`ElasticSpectra` may become a model if exactly one operator kind claims it
through `operator_selectors`. Cross-kind matches are rejected and must use
explicit DSL syntax.
Rust validation remains the semantic authority: it lowers
compatibility JSON into canonical DSL, compiles the graph/campaign/generation
artifact, rejects unsafe augmentation/shape contracts and enforces OOF graph
edges.

External tuner/finetune controllers are canonical operator steps.
`kind: "tuner"` and its alias `kind: "finetune"` compile to
`NodeKind::Tuner`, preserve public tuning metadata and produce fold-aligned OOF
prediction outputs like model nodes; the actual search implementation remains in
the host controller.

Runtime data generators are canonical operator steps, separate from
compile-time search-space generators. `kind: "data_generation"` and its alias
`kind: "generation"` compile to `NodeKind::Generator` and require a public
`shape` contract so synthetic samples/features can be scoped to fold-train
data, audited through origin/group/target inheritance and executed by an
external controller.

## CampaignSpec v1

Schema: `campaign_spec.schema.json`

Canonical fixture: `examples/campaign_oof_generation.json`

Runtime type: `CampaignSpec`

C ABI: `DAG_ML_CAMPAIGN_SPEC_SCHEMA_VERSION`,
`dagml_campaign_spec_contract_json`, `dagml_campaign_validate_json`

This is the portable experimental-plan contract layered beside the graph. It
keeps split invocation, concrete fold sets, leakage-unit policy, repeated-sample
aggregation policy, generation/search dimensions, data/model shape plans and
data bindings outside operator nodes. Selector-driven separation branches are
recorded here as `branch_view_plans`, so source/metadata/tag/filter branch views
can be materialized by data-provider bindings without turning splits or filters
into graph operators. Rust validation remains the semantic authority for fold
membership, leakage guards, generation consistency, shape-plan/key alignment,
branch-view selector sanity and data-binding fingerprint requirements.

## ExecutionPlan v1

Schema: `execution_plan.schema.json`

Canonical fixture:
`examples/fixtures/runtime/execution_plan_branch_merge_executable.json`

Runtime type: `ExecutionPlan`

C ABI: `DAG_ML_EXECUTION_PLAN_SCHEMA_VERSION`,
`dagml_execution_plan_contract_json`,
`dagml_execution_plan_validate_json`

This is the compiled, scheduler-ready DAG contract. It binds the validated
graph, campaign, resolved controller manifests, per-node execution policies,
generation variants, fold set and canonical fingerprints used later by bundles,
replay and provenance exports. The schema documents the portable envelope and
critical coordination fields; Rust validation remains the authority for DAG
topology, controller-policy consistency, OOF capability checks, fold semantics,
shape/data binding checks and fingerprint consistency.

## ModelInputSpec v1

Schema: `model_input_spec.schema.json`

Canonical fixture:
`examples/fixtures/data/model_input_spec_tabular_regressor.json`

Runtime type: `ModelInputSpec`

C ABI: `DAG_ML_MODEL_INPUT_SPEC_SCHEMA_VERSION`,
`dagml_model_input_spec_contract_json`,
`dagml_model_input_spec_validate_json`

This neutral contract is the data/model compatibility request declared by a
controller or binding. It lists required input ports, accepted
representations/types, tensor rank expectations, multi-source support and the
default fusion policy to ask from a data planner.

For a multi-source `concatenate_features` request, the concrete
`DataBinding.metadata.source_index` map is the required source-concat layout
hint. It maps each source id to its feature-axis block index and is propagated
into `DataProviderViewSpec.extra.source_index`; without it the planner refuses
the binding with a structured `dagml.data_requirement.*` code instead of
silently treating early fusion as ordinary flat features. `by_source` branches
remain single-source per branch unless a future contract explicitly supports
grouped source branches.

## DataPlan v1

Schema: `data_plan.schema.json`

Canonical fixture: `examples/fixtures/data/data_plan_tabular_fusion.json`

Runtime type: `DataPlan`

C ABI: `DAG_ML_DATA_PLAN_SCHEMA_VERSION`, `dagml_data_plan_contract_json`,
`dagml_data_plan_validate_json`

This neutral contract is the data-planner answer to a `ModelInputSpec`: a
deterministic sequence of materialize/adapt/align/join/collate steps plus the
named outputs that feed model ports. DAG-ML validates ordering, output
references and refusal metadata before such a plan can become part of an
execution plan or bundle.

## ControllerManifest v1

Schema: `controller_manifest.schema.json`

Canonical fixture:
`examples/fixtures/runtime/controller_manifest_data_aware_model.json`

Runtime type: `ControllerManifest`

C ABI: `DAG_ML_CONTROLLER_MANIFEST_SCHEMA_VERSION`,
`dagml_controller_manifest_contract_json`,
`dagml_controller_manifest_validate_json`,
`dagml_controller_manifest_list_validate_json`

This is the binding-facing contract each external controller registry must
publish. It declares the controller id/version, operator kind, phase support,
ports, deterministic/replay capabilities, fit scope, RNG policy, artifact
policy and optional `ModelInputSpec` data requirements. The schema is the
portable shape; Rust validation remains the authority for registry uniqueness,
phase/fit-scope consistency, capability/port consistency and typed
`data_requirements` semantics.

`operator_selectors` are the minimal-alias bridge used by bindings. A host can
publish a `TransformerMixin` controller that matches aliases such as `SNV`,
plain strings such as `StandardScaler`, a tuner controller that matches
`OptunaTuner`, or class/function/ref/type descriptors; Rust keeps the operator
payload opaque, uses selectors to classify bare aliases when manifests are
available at compile time, and routes the node to the matching controller before
execution.

## NodeTask / NodeResult v1

Schemas: `node_task.schema.json`, `node_result.schema.json`

Canonical fixtures:
`examples/fixtures/runtime/node_task_transform_scale.json`,
`examples/fixtures/runtime/node_result_transform_scale.json`

Runtime types: `NodeTask`, `NodeResult`

C ABI: `DAG_ML_NODE_TASK_SCHEMA_VERSION`,
`DAG_ML_NODE_RESULT_SCHEMA_VERSION`, `dagml_node_task_contract_json`,
`dagml_node_result_contract_json`,
`dagml_node_result_validate_for_task_json`

These are the direct wire contracts between the Rust coordinator and external
operator controllers. `NodeTask` carries the resolved node plan, phase,
variant/fold context, handles, data views, OOF prediction inputs, refit artifact
inputs and deterministic seed. `NodeResult` returns output handles,
sample predictions, optional observation-level predictions, optional aggregated
sample/target/group predictions, shape deltas, artifacts and lineage. Rust validates every result
against the exact task before committing it, including node/run/phase/fold,
variant, controller, seed, params fingerprint, shape fingerprints, output
ownership and artifact handle consistency.

## SelectionPolicy / SelectionDecision v1

Schemas: `selection_policy.schema.json`, `selection_decision.schema.json`

Canonical fixtures: `examples/fixtures/bundle/selection_policy_rmse.json`,
`examples/fixtures/bundle/selection_decision_branch_b0.json`

Runtime types: `SelectionPolicy`, `SelectionDecision`

C ABI: `DAG_ML_SELECTION_POLICY_SCHEMA_VERSION`,
`DAG_ML_SELECTION_DECISION_SCHEMA_VERSION`,
`dagml_selection_policy_contract_json`,
`dagml_selection_decision_contract_json`,
`dagml_selection_policy_validate_json`,
`dagml_selection_decision_validate_json`

These contracts preserve the selection boundary used before refit/replay:
metric name/objective, optional required prediction level
(`observation`/`sample`/`target`/`group`), selected candidate, selected score
and the deterministic ranked candidate list. Rust validation remains the
semantic authority for rank continuity, selected-candidate consistency,
duplicate candidates and finite selected scores.

## DAG-ML OpenLineage Facets v1

Schema: `openlineage_dagml_facets.schema.json`

This is a DAG-ML-specific publication contract, not a shared `dag-ml-data`
wire contract. `export-open-lineage` derives an OpenLineage `RunEvent` from an
already validated research provenance package and uses these custom `dagml_*`
facets to preserve DAG-ML fingerprints, OOF coverage counters, unsafe flags and
bundle/plan identifiers that OpenLineage does not model natively.

## Prediction Cache Tensor Metadata v1

Schema: `prediction_cache_tensor_metadata.schema.json`

This C ABI metadata contract accompanies
`dagml_prediction_cache_payload_f64_tensor_json`. The tensor carries contiguous
row-major F64 prediction values; the metadata carries the stable requirement
key, cache id, prediction level, block offsets, fold ids, sample ids, unit ids
and target names required to interpret rows without hiding traceability inside
the value buffer.

## Prediction Cache Columnar Tensor Metadata v1

Schema: `prediction_cache_columnar_tensor_metadata.schema.json`

This C ABI metadata contract accompanies
`dagml_prediction_cache_payload_f64_columnar_tensor_json`. It keeps the same
traceability fields as the row-major export and adds `layout:
column_major_f64` plus `column_offsets` so host bindings can read each target
column contiguously without guessing buffer order.

## Data Output Provenance v1

Schema: `data_output_provenance.schema.json`

Canonical fixture:
`examples/fixtures/runtime/data_output_provenance_augmented_view.json`

Runtime type: `DataOutputProvenance`

C ABI: `DAG_ML_DATA_OUTPUT_PROVENANCE_SCHEMA_VERSION`,
`DAG_ML_DATA_OUTPUT_PROVENANCE_EXTRA_KEY`,
`dagml_data_output_provenance_contract_json`,
`dagml_data_output_provenance_validate_json`

This DAG-ML runtime contract is embedded under the reserved
`DataProviderViewSpec.extra["dag_ml_output"]` key when a data-producing DAG node
emits a downstream data view. It records the producer node/port/phase,
variant/fold scope, shape-plan and aggregation fingerprints, current feature
schema fingerprint and emitted shape deltas. D6/D7 add optional representation
plans, replay manifests, relation-delta fingerprints and train/predict
compatibility reports so serve-time missing source or repetition differences
are explicit, policy-bound and replayable. Controllers and host bindings can
discover and validate this metadata without reverse-engineering free-form JSON
or hardcoding Rust-only constants.

## Process Adapter Description v1

Schema: `process_adapter_description.schema.json`

Canonical fixture:
`examples/fixtures/runtime/process_adapter_description_python.json`

Runtime shape returned by process adapters from `--describe`:
`{ schema_version, protocol, adapter_id, supported_modes, capabilities }`.

C ABI: `DAG_ML_PROCESS_ADAPTER_DESCRIPTION_SCHEMA_VERSION`,
`dagml_process_adapter_description_contract_json`

This CLI/runtime contract lets the coordinator reject unsupported process
adapters before any `NodeTask` is sent. Version 1 requires protocol
`dag-ml-process-adapter`, mode declarations for `one_shot`/`jsonl` support and
explicit JSON task/result capabilities. Persistent worker and parallel
scheduler features remain opt-in capabilities layered on the same description
object.

## Process Adapter Frame v1

Schema: `process_adapter_frame.schema.json`

Canonical fixtures:
`examples/fixtures/runtime/process_adapter_frame_init.json`,
`process_adapter_frame_task_transform_scale.json`,
`process_adapter_frame_result_transform_scale.json`,
`process_adapter_frame_ack_initialized.json`,
`process_adapter_frame_error_retryable_timeout.json`,
`process_adapter_frame_close.json`

Runtime shape used by persistent JSONL process adapters:
`init | task | close` coordinator request frames and
`ack | result | error` adapter response frames.

C ABI: `DAG_ML_PROCESS_ADAPTER_FRAME_SCHEMA_VERSION`,
`dagml_process_adapter_frame_contract_json`

This contract is enabled only when the adapter description declares
`control_frames_v1`. It gives host adapters a stable lifecycle and error
surface: `init` carries controller and worker identity, `task` wraps a
published `NodeTask`, `result` wraps a published `NodeResult`, `error` carries
typed retryability, and `close` gives the coordinator a bounded shutdown path.

## Aggregation Controller Task/Result v1

Schemas: `aggregation_controller_task.schema.json` and
`aggregation_controller_result.schema.json`

C ABI: `DAG_ML_AGGREGATION_CONTROLLER_TASK_SCHEMA_VERSION`,
`DAG_ML_AGGREGATION_CONTROLLER_RESULT_SCHEMA_VERSION`,
`dagml_aggregation_controller_task_contract_json`,
`dagml_aggregation_controller_result_contract_json`,
`dagml_aggregation_controller_task_validate_json`,
`dagml_aggregation_controller_result_validate_for_task_json`

These contracts define the leakage-sensitive payloads used when aggregation is
delegated to an external controller through `AggregationMethod::CustomController`.
The task carries the custom aggregation policy, controller id, repeated
observation or sample-to-unit inputs, relation metadata and requested output
order. The result is validated against the exact task so custom reducers cannot
change sample/unit coverage, fold scope, target names or prediction level.

## Research Provenance Package Profile v1

Profile: `research_provenance_package_profile.v1.json`

This publication profile declares the required files, optional files, checksum
rules, PROV JSON-LD sections, RO-Crate file properties, OpenLineage facets and
CLI tests for a DAG-ML research package. It is validated by
`scripts/validate_contracts.py` so the human-facing publication contract stays
aligned with the Rust/CLI validator.

## Data Provider C ABI v2

The shared provider surface is `DagMlDataVTable` guarded by
`DAG_ML_DATA_VTABLE_DEFINED` and versioned by
`DAG_ML_DATA_PROVIDER_VTABLE_ABI_VERSION == 2`. `scripts/validate_contracts.py`
and the C ABI tests verify that `dag_ml.h` and `dag_ml_data.h` can be included
together in either order when the sibling checkout is available.