# DSL Parity With nirs4all Pipelines

Status: working contract.

Goal: the dag-ml DSL must be at least as expressive as the current nirs4all
pipeline surface, while keeping dag-ml's architecture: operators stay external,
controllers live in bindings/hosts, and Rust compiles, validates, schedules and
audits the DAG, campaign, OOF and leakage contracts.

This document tracks semantic parity first. The strict `PipelineDslSpec` is the
portable canonical form. A JSON compatibility importer now lowers serialized
nirs4all-style list/dict pipelines into that canonical form; Python object and
YAML frontends should be thin host-side serializers around the same importer.

## Non-Negotiable Design Rules

- Splits are campaign/controller invocations (`split_invocation`), not graph
  operators.
- Any data/model shape mutation must be visible in public DSL fields:
  `shape`, `aggregation_policy`, `augmentation_policy`, `selection_policy`,
  `train_params` and `tuning`.
- OOF safety is graph-visible through prediction edges with `requires_oof` and
  fold alignment; refit/replay consumes validated caches/bundles.
- Branch, merge, concat, source fusion and stacking are DAG structure, not
  Python call-order side effects.
- Runtime behavior belongs to controllers. The DSL records enough intent and
  contracts for Rust to reject unsafe execution and for bindings to route tasks.
- Minimal aliases are the preferred frontend surface when an operator is
  unambiguous. `SNV` should compile as an opaque transform payload; the Python
  or native controller registry resolves it through manifest
  `operator_selectors` to the matching controller. Verbose forms are reserved
  for parameters, explicit ids, controller hints or ambiguous routing.
- Bindings should therefore expose registry aliases as first-class UX, not as a
  compatibility afterthought. A bare alias such as `SNV` or `StandardScaler`
  remains the representation in the graph; a `TransformerMixin` controller may
  build the concrete object and execute it after Rust has selected that
  controller by alias/class/function/ref/type selectors.
- When a controller registry is available during DSL compilation
  (`compile-pipeline-dsl --controllers` or `build-pipeline-dsl-plan`), aliases
  that are not obvious by Rust heuristics may still be reclassified by matching
  `operator_selectors` before ports are frozen. For example, `ElasticSpectra`
  can stay a bare string while a model controller declares that alias; if two
  operator kinds claim the same alias, compilation requires explicit syntax.

## Parity Matrix

| nirs4all surface | dag-ml DSL representation | Contract status |
|---|---|---|
| plain preprocessing/transform step | `kind: "transform"` | compiled to `NodeKind::Transform` |
| model step, named model, explicit params | `kind: "model"`, `operator`, `params`, `metadata` | compiled to `NodeKind::Model` |
| tuner / finetune controller | `kind: "tuner"` or alias `kind: "finetune"`, `operator`, `params`, `tuning` | compiled to external `NodeKind::Tuner`; Rust treats it as an OOF-producing predictive node while the tuning engine and concrete estimator lifecycle remain controller-owned |
| target/y processing | `kind: "y_transform"` | compiled to `NodeKind::YTransform` with target ports |
| splitters (`KFold`, `GroupKFold`, SPXY, fold files) | top-level `split_invocation` in campaign template | deliberately outside graph nodes |
| sequential grouping (`[...]`) | `kind: "sequential"` | inlined during compilation while preserving child node contracts |
| sample augmentation | `kind: "sample_augmentation"` plus mandatory `shape.augmentation_policy` | compiled as augmentation with `dsl_augmentation_kind=sample`; unsafe scopes refused |
| feature augmentation | `kind: "feature_augmentation"` plus `shape` | compiled as augmentation with `dsl_augmentation_kind=feature` |
| runtime data generation / synthetic samples | `kind: "data_generation"` or alias `kind: "generation"` plus mandatory `shape` | compiled as `NodeKind::Generator` with `dsl_generation_kind=data`; actual generation remains external, while Rust validates seed, fold scope, origin/group/target inheritance and shape contracts |
| feature selection / shape-changing processing | `shape.selection_policy`, `feature_namespace`, schema fingerprints | shape plan validated and attached to campaign |
| sample filters | `kind: "sample_filter"` or `kind: "filter"` | compiled to exclusion/filter nodes with explicit `dsl_filter_kind` metadata |
| concat transform / multi-view feature fusion | `kind: "concat_transform"` with branch transforms | compiled to `NodeKind::FeatureJoin` |
| duplicated branches | `kind: "branch"`, `mode: "duplication"` | multiple branch predictions retained |
| separation branches by source/metadata/tag/filter | `branch.mode`, `branch.selector`, per-branch selectors | compiled to graph intent plus campaign `branch_view_plans`; provider materialization remains host-side |
| multiple models per branch | multiple `kind: "model"` steps inside a branch | compiled into distinct OOF inputs for downstream merge |
| merge predictions/features/original data | `kind: "merge"`, `merge_mode`, `output_as`, `include_original_data`, `selectors` | compiled to join nodes with OOF prediction edges and branch data edges; `features`/`sources` consume transformed branch outputs, `all`/`mixed` can consume branch data plus OOF predictions plus original data |
| merge plus immediate meta-model | `kind: "merge_model"` convenience | compiled as model consuming OOF prediction inputs and optional original data |
| stacking, multi-model top-level stacks | repeated `model` steps then `merge`/`merge_model` | pending predictions are preserved until consumed |
| per-branch/per-model selection (`best`, `top_k`, `all`) | `merge.selectors` with branch/model/input scopes | selector targets and `top_k`/metric requirements are compile-validated; scoring remains controller policy |
| finetune / hyperparameter search metadata | `tuning` or `finetune_params`, plus generation dimensions/variants | tuning intent is graph-visible on model/tuner nodes and generation dimensions; concrete search remains controller-side |
| final train params | `train_params` | preserved as `dsl_train_params` metadata |
| `_range_`, `_log_range_`, `_grid_`, param `_or_`, `pick`, `arrange`, `count` | `variants`, explicit `generation_dimensions`, or compact `generators` on DSL nodes | compiled into deterministic `GenerationSpec` dimensions |
| structural `_or_` over step chains | `kind: "generator"`, `mode: "or"`, `branches`, `pick`/`arrange`/`count` | expanded into explicit OOF-producing choices with namespaced node ids and generator metadata |
| structural `_cartesian_` over pipeline stages | `kind: "generator"`, `mode: "cartesian"`, `stages` | expanded into explicit Cartesian OOF-producing choices with namespaced node ids and fold-safe downstream merge inputs |
| serialized list/dict nirs4all surface | top-level `pipeline` array with `preprocessing`, `model`, `branch`, `merge`, `_or_`, `_cartesian_`, `_chain_`, `_grid_`, `_range_`, `_log_range_`, `_zip_`, `_sample_` | compatibility importer lowers to canonical DSL; data-only branch feature merges and merge dicts are compiled, and data-only generator stages are fused with downstream model generators so OOF choices stay complete |
| minimal aliases / plain operator refs | short strings plus `{"class": ...}`, `{"function": ...}`, `{"ref": ...}`, `{"type": ...}` and `{"name": ..., "step": ...}` wrappers | Rust infers only safe planning class: splitters become campaign split invocations, obvious estimators become model nodes, obvious tuners such as `OptunaTuner` become tuner nodes, chart aliases become chart nodes. With a controller registry, selector-only aliases can reclassify unknown transform defaults to model/tuner/etc. before graph planning; cross-kind matches are rejected as ambiguous |
| multiple nirs4all splitter declarations | one campaign `split_invocation` with `params.compat_split_chain` | splitters remain outside graph nodes while preserving train/test + CV chains for host split controllers |
| multisource data | `data_bindings.source_ids`, branch/source selectors, source joins | contract surface present; richer materialization belongs to dag-ml-data |
| repetition/sample/group aggregation | top-level/shape `aggregation_policy`, target/group OOF cache contracts | core runtime implemented for sample/target/group OOF |
| tag/exclude filters | `kind: "tag"` and `kind: "exclude"` | compiled to graph nodes |
| charts/reports | `kind: "chart"` | compiled as a chart node; host controller decides side effects |

## Current Gaps

- Native JSON compatibility import exists for serialized nirs4all-style
  list/dict syntax, including minimal aliases and plain `class`/`function`
  descriptors. Direct Python object/YAML parsing is still a binding-layer task:
  hosts must serialize live objects and splitters into portable descriptors
  before handing the DSL to Rust.
- The DSL node kinds compile and validate graph contracts. Smoke controllers now
  execute transform/model/tuner/data-generation/mixed-join paths, but production
  execution still needs host controller support for each operator family.
- Separation branch materialization by source/metadata/tag/filter now has an
  explicit campaign `branch_view_plans` contract. Production runtime-complete
  support still requires dag-ml-data providers and host controllers to consume
  those selectors when creating branch-local views.
- Merge selector scopes and basic selection contracts are compile-validated, and
  OOF edges are enforced; actual metric scoring and ranking remain the
  responsibility of selection and merge controllers.
- Runtime generation is represented as external generator nodes/controllers for
  data-producing generators. The next open boundary is richer generated-model
  and provider-native generation backends; actual generator implementations
  should stay outside Rust unless they are pure DAG/campaign coordination
  primitives.

## Required Regression Coverage

- Compile every nirs4all canonical sample category into strict DSL equivalents:
  linear, feature augmentation, sample augmentation, branch, stacking/merge,
  concat transform, filters/splits, finetune/tuner, multisource and all-features.
- For each shape-changing step, assert that `DataModelShapePlan` exists and
  rejects unsafe augmentation/selection scopes.
- For every stacking/merge pattern, assert that upstream prediction edges carry
  `requires_oof=true` and fold alignment.
- For repeated/grouped data, assert that target/group OOF cache requirements
  survive bundle capture and replay.