ADR-04: Tag / exclude mask materialization

Status: accepted (2026-05-28) Blocks: workstream E (bridge data-provider), workstream G (lineage taxonomy).

Context

nirs4all’s {"tag": filter} adds a column without removing samples. {"exclude": filter} removes samples from training. {"exclude": [filter_a, filter_b], "mode": "any" | "all"} produces a union or intersection mask. The bridge must reproduce and audit the resulting masks so leakage refusal still applies.

Decision

The bridge pre-computes masks in the adapter layer (i.e. inside nirs4all/bridge/dag_ml_provider.py), then ships the materialized mask + an audit record to dag-ml-data. The materialization itself is offline; the audit is part of the lineage envelope.

Mask materialization contract

  1. The adapter walks every tag / exclude step at compile time, calls the corresponding filter operator’s fit_predict on the training partition, and records the resulting boolean mask in the SampleRelationTable as a sample-level metadata column (__tag_<tag_name>__ / __exclude_<mode>_<hash>__).

  2. Multi-filter exclude masks are materialized via Python set ops (mask_any = mask_a | mask_b, mask_all = mask_a & mask_b). The combined mask carries the canonical key __exclude_{mode}_{sha256(sorted(filter_ids))[:12]}__.

  3. tag masks are additive — they create new metadata columns but never alter the train/test partition. exclude masks set partition = "excluded_from_training" for the matching rows; the rows remain in the dataset for prediction.

Lineage audit fields (ADR-11 / ADR-12)

Every materialized mask is recorded in the lineage envelope with:

  • expression — the original Python expression (operator class name + constructor args).

  • mode — "single" / "any" / "all" for exclude.

  • source_columns — which columns the filter consumed (X, y, metadata key).

  • sample_id_set — the list of SampleId values the mask flagged (full list, not just count, so audits are reproducible).

  • mask_fingerprint — SHA-256 of the canonical mask blob; lets the bridge’s bundle replay assert masks are identical.

Consequences

  • The bridge’s data provider materializes masks before the DAG runs; the runtime never re-computes them.

  • dag-ml-data’s SampleRelationTable learns an MaskAuditRecord reference per materialized mask.

  • The lineage taxonomy (ADR-11) carries mask_* fields that survive RO-Crate / PROV / OpenLineage export.

  • “Tagged but not excluded” remains expressible — downstream branch.by_tag consumes the tag column without partitioning fixed rows.

Open follow-ups

  • The parity oracle’s exclude_multi_any_y_and_x case skips today because sample_data/regression is too small for a 2-filter UNION exclusion to leave a viable train set. Pick regression_2 once available — fixture-size, not contract.