Native Persistence Layer — architecture impact & merit report

Scope. Evaluates a proposed new project: a native (Rust), fast, light-dependency library sitting “between nirs4all-io and dag-ml-data” that owns all persistence of predictions and scores (save/load prediction tables + metric/score records) so this persistence is available cross-language (Python/R/MATLAB/WASM) without heavy Python deps.

Method. Grounded in the actual code of all four candidate layers (cited file:line). No speculation. Aligns to the Migration Mandate (dag-ml/AGENTS.md:12-26) and the war-room target split (TARGET_RESPONSIBILITY_SPLIT.md).

Assembled 2026-06-23.

Executive summary (read this first)

  1. dag-ml already persists prediction tables natively in Rust, light-dep. FilePredictionCacheStore (dag-ml-core/src/runtime.rs:1113) writes per-fold + OOF + aggregated (avg/w_avg) prediction blocks to a JSON directory store with sha256 fingerprint validation, plus an export-prediction-cache-store / validate-prediction-cache-store CLI pair (dag-ml-cli/src/main.rs:1836-1862). Its only deps are serde_json + sha2. The “predictions” half of the proposal is ~80% already built.

  2. The real gap is SCORES, not predictions. dag-ml computes full per-fold/per-target/multi-metric reports (RegressionMetricReport, metrics.rs:100) but never persists them — they collapse to a winning scalar + rank list inside the bundle (SelectionDecision, selection.rs:254). There is no queryable metrics table. That, plus all-partition (test/holdout/final) persistence and a columnar/queryable surface, is the genuine missing capability.

  3. The proposed placement is wrong. The nirs4all-io ↔ dag-ml-data seam is the input path (io emits a CoordinatorDataPlanEnvelope — schema + plan + identity, no outcomes). Predictions/scores are outputs and never flow through it. io explicitly disclaims storage (nirs4all-io/CLAUDE.md:156-158); dag-ml-data explicitly disclaims “OOF prediction blocks” and model outputs (dag-ml-data/docs/ARCHITECTURE.md:53-58). Output persistence belongs on the dag-ml side, downstream of the model run.

  4. A separate repo is not justified. The capability is generic ML-coordination output — exactly dag-ml’s charter (AGENTS.md:17 lists “prediction/score persistence” as a dag-ml mandate). A new repo would duplicate bundle.rs/runtime.rs/provenance.rs, add a release-train hop (ADR-10), fork the fingerprint/schema contract, and split a single replay story across two codebases. Recommendation: extend dag-ml’s existing prediction-cache into a queryable predictions+scores store, in a dag-ml-core submodule (or a thin dag-ml-store crate inside the dag-ml workspace). No new project.

  5. Back-compat is a one-way bridge, not a shared format. The frozen 0.9.x nirs4all workspace (SQLite + Parquet + joblib) and .n4a (nirs4all/CLAUDE.md:5) must stay as-is. The native store does not read/write that schema; nirs4all keeps a thin Python writer that mirrors native records into the legacy workspace during the migration window (ADR-02 bundle-readability SLA). The joblib/pickle artifact format is the hard cross-language wall and is out of scope for a predictions/scores store regardless.

1. Current-state map — who persists predictions/scores today

Two independent, overlapping persistence systems exist.

1a. nirs4all (Python, the production system) — nirs4all/pipeline/storage/

A hybrid SQLite (metadata + scalar scores) + Parquet (dense arrays) + content-addressed joblib (fitted artifacts) workspace, schema SCHEMA_VERSION = 2 (store_schema.py:28).

What

Where

Format

Evidence

Runs / pipelines / chains / artifacts / logs / projects

SQLite tables

rows + JSON-in-TEXT

store_schema.py:44-186 (7 tables)

Prediction rows (per-fold fold_0.., ensemble avg, refit final) + scalar scores

predictions table

SQLite, upsert on natural key

store_schema.py:111-148; fold/refit semantics workspace_store.py:1216-1238

CV score summaries (cv_val_score, cv_scores JSON), refit + aggregated-ensemble scores

chains table

SQLite columns

store_schema.py:77-109

Dense arrays: y_true, y_pred, y_proba(+shape), sample_indices, weights, per-sample metadata (JSON)

arrays/<dataset>.parquet

Parquet (pyarrow write / polars read), Zstd-3

array_store.py:124-140, :292-297, :397-399

Fitted models

artifacts/ab/<sha256>.joblib

joblib / pickle, content-addressed, ref-counted

workspace_store.py:190-210, :1353-1379

Portable export

exports/*.n4a (ZIP)

JSON manifest/trace/chain + joblib artifacts

workspace_store.py:2271-2383; richer bundle/generator.py:305-378

  • WAL mode is enabled (store_schema.py:606-607), with an RLock + retry-on-lock decorator (workspace_store.py:116-157) and POSIX/Windows file locking on the Parquet dir (array_store.py:202-243).

  • Deps (persistence-relevant): sqlite3 (stdlib), pyarrow>=14 (Parquet write — heavy), polars>=1.0 (Parquet read + all DataFrame returns — heavy Rust binary), numpy>=2, joblib>=1.2 (Python-only), pandas (convenience only), pyyaml. The Predictions facade (data/predictions.py:183) buffers rows during a run and flushes via WorkspaceStore.save_prediction (row-by-row SQLite) + array_store.save_batch (one Parquet write per dataset) inside a transaction (predictions.py:927-1044). Store is injected via register_store_backend (predictions.py:496-514) — data layer never imports pipeline layer.

1b. dag-ml (Rust, the migration target) — dag-ml-core/

A JSON-only, fingerprint-validated, replay-grade persistence layer. All persisted types derive serde.

What

Type (file:line)

Persisted?

Per-fold / OOF sample-level predictions

PredictionBlock (oof.rs:29-40)

✅ in cache store

Aggregated avg / w_avg / median predictions

AggregatedPredictionBlock (aggregation.rs:67-78); methods policy.rs:91-100

✅ in cache store

y_true target block

RegressionTargetBlock (metrics.rs:39-46)

serde-able

Prediction-cache payload (values) + manifest (metadata)

BundlePredictionCachePayloadSet (bundle.rs:753), BundlePredictionCacheRecord (bundle.rs:319)

✅ JSON files

Top-level run manifest

ExecutionBundle (bundle.rs:914-938) — schema-versioned (bundle.rs:121, SchemaMigrationPolicy bundle.rs:41)

✅ execution_bundle.json

Selection outcome (winning candidate + ranks)

SelectionDecision / RankedCandidate (selection.rs:254, :248)

✅ in bundle

Full per-fold/per-target/multi-metric scores

RegressionMetricReport (metrics.rs:100-113)

❌ computed, never persisted

Lineage / provenance

ResearchProvenancePackage (provenance.rs:35), W3C PROV-JSONLD + RO-Crate + OpenLineage (provenance.rs:17-26, :294, :396+)

✅ JSON files

  • The prediction-cache store is FilePredictionCacheStore (runtime.rs:1113): a directory of prediction-cache-<hex>.json payloads + a prediction_cache_manifest.json (runtime.rs:939, :1024-1031), written by write_payload_set (runtime.rs:1121) and re-validated against the bundle (fingerprint match) by validate_file_prediction_cache_store. CLI: export-prediction-cache-store (main.rs:1836-1853), validate-prediction-cache-store (main.rs:1854-1862). It is consumed via the RuntimePredictionCacheStore trait (runtime.rs:925) with File / Columnar / InMemory impls.

  • Deps: dag-ml-core = serde, serde_json, yaml_serde, indexmap, sha2, thiserror, tracing. No arrow, no parquet, no sqlite, no DB. The columnar f64 representation (ColumnarPredictionCacheStore, runtime.rs:1724; layout: "column_major_f64", capi/src/lib.rs:3537) is an in-memory ABI hand-off only — never serialized to disk.

1c. Quantified overlap (nirs4all SQLite+Parquet ⟷ dag-ml bundle/cache)

Capability

nirs4all (Python)

dag-ml (Rust)

Overlap

Per-fold predictions persisted

✅ Parquet+SQLite

✅ PredictionBlock JSON

Full

OOF predictions persisted

✅ (query over rows)

✅ cache store (Validation-only)

Full

Ensemble avg / w_avg persisted

✅ avg rows + final_agg_*

✅ AggregatedPredictionBlock

Full

Final refit predictions persisted

✅ final rows

⚠️ representable, but cache validate() forces partition==Validation (bundle.rs:353)

Partial

Test/holdout predictions persisted

✅ partition column

⚠️ block type exists; cache store rejects non-validation

Partial

Full metric/score records (per fold/target/metric)

✅ scores JSON + cv_scores

❌ collapsed to scalar+rank

None — gap

Manifest / run metadata

✅ runs/pipelines/chains

✅ ExecutionBundle

Conceptual

Lineage / provenance / PROV / RO-Crate

❌ (trace.json only)

✅ full

dag-ml only

Fitted-model artifacts

✅ joblib (Python-only)

handles/refs only (RefitArtifactRecord bundle.rs:846)

Different concern

Queryable analytical surface (“RMSE for target X across folds”)

✅ SQL / polars

❌ load-whole-payload-by-key

None — gap

Net: prediction-table persistence is substantially duplicated already. The non-overlapping, genuinely-missing pieces are: (a) a persisted metric/score table, (b) all-partition (not OOF-only) persistence, (c) a columnar-on-disk + queryable surface.

2. The gap — what actually blocks cross-language persistence

The cross-language problem is not “predictions can’t be persisted natively” — dag-ml already does that with serde_json + sha2. The real blockers are narrower and concrete:

  1. The production persistence is Python-only by construction. Every read/write in nirs4all goes through sqlite3 (stdlib API), pyarrow (Parquet write), polars (Parquet read, and every query_* returns a polars.DataFrame — store_protocol.py:20, workspace_store.py:394). The file formats (SQLite, Parquet, JSON) are language-neutral, but the access layer must be reimplemented per language. (Evidence: §1a deps; predictions.py:927-1044 flush path.)

  2. Fitted artifacts are joblib/pickle (workspace_store.py:190-210) — fundamentally Python-object serialization. R/MATLAB/WASM can read the bytes but cannot deserialize the models. This is the true cross-language wall, and it is orthogonal to a predictions/scores store — predictions and scores are plain numbers; models are the unportable part. A predictions/scores persistence layer does not solve, and need not touch, the artifact problem.

  3. dag-ml persists for replay/lineage, not for analytical query. The cache store is keyed by requirement_key; you load a whole payload, you cannot query across folds/targets (runtime.rs:1113-1230). And the scores never get persisted at all (RegressionMetricReport → CandidateScore → discarded except winner; metrics.rs:154, selection.rs:248). So even on the native side, there is no queryable predictions+scores artifact.

Conclusion of the gap analysis: the bottleneck is not “no native persistence.” It is (i) the production path is Python-coupled, and (ii) the native path persists predictions-for-replay but not scores, not all partitions, and not in a queryable form. A focused effort closes (ii) and lets every binding inherit it via the ABI; (i) dissolves automatically once the native store is the source of truth.

3. Proposed-layer design (evaluated against the existing code)

3a. Scope — what it should own

Candidate scope

Verdict

Why

Save/load prediction tables (per-fold/OOF/agg/final/test)

In (mostly exists)

PredictionBlock / AggregatedPredictionBlock already serialize; needs the all-partition relaxation (lift bundle.rs:353 Validation-only constraint for a results store)

Save/load metric/score records (per fold/target/metric)

In (the real new work)

RegressionMetricReport (metrics.rs:100) is serde-ready but unpersisted; add a MetricRecord store

Queryable/indexed read surface

In

the missing analytical capability

Lineage / provenance

Out — already dag-ml’s provenance.rs

don’t fork it

Fitted-model artifacts

Out

joblib/pickle/ONNX is a separate, harder problem; predictions/scores are numbers

Dataset/feature persistence

Out

dag-ml-data owns .n4d feature buffers (buffer_file_store.rs:1-9); not this layer

Scope should be predictions + scores ONLY, exactly as the proposal states. That is correct and matches the gap.

3b. Placement — three options, ranked

Option

Fit to charters

Cost

Verdict

A. Extend dag-ml’s prediction-cache into a predictions+scores store (submodule in dag-ml-core, or a thin dag-ml-store crate inside the dag-ml workspace)

✅ AGENTS.md:17 lists “prediction/score persistence” as a dag-ml mandate; reuses bundle.rs/runtime.rs/fingerprints; one replay story

Lowest — incremental

RECOMMENDED

B. New crate inside dag-ml-data

❌ Violates charter: dag-ml-data “must not expose OOF prediction blocks” (docs/ARCHITECTURE.md:53-58), “must not own … model execution” (CLAUDE.md:11); predictions/scores are outputs

Medium + boundary breach

Reject

C. Standalone new repo “between nirs4all-io and dag-ml-data”

❌ Wrong seam (io→dag-ml-data is the input path; predictions/scores never traverse it — nirs4all-io/CLAUDE.md:156-158, io emits only a CoordinatorDataPlanEnvelope); duplicates dag-ml persistence; +1 release-train hop (ADR-10); forks the fingerprint/schema contract

Highest — new CI, license, repo, cross-repo contract sync

Reject

The proposal’s stated placement (“between nirs4all-io and dag-ml-data”) is architecturally mis-located: that seam carries the input CoordinatorDataPlanEnvelope (schema + plan + sample/target/group/origin identity, no outcomes — confirmed: the envelope schema has no prediction/score/metric field). Output persistence sits downstream of the model run, on the dag-ml side.

3c. Storage tech for “fast + light deps”

dag-ml’s hard constraint is no host runtime deps in the core (dag-ml/CLAUDE.md crate-direction note: dag-ml-core = “no host runtime deps”). Compare:

Tech

Deps added

Cross-lang read

Query

Fit to dag-ml

serde_json (today’s cache store)

none (already in)

trivial

none

✅ current baseline

Arrow IPC

arrow-array/arrow-ipc/arrow-schema (~1 family)

excellent (R/MATLAB/WASM/Python all read Arrow)

columnar scan

✅ already precedented & isolated in dag-ml-data-arrow behind a non-default feature (dag-ml-data-arrow/Cargo.toml:13-16)

Parquet-rs

parquet + arrow

excellent + compression

predicate pushdown

⚠️ heavier; matches nirs4all’s Parquet so easy bridge

redb / sqlite-rs

embedded DB

sqlite=universal; redb=Rust-only

indexed

⚠️ adds a DB dep to a JSON-pure core

custom columnar

none

must re-implement readers everywhere

custom

❌ reinvents Arrow

Recommendation: keep serde_json as the canonical/replay format (zero new deps, already validated), and add Arrow IPC as the columnar/queryable representation — gated behind an optional feature, mirroring the existing dag-ml-data-arrow pattern and the in-memory ColumnarPredictionCacheStore (runtime.rs:1724) that already produces column-major f64. Arrow gives R/MATLAB/WASM native read with one isolated dependency family and aligns with nirs4all’s Parquet (Parquet = Arrow-on-disk), easing the legacy bridge. Avoid adding an embedded DB to dag-ml-core.

3d. Cross-language binding surface

Reuse the proven ecosystem pattern (one Rust core + C ABI + thin per-language wrappers) already shipped by nirs4all-io (Python/R/MATLAB/WASM/C — nirs4all-io/COMPAT.md) and dag-ml-capi. The store surface crosses the ABI as: canonical JSON descriptors + dense f64 tensors (released via dagml_f64_columnar_tensor_free, capi/src/lib.rs) — predictions/y_true/scores as data, never as host handles (matches AGENTS.md:32-36). No new binding mechanism is needed.

3e. Relationship to dag-ml’s existing bundle/prediction-cache

Layer on top / extend — do not merge-away or supersede. The ExecutionBundle + FilePredictionCacheStore are the replay manifest + OOF cache and must stay (replay/lineage depend on them). The new capability is additive:

  1. Relax the Validation-only constraint (bundle.rs:353) — or add a sibling ResultsStore — so test/holdout/final partitions persist.

  2. Add a persisted MetricRecord table (from the already-computed RegressionMetricReport, metrics.rs:100) — the one genuinely new artifact.

  3. Add the Arrow/columnar on-disk write + an index for analytical queries.

4. Impact

4a. On the frozen 0.9.x nirs4all contract (nirs4all/CLAUDE.md:5)

The workspace SQLite/Parquet schemas, run-manifest layout, and .n4a format are stable within 0.9.x. The native store must not attempt to be that format. Migration path (ADR-02 schema-evolution SLA + ADR-17 cutover): native store is the source of truth; nirs4all keeps a thin Python shim that mirrors native predictions/scores into the legacy SQLite+Parquet workspace during the dual-run window, so Studio and existing .n4a consumers see no change. The joblib artifact wall is untouched (out of scope). This is a bridge, consistent with ADR-14’s managed-debt exception — but note the conflict with nirs4all’s no-shims rule (war-room open decision #3) is the maintainer’s call.

4b. On dag-ml’s bundle.rs

Additive: lift one validation constraint, add a metric-record store, add an optional Arrow writer. ~No churn to the existing serde types or the replay path. Consolidation toward dag-ml, not a fork.

4c. On the migration timeline

Option A rides the existing ADR-10 release train (dag-ml-data → dag-ml → nirs4all). Option C adds a fourth package to that train + a new cross-repo contract to keep JSON-identical — directly contrary to the war-room’s “fewer moving parts” posture and the “don’t reinvent dag-ml” warning (Tier-3 trap, war-room README:86-95).

4d. On performance

Today’s cache store is serde_json (text); Arrow/columnar would be faster than both JSON and the nirs4all read-modify-write-whole-Parquet append (array_store.py:350-357). But note the war-room caveat: dag-ml perf is only sanity-probed, not benchmarked (war-room README:129). Any “fast” claim must be benchmarked against the SQLite+Parquet baseline before it is asserted.

4e. On dependency weight

Option A in dag-ml-core keeps the core pure (serde_json + sha2); Arrow is opt-in and isolated (precedent: dag-ml-data-arrow). A standalone repo would re-pull serde/sha2/arrow and re-implement the fingerprint contract — more total dependency surface across the ecosystem, not less.

4f. On maintenance / another-repo cost

A new repo = new CI green-gate, license headers (the dual-CeCILL/AGPL + commercial policy), validate_contracts.py cross-repo parity job, release mechanism, RTD docs, and a contract that must stay byte-identical with dag-ml’s prediction-cache schemas. This is the single largest hidden cost and the strongest argument against Option C.

5. Merit / bien-fondé — recommendation

The capability is justified; a separate project is not.

  • The cross-language predictions+scores persistence goal is real and aligns with the Migration Mandate, which explicitly assigns “prediction/score persistence” to dag-ml (AGENTS.md:17) so “every binding (Python/R/MATLAB/WASM) gets it for free” (AGENTS.md:20-21).

  • But ~80% of it (prediction tables, manifests, fingerprints, replay) already exists in bundle.rs + runtime.rs + provenance.rs with the exact light-dep profile the proposal wants (serde_json + sha2). A new repo would duplicate this.

  • The genuinely-new work — persisted score/metric records, all-partition predictions, a columnar/queryable surface — is small, additive, and lands cleanly inside dag-ml.

  • The proposed placement (io ↔ dag-ml-data) is on the input seam; predictions/scores are outputs and belong downstream of the model run. Both neighbors’ charters explicitly disclaim this (nirs4all-io/CLAUDE.md:156-158; dag-ml-data/docs/ARCHITECTURE.md:53-58).

Recommendation (ranked)

  1. DO (recommended): Extend dag-ml’s prediction-cache into a predictions+scores store, as a submodule of dag-ml-core or a thin dag-ml-store crate inside the dag-ml workspace. Add (a) a persisted MetricRecord from the existing RegressionMetricReport, (b) all-partition support (relax bundle.rs:353), (c) an optional Arrow-IPC columnar/queryable representation (feature-gated, like dag-ml-data-arrow). Expose via the existing C ABI; bind per-language with the io/dag-ml pattern. nirs4all keeps a thin legacy-workspace mirror shim for the 0.9.x contract.

  2. Acceptable alternative: same code as a separate crate within the dag-ml workspace (not a separate repo) if the maintainer wants clearer module boundaries — keeps one repo, one release-train slot, one contract.

  3. Reject: a crate inside dag-ml-data (charter breach) or a standalone new repo “between io and dag-ml-data” (wrong seam, duplication, +1 release/contract/CI hop).

6. Risks & open questions for the maintainer

  1. Scores schema. No score/metric persistence schema exists yet (only node_result.schema.json has an optional metrics map). A new metric_record.schema.json must be designed and added to the conformance pack — decide its grain (per fold × target × metric).

  2. Validation-only constraint. Lifting bundle.rs:353 (cache = Validation-only) for a results store risks blurring the OOF-cache (a leakage-safety boundary) with a general results store. Keep them as distinct stores, or gate the relaxation carefully — do not weaken the OOF invariant (dag-ml/CLAUDE.md engineering rules).

  3. Arrow vs JSON canonical form. Two on-disk forms (JSON for replay/fingerprint + Arrow for query) risk drift. Decide which is canonical and whether the Arrow form is derived-and-disposable or first-class.

  4. Perf is unmeasured. The “fast” claim is unproven (war-room README:129). Benchmark against SQLite+Parquet before asserting it.

  5. Back-compat posture. The legacy-workspace mirror shim conflicts with nirs4all’s absolute no-shims rule (war-room open decision #3) — only the maintainer can sanction the ADR-14 managed-debt exception.

  6. Artifacts stay out. Confirm the layer is predictions/scores only; the joblib→neutral-model-format problem (ONNX / nirs4all-methods / dag-ml handles) is separate and must not be smuggled in.

  7. Reducer-vocabulary drift. robust_mean/exclude_outliers exist data-side only (war-room README:132); a scores/agg store must reconcile reducer names on the release train.

Provenance: grounded in nirs4all/pipeline/storage/*, nirs4all/data/predictions.py, dag-ml-core/src/{bundle,runtime,aggregation,metrics,oof,selection,provenance,policy}.rs, dag-ml-cli/src/main.rs, dag-ml-data/{CLAUDE.md,docs/ARCHITECTURE.md,crates}, nirs4all-io/{CLAUDE.md,COMPAT.md,crates}. All file:line cited inline. 2026-06-23.