C ABI

The ABI is designed around opaque host handles. Rust owns the control lifetime; the host owns the underlying object behind each handle.

When a C entry point fails with a DagMlError, error_out carries the ADR-11 descriptor JSON shape: category, code, severity, message, remediation_hint and context. Null-pointer and malformed-input preflight errors still use the legacy human string because callers often branch on the status code before parsing a payload.

Current Scaffold

crates/dag-ml-capi/include/dag_ml.h exposes:

• version and string-free helpers;

• owned byte release helper for JSON outputs returned by Rust;

• owned row-major DagMlF64Tensor and column-major DagMlF64ColumnarTensor release helpers for Rust-allocated prediction buffers returned to host bindings;

• dagml_graph_spec_contract_json and dagml_graph_validate_json for GraphSpec contract discovery and graph validation before plan building;

• dagml_model_input_spec_contract_json, dagml_model_input_spec_validate_json, dagml_data_plan_contract_json and dagml_data_plan_validate_json for non-Rust bindings that need to exchange neutral data/model compatibility requests and data-planner answers;

• dagml_controller_manifest_validate_json and dagml_controller_manifest_list_validate_json so bindings can preflight controller manifests and registry uniqueness before plan building;

• custom aggregation-controller task/result contract discovery and validation, including result-vs-task checks that preserve requested sample/unit order, fold scope, prediction level and target names before a host aggregation controller result can be trusted by the coordinator;

• the generic controller vtable invoke path is also used for runtime custom aggregation dispatch: bindings receive an AggregationControllerTask JSON and return an AggregationControllerResult JSON, after which Rust validates the result against the exact task;

• dagml_data_output_provenance_contract_json and dagml_data_output_provenance_validate_json so bindings can discover the reserved DataProviderViewSpec.extra key and validate propagated data-view provenance before using it;

• dagml_node_result_validate_for_task_json so host bindings can preflight a controller-produced NodeResult against the exact NodeTask before handing the JSON back to the scheduler;

• dagml_pipeline_dsl_contract_json and dagml_pipeline_dsl_validate_json so bindings can discover and preflight the canonical PipelineDslSpec and serialized nirs4all-compatible list/dict DSL input profiles;

• dagml_pipeline_dsl_compile_json for pure compilation of that DSL input surface into canonical GraphSpec JSON, plus dagml_pipeline_dsl_compile_artifact_json when bindings also need the extracted GenerationSpec including coordinated override dimensions, validated shape-plan fragments, validated data-binding fragments, a campaign template and search-space fingerprint, and dagml_pipeline_dsl_execution_plan_build_json when bindings want the Rust planner to build a validated ExecutionPlan directly from DSL and controller manifests;

• dagml_graph_parallel_levels_json for deterministic node batches that bindings can use to prepare parallel schedulers;

• dagml_execution_plan_build_json for compiling graph/campaign/controller manifests into an ExecutionPlan while Rust owns planner validation;

• dagml_execution_plan_schedule_json for exporting deterministic phase/variant/fold node-level schedules from a compiled ExecutionPlan;

• selection policy/decision validation and candidate selection JSON helpers;

• sample-level and target/group aggregated prediction block conversion helpers that validate canonical JSON blocks and return contiguous row-major double buffers with explicit rows, cols, len and capacity;

• prediction-cache payload tensor export that validates a cache payload set against its bundle before returning contiguous row-major F64 values plus JSON metadata for requirement key, prediction level, block offsets, folds and sample/unit ids. The metadata JSON is versioned by DAG_ML_PREDICTION_CACHE_TENSOR_METADATA_SCHEMA_VERSION and documented in docs/contracts/prediction_cache_tensor_metadata.schema.json;

• prediction-cache payload columnar tensor export that returns contiguous column-major F64 values with explicit column_offsets, versioned by DAG_ML_PREDICTION_CACHE_COLUMNAR_TENSOR_METADATA_SCHEMA_VERSION and documented in docs/contracts/prediction_cache_columnar_tensor_metadata.schema.json;

• execution bundle validation, replay-envelope validation, replay-request validation and prediction-cache payload validation helpers;

• dagml_research_provenance_export_json for building the standards-facing ResearchProvenanceExport JSON over validated execution plans, bundles, optional lineage records, replay envelopes, prediction-cache manifests and artifact manifests; null pointer plus zero length denotes an omitted optional input;

• dagml_openlineage_run_event_json for building the same validated provenance evidence as an OpenLineage RunEvent JSON with explicit namespace and event time byte views;

• vtable replay execution helper that composes host controllers, host data provider, host artifact store and optional host prediction-cache store while Rust owns scheduling and validation;

• replay-request validation can optionally include an OOF prediction-cache payload set, which is required for OOF-dependent REFIT replay;

• mock replay execution helper that returns a JSON summary while exercising Rust-side data handle materialization, data view creation and artifact handle materialization;

• Arrow C Data ArrowArray and ArrowSchema structs for controller predictions and data-provider identity/target/feature exports;

• DagMlControllerVTable for host operator controllers, including generic invoke over NodeTask/NodeResult JSON and explicit returned-byte release. Controller vtable ABI v2 keeps borrowed user_data semantics; v3 is opt-in owned semantics where Rust calls destroy(user_data) after releasing controller-owned result handles. The planned multitask extension keeps the same ownership shape: an optional invoke_batch carries a validated TaskBatchRequest JSON and returns one logical NodeResult per member task plus optional parent batch provenance;

• DagMlDataVTable for host data providers, including materialize, make_view, view_identity, target_arrow and feature_arrow. feature_arrow remains ABI-compatible: hosts may receive either a plain feature-set id or a JSON fusion selector understood by dag-ml-data providers. The vtable uses the shared DAG_ML_DATA_PROVIDER_VTABLE_ABI_VERSION macro and guarded DagMlDataVTable definition so dag_ml.h and dag_ml_data.h can be included together by bindings.

• DagMlArtifactStoreVTable for host replay artifact stores, returning typed DagMlHandleRef values for model/artifact handles. Artifact references are JSON-level Rust contracts with optional typed backend, URI, content fingerprint and plugin/version metadata; the ABI still transports them inside owned JSON payloads so C structs do not freeze a storage layout too early. Artifact-store vtable ABI v1 is borrowed; v2 is opt-in owned lifecycle with destroy(user_data) after materialized artifact handles are released.

• DagMlPredictionCacheVTable for host prediction-cache stores, including load_blocks, materialize and explicit returned-byte release. load_blocks is the single JSON load callback for replay prediction requirements: sample-level requirements return PredictionBlock[], while target/group requirements return AggregatedPredictionBlock[] keyed by typed PredictionUnitId values. Rust selects and validates the expected block shape from the bundle requirement before materializing a prediction handle. Prediction-cache vtable ABI v1 is borrowed; v2 is opt-in owned lifecycle with destroy(user_data) after materialized prediction handles are released.

• docs/contracts/process_adapter_description.schema.json documents the required process-adapter --describe JSON used by CLI-managed host adapters. The C ABI exposes its schema version and schema id through dagml_process_adapter_description_contract_json.

• docs/contracts/process_adapter_frame.schema.json documents the control_frames_v1 JSONL protocol used by persistent process adapters: coordinator init/task/close request frames and adapter ack/result/error response frames wrapping the published NodeTask/NodeResult contracts. The C ABI exposes the frame schema version and schema id through dagml_process_adapter_frame_contract_json.

• docs/contracts/graph_spec.schema.json documents the canonical graph JSON consumed by the planner and C ABI.

• docs/contracts/model_input_spec.schema.json and docs/contracts/data_plan.schema.json document the neutral data-shape contracts exchanged between controller descriptions, data planners and host bindings.

The vtables are intentionally small in this scaffold. They establish shape, ownership and naming before full execution is implemented.

DagMlStatusCode is a fixed uint32_t ABI value rather than a C/Rust enum boundary type. Unknown host status codes are treated as runtime validation errors instead of being decoded as Rust enum discriminants.

Vtable user_data lifetime is explicit per ABI surface. Controller vtable v2 is borrowed for backwards compatibility, while controller vtable v3 opts into Rust-owned lifecycle and calls destroy(user_data) after handle release on drop. Artifact-store and prediction-cache vtable v1 are also borrowed; their v2 surfaces opt into Rust-owned lifecycle with destroy(user_data) after materialized handles are released. Data-provider vtables remain borrowed in the current replay API because that ABI is shared with dag-ml-data. Rust releases controller-result, data/view, replay-artifact and prediction-cache handles that it receives or materializes through the vtables.

Ownership Rules

Object	Owner	Release path
Host data block	Host	DataVTable.release
Host controller result JSON	Host allocation returned through controller vtable	ControllerVTable.release_bytes
Host fitted model	Host	ControllerVTable.release
Host replay artifact handle	Host	ArtifactStoreVTable.release
Host prediction cache handle	Host	PredictionCacheVTable.release
Rust error string	Rust allocation returned through ABI	dagml_string_free
Rust JSON byte output	Rust allocation returned through ABI	dagml_owned_bytes_free
Rust row-major F64 tensor output	Rust allocation returned through ABI	dagml_f64_tensor_free
Rust column-major F64 tensor output	Rust allocation returned through ABI	dagml_f64_columnar_tensor_free
Host JSON byte output	Host allocation returned through prediction-cache vtable	PredictionCacheVTable.release_bytes
Arrow arrays	Producer of the Arrow array	Arrow C Data Interface release callback
JSON blobs	Caller-provided view unless returned as owned bytes	ABI-specific free function

ABI Roadmap

1. Freeze DagMlBytesView, DagMlOwnedBytes, DagMlF64Tensor, DagMlF64ColumnarTensor, handle and status conventions.

2. Add schema coverage for the remaining execution-plan and bundle sub-blobs that are still Rust-implicit.

3. Add conformance tests that call the C ABI from a small C program.

4. Add C conformance tests that drive non-mock replay through the vtable surface.

5. Keep shared dag-ml-data header inclusion and vtable ABI conformance in CI.

6. Add host adapters for Python and native C++ controllers and artifact stores.