ADR-15: Python GIL / async policy

Status: accepted (2026-05-29) Blocks: workstream D (Python packaging), workstream E (bridge)

Context

The PyO3 bindings (workstream D) expose dag-ml and dag-ml-data to Python. Long-running Rust calls (FIT_CV, REFIT, PREDICT) must not block other Python threads holding the GIL, and host controllers re-entering Python from Rust worker threads must do so safely. nirs4all uses joblib.Parallel and sklearn/BLAS thread pools; uncoordinated nesting oversubscribes the CPU (Codex hidden risk).

Decision

  1. Release the GIL on long-running calls — every binding entry point that runs a phase (fit_cv, select, refit, predict, explain) or builds/validates a bundle wraps the Rust call in py.allow_threads(...). Short metadata/validation calls keep the GIL (the acquire/release overhead would dominate).

  2. Controller re-entry contract — a host controller invoked from a Rust worker thread re-acquires the GIL (Python::with_gil) before touching Python objects. The contract is documented for controller authors: your invoke may run on a non-main thread; acquire the GIL yourself.

  3. No asyncio in v1 — nirs4all.run / predict / explain / retrain stay synchronous. An async facade is explicitly out of scope (descoped in the roadmap). Concurrency is achieved by GIL-released Rust scheduling, not by Python coroutines.

  4. Thread-pool ceiling — nirs4all.run(n_jobs=N) maps to the dag-ml scheduler worker count. The docs instruct operators to pin BLAS/OpenMP pools (OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS) and to avoid stacking joblib.Parallel(prefer="threads") on top of an already-parallel scheduler. The bridge logs the effective worker count and detected BLAS thread count at startup so oversubscription is visible.

Consequences

  • Workstream D wraps phase calls in allow_threads; the controller ABI doc (ADR-13 worker-process context) states the GIL re-entry contract.

  • The observability spans (ADR-12) carry the worker-thread id so cross-thread controller invocations are traceable.

  • nirs4all’s existing synchronous API surface is preserved exactly.

Risk

  • Releasing the GIL exposes any unsafe shared state in host controllers. The contract is documented and the default sklearn/R adapters are process-isolated (ADR-13), so the GIL-release path only touches in-process PyO3 controllers, which are the advanced case. In-process controllers that are not thread-safe must declare thread_safe = false in their manifest; the scheduler then serializes them.