DF Export Internals

This page describes how the current DF producer assembles ordinary integrals, accounts for memory and owns temporary storage. For supported inputs, public settings, examples and operational limits, start with DF-Expanded Integral Export.

Assembly Contract

The shared exporter prepares the MO order, authoritative one-electron Hamiltonian, scalar constant, SO data and orbital analysis before DF expansion. The DF producer changes only the Coulomb two-electron approximation. All ERI families, full J/K slices, aj/ak, Fock, vacuum energy, frozen-core Hamiltonian/constant and native v_pair derive from the same whitened MO factors after the common trimming/reordering. Source SCF energies are provenance, never fitted Fock input; Fock uses downstream occupations.

Let N be the AO dimension, M the retained MO dimension, A the selected count, and Q the actual whitened auxiliary rank. Write T(n) = n(n+1)/2. With coefficient matrix C and whitened AO factors B, the transform and expansion are:

L[q, i, j] = sum(mu, nu) C[mu, i] * B[q, mu, nu] * C[nu, j]
(ij|kl)_DF = sum(q) L[q, i, j] * L[q, k, l]
J[p, pair(i,j)] = (ij|pp)_DF
K[p, pair(i,j)] = (ip|jp)_DF

MO storage is pair-major float64, with shape (T(M), Q). AO batches are unpacked, transformed on both MO sides and packed into that store. Pair products expand bounded panels; they do not call exact AO4 or direct J/K. Complete J/K slices feed the common compact-intermediate assembler.

Selected export writes ppaa_pair, papa_pair, pacv_pair to ordinary flat destinations and retains cvcv_factors as a contiguous (C*V,Q) dataset. Bounded row writes avoid both CVCV reconstruction and chunk-padding overhead. Output disk admission includes 8*C*V*Q using the source’s auxiliary rank. Full output preserves int2e_full’s DIAGPT ordering and native v_pair’s final-MO ordering. The native integral group retains its pair_symmetric_prqs_v1 layout and zero-based orbital-index declaration; root selected_idx remains one-based. See the physical families and lengths in Selected Route.

The added artifact group is /mppt/approximation, with integral_mode=df, metric=coulomb, factor_rank, auxbasis, ao_cache_reused and mo_factor_storage=owned_ram or owned_hdf5. The complete MO factor store is internal; only CV rows survive as a consumer dataset. Root CVCV layout/rank attributes are validated and fingerprinted by the workflow. Existing exact artifacts continue using packed CVCV.

Native Factor Validation

DIAGPT checks each factor panel for nonfinite values in read_cv_panel. Keep its ieee_arithmetic import local to that procedure: a module-level import can propagate through ERI-provider dependencies and make GNU Fortran insert IEEE-state save/restore calls into every scalar ai() access. Rebuild dependent Fortran modules after changing this import scope.

Source Resolution

validate_df_source() accepts only native CPU molecular pyscf.df.DF on the identical reference molecule with the full Coulomb operator. Validation rejects periodic, spinor/GPU/custom sources, range-separated caches and incompatible Cartesian settings. Cache inspection requires a positive common rank, real float64 rank-by-AO-pair arrays and the correct packed AO-pair count. Transform reads reject nonfinite factors and a changing rank.

When no source is supplied, the exporter creates scf.RHF(reference.mol).density_fit().with_df without SCF. This preserves PySCF’s normal RHF auxiliary choice instead of substituting the different bare df.outcore default. A supplied resolved auxiliary molecule or atom-label mapping is preserved, including an explicitly configured unbuilt source.

Source	Behavior
Native in-memory AO factors	Read bounded batches without replacing or deleting the caller’s cache; its resident memory belongs to the process base
Native HDF5 AO factors	Read the caller’s packed dataset or native block group; leave the file and provider unchanged
Unbuilt or absent source	Build owned outcore native Coulomb factors with the resolved auxiliary molecule, then transform; never call the borrowed object’s build/reset

ReferenceData detaches its array snapshots but intentionally borrows with_df; that field is excluded from Pydantic serialization and repr. Object identity and layout checks are not a cache-content provenance certificate. Native caches cannot retrospectively certify manual molecule/auxbasis mutations or unrelated raw factors attached by a caller.

Storage Selection

DfMemoryPlan describes phase lifetimes without allocating factor arrays. Before fresh factor construction, minimum downstream work is checked using the auxiliary AO count as an upper bound. After construction, the actual whitened rank determines the MO payload and storage decision. Cached sources supply their actual rank directly.

RAM is chosen only when every minimum phase fits with the complete owned MO store: transform, J/K and compact assembly, property/default copies, native output and selected-family or full-output work. Otherwise the HDF5 alternative must fit before the MO store is created. Unrelated phase peaks are not summed. This is not a factor-payload-only decision.

AO-to-MO Q batches remain capped at 128 and pair tiles at 1024. Admission may reduce either, but never rank, precision or the MO space. These are internal maxima, not user controls. The HDF5 store uses a 1-MiB raw data cache and chunks of at most 128 pair rows by the admitted Q batch. Family caches and a rewritten stripe writer are not implemented.

Consumer phases recheck availability before allocation and may reduce the pair tile. There is no spilling, export restart or scientific fallback after partial output. force_transform_outcore and force_reorder_stream are exact full-route controls, not DF backend overrides; DF full output already requests streaming reorder but still materializes its native and packed arrays.

Phase Admission

The shared planner uses mol.max_memory in decimal MB, host availability, visible cgroup limits and a separate 10% reserve. max_mem_bytes remains an optional new-work cap, not a total-RSS or persistent-payload cap. The public memory guidance applies to both modes; the exact scheduling ledger is not reused as the DF live/pending model described here.

DF captures its base after common SO/orbital preparation, before owned DF allocations. Retained caller AO caches, coefficients and common reference data are already part of that base. IntegralMemoryBudget.phase_workspace_bytes() uses integer bytes:

peak = max(current_RSS, base + live_owned) + pending_persistent + peak_new_work
peak <= fresh_usable_total_after_reserve

Pending memory cannot disappear behind elevated RSS. The temporary cap limits new work separately, not persistent MO/J/K payloads. Existing workspace_mb() and exact-route behavior are unchanged. Removing dead references is not assumed to reduce RSS: there is no forced GC, allocator trim or subtraction from measured RSS. Availability observations are not reservations against other jobs.

The ledger distinguishes these coexisting arrays and temporary peaks:

Phase	Ownership and peak work
Transform	Owned MO store plus current/prefetched/group-read AO buffers, unpacked AO data, matrix products, packing/layout copies, indices and I/O/native allowance
J/K and compact	Pending complete J/K, packed vectors and source map; use the maximum of J contraction, per-orbital K and compact assembly, not their sum
Properties	Live compact data plus AO/MO tensors, transformation/layout copies, defaults, validation and disk-order copies
Native output	Live compact data, pending native pair matrix and bounded pair work
Selected families	MO factors, pair maps and bounded pair products; compact/property/native selected arrays have already expired
Full output	Live native pair matrix, conservative pending packed/reorder arrays and bounded pair work

open_df_factors() takes nact and write_properties rather than the old aggregate fixed_bytes estimate. Its df_memory_plan profile records backend, actual rank, Q/tile, captured base and phase (live, pending, work) tuples excluding the MO store. Phase admission adds resident MO ownership separately.

Native Source Workspace

Fresh native source construction additionally admits six auxiliary-metric matrices, four whole-shell three-center tiles and estimated per-thread integral workspace. Six matrices cover Cholesky and PySCF’s unchanged eigenvalue fallback; the shell allowances include prefetch/solve buffers and shell-balancing overshoot. The source construction follows native PySCF outcore Coulomb factorization without mutating a borrowed provider.

Transform/pair phases include a 16-MiB I/O allowance and an estimated 8 MiB per native thread. Fresh source construction uses a shell-dependent native estimate with the same per-thread floor. These are source-derived scheduling allowances, not proved libcint cache bounds or measured RSS. The native metric fallback is not a fallback to a different approximation.

Lifetimes And Reads

An exporter-built AO file is retired after the AO iterator closes. Transform indices/buffers do not survive that boundary. Per-orbital exchange panels are released before the next K iteration; property/default results after writing; compact J/K after native h1/Fock/frozen-core output and before selected families. The selected native pair result also expires before family expansion.

DfFactors.storage is ram or hdf5; .data is an ndarray or Dataset valid only inside its context. Observers must not assume .data.file exists. Contiguous RAM pair reads are read-only views, including empty reads. Arbitrary/repeated RAM maps use one order-preserving gather. HDF5 uses contiguous hyperslabs where possible and unique/inverse gathering for general maps, preserving requested order and repetitions.

The family writer visits bounded row/column pair tiles and writes row-major or lower-triangular spans into flat destinations. It releases each product before the next tile. Neither bounded pair work nor HDF5 MO storage removes complete J/K or native-output allocations from admission.

Owned AO/MO scratch is removed on success and exceptions; caller-owned caches are never removed. The public reference exporter retains atomic publication and temporary-artifact cleanup on errors. A failed overwrite leaves the old artifact intact. There is no retry/recovery framework implied by this cleanup.

Dimensional Estimates

For N611/M500/Q6637, nominal float64 payloads are shown below as a scaling example, not a recommended truncation. The full-MO target for that reference is M611/A20; its resource validation is a separate experiment.

Payload	Decimal GB
Full packed AO factors, if built	9.927
MO factors in RAM or HDF5	6.650
Both full J/K slices in RAM	1.002
Four final packed families, C106/A20/V374	9.911
Six metric matrices reserved during fresh source construction	2.114

Arrays from different phases do not all coexist. These are payload and scheduling estimates, not measured peak RSS or a 50-GB certification. Scratch and output capacity are checked with 10% free-space headroom, combining requirements when they share a filesystem. Separate checks cover owned AO construction and the chosen MO store; concurrent jobs can change free capacity after admission. Full-route reorder storage is charged to the scratch filesystem at both checks, not to the final-output filesystem; selected export has no reorder scratch.

Full export still materializes native pair and packed/reorder arrays. The conservative full-output ledger reserves a native pair matrix plus three packed vectors:

native_and_packed_bytes = 8 * T(M)**2 + 3 * 8 * T(M) * (T(M) + 1) / 2

At M500 these payloads alone total 313.753 GB, before pair work, any resident MO store, process base and reserve. Admission therefore rejects that full route under a 50-GB budget before factor construction or large output allocation. There is no exact fallback. Selected export bounds producer work, not consumer capacity or total output size; it is not a constant-memory route.

Coverage Contract

Scenarios in pyscf2mppt/tests/features/df_expanded_dump.feature precede the implementation and executable tests. Pytest is the runner, not a BDD framework. The IDs and observables below describe the source coverage contract, not a claim that a particular publication pipeline has passed.

ID	Purpose	Existing overlap	Distinct observable
DF-01	Explicit approximation and native source forwarding	Exact slab configuration and reference adapters	Omitted normalized identity, strict mode, DF/slab rejection, unchanged provider identity
DF-02	Complete matched-DF scientific assembly	Exact reference/HDF5 tests	Every final integral/energy float on both RAM/file full and C/A/V selected spaces, correct backend metadata, without exact AO4/J/K
DF-03	Bounded production and ownership	Reference atomic publication and exact scratch tests	Both owned MO stores close/clean correctly; caller RAM/file caches persist; fresh RHF default and explicit unbuilt label map
DF-04	Honest admission and source validation	Exact memory planner arithmetic	Same-size wrong AO rejection, non-Coulomb rejection, metric and full-output rejection before allocation
DF-05	Live/pending arithmetic	Exact shared-budget tests	Elevated-RSS counterexample, no double counting or fake release, fresh headroom and separate temporary cap
DF-06	Whole-phase storage choice	DF-04 impossible-budget rejection	Allocation-free dimension planning, actual-rank RAM/file thresholds, all later phase peaks included, unchanged Q/tile maxima
DF-07	Pair read semantics	DF-02 artifact parity	Read-only contiguous RAM views, empty reads, one RAM gather, repeated/nonmonotone order from both stores
DF-08	Last-use release	DF-03 file cleanup	Weak references prove J/K survive native writing but not family expansion; property/default objects also expire

These are assembly and contract checks, not DF-versus-exact accuracy bounds or a large-cluster performance certification. The two DF-02 routes are necessary: selected packing/frozen-core and native full ordering have different contracts.

The oracle uses native DF.ao2mo on tiny rotated/trimmed water MOs and the existing consumer writers, independently of this producer’s factor transform. Every final dataset and attribute is compared, with only the new approximation group added. Forced Q/pair tails exercise bounded loops. Exact AO4, direct J/K and SCF entry points fail the test if entered during DF export.

Documentation Verification Intent

The documentation-only review uses this Gherkin intent before verification; it does not add an executable test or replace DF-01 through DF-08.

Feature: Functional and technical DF documentation stay consistent
  Scenario: DOC-DF-01 Split guidance without changing the export contract
    Given the existing DF source and DF-01 through DF-08 coverage contract
    When the combined page is split into functional guidance and internals
    Then the functional page states supported controls, prerequisites and limits
    And internals retain formulas, lifetimes, admission and coverage semantics
    And both pages distinguish assembly roundoff from approximation error
    And only the two documentation pages change

  Scenario: DOC-DF-02 Integrate DF guidance with current main documentation
    Given current main CLI, AVAS and exact-export guidance and the DF contracts
    When documentation conflicts are resolved without editing production or tests
    Then exact-only controls cannot be mistaken for DF storage overrides
    And public defaults, auxiliary sources, maps and output contracts persist
    And current output-dir, resume and logging guidance remains authoritative
    And old anchors and both technical pages remain reachable from the toctree
    And strict HTML and link checks distinguish internal from external failures
    And the documentation does not claim DF is already merged or deployed

Purpose: keep operational decisions visible without requiring implementation knowledge. Existing overlap: DF-01 through DF-08 check executable behavior. Distinct observable: source-consistent prose and cross-page ownership, with unchanged scenario IDs and no new configuration or numerical claims. DOC-DF-01 covers the initial two-page split. DOC-DF-02 extends navigation review to the integrated pages; it overlaps DOC-EXACT-02/04 on links and builds, but additionally checks that DF source forwarding and mode-specific limits survive the port. Documentation build logs are validation evidence, not new quantum-chemistry or executable-test results.