Exact Export Internals

Use Selected Route, External PySCF References and the YAML input reference to configure an export. This page owns the memory model, scratch lifetimes, numerical screening and acceptance coverage behind those controls.

The exporter uses exact Coulomb integrals with numerical screening, not a density-fitted export. Scheduling changes preserve the retained MO space, Hamiltonian, requested SO/property providers and ordinary materialized HDF5 contract. These implementation notes and small tests do not certify large-Ce accuracy, resource use, performance or downstream capacity.

Documentation Validation Scenarios

These scenarios specify the documentation-only verification before it is run. They do not add or change numerical tests or introduce a BDD runtime.

ID	Purpose and existing overlap	Distinct observable
DOC-EXACT-01	Product config tests cover forwarding, not instructions a reader can follow	YAML/API guidance agrees with source on units, defaults, retained MOs, optional caps/slabs and restart behavior
DOC-EXACT-02	Product lifetime tests do not check hosted navigation	Deep details and both existing feature inclusions are reachable from functional pages and the primary toctree; old section anchors remain usable
DOC-EXACT-03	Numerical parity does not validate claims about supported modes or resources	No unsupported export switch, automatic MO truncation, hard-RSS promise or CI/production certification is introduced
DOC-EXACT-04	Source review does not detect Sphinx reference or link errors	Strict HTML build and linkcheck report new internal failures separately from external-site policy failures
DOC-EXACT-05	SLAB-03 covers runtime ranges and lifetime, not the illustrative documentation arithmetic	Single-file float64 payloads, decimal GB and sequential build counts agree with the formula without allocating integral arrays or running quantum chemistry

Feature: Usable exact-export documentation without implementation changes
  Scenario: DOC-EXACT-01 Configure an export from the functional pages
    Given the supported YAML schema and public reference API
    When the configuration fragments and operational warnings are reviewed
    Then their defaults, units, route selection and failure guidance match source
    And no calculation needs to run to validate the documentation

  Scenario: DOC-EXACT-02 Reach the engineering details
    Given the functional pages and primary documentation toctree
    When their links and the previous public section anchors are followed
    Then formulas, lifetimes, screening and existing coverage maps remain reachable

  Scenario: DOC-EXACT-03 Keep promises within the implemented scope
    Given exact export and the existing HDF5 and consumer contracts
    When supported modes and limitations are reviewed
    Then memory controls do not imply DF export, MO truncation or a hard RSS cap
    And whole-process admission is distinguished from full-route routing hints

  Scenario: DOC-EXACT-04 Build and check the hosted pages
    Given only the intended documentation changes
    When strict Sphinx HTML and the documentation linkcheck target run
    Then build results and link diagnostics are retained
    And external HTTP policy failures are distinguished from broken internal links

  Scenario Outline: DOC-EXACT-05 Check the illustrative slab estimates
    Given illustrative dimensions N=1000, M=500, C=100 and A=20
    And a first-side width <b>, with None resolving to C+A
    When only integer payload and sequential build-count arithmetic is evaluated
    Then the largest single half-file payload is <bytes> bytes or <GB> decimal GB
    And sequential consumption requires <builds> half-transform builds
    And these are not measured RSS, total peak disk or wall-time estimates

    Examples:
      | b    | bytes        | GB       | builds |
      | 16   | 25625600000  | 25.6256  | 8      |
      | 32   | 51251200000  | 51.2512  | 4      |
      | 64   | 102502400000 | 102.5024 | 2      |
      | None | 192192000000 | 192.1920 | 1      |

Shared Memory Model

pyscf2mppt/integrals/memory_budget.py owns IntegralMemoryBudget. The public units, automatic defaults and optional cap are described in Shared Memory Budget; they are not separate budgets for each export phase.

The factory converts numeric mol.max_memory from decimal MB to bytes and intersects it with the lowest readable finite cgroup-v2 memory.max. It captures current process RSS as the base, not lifetime peak RSS. Reference coefficients, Hamiltonian arrays and caller-owned SCF/DF caches already in that base are not added again or deleted by the exporter.

The internal reserve is ceil(effective_total_bytes / 10). This 10% policy is separate from modeled native memory, not a user knob or a universal allocation bound. At scheduling boundaries, current host MemAvailable and visible cgroup headroom constrain additional allocation. Workspace subtracts max(current_rss, base_bytes + fixed_bytes) from the usable total rather than counting resident fixed arrays twice. An explicit buffer cap further limits that remaining workspace.

For visible cgroup-v2 ancestors, estimated reclaimable clean disk cache is min(current, max(0, file - shmem - file_dirty - file_writeback)). Anonymous memory, tmpfs/shmem, dirty pages and writeback remain charged. Missing or invalid cache statistics receive no credit. Mandatory Linux procfs reads and malformed cgroup limit/current counters fail rather than inventing a machine size. This targets the visible unified hierarchy, not arbitrary cgroup-v1 or mount layouts.

These are observations, not reservations against other processes. PySCF hints do not enforce an OS limit. A separately configured cgroup such as MemoryMax=50000000000 with MemorySwapMax=0 provides enforcement; the exporter does not create it. Process RSS and cgroup peak, which includes charged cache, are different measurements.

J/K Peaks And Admission

Let N be the AO count, M the retained full MO count, B the projector batch size, and T(M) = M(M+1)/2. The modeled fixed J/K bytes beyond the base are:

16*M*T(M) + 16*T(M) + 8*M + 24*N^2

The terms cover final J/K slices, int64 packing/source indices and conservative optimizer bounds, including overlapping old/new density bounds. The AO phase adds 40*B*N^2 plus thread-private native bytes. The MO/packing phase instead adds:

8*B*(2*N^2 + 2*M^2 + max(M*N, 2*T(M)))

Admission uses the larger phase peak plus the fixed/base bytes and the separate reserve. The legacy cap independently requires at least 40*max(N^2, M^2) bytes for one projector. Thus 16 * 1024**2 remains a valid integer cap with unchanged meaning, but cannot admit even one projector when max(N, M) >= 648. Passing configuration validation does not promise runtime admission; removing a small cap cannot fix an insufficient total budget.

The scheduler finds the largest modeled fitting batch, then rounds down to an even size when at least two items fit (67 -> 66, 66 -> 66). It retains one item when only one fits and raises ValueError before the affected allocation when none fits. Every actual tail is processed. Only scheduling dimensions change: physical N, M, selected count A, occupations, MO coefficients, maps and output dtypes are not rounded or reduced.

Native Assumptions And Lifetimes

The native model follows PySCF 2.14 VHF direct contraction. It counts touched private output tiles, index structures, shell buffers and an estimated cache allowance per thread. The 400,000,000-double stack allowance is shared across the 2*B operators, not multiplied by that operator count. The estimated cache allowance max(8 MiB, 64*S^4 bytes), with S the largest AO shell dimension, is not a verified libcint cache bound: AO counts alone do not determine primitive/contraction-dependent caches. Nominal virtual allocations are not measured RSS, and strict OS overcommit can reject untouched virtual allocations.

compact_intermediates.py constructs one C-contiguous density batch. AO density/native temporaries expire before MO transformation; AO outputs then coexist with the MO matrices and either a matrix-product intermediate or packing copies. The matrix-product intermediate expires before packing. Batch work arrays do not remain owned by the returned compact result, but full J/K slices, packed Fock/pair terms and their maps do remain live for subsequent export work.

Later half slabs must include already allocated ERI outputs and the reusable row reader. eri_live_bytes() models PACV, AAPP, PAPA and transformation work arrays, plus the row reader; CV-CV is assumed file-backed. Compact results, base-owned arrays and backend workspace are accounted for separately. The final PPAA transpose copy and writer workspace also need admission.

Memory-plan/profile events record budgets, modeled peaks/workspaces, selected blocks and ranges, screening, timings and sampled process memory. Profile evidence, actual tails and measured native peaks must be reviewed together; static byte arithmetic or weak-reference lifetime tests are not measured Ce resource acceptance.

Full-Route Admission And Routing

Both routes reject a total already exhausted by resident Python/reference memory and the reserve. The budget is captured after hcore validation and MO projection, before SO allocation. Full-route selection then intersects the legacy conservative Python-array budget with int(1_000_000 * workspace_mb); that workspace has already passed admission and includes any explicit max_mem_bytes cap. Backend flags and their diagnostic messages use this same intersection. The selected-route budget capture and projector minimum are unchanged.

The scf_hocl_low_memory regression now supplies a feasible 512-MB total and retains its 16KB (16384-byte) cap. Its unchanged expected.json checks automatic outcore transformation, streaming reorder, energy, scalar constant, HDF5 shapes and orbital maps. A 1-MB total below process RSS is still rejected; raising only the total without the routing intersection incorrectly selects incore work.

This is a routing fix, not a full-route planner rewrite. Full exports still produce int2e_full. Existing packing/reorder planning floors (64 MiB and 16 MiB respectively) and PySCF native minima can exceed the small work hint. Neither the 16-KiB cap nor a passing small fixture proves an allocation ceiling, bounded full-route peak RSS, green remote CI or large-system feasibility.

Screened Direct J/K

The shared-budget J/K path uses functional hf.get_jk with one reusable VHF optimizer for the molecule/basis/operator. Integral bounds and the libcint optimizer are reused; density bounds are refreshed for each actual batch. This is the exact Coulomb route with numerical screening, not DF or a call through a density-fitted mean-field object.

The internal screening tolerance is 1e-14, not an additional user control. Record the actual tolerance and validate full J/K slices, Fock, pair terms and scalar energy on identical fixed MOs against an independent unscreened oracle and a stricter-threshold check. The threshold is not a global bound on accumulated MO-integral or energy error. Small-molecule parity does not certify large-Ce accuracy, memory use or speedup; end-to-end and measured resource acceptance remain separate gates.

Exact Half-Transform Slabs

HalfTransformSource in pyscf2mppt/integrals/half_transform.py supplies rows to the existing second-half contractions while owning one monolithic or slab scratch file. For N AOs, core count C, selected count A, full retained count M and requested first-side width b, the half-transform payload is at most:

8 * min(b, C+A) * (M-C) * T(N) bytes, where T(N) = N(N+1)/2

The monolithic payload is 8 * (C+A) * (M-C) * T(N) bytes. These formulas exclude HDF5 overhead and every other file. Final pair-packed lengths remain those in the public block contract.

Each first-side MO contributes (M-C) * T(N) float64 values, at eight bytes each. Thus the largest single half-file payload grows linearly with b up to C+A; it does not estimate process RSS or the whole export’s peak disk use. For the illustrative N=1000, M=500, C=100, A=20 case, T(N)=500500 and each first-side MO contributes 1601600000 bytes, or 1.6016 decimal GB.

For sequential row consumption, the number of half-transform builds is ceil((C+A)/b). The source keeps b fixed, clips each end to stop = min(start+b, C+A) and calls ao2mo.outcore.half_e1 separately for each slab. None resolves to b=C+A; an explicit b >= C+A also needs only one build. Smaller widths repeat more AO integral passes; larger widths can therefore be faster, but build counts are not wall-time multipliers and do not guarantee a speedup.

The first slab is built before ERI result allocation. On each later slab, the old half file is closed and removed before the next is generated; the reusable row buffer remains live. Actual final ranges can be shorter than b. Context manager cleanup handles ordinary closure and exceptions without introducing tile resume or a retry/recovery framework.

Only the half-transform swap is bounded by this option. Dense CV-CV staging, PPAA/PAPA/PACV arrays and J/K data remain, as do their ordinary output writers. The half-slab option does not change second-half contractions, J/K, integral values, MO ordering or the HDF5 contract. The width remains explicit and resource-dependent, separate from automatic J/K and reader/transform work blocks: the RAM planner does not silently resize it. Smaller slabs can repeat AO integral passes and slow the dump. Disk and native-memory preflight are still required.

Omitting the YAML slab key leaves it absent from normalized input, preserving legacy checkpoint identity. Explicit null or a positive size remains in normalized input; null still selects the monolithic transform. Existing explicit memory values and aliases are not rewritten. Direct checkpoint resume restores saved input/workdir and may return immediately for a completed dump, as described in the YAML checkpoint reference.

Exact Half-Transform Acceptance Scenarios

The scenarios below were specified before the slab implementation. Tests use ordinary pytest with Gherkin intent in their docstrings, not a separate BDD runtime. Each scenario protects a different observable. Test paths below are under pyscf2mppt/tests/.

ID	Necessity and overlap	Test location
SLAB-01	Existing ERI tests check dense block parity; this checks the complete selected HDF5 and observes the real source option, so a silent no-op cannot pass on parity alone	test_exact_half_slabs.py
SLAB-02	SLAB-01 starts at the public API; this checks the separate YAML/runtime wiring	test_exact_half_slab_config.py
SLAB-03	Equal integrals do not prove bounded disk; this observes source ranges, tail, file lifetime and buffer reuse	test_exact_half_lifetime.py
SLAB-04	Normal closure does not cover an interrupted half build or second transform; no retry/recovery framework is implied	test_exact_half_lifetime.py
SLAB-05	Reject zero/negative/noninteger sizes and the unsupported full route before expensive work; no speculative bad-input grid	test_exact_half_slab_config.py

Feature: Bounded exact half-transform for selected integral dumps
  The approximation, second-half contractions and final packed format stay unchanged.
  Omitting exact_half_block_size keeps one monolithic half-transform.
  A positive size limits the first MO side of each produced half-transform.
  No DF, automatic fallback, tile-resume or full-route slab backend is introduced.

  @SLAB-01
  Scenario Outline: A selected dump preserves its complete numerical content
    Given one fixed small water RHF reference and a nonempty core, selected and virtual partition
    And an HDF5 dump made without the slab option
    When the same reference is dumped with exact_half_block_size <size>
    Then the real half-transform source receives the requested block size
    And all HDF5 integral datasets and physical metadata match within 1e-10 absolute and relative tolerance
    And the selected route still has no int2e_full dataset

    Examples:
      | size |
      | 1    |
      | 4    |
      | 99   |

  @SLAB-02
  Scenario: YAML selection reaches the existing reference exporter
    Given a selected-route YAML input with exact_half_block_size 4
    When the input is parsed and its dump stage is prepared
    Then the existing public reference exporter receives exact_half_block_size 4

  @SLAB-03
  Scenario: The producing source bounds storage and consumes the partial tail
    Given fixed water MOs with seven first-side orbitals and block size 4
    When every half-transform row is consumed in order
    Then the half-transform is built for ranges 0:4 and 4:7
    And each half payload is at most 8 times 4 times the second-side count times the AO-pair count bytes
    And the previous half file is deleted before the next file is built
    And the same row buffer is reused
    And the source leaves no half file when it closes

  @SLAB-04
  Scenario Outline: An interrupted transform releases its owned scratch
    Given an exact selected block transformation with block size 4
    When a controlled exception is raised during <stage>
    Then the original exception propagates
    And neither an owned half file nor an owned dense CV-CV file remains open or on disk

    Examples:
      | stage       |
      | half build  |
      | second half |

  @SLAB-05
  Scenario Outline: Invalid or unsupported slab requests are rejected
    Given an export request with exact_half_block_size <size> and N_select <selected>
    When the request is validated
    Then it is rejected before integral generation

    Examples:
      | size | selected |
      | 0    | 5        |
      | -1   | 5        |
      | true | 5        |
      | 1.5  | 5        |
      | 4    | 0        |

Shared-Budget Acceptance Scenarios

shared_integral_memory.feature was written before the new tests. The map below states purpose, old coverage overlap and the distinct observable required for acceptance; it is not a claim that every integration/benchmark gate has passed. Keep the existing SLAB-01 through SLAB-04 coverage instead of duplicating slab size or cleanup matrices. Test paths below are under pyscf2mppt/tests/.

ID	Purpose and old overlap	Distinct observable and coverage owner
SMEM-01	Preserve automatic defaults and legacy caps; SLAB-02 covers half-size forwarding	test_shared_memory_config.py checks omitted/null/explicit caps through YAML/runtime and the public API with a stubbed integral writer, including full-route/default API calls, no injected 2 GiB and unchanged normalized input. Unchanged IG-09 tests in test_scf_initial_guess_compatibility.py reuse the genuine old fixture to verify that an omitted slab key preserves its hash and checkpoint eligibility, which forwarding alone cannot establish. Planner tests own the 50000 MB conversion and lower external ceilings, not these forwarding stubs.
SMEM-02	Prove allocation-free byte selection rather than only J/K formulas	Planner arithmetic checks hand-worked fixed/native/reserve bytes, Python-integer shape products, even alignment, a one-item minimum and pre-allocation rejection; physical dimensions remain unchanged.
SMEM-03	Extend the old two-MO H2 one-batch oracle, not another slab-size sweep	The J/K numerical test uses seven fixed water projectors in 0:2, 2:4, 4:6, 6:7; it observes one reused optimizer with current density bounds and checks slices, Fock, pair terms and scalar energy at explicit tolerances, including stricter screening.
SMEM-04	Add live-memory ownership to SLAB-03/04 lifetime and failure-cleanup coverage	Extend test_exact_half_lifetime.py observations to include later-slab ERI/reader budgets; J/K lifetime checks distinguish released work arrays from retained compact results. Keep the existing cleanup matrix; weak-reference lifetime is not measured RSS.
SMEM-05	Reuse SLAB-01 complete-artifact parity and the existing full-route regression tests	Extend test_exact_half_slabs.py with automatic versus explicit-cap export, not another size matrix: names/shapes/dtypes and integer maps are exact, scientific metadata is preserved, floating fields use declared tolerances and selected output has no int2e_full.
SMEM-06	Planner and forwarding tests cannot prove actual full-route backend selection	The existing test_suite/pyscf2mppt/integral_dump/scf_hocl_low_memory fixture uses a feasible total with its unchanged 16KB cap, automatic outcore/stream assertions and numerical/HDF5 oracle. test_full_route_memory.py separately rejects an exhausted total at the public entrypoint before two-electron work.
SMEM-07	Existing AVAS export and slab tests cover each feature separately	test_projection_basis_dump.py adds one selected/slab variant to its full-route case, reusing the same final-MO projection oracle and basis metadata checks to exercise both public config and writer arguments together.

Feature: One-budget exact integral export
  Use the existing total memory setting without adding stage-specific user knobs.
  Keep physical dimensions, float64 ERIs, exact maps and every final dataset meaning.
  Count simultaneous live and native workspaces, reserve first and align work blocks last.

  @SMEM-01
  Scenario: Automatic memory reaches the exporter without a hidden buffer cap
    Given a molecule memory limit of 50000 MB
    When YAML and the reference API omit max_mem_bytes
    Then the exporter receives an automatic buffer request
    And the planner resolves 50000000000 bytes before lower external caps
    And an explicit old max_mem_bytes retains its temporary-buffer meaning

  @SMEM-01 @SLAB-02 @IG-09
  Scenario: An omitted slab option preserves legacy checkpoint identity
    Given a genuine legacy normalized input and its saved input hash
    And the SCF initial guess is omitted or null
    When the input is prepared without exact_half_block_size
    Then normalized MPPT input does not gain an exact_half_block_size key
    And the saved input hash and checkpoint reuse eligibility remain unchanged

  @SMEM-02
  Scenario Outline: Byte planning rounds only a fitting work block
    Given hand-checkable shapes and live native and reserved byte counts
    And the largest fitting block is <raw>
    When the scheduler applies even alignment
    Then the result is <result> without changing physical dimensions
    And modeled peak plus reserve does not exceed the effective total budget
    Examples:
      | raw | result               |
      | 67  | 66                   |
      | 66  | 66                   |
      | 1   | 1                    |
      | 0   | pre-allocation error |

  @SMEM-01
  Scenario: Reclaimable disk cache is not counted as pinned workspace
    Given a finite cgroup limit and disk-backed scratch in its clean file cache
    When available additional memory is calculated
    Then clean reclaimable file cache can be reused
    And anonymous memory dirty pages writeback and shmem stay charged
    And absence of cache counters does not invent reclaimable memory

  @SMEM-03
  Scenario: Screened projector batches match an independent fixed-MO oracle
    Given seven fixed small-water MOs and a direct packed-ERI oracle
    And the molecule total budget covers the Python process while a separate explicit buffer cap forces batching
    And a planned J/K batch of two
    When every projector is contracted with a reusable VHF optimizer
    Then ranges 0:2, 2:4, 4:6 and 6:7 are processed
    And one optimizer is reused with current density bounds for every batch
    And full J/K slices Fock pair terms and scalar energy match explicit tolerances

  @SMEM-04
  Scenario: Sequential and interleaved stages account for live ownership
    Given small controlled workspaces and retained compact results
    And a feasible molecule total budget without changing the fixture chemistry
    When J/K returns and later half slabs are generated
    Then obsolete J/K work arrays are not retained by the result
    And compact results remain available
    And later slab budgets include live ERI outputs and the row reader

  @SMEM-05
  Scenario: Automatic planning preserves a complete selected artifact
    Given the existing fixed-reference selected dump fixture
    When automatic and explicit-cap exports are compared
    Then dataset names shapes dtypes and physical metadata remain unchanged
    And integer maps are exact and floating data satisfy declared tolerances
    And the selected route does not create int2e_full

  @SMEM-06
  Scenario: Full-route automatic backends honor the remaining capped workspace
    Given the existing HOCl full-route low-memory fixture with a feasible 512 MB total
    And its unchanged 16KB workspace cap and automatic backend assertions
    When the dump chooses its transform and reorder backends
    Then both use the smaller of the legacy array budget and checked capped workspace
    And outcore transformation and streaming reorder preserve the existing numerical and HDF5 expectations
    But a total below the resident process is still rejected before two-electron work
    And the cap is a routing and workspace hint rather than a strict native allocation ceiling

  @SMEM-07
  Scenario Outline: Independent AVAS projectors survive the exact export route
    Given the existing four-AO H2 reference with occupied sto-3g and virtual cc-pvdz projectors
    When the public API exports n_select <selected> with half slab <slab>
    Then both basis-labelled weights match the existing final-MO projection oracle
    And orbital metadata retains both independent reference bases
    And int2e_full is present only for the full route
    Examples:
      | selected | slab |
      | 0        | None |
      | 2        | 1    |

  # Coverage map:
  # SMEM-01 extends SLAB-02 forwarding with auto/explicit memory and byte units.
  # Its omitted-slab identity check reuses unchanged IG-09 tests and saved fixtures:
  # forwarding alone cannot detect changed managed hashes or lost checkpoint reuse.
  # SMEM-02 adds pure allocation-free peak arithmetic/alignment and minimum-fit checks.
  # SMEM-03 extends the existing H2 one-batch oracle with a real odd-tail water case.
  # SMEM-04 extends SLAB-03 lifetime observations, not its whole cleanup failure matrix.
  # SMEM-05 reuses SLAB-01 artifact coverage; it is not a second slab-size sweep.
  # SMEM-06 reuses the HOCl CLI/checker oracle without changing expected.json or
  # tolerances. A focused public-entrypoint test adds deterministic exhausted-total
  # rejection; planner/forwarding tests alone cannot detect incorrect route flags.
  # SMEM-07 extends test_projection_basis_dump.py, not the numerical slab matrix:
  # its added selected/slab case exercises the merged ExportConfig/writer arguments
  # while reusing the independent final-MO weight oracle and HDF5 metadata checks.