Selected-Orbital Integrals#

Use mppt.N_select > 0 to export the compact selected-route HDF5 data needed by DIAGPT without writing the full-space int2e_full table. YAML and Python inherit exact/DF from the selected reference. A density-fitted source uses DF-expanded export; memory and scratch controls do not change the approximation.

To keep all reference MOs and choose a selected window inside that space, leave delete_from: 0. For an input whose orbital count and occupations support a five-orbital window, the fragment is:

mppt:
  delete_from: 0
  N_select: 5

The selected orbitals are part of the retained full MO space, not additional orbitals. The exporter does not automatically truncate a virtual tail to fit memory. N_select: 0 instead requests the full-ERI route, not a way to retain more MOs. Verify the chosen window and the downstream orbital numbering before running selection.

Set the shared memory hint for the resources actually available. Add exact half-transform slabs only in exact mode when their scratch tradeoff is useful. The reference API accepts completed PySCF data; Exact Export Internals and DF Export Internals contain the mode-specific implementation and validation details.

In MPPT, the selected route means MPPT_SELECTED=true: determinants produced for a selected-orbital PySCF/CASSCF dump are remapped into full-orbital numbering before DIAGPT. It does not mean a CIPSI-selected subset of individual integrals. DIAGPT always reads scalar and two-electron integrals from the HDF5 file selected by MPPT_HDF5_FILE (default pyscf_dump.h5).

MPPT_SELECTED selects only the HDF5 contract. The accepted true values are 1, T, t, TRUE, True, true, YES, Yes, yes, Y, and y. An unset or other value selects the full route.

Current DIAGPT Inputs#

Selected-mode DIAGPT consumes these root datasets:

int1e_full
aj_full_packed
ak_full_packed
jk_slice_indices_full
j_slice_full
k_slice_full
ppaa_pair
papa_pair
pacv_pair

It also requires root metadata including num_irrep, PotNEcor, E_frozen, n_select, selected_idx, n_orb_per_irrep, and the eri_blocks_selected_ncore/nact/nvir/nmo attributes.

The three pair-packed families above are required, plus exactly one CVCV representation: cvcv_pair (packed) or cvcv_factors (factorized). Providing both, or neither, is an error. Exact export uses the packed representation; selected DF export now retains the factors.

Selected DIAGPT rebuilds Fock quantities from int1e_full and the compact J/K slices. It does not consume fock_full_packed or int2e_full. e_vide_selected is not part of this contract and is not written.

Full-Route Contract#

Full mode requires root attributes num_irrep, n_orb_per_irrep, n_occ_per_irrep, and e_vide_full, plus aj_full_packed, ak_full_packed, fock_full_packed, and int2e_full. The model-space orbital count and numbering must agree with that full orbital universe. DIAGPT loads int2e_full into dense packed bijkl; ai() remains the accessor for that storage. Full mode does not load selected pair-packed blocks.

Pair-Packed Blocks#

Let M be the full orbital count, C the core count, A the active/selected count, and V the virtual count. Define T(n) = n(n+1)/2, N_PA = (C+V)A, and N_CV = C*V.

Dataset	Integral family	Stored length
`ppaa_pair`	`(pq	ab)`, full pair by active pair
`papa_pair`	triangular nonactive-active pair table	`N_PA(N_PA+1)/2`
`pacv_pair`	nonactive-active by core-virtual pairs	`N_PA * N_CV`
`cvcv_pair`	triangular core-virtual pair table	`N_CV(N_CV+1)/2`

These families contain the ERIs required by selected-mode DIAGPT while omitting unrelated full four-index combinations.

DF CVCV factors#

cvcv_factors is a real float64 matrix of shape (C*V, Q), with CV rows in core-major, external-virtual-minor order. It contains whitened Coulomb factors: (cv|c'v') = sum_Q B[cv,Q] * B[c'v',Q] within the chosen DF approximation. Required root attributes are cvcv_factor_layout="cv_aux_v1" and the positive integer cvcv_factor_rank=Q. Empty C/V spaces have zero rows and retain Q.

DIAGPT validates shape, type and layout, leaves the factors on disk, and reads bounded hyperslabs. The isolated cvcv_contribution kernel reconstructs direct and exchange tiles using GEMM and immediately applies the original a0006 weights/denominators. No complete CVCV matrix is allocated. Other inactive selected-orbital contributions still belong to a0006; its entire inactive branch is not synonymous with pure CVCV.

Environment variable	Default	Meaning
`DIAGPT_CVCV_WORK_MB`	64	MiB budget for four explicit factor panels and two ERI tiles
`DIAGPT_CVCV_TILE`	256	Upper bound for the virtual tile side
`DIAGPT_CVCV_QBLOCK`	Q	Upper bound for each auxiliary chunk; all Q chunks are included

These are positive integer scheduling controls, not truncation thresholds. The workspace budget excludes other DIAGPT arrays and library/HDF5 caches; it is not a process RSS limit. HDF5 reads are serial; GEMMs use the configured BLAS threading. OPENBLAS_NUM_THREADS can control OpenBLAS separately from the OpenMP loops elsewhere in DIAGPT.

[DIAGPT CVCV] reports the common contribution and wall time. Factor mode also prints tile/Q sizes, explicit workspace bytes, requested factor-read bytes, and I/O/GEMM time. Repeated panel reads are included in read bytes.

Workflow fingerprints include the actual CVCV dataset and factor layout/rank. Rebuild DIAGPT and update the Python package together before consuming new DF artifacts. Existing packed artifacts remain supported. Generic scalar/indexed CVCV lookups fail explicitly on factorized input; the bounded kernel owns that access path.

Indexing Invariants#

The built-in writer stores selected_idx as one-based full-orbital indices.
HEFFSO preflight accepts an external raw selected_idx attribute whose range is consistently zero-based or one-based when length and uniqueness are valid. It does not rewrite that stored attribute, but workflow identities hash its normalized canonical one-based values. DIAGPT applies the same normalization before remapping determinants, and the DIAGPT-to-HEFFSO handoff always stores a one-based signed-int64 map.
The pair-block reader currently expects selected orbitals to be the contiguous interval immediately after the core orbitals.
C + A + V must equal M.
Every pair-packed vector length must match the formulas above.
YAML orbital labels are human-facing and one-based; cipsixx configuration is normalized to zero-based indices inside its native HDF5 group.

/cipsixx@orbital_space is authoritative when present. If it is absent, workflow preflight infers the route from the model-space, full-space, and selected-space orbital counts. A full-count match takes precedence, a selected-count match selects the selected route, and inconsistent counts are rejected.

The DIAGPT-to-HEFFSO handoff records this distinction explicitly with the required scalar integer attribute selected_remap. Its value is 1 exactly when the canonical selected_idx dataset is present, and 0 for an unremapped full-route handoff. Route selection never relies only on comparing the two orbital counts. The selected map and final remapped determinant vectors are included in the required determinant_content_hash_fnv1a64, so a stale or modified catalog is rejected even when all dimensions still match.

Use mppt.diagpt.integral_space: auto for normal workflow operation. It follows the orbital-space contract stored in HDF5. Explicit full or selected is intended for controlled validation and must match the model-space numbering.

Model-Space Limits#

The compiled spatial-orbital capacities are doa=3000 in DIAGPT and kzo=3000 in HEFFSO. Both consumers reject larger orbital counts before loading the large integral payloads or writing bounded orbital arrays. HEFFSO reserves a separate bookkeeping slot for the odd-electron infinity marker; it is not a physical MO. Rebuild both consumers to use the increased capacities.

Packed ERI extent validation uses int64 size queries, matching the existing int64 counts and offsets. DIAGPT loads the three common packed vectors and, for packed inputs, CVCV into RAM; factorized CVCV stays on disk. The orbital capacity is not a memory-fit or runtime guarantee. Package unit tests cover the capacity boundary, infinity bookkeeping and bounded synthetic integral reads. A 3000-orbital production calculation is not implied by this compile-time bound or by those tests.

Current preflight rejects excitation rank above eight, DIAGPT dense model spaces with ncf > 46340, metat > 1000, or metat > ncf. When HEFFSO is enabled, it additionally rejects ncf > 35000 and packed excitation-slot count above 280001. DIAGPT also requires 2*norb + 1 and its packed excitation-slot count to fit signed 32-bit integers. These are hard validation errors; the workflow never truncates the determinant or state space to fit.

Producer Behavior And Limits#

The selected pyscf2mppt writer does not build or store the full-space s4/s8 MO integral table and does not create int2e_full. For an exact source, it obtains the full-orbital J/K slices from bounded AO J/K batches, transforms the selected subspace integrals, and builds only the four required pair-packed families using exact Coulomb integrals. The full route writes int2e_full.

A DF source uses automatically selected RAM or HDF5 MO factor storage for every two-electron-derived quantity and writes PPAA/PAPA/PACV plus retained CV factors, without materializing CVCV. The integral payloads remain float64. See DF-expanded export for compatible sources, memory hints and validation limits. Approximation selection is separate from the full/selected consumer route; there is no flag to force DF MO storage.

The built-in writer emits the canonical selected-route inputs and root metadata listed above. External producers are validated against the same contract.

The selected route omits the full T(M)(T(M)+1)/2 ERI vector from RAM, scratch and final HDF5 storage, but is not constant-memory. The J/K slices contain 2*M*T(M) values; native selected integrals and final packed families still grow with the retained/selected spaces. Persistent arrays can coexist with transformation buffers. Both memory and destination capacity must fit; see the exact memory model or DF dimensional estimates for engineering details.

Shared Memory Budget#

Start with molecule.max_memory, the total memory hint used by the dump. The public reference API uses reference.mol.max_memory. The units and cap semantics below apply to exact and DF, on both selected and full export routes.

Setting	What to supply	What it controls
`molecule.max_memory` / `reference.mol.max_memory`	Numeric decimal PySCF MB; `50000` means 50,000,000,000 bytes, not 50000 MiB	Total hint, constrained by visible cgroup limits and current host availability
`mppt.max_mem_bytes` / `ExportConfig.max_mem_bytes`	Omit, or use YAML `null` / Python `None`	Automatic workspace from the total hint; no implicit 2-GiB cap
Explicit `max_mem_bytes`	Positive integer bytes; YAML also accepts legacy unit strings	Optional temporary-buffer cap within the remaining workspace, not a replacement total
`scf.max_memory` / `casscf.max_memory`	Stage-specific PySCF MB	Override those calculation stages, not the dump’s molecule total hint

Historical YAML GB, MB and KB suffixes on mppt.max_mem_bytes use powers of 1024: "2GB" is 2,147,483,648 bytes. Python ExportConfig requires integer bytes, not a unit string. Existing explicit values keep their type and meaning; for example, 16 * 1024**2 is still a valid cap value, but runtime admission can reject it if it is too small for even one unit of work.

Neither setting is a hard process limit or an output-size limit. Use separate OS enforcement when required and preflight scratch and destination capacity. Caller-owned SCF/DF caches still consume RAM; the exporter does not delete them to make a request fit. Exact full-route automatic transform/reorder choices use the remaining workspace, including an explicit cap, but still materialize int2e_full. Native and packing buffer floors can exceed a tiny cap; it is not a strict per-allocation ceiling. Full DF also materializes int2e_full and retains large native pair and packed/reorder allocations even with HDF5 MO factors; force_transform_outcore cannot change that DF behavior.

If The Budget Does Not Fit#

The planner raises ValueError before the affected allocation when modeled work cannot fit. This can happen after earlier export stages have completed; it is not a guarantee against a later native allocation failure or OS OOM.

Diagnostic or situation	What to change
Exact: `One J/K projector does not fit before allocation`, with `legacy_buffer_cap` below `legacy_minimum`	Remove the cap with `null`/`None`, or increase it, only if the total budget has room. The error reports byte counts.
`Integral workspace does not fit`, `DF phase workspace does not fit`, or an exact projector still cannot fit without a cap	Check current process memory, available host RAM and the worker’s cgroup limit. Raise the molecule total only when that RAM is genuinely available within the allocation. Removing a cap alone cannot create memory.
Scratch fills during selected exact export	Consider a smaller optional half slab and more scratch capacity. Other temporary files and final HDF5 storage still need space.
DF scratch/output admission fails	Check scratch and final-output free capacity. DF storage is automatic; an exact-only force flag or half-slab setting is not a remedy.

A nominal 1-MB total can be below the already resident Python/reference process and is rejected on both routes. To request small work buffers, keep a feasible total and set a separate cap. The HOCl full-route regression uses a 512-MB total and unchanged 16KB cap to select automatic outcore transformation and streaming reorder, without force flags or relaxed numerical expectations. See the exact memory model and routing limits and the DF phase model; small tests are not large-Ce resource or production certification.

Screened Direct J/K#

This applies only to exact export; no additional J/K setting is needed. See the screening implementation and accuracy limits. DF export instead derives J/K consistently from its chosen approximation.

Exact Half-Transform Slabs#

Optionally set mppt.exact_half_block_size to a strict positive integer to reduce the half-transform scratch file for selected blocks in exact mode:

mppt:
  N_select: 5
  exact_half_block_size: 4

The default is null (None in Python), which keeps one monolithic half-transform file. Booleans, fractional values and nonpositive sizes are rejected. The option requires N_select > 0; YAML, the public ExportConfig API and direct dump_integrals calls reject full-route requests rather than silently ignoring the option. Python callers use ExportConfig(n_select=5, exact_half_block_size=4), or the keyword-only exact_half_block_size argument of dump_integrals. The option is incompatible with integral_mode="df" and is rejected, not ignored, in DF mode.

Omitting the YAML key leaves it absent from normalized input, preserving legacy checkpoint identity. An explicit null or positive size remains in normalized input; null still selects the monolithic transform.

Choose the slab width manually; the RAM planner neither chooses nor resizes it. Below the core-plus-selected width, a smaller width reduces the largest single half-transform scratch payload approximately in proportion to the width for fixed orbital dimensions, but requires more half-transform builds and repeated AO integral passes. A larger width needs more scratch for that file and fewer repeated passes, so it can be faster, but a speedup is not guaranteed. Do not assume wall time scales linearly with width or build count.

Illustrative dimensions, not a Ce calculation: N=1000 AOs, M=500 retained MOs, C=100 core and A=20 selected MOs. The table counts float64 payload only, in decimal GB (1 GB = 1,000,000,000 bytes), and builds for one sequential traversal of the core-plus-selected range.

`exact_half_block_size`	Largest single half-file payload (GB)	Half-transform builds
`16`	25.6256	8
`32`	51.2512	4
`64`	102.5024	2
`null` / `None` (monolithic width 120)	192.1920	1

These are storage estimates, not measured timings, RSS or whole-export peak disk use. Allow additional capacity for HDF5 overhead, dense CV-CV staging, other scratch and the final output, plus RAM for live arrays and workspaces. Changing the width does not reduce J/K work or final dataset payloads.

Changing the width preserves integral values, retained MOs, ordering and the ordinary HDF5 contract, including the complete Hamiltonian and requested SO/property data. It is not a DF switch or an automatic fallback. See the slab formula and lifetime model for storage estimates.

Changing Settings After A Checkpoint#

CLI --resume validates the saved normalized input and resources in the run directory; it does not turn a completed checkpoint into a new export with different memory, slab or approximation settings. Use the existing reference API with an explicit ExportConfig and a new destination for a separate export of a completed reference, or start a fresh calculation with a new --output-dir when that is intended. See the checkpoint rules before reusing saved work.

Exact Half-Transform Acceptance Scenarios#

The existing slab scenarios and coverage map now live with the engineering details in Exact Export Internals.

Shared-Budget Acceptance Scenarios#

The existing exact memory scenarios and coverage map are retained separately from configuration guidance. DF has its own coverage contract. Neither implies that every integration or resource gate has passed.