Runs & Sweeps#

Every CLI calculation owns the same durable artifact layout. The run spelling is optional; both commands below launch an isolated calculation:

pyscf2mppt input.yaml
pyscf2mppt run input.yaml

Managed single point#

Run one complete configured calculation in an isolated directory:

pyscf2mppt run input.yaml --output-dir runs/h2

When --output-dir is omitted, the default is runs/<input-stem> below the current directory. A managed point contains:

runs/h2/
├── .pyscf2mppt.lock
├── summary.md
├── run.json
├── run.log
├── input/       # original.yaml, effective.yaml, resources/, generated/
├── logs/        # raw stage logs, events.jsonl, attempts/
├── reports/     # cipsixx/ CSVs and heffso/ analysis/report outputs
├── data/        # HDF5 and binary results
├── checkpoints/
└── scratch/

input/effective.yaml is the normalized input actually executed. File-based basis, ECP, GRPP, auxiliary-basis and SCF initial-guess resources are resolved relative to the source input YAML directory and copied into input/resources/ before execution. run.json schema v2 with layout_version: 2 records the input hash, run directory, attempt number, exact child command, timestamps, return code, terminal status, executable versions and SHA-256 identities, and a validator-owned artifact inventory. Every artifact record carries its role, producing stage, run-relative path, exact size, SHA-256, required flag, and schema. The input hash includes the contents of those file resources, not only their paths. run.log combines every stage’s standard output and standard error, labels the producing stage and separates repeated attempts. Raw stage logs remain unchanged for scientific parsers. summary.md and logs/events.jsonl update during execution; terminal stage evidence is included in run.json.

A zero child exit is not sufficient for complete. Runs that enable integral dumping require the configured full- or selected-route HDF5 contract and bind it to the effective-input hash. Enabled selection, DIAGPT, and HEFFSO stages are revalidated through their existing HDF5 checkpoints and output contracts before their files are hashed. HEFFSO completion inventories the primary, occupancy, and configuration logs as distinct artifacts. Failed and interrupted manifests may inventory safe logs as diagnostics, but mark every such record nonrequired and never claim a complete result bundle.

A new run rejects an existing output directory. Resume a matching run with:

pyscf2mppt run input.yaml --output-dir runs/h2 --resume

Resume requires the same normalized effective input and the original managed directory location. Each resume creates a new attempt and launches the child pipeline again. Direct PySCF checkpoint resume is used only when all declared checkpoint payloads are readable. A no_dump: true run needs only those payloads. A dump-complete checkpoint additionally requires a structurally valid configured HDF5 file carrying the matching managed-input hash. That hash is written only after a successful managed attempt; failed or interrupted finalization removes the new hash or restores the identity from the previous successful attempt. An interruption before that point, or a downstream failure after the dump was created, can therefore make the next attempt rerun SCF/CASSCF instead of resuming those checkpoints. That fallback opens a new dump in replacement mode, so prior /mppt/workflow stage records in the HDF5 are lost rather than reused. Even when direct resume is available, a pre-dump checkpoint can continue into a replacement dump and erase prior workflow state. Downstream reuse survives only when the resumed direct path does not rewrite the HDF5, normally for a dump-complete checkpoint. Use --force-workflow to rerun otherwise reusable downstream stages deliberately.

A copied or moved managed directory cannot be resumed in place because run.json records its original absolute location. Keep the copy as an archive, choose a new nonexistent output path, and start a managed run from the source YAML. A fresh run rejects every already existing path, including an empty directory.

Managed directories are single-writer resources. A concurrent run or sweep attempt against a directory already in use is rejected.

Enabled downstream executables are checked after the initial run record is created, so a failed executable preflight retains diagnostic logs and status. They may be selected explicitly with --selection-executable, --diagpt-executable, and --heffso-executable or through the existing environment-variable and PATH resolution.

Request the installed Caspy eight-file report as part of terminal validation with:

pyscf2mppt run input.yaml \
  --output-dir runs/pr3 \
  --heffso-report

This option requires mppt.no_dump: false, enabled mppt.heffso, enabled mppt.heffso.properties, and a regular non-editable Caspy installation. Caspy validates the shared HDF5, DIAGPT handoff, all three HEFFSO logs, and properties HDF5 before writing the report. The manifest records Caspy’s installed version, the report-process interpreter bytes, the rendering controls, and every output hash. Archive validation checks those exact bytes and does not render the report again. Sweeps accept the same flag for every point; repeat it when resuming because report policy is part of persisted sweep state.

Install Caspy with the regular package command shown in outputs and restart state.

Managed output paths must remain below the point directory. In particular, mppt.prefix cannot contain /, \, :, or a Windows drive prefix. mppt.output_hdf5 and configured cipsixx report paths must be portable relative paths, cannot escape through .., and cannot collide case-insensitively with each other, managed control names, or the currently reserved prefix-derived h0.1, heffso.h5, sovec, heffso_properties.h5, heffso_occupancy_analysis.log, and heffso_configuration_analysis.log artifacts. Eight report suffixes are reserved for every prefix: state_properties.csv, orbital_character.csv, dominant_configurations.csv, properties_summary.md, line_strength_stick_spectrum.png, line_strength_broadened_spectrum.png, oscillator_strength_stick_spectrum.png, and oscillator_strength_broadened_spectrum.png. Reserved top-level names include run.json, run.log, summary.md, input, logs, checkpoints, and scratch. Retired control names remain reserved. Calculation, base-input, and sweep files must use a .yaml or .yml extension.

Initial-Guess Resources And Identity#

scf.initial_guess selects a native starting density for a fresh RHF/ROHF stage. See the YAML initial-guess reference for the method Enum, strict SAP fit format, checkpoint projection, and native density/Hamiltonian semantics. A native checkpoint guess starts a new SCF calculation; it is not an MPPT --resume checkpoint.

Managed preparation handles these additional input resources:

Initial-guess field	Captured resource
`method: chkfile`, `path`	Read-only native PySCF checkpoint, not MPPT `scf.npz`.
`method: sap`, common `sap_basis: [file, path]`	SAP potential-fit file.
`method: sap`, per-label `sap_basis` map	Every referenced `[file, path]` fit.

Relative paths resolve from the source calculation YAML directory. In a sweep this is the base_input YAML directory, not the sweep YAML directory or the point directory. Low-level Python calculation API calls use their explicit working directory instead.

Preparation captures file bytes and hashes them into the managed input identity. Materialization writes those captured bytes under input/resources/<first-16-SHA256-characters>-<original-name> and rewrites the corresponding fields in input/effective.yaml to the managed copies’ absolute paths. It does not reread the original files to make those copies. The copies are inventoried as required input_resource artifacts with exact sizes and SHA-256 hashes. Capturing bytes does not validate native checkpoint records or SAP fit contents; those are checked when the fresh SCF stage loads the guess.

Library SAP sources remain library references and create no file-resource copy. At runtime, both explicit library names and inherited native string defaults resolve through PySCF’s ALIAS registry to bundled NWChem files, without basis.load pathname or current-directory fallback. A same-named local file cannot shadow either choice. Declare custom fits as [file, path] so their bytes are captured and inventoried; arbitrary paths in library sources are rejected. A native in-memory default fit dictionary retains its data after validation, but is not an inline-array YAML input format.

Fresh-stage validation follows the actual mean-field family, including native RHF-to-ROHF promotion for open shells. Orbital tags must represent the density under the density-fitting spin interpretation; inconsistent native UHF-to-ROHF checkpoint tags fail clearly rather than being stripped or repaired. The native checkpoint input must also differ from mf.chkfile, including symlink and hard-link aliases. These runtime checks do not normalize starting populations or change managed identity or resume policy.

Omitted initial_guess and initial_guess: null normalize to the same scientific input without a new default key. Within one output-layout version, identity and checkpoint eligibility are unchanged by omission/null alone. The layout migration itself changes output paths and prepared identities. An explicit method, including explicit minao, and its effective parameters are part of identity. Method case and the 1e alias are canonicalized; null optional fields are omitted. A changed effective selection or changed file contents changes identity and rejects incompatible managed resume. Use a new output directory to compare guesses.

When managed resume reuses a valid completed direct SCF checkpoint, it restores the saved arrays without evaluating the initial guess or rerunning SCF, even when the saved result is unconverged. Restoring those arrays does not read the starting-guess file, but managed preparation still resolves and hashes declared source resources, and artifact validation still requires their managed copies. The existing fallback rules above can restart SCF when direct-checkpoint reuse is unavailable; initial-guess selection does not change that policy.

Auxiliary Resources And Identity#

Atom-specific scf.auxbasis and casscf.auxbasis maps use [library, name] or [file, path] entries. Their YAML syntax preserves full atom labels such as O1.

Every explicit auxiliary file participates in source-YAML-relative resolution, immutable byte capture, input hashing, copying, effective-input rewriting and the required input_resource inventory. The same handling applies when the setting uses aux_basis, auxiliary_basis or df_auxbasis. Editing an auxiliary file changes the prepared identity; materializing an existing preparation uses its captured bytes rather than rereading an edited source file.

Library entries and legacy whole-setting strings are not captured as files. Use explicit file entries when transporting a fixed auxiliary basis. Content capture does not validate Gaussian basis data: native PySCF loads the files when the enabled DF object is configured. Direct checkpoint restoration can therefore need these files even though it does not rerun SCF. Existing string/null input and managed identity behavior are retained.

Manifest and archive validation#

Scientific archive publication requires complete schema-v2, layout-v2 provenance with recorded artifact and executable evidence. The archive copies exact declared bytes and does not render reports again.

Publish a complete run into a new exact directory with:

pyscf2mppt archive runs/pr3/run.json --output-dir archives/pr3

The collector holds the run lock, validates one immutable run.json snapshot, and copies only that snapshot plus its declared artifacts through a private staging directory followed by an atomic no-replace rename. It rejects incomplete or unsupported manifest provenance, moved or copied source manifests, absolute and escaping paths, symlinks, hard-link aliases, case aliases, unknown role/schema combinations, changed file sizes or hashes, and changed or missing executables. The destination must not exist and must be outside the source run. The copied manifest retains the original absolute run location as provenance, so an archive is for publication and inspection, not in-place resume. Archive publication currently requires Linux libc support for renameat2(RENAME_NOREPLACE) and fails closed when that primitive is absent.

--diagnostics also accepts failed or interrupted attempts and preserves that status. It copies their diagnostic inventory without claiming a complete scientific result. For downstream-only runs, an explicitly referenced external HDF5 is copied only when a scientific archive is requested.

Old-layout directories are identified by their missing/older layout_version. Use their original release for resume/archive, or start a new directory with --from-hdf5 to reuse an existing integral dump. Selection reuse still requires matching configuration/executable fingerprints; set mppt.cipsixx.enabled: false when explicitly consuming an already selected model space without reselection. Automatic in-place directory migration is not provided.

Sweep definition#

A sweep file points to one normal calculation input and maps dotted input paths to non-empty value lists:

base_input: h2.yaml
parameters:
  molecule.atoms:
    - [[H, 0.0, 0.0, 0.0], [H, 0.0, 0.0, 1.2]]
    - [[H, 0.0, 0.0, 0.0], [H, 0.0, 0.0, 1.4]]
  scf.conv_tol: [1.0e-8, 1.0e-10]

base_input is resolved relative to the sweep file. The example expands to four points in YAML parameter order. Each value replaces the complete value at its dotted mapping path, so molecule.atoms replaces the complete geometry. List-index paths such as molecule.atoms[1] are not supported.

To compare initial guesses with different method-specific fields, replace the whole mapping so fields from the base input do not leak into another method:

base_input: h2.yaml
parameters:
  scf.initial_guess:
    - {method: hcore}
    - {method: sap, sap_basis: [library, sapgraspsmall]}

Each point has its own initial-guess selection and input identity. This does not add cross-point checkpoint reuse or change completed-stage resume behavior.

Validate and inspect the expansion without creating artifacts:

pyscf2mppt sweep sweep.yaml --dry-run

Dry-run parses and normalizes every expanded input, resolves file resources, and prints deterministic point IDs and overrides. It returns before executable preflight and does not use --output-dir, --resume, --force-workflow, or executable overrides.

Execute the points sequentially:

pyscf2mppt sweep sweep.yaml --output-dir runs/h2-sweep

When --output-dir is omitted, the default is runs/<sweep-stem> below the current directory.

The sweep directory contains sweep.json and one managed directory per point:

runs/h2-sweep/
├── .pyscf2mppt.lock
├── sweep.json
├── sweep.log
├── summary.csv
└── points/
    ├── point-000001/
    ├── point-000002/
    ├── point-000003/
    └── point-000004/

sweep.json records the definition hash, ordered point IDs, overrides, input hashes, statuses, return codes, errors, and the current count of points whose status is interrupted. That count is recomputed and can decrease after resume; it is not cumulative history. Child-process failures are recorded and later points continue. The command returns 1 when any point is incomplete and prints a point’s run.log path when that log was created. Preflight validates every expanded input and every enabled executable before a new or resumed non-dry-run sweep starts.

Resume a sweep only when its expanded definition is unchanged:

pyscf2mppt sweep sweep.yaml --output-dir runs/h2-sweep --resume

Completed points are not simply skipped. They receive another managed attempt and are revalidated through the same single-point resume rules. With no_dump: true, valid direct checkpoint payloads enable resume. A dumping point also needs the matching managed HDF5 identity. Otherwise it restarts the direct calculation path and a replacement dump removes old workflow state. Downstream stages are reusable only when the resumed path preserves that HDF5. Failed or interrupted points are attempted again through the same path.

Output and exit behavior#

Single points retain full pipeline output in run.log and raw stage logs. Their terminal output is concise unless --full-output is selected. Sweeps keep per-point child output in each point’s run.log and print only point-level progress and summaries. A complete single point returns 0; a failed point returns the nonzero child code when available and otherwise 1. An all-complete sweep returns 0, while any incomplete sweep returns 1. While a managed child is running, SIGINT or SIGTERM records interrupted, terminates the child process group, escalates after five seconds when necessary, and returns 130. A signal during expansion, preflight, materialization, or between sweep points is not guaranteed to update a manifest. Top-level SIGINT still returns 130.

Current limitations#

Sweep execution is sequential and does not provide a parallel jobs option. Sweeps are independent single points, not a potential-energy-surface model: there is no state tracking, geometry continuation, or cross-point checkpoint reuse. summary.csv adds explicitly labelled SCF/CASSCF energies in Hartree and point log paths; HEFFSO state tables and other observables remain per-point. The complete Cartesian product and every normalized point are materialized in memory before execution or dry-run. There is currently no point-count limit, streaming expansion, or large-sweep confirmation prompt.