fix(security)!: harden artifact serialization and loading

[XianBW (XianBW)]

Summary

HIST metadata and default MLflow artifact loading could deserialize executable pickle content. The restricted unpickler also trusted entire pandas/NumPy namespaces, allowing I/O helpers such as pandas.read_pickle to bypass its restrictions. This PR replaces HIST object-NPY metadata with validated JSON and makes artifact loading restricted by default.

Breaking upgrade: public Python APIs and CLI entry points consistently use trusted=False / --trusted=True. The upgrade guide covers required changes and unaffected workflows; its source is docs/start/artifact_migration.rst. These changes are unreleased until included in a tagged package release.

Issue coverage

Area	Result
HIST fit / predict	Require JSON stock indices, validate index bounds, and disable NumPy object-pickle loading. The bundled mapping retains all 735 entries.
RestrictedUnpickler	Replace namespace-prefix trust with an explicit global allowlist. Reject pandas.read_pickle, numpy.load, arbitrary execution globals, and nested/two-stage payloads.
MLflowRecorder.load_object	Use restricted loading by default. Arbitrary model objects require explicit trusted=True; custom unpicklers are explicit and emit a warning. The recorder facade forwards the trust option.
HighFreqProvider paths	Resolve configured and derived paths beneath artifact_root, including split, temporary, daily, and per-instrument files. Reject parent and symlink escapes, and preserve caller configurations.

Expression execution and file-module loading are handled separately in #2340.

Compatibility and trust boundaries

• HIST stock_index must be JSON; object .npy metadata is no longer supported.
• Supported DataFrame, Series, ndarray, and Qlib data artifacts use restricted loading, including pickle protocols 4/5, datetime frequencies/timezones, nullable arrays, Period/Interval/Sparse data, and masked arrays.
• Low-level arbitrary model/reweighter loading requires explicit trusted=True. High-level online, delayed-training and meta-workflows use the same default-off trusted=False option, rather than silently granting trust internally. Their artifact sources and storage must remain trusted; this option does not make an untrusted shared MLflow store safe.
• Arrow-backed pandas arrays and ZoneInfo-backed timezones remain outside the default allowlist. Their MLflow artifacts require explicit trusted=True from a trusted store, or conversion to supported dtypes/timezones. StaticDataLoader already rejected these formats on main.
• High-frequency paths default to the current directory as artifact_root. Set an explicit root for data stored elsewhere. Cached high-frequency datasets still use Python pickle and require trusted contents; path containment is not a filesystem sandbox against concurrent replacement.

CI dataset download follow-up

• The inspected CI failures stopped at dataset downloads with HTTP 504, before reaching the tests. A nonexistent version-specific archive can also be misclassified as present when its availability probe returns 504.
• All three workflows now request the existing v2 archives directly: the same simple/full datasets used by successful jobs. Download setup gets five attempts, 60 seconds apart, with the existing 15-minute per-attempt timeout and failure propagation.
• No application code, OS/Python combinations, data sizes, training assertions or test selection changes. The PyPI workflow still invokes the installed CLI outside the checkout.
• Validation: all 16 offline CI-policy checks pass. Both updated archive commands downloaded and extracted the real datasets locally, and Qlib read finite, positive features from each. New cross-platform CI results remain the final readiness check.

Unified API and discoverable migration

• All public trust parameters, CLI flags, internal routing, generated task configurations, examples and current documentation use trusted. Prediction/label loading, default refusal and the absence of global consent are unchanged. The four example CLIs expose --trusted in generated help.
• Compatibility targets users upgrading from Qlib before this PR. Saved workflow components without a trust field default to restricted loading; no field-renaming migration is required. Each restored strategy, tool and delayed trainer must be configured separately after reviewing its artifact sources.
• A single canonical upgrade guide is directly linked from the repository README, installation page, documentation navigation, component docs, and example READMEs. Both changelog channels include an Unreleased breaking-change notice. Recorder documentation links to the same guide instead of duplicating the instructions.
• MLflow/restricted-pickle refusals, DDG-DA cache errors, HIST mapping errors and high-frequency path errors link to the guide with context-specific instructions, not blanket advice to enable trust.
• The guide covers old HIST model paths, cached OnlineManager state, saved DDG-DA task consent, high-frequency roots, custom extensions, unsupported data representations and the difference between source upgrades and package releases.
• Worked migration examples show how to re-authorize selected restored manager components without overwriting the original, convert Arrow-backed columns without losing large integers or missing values, preserve ZoneInfo index semantics across a daylight-saving transition, and save converted data in a new recorder with restricted loading still enabled.

Trust-workflow validation: 263 scoped security/API/full-workflow cases, including four CLI-help cases, passed on both NumPy 1.24.4 / pandas 2.0.3 and NumPy 2.2.6 / pandas 2.3.3. Production lint, Python 3.8 syntax and the Sphinx warnings-as-errors build passed. The two original full-sized online prediction/label workflows also passed on the preceding revision. Earlier validation details are recorded below.

Explicit workflow trust follow-up

Commit 1039e70e moves recorder-artifact consent to caller-facing workflow entry points without replacing serialization formats or splitting this PR:

• RollingStrategy passes its selected policy through OnlineToolR, the updater and RMDLoader. Direct updaters/loaders expose the same default-off option. Prediction and label artifacts stay restricted even when model/dataset loading is trusted.
• DelayTrainerR / DelayTrainerRM pass explicit consent to training completion and the delayed worker, with per-call overrides. Ordinary data-only task configurations remain usable without opting in.
• DDGDA, MetaDatasetDS and InternalData.setup expose recorder consent. The subsequent full-workflow follow-up below also covers DDG-DA's necessary local handler/internal-data caches; lower-level MetaDatasetDS / InternalData.setup consent remains scoped to recorder tasks.
• Legacy R.load_object(name) calls preserve custom recorders' original default signature. Restored workflow objects without a saved trust field default to restricted loading. Custom implementations must enforce the policy themselves.
• Migration guidance covers HIST JSON mapping, supported/unsupported data representations, custom recorder/loader signatures, persisted online managers, and writer/store permissions. Online example CLIs expose the option rather than enabling it implicitly.

Follow-up validation on Linux / Python 3.10:

• 202 security/trust tests passed with both NumPy 2.2.6 / pandas 2.2.3 and NumPy 1.24.4 / pandas 2.0.3.
• Both original full-sized LightGBM online prediction/label update tests passed, retaining their original datasets and assertions and adding default-refusal checks.
• Compared 78abf829 with the follow-up using identical sample data and seeds: actual Alpha158/LightGBM training with a serialized TimeReweighter, IC generation, a 20-day portfolio backtest, model/dataset reload, and online prediction/label updates. All 6,000 initial predictions, 8,700 updated predictions and labels, and daily IC values matched exactly. The backtest table's maximum absolute numeric difference was 1.49e-8. The shortened training run checks execution/output compatibility, not model performance.
• Default delayed-task/model/dataset loading was refused before execution; explicitly authorized loading completed the same workflow.
• Sphinx HTML build with warnings as errors, changed-code Black/Flake8/Pylint, and Python 3.8 syntax checks passed.

The follow-up retains the existing trust assumptions of unrelated local pickle APIs and executable task/YAML configurations. Broader serialization-format changes remain deferred pending additional actual Qlib workflow coverage.

Full-workflow coverage follow-up

Added repeatable offline integration tests using actual Qlib binary-data providers, Alpha158/Alpha360 handlers, model training, MLflow serialization, signal analysis, strategy and backtest execution. No workflow stage is mocked, and fixtures use isolated temporary stores without market/model downloads. Existing slow CI discovers these tests on its Python/OS matrix.

Workflow	Executed stages and assertions
DDG-DA, linear similarity	LightGBM feature selection, seven similarity-training windows, daily rank IC and cache restoration, all 30 meta-training epochs with changing loss, inferred time weights, two rolling training windows, 320 prediction/label rows, 40-day IC and portfolio reports.
DDG-DA, LightGBM similarity	The same full chain with 192 instruments and 7,680 prediction/label rows; all seven similarity models build multiple trees with early stopping disabled.
Saved DDG-DA tasks	Both variants retain lightweight handler references (task file below 10 KB), reload saved reweighted tasks, refuse default delayed completion, and complete explicitly authorized replay with predictions exactly matching the original rolling models.
HIST on CPU	Real Alpha360/DatasetH and task_train, one epoch and 12 optimizer updates, 48 prediction/label rows, signal analysis and a six-day portfolio backtest. Default model/dataset reads fail; trusted reload reproduces predictions exactly in both the same and a fresh Python process.

Full execution exposed and fixed tightly coupled blockers that routing-only tests did not reveal:

• Preserve the daily grouping index when computing DDG-DA's rank IC.
• Scope DDG-DA's workflow consent (now uniformly named trusted) to its required local handler/internal-data caches as well as recorder artifacts. Default loading stays restricted, actual booleans are required, and explicit unrestricted cache loads warn. There is no unsafe retry or global allowlist expansion.
• Keep cached handlers as reloadable task configurations instead of embedding market data; preserve selected cache consent in saved tasks and document the requirement to trust those executable configurations.
• Correct duplicated relative working_dir paths and avoid mutating shared proxy-model/processor templates across runs.
• Do not register LightGBM's early-stopping callback when the model's effective setting is None.
• Use the tensor's device in HIST so its forward pass works on CPU as well as GPU.

Validation on Linux / Python 3.10:

• 214 tests passed (security suite plus the new complete workflows and cache regressions) with NumPy 1.24.4 / pandas 2.0.3 and NumPy 2.2.6 / pandas 2.3.3.
• Both original, full-sized LightGBM online prediction/label tests passed again using downloaded Qlib sample data; their original assertions and dataset sizes were retained.
• Changed-code Black/Flake8/Pylint, Python 3.8 syntax checks, and the Sphinx warnings-as-errors build passed.

The new market fixtures are deterministic synthetic daily prices passed through real Qlib components, not paper-scale market-performance experiments. These tests establish full small-scale execution, meaningful training and artifact round-trip behavior; they do not establish model profitability or justify a repository-wide serialization-format migration. The migration guide and both model example READMEs document the scope and reproduction commands.

Earlier validation

Verified locally on Linux / Python 3.11, NumPy 2.2.6, pandas 2.2.3, and PyTorch 2.8:

• python -m pytest tests/security -q: 140 passed.
• Existing MLflow-client and nested-data-loader tests passed.
• Combined with fix(security): constrain config-driven code execution #2340: 224 passed, covering both security suites, original operator/custom-operator tests, utilities, workflow/MLflow, and CI configuration tests.
• Verified the migrated JSON exactly matches all 735 entries of the original mapping using restricted deserialization.
• Black for changed Python files, Flake8 for affected Qlib code, Pylint for the newly modified production code, Python 3.8 syntax checks, YAML parsing, and CRLF-aware git diff --check passed.

Executed workflow comparisons

Compared main be725493 with both PRs combined, using the same data and seeds:

• Actual LightGBM task training, signal/IC generation, 23-day portfolio backtest, fresh-process model/dataset reload, and online prediction/label updates passed. All 460 initial predictions and all 840 updated predictions/labels matched exactly; the backtest table's maximum absolute difference was 2.33e-10.
• HIST completed one GPU training epoch, inference, and artifact reload using real Alpha360 inputs and a deterministic 735 x 8 numeric concept fixture. All 120 predictions matched main exactly.
• HighFreqProvider generated, reloaded, and split three days of 240-minute data for two instruments using an isolated binary-data fixture and two process workers. All six feature/label splits matched main exactly. Repeating get_pre_datasets() now succeeds without losing configuration paths.
• Runtime artifact comparisons identified and fixed missing PeriodDtype, Interval, SparseArray, and MaskedArray reconstruction entries. New tests include real MLflow file-store round trips with a new reader client and rejection of executable objects inside masked arrays.

Reproduction environment: Python 3.11.11, NumPy 2.2.6, pandas 2.2.3, SciPy 1.15.3, LightGBM 4.7.0, MLflow 2.21.3, PyTorch 2.8.0. Shortened model runs verify execution and output compatibility.

Both PR branches incorporate current main (be72549), preserving the current CI setup and action pins. Fresh cross-platform GitHub Actions runs have been triggered; the current runs, not earlier green commits, determine CI readiness. This PR is not yet merged or released.

[XianBW (XianBW)]

Local verification update:

• 23 security regression tests passed
• handler serialization plus full train/backtest pipeline: 4 passed
• repository-wide Black, Flake8, and Pylint passed
• HIST JSON mapping verified identical to all 735 entries in the previous object NPY
• added explicit trusted=True to the pipeline test that intentionally restores an arbitrary model pickle

The branch also contains minimal CI compatibility fixes for current Node/commitlint, Plotly, MLflow, and Windows/Python 3.8 OSQP behavior.

[XianBW (XianBW)]

Final verification: all 76 GitHub checks pass on commit 9d8d63df/latest branch state, including title, CLA, pip compatibility, source builds, notebook execution, full pytest, and slow pytest matrices.