Skip to content

Config helper - #236

Open
rountree wants to merge 8 commits into
llnl:develfrom
rountree:config_helper
Open

rountree wants to merge 8 commits into
llnl:develfrom
rountree:config_helper

Conversation

@rountree

Copy link
Copy Markdown
Collaborator

Separate binary that creates/validates Spindle config files. Work in progress.

Fixes case where all candidate commpaths were added to localprefix
prior to vetting.
The "long" tests were not compiled with `-rpath`, which led to
errors about missing libmodules.  This fix allows cray compilers
to be used without putting the fortran library location in
LD_LIBRARY_PATH.
● Add spindle-config-helper utility (milestone 1)

  Introduces a new tool to help users generate and validate Spindle
  configuration files. This milestone implements the foundational
  parse-and-output functionality.

  Features:
  - Parses configuration from command line, environment variables, and
    config files using existing config_mgr.cc infrastructure
  - Outputs merged configuration in spindle.conf format
  - Supports all 62 Spindle configuration options
  - Command-line help: -h/--help, -V/--version, --usage
  - Application argument is optional (auto-supplies dummy if needed)
  - Man page: spindle-config-helper(1)

  Architecture:
  - New binary: src/config_helper/spindle-config-helper
  - Private static library: src/fe/startup/libconfigparse.a
    Contains config_mgr.cc, config_parser.cc, parseargs.cc, and
    supporting utilities. Library is not installed (noinst_LIBRARIES)
    to avoid distribution complexity in sidecar scenarios.
  - Integrated into autotools build system

  Future milestones will add environment probing (detecting local mount
  points, resource managers, security subsystems) and configuration
  validation.

  Co-Authored-By: Claude <noreply@anthropic.com>
  Add filesystem analysis and path validation to spindle-config-helper (milestone 2)

  Enhances spindle-config-helper to analyze and validate configured paths,
  helping users identify problematic configurations before runtime.

  Features:
  - Analyzes each path in commpaths, cachepaths, and localprefix
  - Detects and warns about remote filesystems (NFS, Lustre, GPFS, etc.)
  - Validates environment variable expansion
    - Detects undefined variables ($BADVAR)
    - Detects empty variables creating double-slashes (/var/$EMPTY/dir)
    - Shows both original and expanded paths
  - Checks parent directory permissions for non-existent paths
  - Resolves relative paths and symlinks to show real locations
  - Hardcodes kernel filesystem paths (/proc, /sys, /dev) as local
  - Treats overlay filesystems as local (container backing storage)

  Output format provides per-path analysis with status (OK/WARNING),
  original path, resolved path, filesystem type, and specific warnings:

    commpaths analysis:
      OK: $TMPDIR -> /var/tmp/rountree (tmpfs, 250GB)
      WARNING: /var/$EMPTY/dir -> /var//dir
        Path contains empty segment (variable may be empty)
        Parent directory (/var) is not writable
      WARNING: /nfs/cache -> /nfs/cache (nfs4)
        Path is on REMOTE filesystem (nfs4)

  This detailed feedback helps users and administrators diagnose
  configuration issues by showing exactly what each path expands to,
  where it resolves, and why it might not work.

  Technical details:
  - Uses wordexp(3) with WRDE_UNDEF for variable expansion
  - Uses realpath(3) for symlink/relative path resolution
  - Parses /proc/mounts with 1GB minimum space threshold
  - Checks write permission on nearest existing parent directory
  - Filters duplicate mounts and system/virtual filesystems

  Co-Authored-By: Claude <noreply@anthropic.com>
  Adds a lightweight integration test suite to validate spindle-config-helper
  functionality and catch regressions.

  Test coverage (10 tests, all passing):
  - Basic execution (smoke test)
  - Command-line flags (--help, --version)
  - Local filesystem detection (/dev/shm, /tmp)
  - Variable handling (undefined, empty)
  - Multiple colon-separated paths
  - Kernel filesystem recognition (/proc, /sys, /dev)
  - Parent directory permission checking
  - Relative path resolution

  Structure:
    testsuite/config_helper/
    ├── run_tests.sh              Test runner script
    ├── README.md                 Documentation
    ├── sample_mounts/            Example /proc/mounts files
    │   ├── simple.mounts         Basic NFS + local
    │   ├── hpc.mounts            Lustre + GPFS + local
    │   └── container.mounts      Overlay + tmpfs
    └── expected/                 (Reserved for golden files)

  Design philosophy:
  - Simple integration tests that run the actual binary
  - Use real filesystem operations where possible
  - No heavy mocking - trust implementation, catch regressions
  - Fast execution (~1 second)
  - Easy to extend with new test functions

  Sample mount files are provided for future enhancements if we add
  deterministic testing against various filesystem configurations.

  Usage:
    cd testsuite/config_helper
    ./run_tests.sh /path/to/spindle-config-helper

  Exit codes: 0 (all passed), 1 (one or more failed)

  Co-Authored-By: Claude <noreply@anthropic.com>
  Commit Message 1: Python Prefix Validation

  Add pythonprefix validation to spindle-config-helper

  Validates that pythonprefix paths point to valid Python installations.

  Background:
  Spindle uses pythonprefix for prefix matching on file paths. The default
  is "/usr/lib/python:/usr/lib64/python" which matches paths like:
    /usr/lib/python3.11/site-packages/numpy/__init__.py
    /usr/lib/python3.11/lib-dynload/_ctypes.so

  The prefix should be the Python installation root (e.g., /usr/tce or
  /opt/cray/pe/python/3.12.12), NOT the lib subdirectory.

  Validation rules:
  - Path must exist (no point in intercepting files that don't exist)
  - No undefined variables (catches $PYTHON_HOME when unset)
  - Must contain bin/python or bin/python3 (validates it's a Python install)

  Example output:
    pythonprefix analysis:
      OK: /usr/tce -> /usr/tce
      WARNING: /tmp -> /tmp
        No bin/python or bin/python3 found (not a Python installation?)
      WARNING: $PYTHON_HOME -> (empty)
        Variable is not set

  This helps users catch common mistakes like:
  - Setting prefix to a non-existent path
  - Using /usr/lib/python/3.11 instead of /usr/lib/python
  - Pointing to a non-Python directory
  - Relying on undefined environment variables

  Co-Authored-By: Claude <noreply@anthropic.com>

  Commit Message 2: Test Suite Updates

  Add pythonprefix tests to spindle-config-helper test suite

  Adds 3 new tests for pythonprefix validation (13 tests total, all passing):

  11. test_11_pythonprefix_valid - Valid Python installation
      Uses /usr/tce if available, skips gracefully otherwise

  12. test_12_pythonprefix_invalid - Path without Python binaries
      Tests /tmp which exists but has no bin/python

  13. test_13_pythonprefix_undefined_var - Undefined variable detection
      Tests $PYTHON_HOME when unset

  Also fixes test_09_permissions which broke when pythonprefix validation
  changed path-doesn't-exist behavior from OK to WARNING.

  All tests are portable:
  - Test 11 skips on systems without /usr/tce
  - Tests 12 and 13 work on any system
  - No searching, no external dependencies
  - Fast execution

  Co-Authored-By: Claude <noreply@anthropic.com>
  Implements comprehensive security validation including:
  - Detection of compiled security mode and current runtime setting
  - Mode-specific validation:
    * none: warns about insecure connections
    * keyfile: validates keydir is on shared filesystem
    * launchmon: warns about deprecated mode
    * munge: verifies daemon availability
  - Sandbox-aware munge testing (detects SANDBOX_ENGINE env var)
  - Always recommends munge when available

  New helper function get_sec_keydir_path() in keyfile.c exposes
  SEC_KEYDIR compile-time constant for keyfile validation.

  Test suite expanded with tests 14-17 covering security mode
  detection, warnings, and munge availability checks.

  All 17 tests passing.

  Co-Authored-By: Claude <noreply@anthropic.com>
@rountree

Copy link
Copy Markdown
Collaborator Author

spindle-config-helper is built using the same config_mgr.cc as Spindle and will pull in and parse runtime parameters from the same sources, including the command line.

The design philosophy is to validate those runtime parameters, issue warnings and validation where needed, and then dump all parameters and their current values to stdout. The user can redirect the output to their spindle.conf file.

Here's a snapshot of current functionality:

# Security configuration:
#   Current mode: munge
#     Munge authentication enabled
#
#   Munge availability:
#     Munge is available and working (recommended security option)
#
# Spindle filesystem recommendations:
#   Local: /dev/shm (tmpfs, 250GB), /tmp (tmpfs, 428GB)
#
#   commpaths analysis:
#     WARNING: /bad_commpath -> /bad_commpath
#       Path does not exist (will be created by Spindle)
#       Parent directory (/) is not writable
#     OK: /tmp/commpath -> /tmp/commpath (tmpfs, 428GB)
#
#   cachepaths analysis:
#     WARNING: /bad_cachepath -> /bad_cachepath
#       Path does not exist (will be created by Spindle)
#       Parent directory (/) is not writable
#     OK: /tmp/commpath/cachepath -> /tmp/commpath/cachepath (tmpfs, 428GB)
#
#   pythonprefix analysis:
#     WARNING: /usr/lib/python -> /usr/lib/python
#       Path does not exist
#     WARNING: /usr/lib64/python -> /usr/lib64/python
#       Path does not exist
#
#   localprefix analysis:
#     WARNING: /var/ -> /var
#       Cannot determine filesystem type
#     OK: /tmp/ -> /tmp (tmpfs, 428GB)
#     OK: /proc/ -> /proc
#     OK: /dev/ -> /dev
#     OK: /sys/ -> /sys
#
port = 21940
numports = 250
commpaths = /bad_commpath:/tmp/commpath
cachepaths = /bad_cachepath:/tmp/commpath/cachepath
pythonprefix = /usr/lib/python:/usr/lib64/python
localprefix = /var/:/tmp/:/proc/:/dev/:/sys/
strip = yes
relocaout = yes
reloclibs = yes
relocpython = yes
relocexec = yes
followfork = yes
stopreloc = no
level = default
pushpull = push
network = cobo
security = munge
launcher = serial
numa = no
audittype = audit
shmcachesize = 0
debug = no
noclean = no
disablelogging = yes
nohide = no
persist = no
msgcachebuffer = 0
msgcachetimeout = 0
cleanupproc = no
execexcludes = gcc:g++:cc:CC:clang:clang++:hipcc:amdclang:amdclang++:craycc:crayCC:ld:ld.lld:mpicc:mpic++:mpicxx:make:gmake:cmake
enablersh = no
patchldso = yes
crashdedup = no
calctls = yes
crashaltstack = no

This branch is waiting to be deployed

1 waiting deployment
Spindle CI — 3c35175a Waiting Oct 23, 2026 by rountree via Testsuite (Serial, Ubuntu) #285
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant