Skip to content

Hook: Include cuDNN 8.9.2.26 in CUDA CC 0.0a exclusion - #323

Open
Flamefire wants to merge 1 commit into
EESSI:mainfrom
Flamefire:patch-2
Open

Flamefire wants to merge 1 commit into
EESSI:mainfrom
Flamefire:patch-2

Conversation

@Flamefire

Copy link
Copy Markdown
Contributor

EESSI/2023.06 uses the older cuDDN version and is now failing the check, even already at the installation of CUDA libraries to host_injections.

EESSI/2023.06 uses the older cuDDN version and is now failing the check, even already at the installation of CUDA libraries to `host_injections`.

@casparvl casparvl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already replicated the issue here https://gitlab.com/eessi/support/-/work_items/283#note_4004603381

To test the fix, I checked out your feature branch, and made a small hard-coded modification to the install_cuda_and_libraries.sh from that branch to overwrite the EASYBUILD_HOOKS by the one from this feature branch:

    EESSI_SITE_CPU_FAMILY_PATH=$(echo "$EESSI_SITE_SOFTWARE_PATH" | sed 's|\(.*\)'"$EESSI_VERSION"/software/"$EESSI_OS_TYPE"/"$EESSI_SOFTWARE_SUBDIR"'|\1'"$EESSI_CPU_FAMILY"'|')
    export EASYBUILD_INSTALLPATH=$EESSI_SITE_CPU_FAMILY_PATH

    # FOR DEBUGGING ONLY!
    export EASYBUILD_HOOKS=/scratch-local/casparl.28074581/software-layer-scripts/eb_hooks.py

    # Install modules in hidden .modules dir to keep track of what was installed before
    # (this action is temporary, and we do not call Lmod again within the current shell context, but in EasyBuild
    # subshells, so loaded modules are not automatically unloaded)
    MODULEPATH=${EASYBUILD_INSTALLPATH}/.modules/all
    echo "set MODULEPATH=${MODULEPATH}"

I then ran this from the local checkout using:

NVIDIA_EASYSTACKS_DIRECTORY=scripts/gpu_support/nvidia/easystacks/2023.06/ scripts/gpu_support/nvidia/install_cuda_and_libraries.sh --accept-cuda-eula --accept-cudnn-eula

In the config output, I see:

cuda-compute-capabilities                (E) = 9.0a
cuda-sanity-check-error-on-failed-checks (E) = True

as expected. Then, I see:

== 2026-10-14 15:48:22,471 config.py:712 WARNING Build option 'cuda_compute_capabilities' was updated to: ['9.0']

In the cuDNN build log, showing this works as intended. Indeed, build now passes:

== Build succeeded for 1 out of 1 (total: 1 min 59 secs)
== Summary:
   * [SUCCESS] cuDNN/8.9.2.26-CUDA-12.1.1
== Temporary log file(s) /scratch-local/casparl.28081376/eb-oztndv2n/easybuild-w79_2z_o.log* have been removed.
== Temporary directory /scratch-local/casparl.28081376/eb-oztndv2n has been removed.
all installations at /cvmfs/software.eessi.io/host_injections/x86_64/software/... succeeded!

@casparvl

Copy link
Copy Markdown
Contributor

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2026.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

@eessi-bot-aws

eessi-bot-aws Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.10/pr_323/207846

date job status comment
Oct 14 13:54:01 UTC 2026 submitted job id 207846 awaits release by job manager
Oct 14 13:54:07 UTC 2026 released job awaits launch by Slurm scheduler
Oct 14 14:20:11 UTC 2026 running job 207846 is running
Oct 14 14:21:22 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-207846.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-17919876150.tar.zstsize: 0 MiB (34872 bytes)
entries: 1
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod
no reprod directories in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
no other files in tarball
Oct 14 14:21:22 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-207846.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 14 16:25:11 UTC 2026 uploaded transfer of eessi-2023.06-software-linux-x86_64-amd-zen2-17919876150.tar.zst to S3 bucket succeeded

@eessi-bot-aws

eessi-bot-aws Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.10/pr_323/207847

date job status comment
Oct 14 13:54:07 UTC 2026 submitted job id 207847 awaits release by job manager
Oct 14 13:55:24 UTC 2026 released job awaits launch by Slurm scheduler
Oct 14 14:22:32 UTC 2026 running job 207847 is running
Oct 14 14:23:40 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-207847.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2025.06-software-linux-x86_64-amd-zen2-17919877280.tar.zstsize: 0 MiB (34873 bytes)
entries: 1
modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2025.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod
no reprod directories in tarball
other under 2025.06/software/linux/x86_64/amd/zen2
no other files in tarball
Oct 14 14:23:40 UTC 2026 test result
😢 FAILURE (click triangle for details)
Reason
EESSI test suite was not run, test step itself failed to execute.
Details
✅ job output file slurm-207847.out
❌ found message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 14 16:25:20 UTC 2026 uploaded transfer of eessi-2025.06-software-linux-x86_64-amd-zen2-17919877280.tar.zst to S3 bucket succeeded

@eessi-bot-aws

eessi-bot-aws Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

New job on instance eessi-bot-mc-aws for repository eessi.io-2026.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.10/pr_323/207848

date job status comment
Oct 14 13:54:15 UTC 2026 submitted job id 207848 awaits release by job manager
Oct 14 13:55:21 UTC 2026 released job awaits launch by Slurm scheduler
Oct 14 14:20:14 UTC 2026 running job 207848 is running
Oct 14 14:25:54 UTC 2026 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-207848.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.* created!
Artefacts
eessi-2026.06-software-linux-x86_64-amd-zen2-17919877550.tar.zstsize: 0 MiB (34826 bytes)
entries: 1
modules under 2026.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2026.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
reprod directories under 2026.06/software/linux/x86_64/amd/zen2/reprod
no reprod directories in tarball
other under 2026.06/software/linux/x86_64/amd/zen2
no other files in tarball
Oct 14 14:25:54 UTC 2026 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ PASSED ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 expected failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-207848.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case
Oct 14 16:25:30 UTC 2026 uploaded transfer of eessi-2026.06-software-linux-x86_64-amd-zen2-17919877550.tar.zst to S3 bucket succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants