EESSI: Difference between revisions

From LWP-Wiki
Jump to navigation Jump to search
No edit summary
No edit summary
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== The EESSI environment ==
[[Category:software-faq]]
You can set up the environment by sourcing the provided init script:
The European Environment for Scientific Software Installations (EESSI, pronounced as "easy") is a collaboration between different European partners in HPC community. The goal of this project is to build a common stack of scientific software installations for HPC systems and beyond, including laptops, personal workstations and cloud infrastructure.<br>
   source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
 
If all goes well, you should see output like this:
For more information visit https://www.eessi.io/docs/.<br>
   Found EESSI pilot repo @ /cvmfs/pilot.eessi-hpc.org/versions/2021.12!
 
  Using x86_64/intel/haswell as software subdirectory.
== Setting up your environment ==
   Using /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/haswell/modules/all as the directory to be added to MODULEPATH.
In Unix-like systems, environment variables are used to configure the environment in which applications and scripts run. To set up EESSI, you need to configure a specific set of environment variables so that your operating system is aware that EESSI exists and is to be used. We have prepared a few automated approaches that do this for you: you can either load an EESSI environment module or source an initialisation script for bash.<br>
   Found Lmod configuration file at /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/haswell/.lmod/lmodrc.lua
 
   Initializing Lmod...
With any of the approaches below, the first time you use them they may seem to take a while as any necessary data is downloaded in the background from a Stratum 1 server (which is part of the CernVM-FS infrastructure used to distribute files for EESSI).<br>
   Prepending /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/haswell/modules/all to $MODULEPATH...
 
  Environment set up to use EESSI pilot software stack, have fun!
=== Loading an EESSI environment module ===
  [EESSI pilot 2021.12] $
There are a few different scenarios where you may want to set up the EESSI environment by loading an EESSI environment module. The simplest scenario is one where you do not already have a environment module tool on your system, in this case we configure the Lmod module tool shipped with EESSI and automatically load the EESSI environment module:
'''!NOTE''': You might see the following error which can be ignored:''<br>
   source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.''
 
This command configures Lmod for your system and automatically loads the EESSI module so that EESSI is immediately available to use. If you would like to see what environment variables the module sets, you can use module show EESSI.<br>
👏 Your environment is now set up, you are ready to start running software provided by EESSI!<br>
 
==== What if I don't use a bash shell? ====
The example above is shown for a bash shell but the environment module approach supports all the shells that Lmod itself supports (bash, csh, fish, ksh, zsh):
   source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
   source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/csh
  source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/fish
   source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/ksh
  source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/zsh
 
==== What if I already have Lmod installed or another module tool is available on the system? ====
You can check if the module command is already defined for your system and what version it has with:
   command -v module && module --version
 
1. If you are already using Lmod (modules based on Lua) with version >= 8.6:<br>
In this case, we recommend resetting $MODULEPATH, because EESSI is not designed to mix modules coming from EESSI and from your system.
   module unuse $MODULEPATH
  module use /cvmfs/software.eessi.io/init/modules
  module load EESSI/2023.06
 
👏 Your environment is now set up, you are ready to start running software provided by EESSI!<br>
 
2. If you are using an Lmod with a version older than 8.6 or any other module tool utilizing MODULEPATH (e.g., [https://modules.sourceforge.net/ Tcl-based Environment Modules]):


Now you're all set up! Go ahead and explore the software stack using '''module avail''', and go wild with testing the available software installations!
It is recommended to unset $MODULEPATH to prevent Lmod from attempting to build a cache for your module tree (as this can be very slow if you have a lot of modules). Again, unsetting the $MODULEPATH should be considered as a good idea in general so you do not mix local and EESSI modules. You then will need to initialise a compatible version of Lmod, for example the one shipped with EESSI:
  unset MODULEPATH
  source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash


=== Testing the EESSI pilot software stack ===
👏 Your environment is now set up, you are ready to start running software provided by EESSI!<br>
Please test the EESSI pilot software stack as you see fit: running simple commands, performing small calculations or running small benchmarks, etc.


Test scripts that have been verified to work correctly using the pilot software stack are available at https://github.com/EESSI/software-layer/tree/main/tests.
===== Why do we recommend to unset MODULEPATH? =====
Unsetting the $MODULEPATH environment variable, which tells Lmod in which directories environment module files are available, may be necessary. The underlying reason to suggest this is that EESSI and your system are most likely based on two different operating system distributions - EESSI uses it's compatibility layer, your system almost certainly uses some other Linux distribution. If you can find a way to ensure that the software stacks from your site and EESSI do not mix (in particular when someone is building new software!), then this should be good enough.<br>


=== Giving feedback or reporting problems ===
=== Sourcing the EESSI bash initialisation script ===
Any feedback is welcome, and questions or problems reports are welcome as well, through one of the EESSI communication channels:
<sup>This is supported exclusively for bash shell users. If you're using a different shell, please use [https://www.eessi.io/docs/using_eessi/setting_up_environment/#loading-an-eessi-environment-module the alternative approach]</sup><br>
   
<sup>You can to see what your current shell is with the command ''echo $SHELL''</sup><br>
    * (preferred!) EESSI software-layer GitHub repository: https://github.com/EESSI/software-layer/issues
    * EESSI mailing list (eessi@list.rug.nl)
    * EESSI Slack: https://eessi-hpc.slack.com (get an invite via https://www.eessi-hpc.org/join)
    * monthly EESSI meetings (first Thursday of the month at 2pm CEST)


=== Available software ===
You can initialise EESSI (in a non-reversible way) by running the command:
(last update: Mar 21st 2022)
  source /cvmfs/software.eessi.io/versions/2023.06/init/bash


EESSI currently supports the following HPC applications as well as all their dependencies:
You should see the following output:
  Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
  archdetect says x86_64/amd/zen2  # (1)
  archdetect could not detect any accelerators
  Using x86_64/amd/zen2 as software subdirectory.
  Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/lmodrc.lua
  Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
  Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2 as the site extension directory for installations.
  Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all as the directory to be added to MODULEPATH.
  Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all as the site extension directory to be added to MODULEPATH.
  Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
  Initializing Lmod...
  Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
  Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
  Environment set up to use EESSI (2023.06), have fun!
  {EESSI 2023.06} [user@system ~]$  # (2)!


    * GROMACS (2020.1 and 2020.4)
What is reported at (1) depends on the CPU architecture of the machine you are running the source command.<br>
    * OpenFOAM (v2006 and 8)
At (2) is the prompt indicating that you have access to the EESSI software stack.<br>
    * R (4.0.0) + R-bundle-Bioconductor (3.11) + RStudio Server (1.3.1093)
    * TensorFlow (2.3.1) and Horovod (0.21.3)
    * OSU-Micro-Benchmarks (5.6.3)
    * ReFrame (3.9.1)
    * Spark (3.1.1)
    * IPython (7.15.0)
    * QuantumESPRESSO (6.6) (currently not available on ppc64le)
    * WRF (3.9.1.1)


For more information go to https://eessi.github.io/docs/pilot/.
👏 Your environment is now set up, you are ready to start running software provided by EESSI!<br>

Latest revision as of 16:50, 16 December 2024

The European Environment for Scientific Software Installations (EESSI, pronounced as "easy") is a collaboration between different European partners in HPC community. The goal of this project is to build a common stack of scientific software installations for HPC systems and beyond, including laptops, personal workstations and cloud infrastructure.

For more information visit https://www.eessi.io/docs/.

Setting up your environment

In Unix-like systems, environment variables are used to configure the environment in which applications and scripts run. To set up EESSI, you need to configure a specific set of environment variables so that your operating system is aware that EESSI exists and is to be used. We have prepared a few automated approaches that do this for you: you can either load an EESSI environment module or source an initialisation script for bash.

With any of the approaches below, the first time you use them they may seem to take a while as any necessary data is downloaded in the background from a Stratum 1 server (which is part of the CernVM-FS infrastructure used to distribute files for EESSI).

Loading an EESSI environment module

There are a few different scenarios where you may want to set up the EESSI environment by loading an EESSI environment module. The simplest scenario is one where you do not already have a environment module tool on your system, in this case we configure the Lmod module tool shipped with EESSI and automatically load the EESSI environment module:

 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash

This command configures Lmod for your system and automatically loads the EESSI module so that EESSI is immediately available to use. If you would like to see what environment variables the module sets, you can use module show EESSI.
👏 Your environment is now set up, you are ready to start running software provided by EESSI!

What if I don't use a bash shell?

The example above is shown for a bash shell but the environment module approach supports all the shells that Lmod itself supports (bash, csh, fish, ksh, zsh):

 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash
 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/csh
 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/fish
 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/ksh
 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/zsh

What if I already have Lmod installed or another module tool is available on the system?

You can check if the module command is already defined for your system and what version it has with:

 command -v module && module --version

1. If you are already using Lmod (modules based on Lua) with version >= 8.6:
In this case, we recommend resetting $MODULEPATH, because EESSI is not designed to mix modules coming from EESSI and from your system.

 module unuse $MODULEPATH
 module use /cvmfs/software.eessi.io/init/modules
 module load EESSI/2023.06

👏 Your environment is now set up, you are ready to start running software provided by EESSI!

2. If you are using an Lmod with a version older than 8.6 or any other module tool utilizing MODULEPATH (e.g., Tcl-based Environment Modules):

It is recommended to unset $MODULEPATH to prevent Lmod from attempting to build a cache for your module tree (as this can be very slow if you have a lot of modules). Again, unsetting the $MODULEPATH should be considered as a good idea in general so you do not mix local and EESSI modules. You then will need to initialise a compatible version of Lmod, for example the one shipped with EESSI:

 unset MODULEPATH
 source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash

👏 Your environment is now set up, you are ready to start running software provided by EESSI!

Why do we recommend to unset MODULEPATH?

Unsetting the $MODULEPATH environment variable, which tells Lmod in which directories environment module files are available, may be necessary. The underlying reason to suggest this is that EESSI and your system are most likely based on two different operating system distributions - EESSI uses it's compatibility layer, your system almost certainly uses some other Linux distribution. If you can find a way to ensure that the software stacks from your site and EESSI do not mix (in particular when someone is building new software!), then this should be good enough.

Sourcing the EESSI bash initialisation script

This is supported exclusively for bash shell users. If you're using a different shell, please use the alternative approach
You can to see what your current shell is with the command echo $SHELL

You can initialise EESSI (in a non-reversible way) by running the command:

 source /cvmfs/software.eessi.io/versions/2023.06/init/bash

You should see the following output:

 Found EESSI repo @ /cvmfs/software.eessi.io/versions/2023.06!
 archdetect says x86_64/amd/zen2  # (1)
 archdetect could not detect any accelerators
 Using x86_64/amd/zen2 as software subdirectory.
 Found Lmod configuration file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/lmodrc.lua
 Found Lmod SitePackage.lua file at /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/.lmod/SitePackage.lua
 Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2 as the site extension directory for installations.
 Using /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all as the directory to be added to MODULEPATH.
 Using /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all as the site extension directory to be added to MODULEPATH.
 Found libcurl CAs file at RHEL location, setting CURL_CA_BUNDLE
 Initializing Lmod...
 Prepending /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
 Prepending site path /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2/modules/all to $MODULEPATH...
 Environment set up to use EESSI (2023.06), have fun!
 {EESSI 2023.06} [user@system ~]$  # (2)!

What is reported at (1) depends on the CPU architecture of the machine you are running the source command.
At (2) is the prompt indicating that you have access to the EESSI software stack.

👏 Your environment is now set up, you are ready to start running software provided by EESSI!