Skip to content

EasyBuild

Description

EasyBuild can install software packages including the related modules. The location will be controlled using our modules, e.g. the Workspace module, see Installing Custom Software. On top of the usual EasyBuild framework we added some extensions which allows you to build for specific architectures or a generic software stack in your user/group space. Therefore, use the eb command to search and try and the eb-install-all or eb-install-generic command to install the package.

The following steps need are necessary:

  • load modules
  • find the package specification
  • decide the desired software stack
  • run EasyBuild installation using eb-install-all or eb-install-generic

Modules

Depending if you want to install the package in user or a group space you need to load the related module and the EasyBuild module, e.g.:

module load Workspace  ### if you want to install into your HOME use Workspace/home
module load EasyBuild

Therewith, our EasyBuild tools and EasyBuild itself are available.

Note

Specify the WorkspaceID if necessary when loading the Workspace module. See module instructions

Package Specification

EasyBuild has a large repository of available packages in different versions. You can use these specifications as is or copy/download and modify the EasyConfigs (see below).

Available packages can be searched using the following command, here for the gatk package

eb --search gatk
[...]
eb --search gatk
== found valid index for /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs, so using it...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-1.0.5083.eb
...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.0.1.2-Java-1.8.eb
...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.0.12.0-foss-2018b-Python-3.6.6.eb
...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.0.4.0-intel-2018a-Python-3.6.4.eb
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.0.5.1-foss-2018a-Python-3.6.4.eb
...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.0.8.1-foss-2018b-Python-2.7.15.eb
...
 * /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.1.8.1-GCCcore-9.3.0-Java-1.8.eb

As shown above there are different versions of GATK and for different toolchains available (foss, intel, GCCcore). Select one

You can list all dependencies using:

eb -Dr /storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs/g/GATK/GATK-4.1.8.1-GCCcore-9.3.0-Java-1.8.eb
...
Dry run: printing build status of easyconfigs and dependencies
CFGS=/storage/software/generic.el7/software/EasyBuild/4.3.3/easybuild/easyconfigs
 * [ ] $CFGS/j/Java/Java-1.8.0_281.eb (module: Java/1.8.0_281)
... 
 * [x] $CFGS/p/Python/Python-2.7.18-GCCcore-9.3.0.eb (module: Python/2.7.18-GCCcore-9.3.0)
 * [ ] $CFGS/g/GATK/GATK-4.1.8.1-GCCcore-9.3.0-Java-1.8.eb (module: GATK/4.1.8.1-GCCcore-9.3.0-Java-1.8)

Dependencies marked with x are already installed, the other dependencies will be installed if using the robot option.

Additional options, e.g. for selecting a specific software version can be found using eb --help.

Using EasyConfig files

You can use the directy selected easyconfig or if necessary copy and adapt it. easyconfig files are text files specifying the software version, toolchain version, dependencies, compile arguments and more. If you need more information see EasyBuild documentation, and if necessary ask our support team for assistance.

Selecting a software stack

Depending on the package and its target usage one or more software stacks should be selected. Therefore, the installation command starts with one for the following command:

  • all architectural software stacks: eb-install-all
  • a specific architectural software stack (e.g. only targeting Broadwell nodes): eb-install-all --archs='broadwell' OR
  • generic software stack: eb-install-generic, CPU architecture independent, like git

Installation

After selecting the package installation recipe and the target software stack, the installation process can be submitted. With the following commands, SLURM job files will be created, and submitted to the desired compute nodes. There the packages are build and module files created. The general syntax is:

eb_install_{all,generic} [options] [easybuild options] <easyconfig>.eb
Additional SLURM arguments can be selected using the --slurm-args option, e.g. --slurm-args='--account=xyz --time=00:10:00 --cpus-per-task'. If specific architectures should be selected use e.g. --arch='broadwell ivy'. After this options, EasyBuild arguments can be provided without prefix, e.g. --robot.

Few examples:

  • for FFTW in all architectural software stacks:
    eb-install-all --robot --software-name=FFTW --toolchain-name=GCC
    
  • for git in the generic software stack:
    eb-install-generic --robot --software-name=git --toolchain-name=GCC
    
  • for a custom EasyConfig myApp.eb only in the Broadwell and Ivybridge software stack:
    eb-install-all --archs='ivy broadwell' --robot myApp.eb
    

This will need time to get scheduled and processed. The job output is presented in the eb_out.* files, one for each architecture.

If the build could not be finished in the default time of 1h, the walltime can be extended using:

eb-install-all --robot --slurm-args='--time=05:00:00' ...

Note

Please check the end of the out file for the COMPLETED: Installation ended successfully statement.

When finished you (and your collaborators) should be able to use use the software, by just loading the user/workspace related module and the module for the installed package.

Meta module and Toolchains

On the one hand a module may load/depend on other modules, which are automatically loaded with that module. As an example the toolchain module foss loads: the GCC compiler, the MPI libary OpenMPI, math libaries OpenBLAS, FFTW and ScaLAPACK. These libraries often have further dependencies.

When building new packages, version consitencies need to be respected. Targeting a build with foss/2020b libraries should be utilized, build with the same version of foss or it underlying packages/libaries, e.g. GCC/10.2.0.

Adapting Easyconfigs

in the following description and example we update an existing old easyconfig for newer versions. In our case we want to update the version of Relion, the toolchain, and dependent libraries it is build with.

  • setup EasyBuild environment

    module load EasyBuild
    module load Workspace   ### OR Workspace/home
    

  • find a suitable easyconfig

    $ eb --search Relion 
    
    alternatively you may find easyconfigs online, e.g. https://github.com/easybuilders/easybuild-easyconfigs

  • copy the easyconfig into a working directory (here .)

    $ cp $EBROOTEASYBUILD/easybuild/easyconfigs/r/RELION/RELION-3.0.4-foss-2017b.eb .
    

  • rename to the targeted versions (here newer relion, newer toolchain)

    $ mv RELION-3.0.4-foss-2017b.eb RELION-3.1.2-foss-2020b.eb
    

  • find the new versions of toolchain and libraries

    • all installed version of a package can be listed using module avail package, e.g. module avail foss
    • available easyconfigs of non-installed packages can be listed using eb --search package. If there is a targeted version available, you can just define that dependency version in the above easyconfig and EasyBuild will find and use it.
  • update the versions settings in the file

    • package version, the toolchain version, and all related libraries
    • Keep in mind that toolchain versions need to match (see toolchains above)
      easyblock = 'CMakeMake'
      
      name = 'RELION'
      version = '3.1.2'                            #### The Relion version was '3.0.4' before
      
      homepage = 'http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page'
      description = """RELION (for REgularised LIkelihood OptimisatioN, pronounce rely-on) is a stand-alone computer
       program that employs an empirical Bayesian approach to refinement of (multiple) 3D reconstructions or 2D class
       averages in electron cryo-microscopy (cryo-EM)."""
      
      toolchain = {'name': 'foss', 'version': '2020b'}   ### the foss toolchain version was 2020b before
      toolchainopts = {'openmp': True}
      
      source_urls = ['https://github.com/3dem/relion/archive']
      sources = ['%(version)s.tar.gz']
      checksums = ['2580d66088923a644bc7d3b02efd154b775a3ec3d010426f382bb3be5db9c98b']
      
      builddependencies = [('CMake', '3.18.4')]    ### was 3.9.5
      
      dependencies = [
          ('X11', '20201008'),                     ### was 20171023
          ('FLTK', '1.3.5'),                       ### was 1.3.4
          ('LibTIFF', '4.1.0'),                    ### 4.0.9
          ('tbb', '2020.3'),                       ### 2018_U5
      ]
      
      configopts = "-DCMAKE_SHARED_LINKER_FLAGS='-lpthread'  -DMPI_INCLUDE_PATH=$EBROOTOPENMPI/include "
      configopts += "-DMPI_C_COMPILER=$EBROOTOPENMPI/bin/mpicc -DMPI_CXX_COMPILER=$EBROOTOPENMPI/bin/mpicxx "
      configopts += "-DCUDA=OFF -DCudaTexture=OFF "
      configopts += "-DALTCPU=ON -DFORCE_OWN_TBB=OFF "
      
      sanity_check_paths = {
          'files': ['bin/relion'],
          'dirs': []
      }
      
      moduleclass = 'bio'
      
  • update the checksum (if package version is changed) The downloaded source packages are typically checked with SHA256 checksums. When we change to a different source code versio, the checksum changes too. And need to be updated.

    $ eb --force --inject-checksums sha256 RELION-3.1.2-foss-2020b.eb
    

  • build the new package as described in Installation above, e.g.

    $ eb-install-all --robot RELION-3.1.2-foss-2020b.eb