A-Cluster

Linux cluster with currently 17 compute nodes (CPUs: 544 cores, GPUs: 8× RTX 2080 + 24× RTX 3090+ 4× RTX 4090) and 2×251TiB disk storage, purchased by Ana Vila Verde and Christopher Stein

Login

External Hostname is a-cluster.physik.uni-due.de (134.91.59.16), internal hostname is stor2.

Queueing system: Slurm

• There are four queues (partitions in Slurm terminology) named:
  • CPUs, the default. The nodes have two Intel CPUs, Xeon Gold 6226R or 6346R. To run on a node with the latter, you need to specify the option -C X6346R.
  • GPUs, to be selected via -p GPUs for jobs which involve a GPU
  • AMD, to be selected via -p AMD, which contains only the node g3pu17 (having an AMD EPYC 9354P instead of an Intel CPU)
  • Test, to be selected via -p Test for test jobs of maximal 10 minutes running time (Compute node gpu01 is reserved exclusively for this queue.)
• In the CPUs queue, 2 cores stay reserved on each node for GPU jobs, resulting in 30 available cores.
• sinfo displays the cluster's total load.
• squeue shows running jobs. You can modify its output via the option -o. To make that permanent put something like alias squeue='squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %C %o"' into your .bashrc.
• In the most simple cases, jobs are submitted via sbatch -n n script-name. The number n of CPUs is available within the script as $SLURM_NTASKS. It's not necessary to pass it on to mpirun, since the latter evaluates it on its own, anyway.
• To allocate GPUs as well, add -G n or --gpus=n with n ∈ {1,2}. You can specify the type as well by prepending rtx2080:, rtx3090:, or rtx4090: to n.
• Don't use background jobs (&), unless you wait for them before the end of the script.
• srun is intended for interactive jobs (stdin+stdout+stderr stay attached to the terminal) and its -n doesn't only reserve n cores but starts n jobs. (Those shouldn't contain mpirun, otherwise you'd end up with n² busy cores.)
• For an interactive shell with n reserved cores on a compute node: srun --pty -cn bash
• If you want to avoid certain nodes, you can specify their names to the option -x.
• The assignment of cores can be non-trivial (cf. also task affinity), some rules:
  • gromacs: Don't use its -pin options.
• There are restrictions per user:
  • You cannot use more than 384 CPU cores simultaneously.
  • You cannot have more than 128 submitted jobs. If you have many, many runs with just varying parameters, consider using job arrays.

GPUs

There are GPUs on each node (2× RTX2080 on gpu01-04, 2× RTX3090 on g3pu05-16, 4× RTX4090 on g4pu17). After having requested GPUs (cf. above), you'll find the ID(s) \(\in\{0,1\}\) of the GPUs(s) assigned to your job in the environment variable SLURM_STEP_GPUS as well as in GPU_DEVICE_ORDINAL.

The command sgpus (no manpage) displays the number of unallocated GPUs on each node.

Scratch space

If your job makes heavy use of temporary files, you shouldn't have them in your home directory (to avoid too much network traffic). Each node has about 400GiB disk space available in /tmp, where you should create /tmp/$USER/$SLURM_JOBID (to avoid cluttering) and wipe it at the end of your job.

Four nodes (g3pu07-10) have a dedicated scratch directory /scratch of 3.4TiB capacity, where you should create (and later wipe) /scratch/$USER/$SLURM_JOBID. To use it, you have to specify --gres=scratch:X upon submission, where X is the amount of scratch space intended to use in GiB (max 3480). (This amount is not checked during the job's runtime.)

Scientific Software

... installed (on the compute nodes)

AMBER

The module system is not involved. Instead, scripts provided by the software set the environment.

• /usr/local/amber18
• /usr/local/amber20 (provides parmed as well)

Script to source therein (assuming bash): amber.sh

GROMACS

The module system is not involved. Instead, scripts provided by the software set the environment.

Versions (not all tested):

• /usr/local/gromacs-2018.3
• /usr/local/gromacs-2020.4
• /usr/local/gromacs-3.3.4
• /usr/local/gromacs-4.6.4
• /usr/local/gromacs-5.0.1
• /usr/local/gromacs-5.1.1

Script to source therein (assuming bash): bin/GMXRC.bash

Ana provided an example script to be submitted via sbatch.

OpenMM + open forcefield

• source /usr/local/miniconda3/bin/activate
• conda activate openforcefield
• installed openff components: forceBalance, geomeTRIC, openFF toolkit, openFF evaluator, TorsionDrive, pyMBAR
• also installed: jupyterlab

OpenMolcas

(compiled with Intel compiler and MKL)

Minimal example script to be sbatched:

 #!/bin/bash
 
 export MOLCAS=/usr/local/openmolcas
 export MOLCAS_WORKDIR=/tmp/$USER-$SLURM_JOB_NAME-$SLURM_JOB_ID
 mkdir $MOLCAS_WORKDIR
 export PATH=$PATH:$MOLCAS
 export LD_LIBRARY_PATH=/opt/intel/oneapi/compiler/latest/linux/compiler/lib/intel64_lin:/opt/intel/oneapi/mkl/latest/lib/intel64
 export OMP_NUM_THREADS=${SLURM_NTASKS:-1}
 
 pymolcas the_input.inp
 
 rm -rf $MOLCAS_WORKDIR

If you want/need to use the module system instead of setting LD_LIBRARY_PATH manually:

 shopt -s expand_aliases
 source /etc/profile.d/modules.sh
 
 module use /opt/intel/oneapi/modulefiles
 module -s load compiler/latest
 module -s load mkl/latest

Intel Compiler & Co.

• is located in /opt/intel/oneapi
• must be made available via module use /opt/intel/oneapi/modulefiles (unless you include /opt/intel/oneapi/modulefiles in your MODULEPATH), then module avail lists the available modules.
• Module mkl/latest contains also FFT routines.

Backups

A backup of the users' home directories is taken nightly. To access the backups, first log in to the cluster. Then:

• Users in /home/stor.vd1: Last night's backup is in /export/vd1/$USER.
• Users in /home/stor1.lv0: You actually have seven backups corresponding to the last 7 days in /exports/lv1/snapshots/days.D/stor1/home/stor1.lv0/$USER with D \(\in\{0,\dots,6\}\).