A-Cluster - Versionsgeschichte

Brendel: +g4pu18, +g5pu19, /scratch

2025-12-18T16:40:52Z

+g4pu18, +g5pu19, /scratch

← Nächstältere Version		Version vom 18. Dezember 2025, 18:40 Uhr
Zeile 1:		Zeile 1:
	Linux cluster with currently 17 compute nodes (CPUs: ~~544~~ cores, GPUs: 8× RTX 2080 + ~~24×~~ RTX 3090+ 4× RTX 4090) and 2×316TiB disk storage, purchased by Ana Vila Verde and Christopher Stein		Linux cluster with currently 17 compute nodes (CPUs: 800 cores, GPUs: 8× RTX 2080 + 23× RTX 3090+ 6× RTX 4090 + 2× RTX 5090) and 2×316TiB disk storage, purchased by Ana Vila Verde and Christopher Stein

	= Login =		= Login =
Zeile 10:		Zeile 10:
	** ''CPUs'', the default. The nodes have two Intel CPUs each, Xeon Gold 6226R or 6346R. To assure running on a node with the latter, you need to specify the option <code>-C X6346R</code>.		** ''CPUs'', the default. The nodes have two Intel CPUs each, Xeon Gold 6226R or 6346R. To assure running on a node with the latter, you need to specify the option <code>-C X6346R</code>.
	** ''GPUs'', to be selected via <code>-p GPUs</code> for jobs which involve a GPU		** ''GPUs'', to be selected via <code>-p GPUs</code> for jobs which involve a GPU
	** ''AMD'', to be selected via <code>-p AMD</code>, which contains only the nodes g4pu17 ~~and~~ g4pu18 and (having AMD CPUs [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9354p.html EPYC 9354P] ~~and~~ [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html EPYC 9554], respectively, instead of Intel CPUs).		** ''AMD'', to be selected via <code>-p AMD</code>, which contains only the nodes g4pu17, g4pu18 and g5pu19 (having AMD CPUs [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9354p.html EPYC 9354P], [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html EPYC 9554] and [https://www.amd.com/de/products/processors/server/epyc/9005-series/amd-epyc-9745.html EPYC 9745], respectively, instead of Intel CPUs).
	** ''Test'', to be selected via <code>-p Test</code> for test jobs of maximal 10 minutes running time (Compute node ''gpu01'' is reserved exclusively for this queue.)		** ''Test'', to be selected via <code>-p Test</code> for test jobs of maximal 10 minutes running time (Compute node ''gpu01'' is reserved exclusively for this queue.)
	* In the ''CPUs'' queue, 2 cores stay reserved on each node for GPU jobs, resulting in 30 available cores.		* In the ''CPUs'' queue, 2 cores stay reserved on each node for GPU jobs, resulting in 30 available cores.
Zeile 30:		Zeile 30:
	= GPUs =		= GPUs =

	There are GPUs on each node (2× RTX2080 on gpu01-04, 2× RTX3090 on g3pu05-16, 4× RTX4090 on g4pu17). After having requested GPUs (cf. above), you'll find the ID(s) $\in\{0,...,3\}$ of the GPUs(s) assigned to your job in the environment variable <code>SLURM_STEP_GPUS</code> as well as in <code>GPU_DEVICE_ORDINAL</code>.		There are GPUs on each node (2× RTX2080 on gpu01-04, 2× RTX3090 on g3pu05-16, 4× RTX4090 on g4pu17, 2× RTX4090 on g4pu18, 2× RTX5090 on g5pu19). After having requested GPUs (cf. above), you'll find the ID(s) $\in\{0,...,3\}$ of the GPUs(s) assigned to your job in the environment variable <code>SLURM_STEP_GPUS</code> as well as in <code>GPU_DEVICE_ORDINAL</code>.

	The command <code>sgpus</code> (no manpage) displays the number of unallocated GPUs on each node.		The command <code>sgpus</code> (no manpage) displays the number of unallocated GPUs on each node.
Zeile 36:		Zeile 36:
	= Scratch space =		= Scratch space =

	If your job makes heavy use of temporary files, you shouldn't have them in your home directory (to avoid too much network traffic). Each node has ~~about 400GiB~~ disk space available in <code>/~~tmp~~</code>, where you should create <code>/~~tmp~~/$USER/$SLURM_JOBID</code> (to avoid cluttering) and wipe it at the end of your job.		If your job makes heavy use of temporary files, you shouldn't have them in your home directory (to avoid too much network traffic). Each node has at least 375GiB disk space available in <code>/scratch</code>, where you should create <code>/scratch/$USER/$SLURM_JOBID</code> (to avoid cluttering) and wipe it at the end of your job.

	Four nodes (g3pu07-10) have a ~~dedicated~~ scratch directory <code>/scratch</code> of 3.4TiB capacity~~, where you should create (and later wipe) <code>/scratch/$USER/$SLURM_JOBID</code>~~. To use it, you have to specify <code>--gres=scratch:</code>''X'' upon submission, where ''X'' is the amount of scratch space intended to use in GiB (max 3480). (This amount is not checked during the job's runtime.)		Four nodes (g3pu07-10) have a larger scratch directory <code>/scratch</code> of 3.4TiB capacity. To use it, you have to specify <code>--gres=scratch:</code>''X'' upon submission, where ''X'' is the amount of scratch space intended to use in GiB (max 3480). (This amount is not checked during the job's runtime.)

	= Scientific Software =		= Scientific Software =

Brendel: /* Queueing system: Slurm */ NPROCS = NTASKS

2025-02-05T21:49:01Z

Queueing system: Slurm: NPROCS = NTASKS

← Nächstältere Version		Version vom 5. Februar 2025, 23:49 Uhr
Zeile 15:		Zeile 15:
	* <code>[https://slurm.schedmd.com/sinfo.html sinfo]</code> displays the cluster's total load.		* <code>[https://slurm.schedmd.com/sinfo.html sinfo]</code> displays the cluster's total load.
	* <code>[https://slurm.schedmd.com/squeue.html squeue]</code> shows running jobs. You can modify its output via the option <code>-o</code>. To make that permanent put something like <code>alias squeue='squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %C %o"'</code> into your <code>.bashrc</code>.		* <code>[https://slurm.schedmd.com/squeue.html squeue]</code> shows running jobs. You can modify its output via the option <code>-o</code>. To make that permanent put something like <code>alias squeue='squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R %C %o"'</code> into your <code>.bashrc</code>.
	* In the most simple cases, jobs are submitted via <code>[https://slurm.schedmd.com/sbatch.html sbatch] -n</code> ''n'' ''script-name''. The number ''n'' of CPUs is available within the script as <code>$SLURM_NTASKS</code>. It's not necessary to pass it on to <code>mpirun</code>, since the latter evaluates it on its own, anyway.		* In the most simple cases, jobs are submitted via <code>[https://slurm.schedmd.com/sbatch.html sbatch] -n</code> ''n'' ''script-name''. The number ''n'' of CPUs is available within the script as <code>$SLURM_NPROCS</code> and (equivalently) <code>$SLURM_NTASKS</code>. It's not necessary to pass it on to <code>mpirun</code>, since the latter evaluates it on its own, anyway.
	* To allocate GPUs as well, add <code>-G </code>''n'' or <code>--gpus=</code>''n'' with ''n'' ∈ {1,2}. You can specify the type as well by prepending <code>rtx2080:</code>, <code>rtx3090:</code>, or <code>rtx4090:</code> to ''n''.		* To allocate GPUs as well, add <code>-G </code>''n'' or <code>--gpus=</code>''n'' with ''n'' ∈ {1,2}. You can specify the type as well by prepending <code>rtx2080:</code>, <code>rtx3090:</code>, or <code>rtx4090:</code> to ''n''.
	* Don't use background jobs (<code>&</code>), unless you <code>wait</code> for them before the end of the script.		* Don't use background jobs (<code>&</code>), unless you <code>wait</code> for them before the end of the script.
	* <code>[https://slurm.schedmd.com/srun.html srun]</code> can be used as a substitute for <code>mpirun</code> or for interactive jobs (stdin+stdout+stderr stay attached to the terminal), its <code>-n</code> doesn't only reserve ''n'' cores but starts ''n'' jobs. (Those shouldn't contain <code>mpirun</code> again, otherwise you'd end up with ''n''² busy cores.)		* <code>[https://slurm.schedmd.com/srun.html srun]</code> can be used as a substitute for <code>mpirun</code> or for interactive jobs (stdin+stdout+stderr stay attached to the terminal), its <code>-n</code> doesn't only reserve ''n'' cores but starts ''n'' jobs. (Those shouldn't contain <code>mpirun</code> again, otherwise you'd end up with ''n''² busy cores.)
	* If you launch the actual ''n'' workers yourself (instead of using <code>mpirun</code> or <code>srun</code>), read ''n'' from the environment ~~variable~~ <code>SLURM_NPROCS</code>.		* If you launch the actual ''n'' workers yourself (instead of using <code>mpirun</code> or <code>srun</code>), read ''n'' from the environment variables <code>SLURM_NPROCS</code> or <code>SLURM_NTASKS</code>.
	* For an interactive shell with ''n'' reserved cores on a compute node: <code>srun --pty -c</code>''n''<code> bash</code>		* For an interactive shell with ''n'' reserved cores on a compute node: <code>srun --pty -c</code>''n''<code> bash</code>
	* If you want to avoid certain nodes, you can specify their names to the option <code>-x</code>.		* If you want to avoid certain nodes, you can specify their names to the option <code>-x</code>.

Brendel: /* Queueing system: Slurm */ srun, SLURM_NPROCS

2025-02-05T20:33:12Z

Queueing system: Slurm: srun, SLURM_NPROCS

← Nächstältere Version		Version vom 5. Februar 2025, 22:33 Uhr
Zeile 18:		Zeile 18:
	* To allocate GPUs as well, add <code>-G </code>''n'' or <code>--gpus=</code>''n'' with ''n'' ∈ {1,2}. You can specify the type as well by prepending <code>rtx2080:</code>, <code>rtx3090:</code>, or <code>rtx4090:</code> to ''n''.		* To allocate GPUs as well, add <code>-G </code>''n'' or <code>--gpus=</code>''n'' with ''n'' ∈ {1,2}. You can specify the type as well by prepending <code>rtx2080:</code>, <code>rtx3090:</code>, or <code>rtx4090:</code> to ''n''.
	* Don't use background jobs (<code>&</code>), unless you <code>wait</code> for them before the end of the script.		* Don't use background jobs (<code>&</code>), unless you <code>wait</code> for them before the end of the script.
	* <code>[https://slurm.schedmd.com/srun.html srun]</code> ~~is intended~~ for interactive jobs (stdin+stdout+stderr stay attached to the terminal) ~~and~~ its <code>-n</code> doesn't only reserve ''n'' cores but starts ''n'' jobs. (Those shouldn't contain <code>mpirun</code>, otherwise you'd end up with ''n''² busy cores.)		* <code>[https://slurm.schedmd.com/srun.html srun]</code> can be used as a substitute for <code>mpirun</code> or for interactive jobs (stdin+stdout+stderr stay attached to the terminal), its <code>-n</code> doesn't only reserve ''n'' cores but starts ''n'' jobs. (Those shouldn't contain <code>mpirun</code> again, otherwise you'd end up with ''n''² busy cores.)
			* If you launch the actual ''n'' workers yourself (instead of using <code>mpirun</code> or <code>srun</code>), read ''n'' from the environment variable <code>SLURM_NPROCS</code>.
	* For an interactive shell with ''n'' reserved cores on a compute node: <code>srun --pty -c</code>''n''<code> bash</code>		* For an interactive shell with ''n'' reserved cores on a compute node: <code>srun --pty -c</code>''n''<code> bash</code>
	* If you want to avoid certain nodes, you can specify their names to the option <code>-x</code>.		* If you want to avoid certain nodes, you can specify their names to the option <code>-x</code>.

Brendel: /* Queueing system: Slurm */ g4pu18, AMD-Links

2025-02-05T12:51:32Z

Queueing system: Slurm: g4pu18, AMD-Links

← Nächstältere Version		Version vom 5. Februar 2025, 14:51 Uhr
Zeile 10:		Zeile 10:
	** ''CPUs'', the default. The nodes have two Intel CPUs each, Xeon Gold 6226R or 6346R. To assure running on a node with the latter, you need to specify the option <code>-C X6346R</code>.		** ''CPUs'', the default. The nodes have two Intel CPUs each, Xeon Gold 6226R or 6346R. To assure running on a node with the latter, you need to specify the option <code>-C X6346R</code>.
	** ''GPUs'', to be selected via <code>-p GPUs</code> for jobs which involve a GPU		** ''GPUs'', to be selected via <code>-p GPUs</code> for jobs which involve a GPU
	** ''AMD'', to be selected via <code>-p AMD</code>, which contains only the ~~node g3pu17~~ (having an [https://www.amd.com/de/products/~~cpu~~/amd-epyc-9354p~~#product~~-~~specs AMD~~ EPYC ~~9354P~~] instead of an Intel ~~CPU~~)		** ''AMD'', to be selected via <code>-p AMD</code>, which contains only the nodes g4pu17 and g4pu18 and (having AMD CPUs [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9354p.html EPYC 9354P] and [https://www.amd.com/de/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9554.html EPYC 9554], respectively, instead of Intel CPUs).
	** ''Test'', to be selected via <code>-p Test</code> for test jobs of maximal 10 minutes running time (Compute node ''gpu01'' is reserved exclusively for this queue.)		** ''Test'', to be selected via <code>-p Test</code> for test jobs of maximal 10 minutes running time (Compute node ''gpu01'' is reserved exclusively for this queue.)
	* In the ''CPUs'' queue, 2 cores stay reserved on each node for GPU jobs, resulting in 30 available cores.		* In the ''CPUs'' queue, 2 cores stay reserved on each node for GPU jobs, resulting in 30 available cores.

Brendel: /* On the Login Node (stor2) */

2024-12-03T14:49:35Z

On the Login Node (stor2)

← Nächstältere Version		Version vom 3. Dezember 2024, 16:49 Uhr
Zeile 43:		Zeile 43:
	== On the Login Node (''stor2'') ==		== On the Login Node (''stor2'') ==

	* VMD 1.9.3 (which can [[~~VMC~~/create movies\|create movies]])		* VMD 1.9.3 (which can [[VMD/create movies\|create movies]])

	== On the Compute Nodes ==		== On the Compute Nodes ==

Brendel: VMD on stor2

2024-12-03T14:44:30Z

VMD on stor2

← Nächstältere Version		Version vom 3. Dezember 2024, 16:44 Uhr
Zeile 41:		Zeile 41:
	= Scientific Software =		= Scientific Software =

	~~... installed~~ (~~on the~~ ''~~compute nodes~~'')		== On the Login Node (''stor2'') ==

	== AMBER ==		* VMD 1.9.3 (which can [[VMC/create movies\|create movies]])

			== On the Compute Nodes ==

			=== AMBER ===

	The [https://modules.readthedocs.io/en/latest module system] is not involved. Instead, scripts provided by the software set the environment.		The [https://modules.readthedocs.io/en/latest module system] is not involved. Instead, scripts provided by the software set the environment.
Zeile 52:		Zeile 56:
	Script to source therein (assuming [https://en.wikipedia.org/wiki/Bash_(Unix_shell) bash]): <code>amber.sh</code>		Script to source therein (assuming [https://en.wikipedia.org/wiki/Bash_(Unix_shell) bash]): <code>amber.sh</code>

	== GROMACS ==		=== GROMACS ===

	The [https://modules.readthedocs.io/en/latest module system] is not involved. Instead, scripts provided by the software set the environment.		The [https://modules.readthedocs.io/en/latest module system] is not involved. Instead, scripts provided by the software set the environment.
Zeile 69:		Zeile 73:
	Ana provided an [https://wiki.uni-due.de/vilaverde/index.php/File:Gromacs_cpu.sh example script] to be submitted via <code>sbatch</code>.		Ana provided an [https://wiki.uni-due.de/vilaverde/index.php/File:Gromacs_cpu.sh example script] to be submitted via <code>sbatch</code>.

	== OpenMM + open forcefield ==		=== OpenMM + open forcefield ===

	* <code>source /usr/local/miniconda3/bin/activate</code>		* <code>source /usr/local/miniconda3/bin/activate</code>
Zeile 76:		Zeile 80:
	* also installed: jupyterlab		* also installed: jupyterlab

	== OpenMolcas ==		=== OpenMolcas ===

	(compiled with Intel compiler and MKL)		(compiled with Intel compiler and MKL)

Brendel: +Python 3.12

2024-03-11T07:12:37Z

+Python 3.12

← Nächstältere Version		Version vom 11. März 2024, 09:12 Uhr
Zeile 104:		Zeile 104:
	module -s load mkl/latest		module -s load mkl/latest

	= Intel Compiler & Co. =		= Compilers & Interpreters =

			== Intel Compiler & Co. ==

	* is located in <code>/opt/intel/oneapi</code>		* is located in <code>/opt/intel/oneapi</code>
	* must be made available via <code>module use /opt/intel/oneapi/modulefiles</code> (unless you include <code>/opt/intel/oneapi/modulefiles</code> in your <code>MODULEPATH</code>), then <code>module avail</code> lists the available modules.		* must be made available via <code>module use /opt/intel/oneapi/modulefiles</code> (unless you include <code>/opt/intel/oneapi/modulefiles</code> in your <code>MODULEPATH</code>), then <code>module avail</code> lists the available modules.
	* Module ''mkl/latest'' contains also FFT routines.		* Module ''mkl/latest'' contains also FFT routines.

			== Python ==

			The Python version provided by the system is currently 3.9. If you need a newer version, you can make 3.12 available by doing (in bash):

			<code>source /usr/local/miniconda3/bin/activate py312</code>

	= Backups =		= Backups =

Brendel: 316TB

2024-03-10T20:30:09Z

316TB

← Nächstältere Version		Version vom 10. März 2024, 22:30 Uhr
Zeile 1:		Zeile 1:
	Linux cluster with currently 17 compute nodes (CPUs: 544 cores, GPUs: 8× RTX 2080 + 24× RTX 3090+ 4× RTX 4090) and ~~2×251TiB~~ disk storage, purchased by Ana Vila Verde and Christopher Stein		Linux cluster with currently 17 compute nodes (CPUs: 544 cores, GPUs: 8× RTX 2080 + 24× RTX 3090+ 4× RTX 4090) and 2×316TiB disk storage, purchased by Ana Vila Verde and Christopher Stein

	= Login =		= Login =

Brendel: /* Backups */ lv1

2024-03-06T07:13:30Z

Backups: lv1

← Nächstältere Version		Version vom 6. März 2024, 09:13 Uhr
Zeile 115:		Zeile 115:
	backups, first log in to the cluster. Then:		backups, first log in to the cluster. Then:
	* Users in <code>/home/stor.vd1</code>: Last night's backup is in <code>/export/vd1/$USER</code>.		* Users in <code>/home/stor.vd1</code>: Last night's backup is in <code>/export/vd1/$USER</code>.
	* Users in <code>/home/stor1.~~lv0~~</code>: You actually have seven backups corresponding to the last 7 days in <code>/exports/lv1/snapshots/days.''D''/stor1/home/stor1.~~lv0~~/$USER</code> with ''D'' $\in\{0,\dots,6\}$.		* Users in <code>/home/stor1.lv1</code>: You actually have seven backups corresponding to the last 7 days in <code>/exports/lv1/snapshots/days.''D''/stor1/home/stor1.lv1/$USER</code> with ''D'' $\in\{0,\dots,6\}$.

Brendel: /* GPUs */ four GPUs

2024-03-06T07:12:38Z

GPUs: four GPUs

← Nächstältere Version		Version vom 6. März 2024, 09:12 Uhr
Zeile 29:		Zeile 29:
	= GPUs =		= GPUs =

	There are GPUs on each node (2× RTX2080 on gpu01-04, 2× RTX3090 on g3pu05-16, 4× RTX4090 on g4pu17). After having requested GPUs (cf. above), you'll find the ID(s) $\in\{0,1\}$ of the GPUs(s) assigned to your job in the environment variable <code>SLURM_STEP_GPUS</code> as well as in <code>GPU_DEVICE_ORDINAL</code>.		There are GPUs on each node (2× RTX2080 on gpu01-04, 2× RTX3090 on g3pu05-16, 4× RTX4090 on g4pu17). After having requested GPUs (cf. above), you'll find the ID(s) $\in\{0,...,3\}$ of the GPUs(s) assigned to your job in the environment variable <code>SLURM_STEP_GPUS</code> as well as in <code>GPU_DEVICE_ORDINAL</code>.

	The command <code>sgpus</code> (no manpage) displays the number of unallocated GPUs on each node.		The command <code>sgpus</code> (no manpage) displays the number of unallocated GPUs on each node.