HPC Guide - User contributions [en]

Using jupyter on Slurm

2025-12-03T14:48:30Z

Eyal:

'''Using Jupyter Lab on Slurm Cluster'''

'''Step 1''': Start an Interactive Job on the Head Node

First, you need to request resources on the Slurm cluster. Run the following command to start an interactive job on the appropriate partition (<partition-name>) and with your account (<account-name>):

srun -p <partition-name> -A <account-name> --pty bash

Replace:

<partition-name> with your Slurm partition name (e.g., power-general).
<account-name> with your Slurm account name (e.g., power-leahfa-users).

This command will allocate resources for your job and provide you with an interactive shell on a compute node (e.g., compute-0-62).

'''Step 2''': Load the Jupyter Environment

Once inside the interactive session, load the mamba-env158/jupyter module:

module load mamba-env158/jupyter

This will prepare the environment to run Jupyter.

'''Step 3''': Start Jupyter Lab

Now, start the Jupyter Lab server. Run the following command inside the interactive session:

jupyter lab --ip=* --port=8892 --no-browser

The --ip=* option binds Jupyter Lab to all available network interfaces.
The --port=8892 specifies that Jupyter Lab will use port 8892.
The --no-browser option prevents Jupyter Lab from opening a browser automatically.

Once Jupyter Lab starts, you should see output like this:

[I 2025-03-16 09:25:06.190 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2025-03-16 09:25:06.190 ServerApp] http://localhost:8892/lab?token=<token>

Example:

[[File:2025-12-03 161826.jpg]]

'''Step 4''': Set Up SSH Port Forwarding

To access the Jupyter Lab server from your local machine, you need to set up SSH port forwarding.

Open a terminal on your local (cmd on windows) machine and run the following command:

ssh -N -L 8892:<compute-node-name>:8892 <username>@<headnode-name>

Replace:

<compute-node-name> with the name of the compute node where Jupyter Lab is running (e.g., compute-0-62).
<username> with your Slurm username.
<headnode-name> with the hostname or IP address of the Slurm head node (e.g., powerslurm-login.tau.ac.il).

This command forwards port 8892 from the compute node back to your local machine.

Example:

[[File:2025-12-03 162115.jpg]]

'''Step 5''': Open Jupyter Lab in Your Browser

After establishing the SSH tunnel, open your web browser and navigate to:

http://localhost:8892/lab?token=<token>

This URL will give you access to your Jupyter Lab instance.

Example:

[[File:2025-12-03 162302.jpg]]

'''Step 6''': Start Using Jupyter Lab

Once you’ve entered the token, you can start working with Jupyter Lab on your browser, using the resources of the cluster.

'''Step 7''': Closing the Job

When you're finished using Jupyter Lab, return to the interactive session and press Ctrl+C to stop the Jupyter Lab server. Then, exit the interactive session by typing:

exit

Additional Notes:

Ensure that the SSH tunnel (ssh -N -L ...) remains open while you are working with Jupyter Lab.<br>
You can use different ports if 8892 is already in use, just adjust the port number in both the jupyter lab command and the SSH command.<br>
If you encounter any issues with port forwarding, double-check the node name (<compute-node-name>) and the port number.

Using jupyter on Slurm

2025-12-03T14:43:22Z

Eyal: Created page with "'''Using Jupyter Lab on Slurm Cluster''' '''Step 1''': Start an Interactive Job on the Head Node First, you need to request resources on the Slurm cluster. Run the following..."

'''Using Jupyter Lab on Slurm Cluster'''

'''Step 1''': Start an Interactive Job on the Head Node

First, you need to request resources on the Slurm cluster. Run the following command to start an interactive job on the appropriate partition (<partition-name>) and with your account (<account-name>):

srun -p <partition-name> -A <account-name> --pty bash

Replace:

<partition-name> with your Slurm partition name (e.g., power-general).
<account-name> with your Slurm account name (e.g., power-leahfa-users).

This command will allocate resources for your job and provide you with an interactive shell on a compute node (e.g., compute-0-62).

'''Step 2''': Load the Jupyter Environment

Once inside the interactive session, load the mamba-env158/jupyter module:

module load mamba-env158/jupyter

This will prepare the environment to run Jupyter.

'''Step 3''': Start Jupyter Lab

Now, start the Jupyter Lab server. Run the following command inside the interactive session:

jupyter lab --ip=* --port=8892 --no-browser

The --ip=* option binds Jupyter Lab to all available network interfaces.
The --port=8892 specifies that Jupyter Lab will use port 8892.
The --no-browser option prevents Jupyter Lab from opening a browser automatically.

Once Jupyter Lab starts, you should see output like this:

[I 2025-03-16 09:25:06.190 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2025-03-16 09:25:06.190 ServerApp] http://localhost:8892/lab?token=<token>

Example:

[[File:2025-12-03 161826.jpg]]

'''Step 4''': Set Up SSH Port Forwarding

To access the Jupyter Lab server from your local machine, you need to set up SSH port forwarding.

Open a terminal on your local (cmd on windows) machine and run the following command:

ssh -N -L 8892:<compute-node-name>:8892 <username>@<headnode-name>

Replace:

<compute-node-name> with the name of the compute node where Jupyter Lab is running (e.g., compute-0-62).
<username> with your Slurm username.
<headnode-name> with the hostname or IP address of the Slurm head node (e.g., powerslurm-login.tau.ac.il).

This command forwards port 8892 from the compute node back to your local machine.

Example:

[[File:2025-12-03 162115.jpg]]

'''Step 5''': Open Jupyter Lab in Your Browser

After establishing the SSH tunnel, open your web browser and navigate to:

http://localhost:8892/lab?token=<token>

This URL will give you access to your Jupyter Lab instance.

Example:

[[File:2025-12-03 162302.jpg]]

'''Step 6''': Start Using Jupyter Lab

Once you’ve entered the token, you can start working with Jupyter Lab on your browser, using the resources of the cluster.

'''Step 7''': Closing the Job

When you're finished using Jupyter Lab, return to the interactive session and press Ctrl+C to stop the Jupyter Lab server. Then, exit the interactive session by typing:

exit

Additional Notes:

Ensure that the SSH tunnel (ssh -N -L ...) remains open while you are working with Jupyter Lab.
You can use different ports if 8892 is already in use, just adjust the port number in both the jupyter lab command and the SSH command.
If you encounter any issues with port forwarding, double-check the node name (<compute-node-name>) and the port number.

File:2025-12-03 162302.jpg

2025-12-03T14:32:10Z

Eyal:

browser jupyter example

File:2025-12-03 162115.jpg

2025-12-03T14:30:06Z

Eyal:

ssh tunnle example

File:2025-12-03 161826.jpg

2025-12-03T14:25:58Z

Eyal:

example of jupyter interactive job

Submitting a job to a slurm queue

2025-04-08T07:12:49Z

Eyal:

== Accessing the System ==

To submit jobs to SLURM at Tel Aviv University, you need to access the system through one of the following login nodes:

* slurmlogin.tau.ac.il

=== Requirements for Access ===

* '''Group Membership''': You must be part of the "power" group to access the resources.
* '''University Credentials''': Use your Tel Aviv University username and password to log in.

These login nodes are your starting point for submitting jobs, checking job status, and managing your SLURM tasks.

=== SSH Example ===

To access the system using SSH, use the following example:

<syntaxhighlight lang="bash">
# Replace 'your_username' with your actual Tel Aviv University username
ssh your_username@slurmlogin.tau.ac.il
</syntaxhighlight>

Your connection will be automatically routed to one of the login nodes:
powerslurm-login, powerslurm-login2, or powerslurm-login3.

If you have an SSH key set up for password-less login, you can specify it like this:

<syntaxhighlight lang="bash">
# Replace 'your_username' and '/path/to/your/private_key' accordingly
ssh -i /path/to/your/private_key your_username@slurmlogin.tau.ac.il
</syntaxhighlight>

== Environment Modules ==

Environment Modules in SLURM allow users to dynamically modify their shell environment, providing an easy way to load and unload different software applications, libraries, and their dependencies. This system helps avoid conflicts between software versions and ensures the correct environment for running specific applications.

Here are some common commands to work with environment modules:<syntaxhighlight lang="bash">
#List Available Modules: To see all the modules available on the system, use:
module avail

#To search for a specific module by name (e.g., `gcc`), use:
module avail gcc/gcc-12.1.0

#Get Detailed Information About a Module: The `module spider` command provides detailed information about a module, including versions, dependencies, and descriptions:
module spider gcc/gcc-12.1.0

#View Module Settings: To see what environment variables and settings will be modified by a module, use:
module show gcc/gcc-12.1.0

#Load a Module: To set up the environment for a specific software, use the `module load` command. For example, to load GCC version 12.1.0:
module load gcc/gcc-12.1.0

#List Loaded Modules: To view all currently loaded modules in your session, use:
module list

#Unload a Module: To unload a specific module from your environment, use:
module unload gcc/gcc-12.1.0

#Unload All Modules:** If you need to clear your environment of all loaded modules, use:
module purge

</syntaxhighlight>By using these commands, you can easily manage the software environments needed for different tasks, ensuring compatibility and reducing potential conflicts between software versions.

== Basic Job Submission Commands ==

=== Finding Your Account and Partition ===

Before submitting a job, you need to know which partitions you have permission to use.

Run the command `<code>check_my_partitions</code>` to view a list of all the partitions you have permission to send jobs to.

== Submitting Jobs==
sbatch: Submits a job script for batch processing.

'''Example''':<syntaxhighlight lang="bash">

sbatch --ntasks=1 --time=10 -p power-general -A power-general-users pre_process.bash
# This command submits pre_process.bash to the power-general partition for 10 minutes.
# With 1 GPU:
sbatch --gres=gpu:1 -p gpu-general -A gpu-general-users gpu_job.sh

</syntaxhighlight>

===Writing SLURM Job Scripts===

Here is a simple job script example:

==== Basic Script====
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=my_job # Job name
#SBATCH --account=power-general-users # Account name
#SBATCH --partition=power-general # Partition name
#SBATCH --time=02:00:00 # Max run time (hh:mm:ss)
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --nodes=1 # Number of nodes
#SBATCH --cpus-per-task=1 # CPUs per task
#SBATCH --mem-per-cpu=4G # Memory per CPU
#SBATCH --output=my_job_%j.out # Output file
#SBATCH --error=my_job_%j.err # Error file

echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Your application commands go here
# ./my_program

echo "Job completed"
</syntaxhighlight>

To ask for x cores interactively:
<pre>
srun --ntasks=1 --cpus-per-task=x --partition=power-general --nodes=1 --pty bash
</pre>

However, need for now also to set slurm parameters inside the script, or within the interactive job:
<pre>
export SLURM_TASKS_PER_NODE=48
export SLURM_CPUS_ON_NODE=48
</pre>

For defining an array, may add:
<syntaxhighlight lang="bash">
#SBATCH --array=1-300
</syntaxhighlight>

==== Script for 1 GPU ====
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=gpu_job # Job name
#SBATCH --account=my_account # Account name
#SBATCH --partition=gpu-general # Partition name
#SBATCH --time=02:00:00 # Max run time
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --nodes=1 # Number of nodes
#SBATCH --cpus-per-task=1 # CPUs per task
#SBATCH --gres=gpu:1 # Number of GPUs
#SBATCH --mem-per-cpu=4G # Memory per CPU
#SBATCH --output=my_job_%j.out # Output file
#SBATCH --error=my_job_%j.err # Error file

module load python/python-3.8

echo "Starting GPU job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Your GPU commands go here

echo "Job completed"
</syntaxhighlight>
For excluding a node, one may add the following
<syntaxhighlight>
#SBATCH --exclude=compute-0-[100-103],compute-0-67
</syntaxhighlight>

===Importance of Correct RAM Usage in Jobs===

When writing SLURM job scripts, it's crucial to understand and correctly specify the memory requirements for your job.

Proper memory allocation ensures efficient resource usage and prevents job failures due to out-of-memory (OOM) errors.

==== Why Correct RAM Usage Matters ====

* '''Resource Efficiency''': Allocating the right amount of memory helps in optimal resource utilization, allowing more jobs to run simultaneously on the cluster.
* '''Job Stability''': Underestimating memory requirements can lead to OOM errors, causing your job to fail and waste computational resources.
* '''Performance''': Overestimating memory needs can lead to underutilization of resources, potentially delaying other jobs in the queue.

==== How to Specify Memory in SLURM ====

* '''--mem''': Specifies the total memory required for the job.
* '''--mem-per-cpu''': Specifies the memory required per CPU.

'''Example''':<syntaxhighlight lang="bash">
#SBATCH --mem=4G # Total memory for the job
#SBATCH --mem-per-cpu=2G # Memory per CPU
</syntaxhighlight>

===Interactive Jobs===
<syntaxhighlight lang="bash">
#Start an interactive session:
srun --ntasks=1 -p power-general -A power-general-users --pty bash

#Specify a compute node:
srun --ntasks=1 -p power-general -A power-general-users --nodelist="compute-0-12" --pty bash

#Using GUI:
srun --ntasks=1 -p power-general -A power-general-users --x11 /bin/bash

</syntaxhighlight>

=== Submitting RELION Jobs===

To submit a RELION job interactively on the <code>gpu-relion</code> queue with X11 forwarding, use the following steps:<syntaxhighlight lang="bash">
#Start an interactive session with X11:
srun --ntasks=1 -p gpu-relion -A your_account --x11 --pty bash
#Load the RELION module:
module load relion/relion-4.0.1
#Launch RELION:
relion

</syntaxhighlight>

==Running matlab example==
In this example there are 3 files:

myTable.m ⇒ This matlab file calculates something
<pre>
fprintf('=======================================\n');
fprintf(' a b c d \n');
fprintf('=======================================\n');
while 1
for j = 1:10
a = sin(10*j);
b = a*cos(10*j);
c = a + b;
d = a - b;
fprintf('%+6.5f %+6.5f %+6.5f %+6.5f \n',a,b,c,d);
end
end
fprintf('=======================================\n');
</pre>

my_table_script.sh ⇒ This script executes the matlab program. Need just to run qsub with this script
<pre>
#!/bin/bash

#SBATCH --mem=50mg
#SBATCH --partition powers-general
#SBATCH -A power-general-users
hostname

cd /a/home/cc/tree/taucc/staff/dvory/matlab

matlab -nodisplay -nosplash -nodesktop -r "run(myTable());exit;"
</pre>

run_in_loop.sh ⇒ However, one may also generate many jobs with this file
<pre>
#!/bin/bash

for i in {1..100}

do

sbatch my_table_script.sh

done
</pre>

Running my job is with the command (after doing chmod +x 'run_in_loop.sh'):
<pre>
./run_in_loop.sh
</pre>

==AlphaFold==

AlphaFold is a deep learning tool designed for predicting protein structures.

'''Guides:'''

[https://hpcguide.tau.ac.il/index.php?title=Alphafold AlphaFold Guide]

[https://hpcguide.tau.ac.il/index.php?title=Alphafold3 AlphaFold3 Guide]

==Common SLURM Commands==
<syntaxhighlight lang="bash">
#View all queues (partitions):
sinfo
#View all jobs:
squeue
#View details of a specific job:
scontrol show job <job_number>
#Get information about partitions:
scontrol show partition
</syntaxhighlight>

== Troubleshooting & Tips ==

=== Common Errors ===

# <code>srun: error: Unable to allocate resources: No partition specified or system default partition</code> <br />'''Solution:''' Always specify a partition. Example: <code>srun --pty -c 1 --mem=2G -p power-general /bin/bash</code>
# Job failed, and upon doing scontrol show job job_id or when running sacct -j job_id -o JobID,JobName,State%20 <br />you see: <code>JobState=OUT_OF_MEMORY Reason=OutOfMemory</code> or :<syntaxhighlight lang="bash">
JobID JobName State
------------ ---------- --------------------
71 oom_test OUT_OF_MEMORY
71.batch batch OUT_OF_MEMORY
71.extern extern COMPLETED

</syntaxhighlight>it means that the ram requested for the job was not enough, please resubmit the job again with more ram. see [https://wikihpc.tau.ac.il/index.php?title=Slurm_user_guide#Estimating_RAM_Usage below] for help with understanding how much ram your job may need.

=== Chain Jobs ===
Use the <code>--depend</code> flag to set job dependencies.

'''Example:'''
<syntaxhighlight lang="bash">
sbatch --ntasks=1 --time=60 -p power-general -A power-general-users --depend=45001 do_work.bash
</syntaxhighlight>

=== Always Specify Resources ===
When submitting jobs, ensure you include all required resources like partition, memory, and CPUs to avoid job failures.

=== Attaching to Running Jobs ===
If you need to monitor or interact with a running job, use <code>sattach</code>. This command allows you to attach to a job's input, output, and error streams in real-time.

'''Example:'''
<syntaxhighlight lang="bash">
sattach <job_id>
</syntaxhighlight>

To view job steps of a specific job, use the following command:

<syntaxhighlight lang="bash">
scontrol show job <job_id>
</syntaxhighlight>

Look for sections labeled "StepId" within the output.

'''For specific job steps, use:'''
<syntaxhighlight lang="bash">
sattach <job_id.step_id>
</syntaxhighlight>

'''Note:''' <code>sattach</code> is particularly useful for interactive jobs, where you can provide input directly. For non-interactive jobs, it acts like <code>tail -f</code>, allowing you to monitor the output stream.

=== Estimating RAM Usage ===

When writing SLURM job scripts, it's crucial to understand and correctly specify the memory requirements for your job. Proper memory allocation ensures efficient resource usage and prevents job failures due to out-of-memory (OOM) errors.

==== Tips for Estimating RAM Usage ====

* Check Application Documentation: Refer to the official documentation or user guides for memory-related information.
* Run a Small Test Job: Submit a smaller version of your job and monitor its memory usage using commands like `free -m`, `top`, or `htop`.
* Use Profiling Tools: Tools like `valgrind`, `gprof`, or built-in profilers can help you understand memory usage.
* Analyze Previous Jobs: Review SLURM logs and job statistics for insights into memory consumption of past jobs.
* Consult with Peers or Experts: Ask colleagues or experts who have experience with similar workloads.

==== Example: Monitoring Memory Usage ====
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=memory_test
#SBATCH --account=your_account
#SBATCH --partition=your_partition
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --output=memory_test.out
#SBATCH --error=memory_test.err

# Monitor memory usage
echo "Memory usage before running the job:"
free -m

# Your application commands go here
# ./your_application

# Monitor memory usage after running the job
echo "Memory usage after running the job:"
free -m
</syntaxhighlight>

==== General Tips ====

* Start Small: Begin with a conservative memory request and increase it based on observed usage.
* Consider Peak Usage: Plan for peak memory usage to avoid OOM errors.
* Use SLURM's Memory Reporting: Use `sacct` to view memory usage statistics.

'''Example:'''
<syntaxhighlight lang="bash">
sacct -j <job_id> --format=JobID,JobName,MaxRSS,Elapsed
</syntaxhighlight>

Submitting a job to a slurm queue

2025-04-08T07:05:44Z

Eyal:

== Accessing the System ==

To submit jobs to SLURM at Tel Aviv University, you need to access the system through one of the following login nodes:

* slurmlogin.tau.ac.il

=== Requirements for Access ===

* '''Group Membership''': You must be part of the "power" group to access the resources.
* '''University Credentials''': Use your Tel Aviv University username and password to log in.

These login nodes are your starting point for submitting jobs, checking job status, and managing your SLURM tasks.

=== SSH Example ===

To access the system using SSH, use the following example:

<syntaxhighlight lang="bash">
# Replace 'your_username' with your actual Tel Aviv University username
ssh your_username@slurmlogin.tau.ac.il
</syntaxhighlight>

Your connection will be automatically routed to one of the login nodes:
powerslurm-login, powerslurm-login2, or powerslurm-login3.

If you have an SSH key set up for password-less login, you can specify it like this:

<syntaxhighlight lang="bash">
# Replace 'your_username' and '/path/to/your/private_key' accordingly
ssh -i /path/to/your/private_key your_username@powerslurm-login.tau.ac.il
</syntaxhighlight>

You need to that for all 3 login nodes.

== Environment Modules ==

Environment Modules in SLURM allow users to dynamically modify their shell environment, providing an easy way to load and unload different software applications, libraries, and their dependencies. This system helps avoid conflicts between software versions and ensures the correct environment for running specific applications.

Here are some common commands to work with environment modules:<syntaxhighlight lang="bash">
#List Available Modules: To see all the modules available on the system, use:
module avail

#To search for a specific module by name (e.g., `gcc`), use:
module avail gcc/gcc-12.1.0

#Get Detailed Information About a Module: The `module spider` command provides detailed information about a module, including versions, dependencies, and descriptions:
module spider gcc/gcc-12.1.0

#View Module Settings: To see what environment variables and settings will be modified by a module, use:
module show gcc/gcc-12.1.0

#Load a Module: To set up the environment for a specific software, use the `module load` command. For example, to load GCC version 12.1.0:
module load gcc/gcc-12.1.0

#List Loaded Modules: To view all currently loaded modules in your session, use:
module list

#Unload a Module: To unload a specific module from your environment, use:
module unload gcc/gcc-12.1.0

#Unload All Modules:** If you need to clear your environment of all loaded modules, use:
module purge

</syntaxhighlight>By using these commands, you can easily manage the software environments needed for different tasks, ensuring compatibility and reducing potential conflicts between software versions.

== Basic Job Submission Commands ==

=== Finding Your Account and Partition ===

Before submitting a job, you need to know which partitions you have permission to use.

Run the command `<code>check_my_partitions</code>` to view a list of all the partitions you have permission to send jobs to.

== Submitting Jobs==
sbatch: Submits a job script for batch processing.

'''Example''':<syntaxhighlight lang="bash">

sbatch --ntasks=1 --time=10 -p power-general -A power-general-users pre_process.bash
# This command submits pre_process.bash to the power-general partition for 10 minutes.
# With 1 GPU:
sbatch --gres=gpu:1 -p gpu-general -A gpu-general-users gpu_job.sh

</syntaxhighlight>

===Writing SLURM Job Scripts===

Here is a simple job script example:

==== Basic Script====
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=my_job # Job name
#SBATCH --account=power-general-users # Account name
#SBATCH --partition=power-general # Partition name
#SBATCH --time=02:00:00 # Max run time (hh:mm:ss)
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --nodes=1 # Number of nodes
#SBATCH --cpus-per-task=1 # CPUs per task
#SBATCH --mem-per-cpu=4G # Memory per CPU
#SBATCH --output=my_job_%j.out # Output file
#SBATCH --error=my_job_%j.err # Error file

echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Your application commands go here
# ./my_program

echo "Job completed"
</syntaxhighlight>

To ask for x cores interactively:
<pre>
srun --ntasks=1 --cpus-per-task=x --partition=power-general --nodes=1 --pty bash
</pre>

However, need for now also to set slurm parameters inside the script, or within the interactive job:
<pre>
export SLURM_TASKS_PER_NODE=48
export SLURM_CPUS_ON_NODE=48
</pre>

For defining an array, may add:
<syntaxhighlight lang="bash">
#SBATCH --array=1-300
</syntaxhighlight>

==== Script for 1 GPU ====
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=gpu_job # Job name
#SBATCH --account=my_account # Account name
#SBATCH --partition=gpu-general # Partition name
#SBATCH --time=02:00:00 # Max run time
#SBATCH --ntasks=1 # Number of tasks
#SBATCH --nodes=1 # Number of nodes
#SBATCH --cpus-per-task=1 # CPUs per task
#SBATCH --gres=gpu:1 # Number of GPUs
#SBATCH --mem-per-cpu=4G # Memory per CPU
#SBATCH --output=my_job_%j.out # Output file
#SBATCH --error=my_job_%j.err # Error file

module load python/python-3.8

echo "Starting GPU job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Your GPU commands go here

echo "Job completed"
</syntaxhighlight>
For excluding a node, one may add the following
<syntaxhighlight>
#SBATCH --exclude=compute-0-[100-103],compute-0-67
</syntaxhighlight>

===Importance of Correct RAM Usage in Jobs===

When writing SLURM job scripts, it's crucial to understand and correctly specify the memory requirements for your job.

Proper memory allocation ensures efficient resource usage and prevents job failures due to out-of-memory (OOM) errors.

==== Why Correct RAM Usage Matters ====

* '''Resource Efficiency''': Allocating the right amount of memory helps in optimal resource utilization, allowing more jobs to run simultaneously on the cluster.
* '''Job Stability''': Underestimating memory requirements can lead to OOM errors, causing your job to fail and waste computational resources.
* '''Performance''': Overestimating memory needs can lead to underutilization of resources, potentially delaying other jobs in the queue.

==== How to Specify Memory in SLURM ====

* '''--mem''': Specifies the total memory required for the job.
* '''--mem-per-cpu''': Specifies the memory required per CPU.

'''Example''':<syntaxhighlight lang="bash">
#SBATCH --mem=4G # Total memory for the job
#SBATCH --mem-per-cpu=2G # Memory per CPU
</syntaxhighlight>

===Interactive Jobs===
<syntaxhighlight lang="bash">
#Start an interactive session:
srun --ntasks=1 -p power-general -A power-general-users --pty bash

#Specify a compute node:
srun --ntasks=1 -p power-general -A power-general-users --nodelist="compute-0-12" --pty bash

#Using GUI:
srun --ntasks=1 -p power-general -A power-general-users --x11 /bin/bash

</syntaxhighlight>

=== Submitting RELION Jobs===

To submit a RELION job interactively on the <code>gpu-relion</code> queue with X11 forwarding, use the following steps:<syntaxhighlight lang="bash">
#Start an interactive session with X11:
srun --ntasks=1 -p gpu-relion -A your_account --x11 --pty bash
#Load the RELION module:
module load relion/relion-4.0.1
#Launch RELION:
relion

</syntaxhighlight>

==Running matlab example==
In this example there are 3 files:

myTable.m ⇒ This matlab file calculates something
<pre>
fprintf('=======================================\n');
fprintf(' a b c d \n');
fprintf('=======================================\n');
while 1
for j = 1:10
a = sin(10*j);
b = a*cos(10*j);
c = a + b;
d = a - b;
fprintf('%+6.5f %+6.5f %+6.5f %+6.5f \n',a,b,c,d);
end
end
fprintf('=======================================\n');
</pre>

my_table_script.sh ⇒ This script executes the matlab program. Need just to run qsub with this script
<pre>
#!/bin/bash

#SBATCH --mem=50mg
#SBATCH --partition powers-general
#SBATCH -A power-general-users
hostname

cd /a/home/cc/tree/taucc/staff/dvory/matlab

matlab -nodisplay -nosplash -nodesktop -r "run(myTable());exit;"
</pre>

run_in_loop.sh ⇒ However, one may also generate many jobs with this file
<pre>
#!/bin/bash

for i in {1..100}

do

sbatch my_table_script.sh

done
</pre>

Running my job is with the command (after doing chmod +x 'run_in_loop.sh'):
<pre>
./run_in_loop.sh
</pre>

==AlphaFold==

AlphaFold is a deep learning tool designed for predicting protein structures.

'''Guides:'''

[https://hpcguide.tau.ac.il/index.php?title=Alphafold AlphaFold Guide]

[https://hpcguide.tau.ac.il/index.php?title=Alphafold3 AlphaFold3 Guide]

==Common SLURM Commands==
<syntaxhighlight lang="bash">
#View all queues (partitions):
sinfo
#View all jobs:
squeue
#View details of a specific job:
scontrol show job <job_number>
#Get information about partitions:
scontrol show partition
</syntaxhighlight>

== Troubleshooting & Tips ==

=== Common Errors ===

# <code>srun: error: Unable to allocate resources: No partition specified or system default partition</code> <br />'''Solution:''' Always specify a partition. Example: <code>srun --pty -c 1 --mem=2G -p power-general /bin/bash</code>
# Job failed, and upon doing scontrol show job job_id or when running sacct -j job_id -o JobID,JobName,State%20 <br />you see: <code>JobState=OUT_OF_MEMORY Reason=OutOfMemory</code> or :<syntaxhighlight lang="bash">
JobID JobName State
------------ ---------- --------------------
71 oom_test OUT_OF_MEMORY
71.batch batch OUT_OF_MEMORY
71.extern extern COMPLETED

</syntaxhighlight>it means that the ram requested for the job was not enough, please resubmit the job again with more ram. see [https://wikihpc.tau.ac.il/index.php?title=Slurm_user_guide#Estimating_RAM_Usage below] for help with understanding how much ram your job may need.

=== Chain Jobs ===
Use the <code>--depend</code> flag to set job dependencies.

'''Example:'''
<syntaxhighlight lang="bash">
sbatch --ntasks=1 --time=60 -p power-general -A power-general-users --depend=45001 do_work.bash
</syntaxhighlight>

=== Always Specify Resources ===
When submitting jobs, ensure you include all required resources like partition, memory, and CPUs to avoid job failures.

=== Attaching to Running Jobs ===
If you need to monitor or interact with a running job, use <code>sattach</code>. This command allows you to attach to a job's input, output, and error streams in real-time.

'''Example:'''
<syntaxhighlight lang="bash">
sattach <job_id>
</syntaxhighlight>

To view job steps of a specific job, use the following command:

<syntaxhighlight lang="bash">
scontrol show job <job_id>
</syntaxhighlight>

Look for sections labeled "StepId" within the output.

'''For specific job steps, use:'''
<syntaxhighlight lang="bash">
sattach <job_id.step_id>
</syntaxhighlight>

'''Note:''' <code>sattach</code> is particularly useful for interactive jobs, where you can provide input directly. For non-interactive jobs, it acts like <code>tail -f</code>, allowing you to monitor the output stream.

=== Estimating RAM Usage ===

When writing SLURM job scripts, it's crucial to understand and correctly specify the memory requirements for your job. Proper memory allocation ensures efficient resource usage and prevents job failures due to out-of-memory (OOM) errors.

==== Tips for Estimating RAM Usage ====

* Check Application Documentation: Refer to the official documentation or user guides for memory-related information.
* Run a Small Test Job: Submit a smaller version of your job and monitor its memory usage using commands like `free -m`, `top`, or `htop`.
* Use Profiling Tools: Tools like `valgrind`, `gprof`, or built-in profilers can help you understand memory usage.
* Analyze Previous Jobs: Review SLURM logs and job statistics for insights into memory consumption of past jobs.
* Consult with Peers or Experts: Ask colleagues or experts who have experience with similar workloads.

==== Example: Monitoring Memory Usage ====
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=memory_test
#SBATCH --account=your_account
#SBATCH --partition=your_partition
#SBATCH --time=01:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --output=memory_test.out
#SBATCH --error=memory_test.err

# Monitor memory usage
echo "Memory usage before running the job:"
free -m

# Your application commands go here
# ./your_application

# Monitor memory usage after running the job
echo "Memory usage after running the job:"
free -m
</syntaxhighlight>

==== General Tips ====

* Start Small: Begin with a conservative memory request and increase it based on observed usage.
* Consider Peak Usage: Plan for peak memory usage to avoid OOM errors.
* Use SLURM's Memory Reporting: Use `sacct` to view memory usage statistics.

'''Example:'''
<syntaxhighlight lang="bash">
sacct -j <job_id> --format=JobID,JobName,MaxRSS,Elapsed
</syntaxhighlight>

Main Page

2023-09-10T08:54:37Z

Eyal:

'''Welcome to HPC Guide.'''

[[Linux basic commands]]

[[Public queues]]

[[Submitting a job to a queue]]

[[Submitting a job to a slurm queue]]

[[PBS-To-SLURM]]

[[Creaing and using conda environment]]

[[Palo Alto VPN for linux]]

[[Alphafold]]

[[Using GPU]]

This HPC Tutorial is designed for researchers at TAU who are in need of computational power (computer resources) and wish to explore and use our High Performance Computing (HPC) core facilities.
<br>
The audience may be completely unaware of the HPC concepts but must have some basic understanding of computers and computer programming.

'''What is HPC?'''

“High Performance Computing” (HPC) is computing on a “Supercomputer”,
<br>
a computer at the front line of contemporary processing capacity – particularly speed of calculation and available memory.
<br>
A computer cluster consists of a set of loosely or tightly connected computers that work together so that in many respects they can be viewed as a single system.
<br>
The components of a cluster are usually connected to each other through fast local area networks(“LAN”) with each node (computer used as a server) running its own instance of an operating system.
<br>
Computer clusters emerged as a result of convergence of a number of computing trends including the availability of low cost microprocessors,
<br>
high-speed networks, and software for high performance distributed computing.
<br>
Compute clusters are usually deployed to improve performance and availability over that of a single computer, while typically being more cost-effective than single computers of comparable speed or availability.

PBS-To-SLURM

2023-09-10T08:52:29Z

Eyal: Created page with " '''Translating PBS Scripts to Slurm Scripts''' The following table contains a list of common commands and terms used with the TORQUE/PBS scheduler, and the corresponding com..."

'''Translating PBS Scripts to Slurm Scripts'''

The following table contains a list of common commands and terms used with the TORQUE/PBS scheduler, and the corresponding commands and terms used under the Slurm scheduler.
<br>
This sheet can be used to assist in translating your existing PBS scripts into Slurm scripts to be read by the new scheduler, or as a reference when creating new Slurm job scripts.
<br>
[[File:User-commands.jpg|frame|left]]
<br>

<br>
[[File:Environment.jpg|frame|left]]
<br>

<br>
[[File:Job-specification.jpg|frame|left]]
<br>

File:Job-specification.jpg

2023-09-10T08:49:41Z

Eyal:

Job-specification

File:Environment.jpg

2023-09-10T08:41:25Z

Eyal:

Environment

File:User-commands.jpg

2023-09-10T07:51:23Z

Eyal:

user-commands

Submitting a job to a queue

2022-12-14T14:34:03Z

Eyal:

Power is a Linux cluster system running CentOS (version 7.3-8). The cluster consists of a single head node (power9), and more than 400 compute nodes (some with 16GB, others with 36GB or even 600GB memory and even more) 16 to 96 cores each. Users belonging to netgroup 'power' can login and run their batch jobs on it.

The Faculty Computer Coordinators can change their netgroup from general to power.

Users’ jobs are executed on the compute nodes (compute-0-0 – compute-0-500) under control of a queuing system (PBSPRO). Users are able to logon to the head node, power, via ssh (where their home directory is mounted from the CC filer, the same as on the other CC servers) and submit their jobs to the batch system.

Power cluster and pbspro queueing system

===PBSPRO main commands===

A good reference can be found in link http://www.pbsworks.com/documentation/support/PBSProUserGuide10.4.pdf

Start with one of the below commands:
<pre>
ssh <username>@power9login.tau.ac.il
ssh powerlogin9.tau.ac.il -l <username>
</pre>
Create a batch job script, for example, file named script that contains the following lines:
<pre>
#!/bin/bash
cd executables
./a.out
</pre>

Send the script to be executed in one of the existing queues, for example, to queue ‘public’:
<pre>
qsub -q public script
</pre>
The number which is returned from this command is the job id that was assigned to the new job:

'''6770818.power.tau.ac.il'''

You can see the status of your executing jobs by executing:
<pre>
qstat -u <username>
</pre>
Which lists all the jobs running or being queued for the specified user.

Job status may be mainly one of the following:

Q – queued (waiting for its run)
R - running
You can see the status of all the executing jobs by executing:

'''qstat'''

To see the current available queues and their cputime and memory limits, execute:
<pre>
qstat –q
</pre>

To see the status of a specific job, you may run:
<pre>
qstat -f <job number>
</pre>
Some of the queues are private, accessible to a predefined group of users, other are public, open to all the users of power.
More detailed information on any queue limits may be viewed by:
<pre>
qmgr -c "list queue <queuename>"
</pre>

For example:
<pre>
qmgr -c "list queue power-general"
</pre>
Default queue limits are enforced unless specified otherwise (up to max values) on 'qsub' command, using flag ‘-l’ (small ‘L’), according the following format:
<pre>
qsub -q <queue> -l<attribute=limit,attribute=limit,.. <script>
</pre>
For example:
<pre>
qsub -q power-dvory -lpmem=2000mb,pvmem=3000mb <script>
qsub -q power-dvory -lmem=14gb,pmem=5gb,vmem=20gb,pvmem=20gb <script>
qsub -q gpu -lngpus=1 <script>
qsub -q public -lselect=1:ncpus=4 <script>
</pre>
While:

'''mem''' - refers to maximum amount of memory to be allocated

'''pmem''' - refers to maximum amount of memory to be allocated per process

'''vmem''' - refers to maximum amount of virtual memory to be allocated

'''pvmem''' - refers to maximum amount of virtual memory to be allocated per process

'''nodes''' - number of required nodes (servers)

'''ppn''' - number of required cores (within a node)

'''ngpus''' - number of required gpus (exists only for queue gpu)
The more updated syntax includes the word 'select'

'''Quesues list'''
'''gpu''' – this queue’s purpose it to enable running jobs, which require gpu processing

'''public''' - queue that is used for public users, who do not pay. They have the lower priority.

'''power-PI-username''' - this queue is used by PI and her/his group. Jobs are directed to one global queue, named power-general

The standard output and standard error files will be written by default at the end of the execution to files in your home directory: script.o#n and script.e#n (where #n is the job number given to your job by the batch queueing system).

To delete a job, use the qdel command:
<pre>
qdel <job number>
</pre>

PBSPRO file parameters
The script to be run may have additional commands which are directions to the scheduler, instead of adding parameters to the qsub command line.

Explanations regarding PBS script directives can be found at: https://www.osc.edu/supercomputing/batch-processing-at-osc/pbs-directives-summary

For example, instead of specifying ‘qsub –q public …’, one may add ‘#PBS –q public’ to the script to be executed. Like in the below script, named ‘script.sh’, which can be run using the command: ‘qsub script.sh’
<pre>
#!/bin/bash
#PBS -l walltime=1:00:00
#PBS -l select=1:ncpus=4,mem=400mb
./my application
</pre>

Running matlab example
In this example there are 3 files:

myTable.m ⇒ This matlab file calculates something
<pre>
function [] = myTable()
fprintf('=======================================\n');
fprintf(' a b c d \n');
fprintf('=======================================\n');
while 1
for j = 1:10
a = sin(10*j);
b = a*cos(10*j);
c = a + b;
d = a - b;
fprintf('%+6.5f %+6.5f %+6.5f %+6.5f \n',a,b,c,d);
end
end
fprintf('=======================================\n');
</pre>

my_table_script.sh ⇒ This script executes the matlab program. Need just to run qsub with this script
<pre>
#!/bin/bash

#PBS -e /tmp/dvory/matlab/output

#PBS -o /tmp/dvory/matlab/output

#PBS -l mem=5000mb

#PBS -q power-dvory

hostname

cd /a/home/cc/tree/taucc/staff/dvory/matlab

matlab -nodisplay -r "myTable()"
</pre>

run_in_loop.sh ⇒ However, one may also generate many jobs with this file
<pre>
#!/bin/bash

for i in {1..100}

do

qsub my_table_script.sh

done
</pre>

Running my job is with the command:
<pre>
./run_in_loop.sh
</pre>

===Interactive session===
Interactive sessions (line mode) are enabled, using flag ‘-I’ (a big ‘i'):
<pre>
qsub -q <queue name> -I
</pre>
(without adding a script name)

Interactive sessions with X window
To enable opening an x window (such as matlab window, or math window)

This may be enabled using the commands below:

Login to power.tau.ac.il with ‘X’:
<pre>
ssh -X -l <username> powerlogin9.tau.ac.il
</pre>
Then use the qsub command with ‘-X’:
<pre>
qsub -I -X -q <queue>
</pre>
(without adding a script name)
Keep in mind that - running matlab via an X window slows the matlab execution.

For the benefit of matlab, need to allocate more memory than is defined in the default public queues, at least the following memory needs to requested:
<pre>
qsub -q power-dvory –lmem=60gb,pmem=60gb,vmem=60gb,pvmem=60gb -I -X
</pre>

===Parallelism===
Parallel jobs can be executed in the cluster - using up to 96 cores (=ppn) for a job. For example, jobs compiled with mpich can be submitted with the following command:
<pre>
qsub -l select=1:ncpus=8 -q paublic <script-filename>
</pre>
Multithreaded matlab jobs can be submitted with the following command:
<pre>
qsub -l select=1:ncpus=8 -q parallel <matlab-script>
</pre>
‘-l’ refers to a small ‘L’

===Environment modules===
The Environment Modules package provides for the dynamic modification of a user’s environment via modulefiles.

Typically modulefiles instruct the module command to alter or set shell environment variables such as PATH, MANPATH, etc. Modules are useful in managing different versions of applications.
Useful commands:
<pre>
module avail
</pre>
⇒ lists the available modules on the system
<pre>
module load <module>
</pre>
e.g.:
<pre>
module load intel/ifort10
</pre>
⇒ loads the appropriate module and enables to use ifort version 10 without specifying the path to its binaries and libraries
<pre>
module list
</pre>
⇒ lists the loaded modules
<pre>
module unload intel/ifort10
</pre>
⇒ unloads the loaded module