Slurm API

2024-04-07T14:01:24Z

Levk: /* Python Example: */

This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.

For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html
= Authentication =

==== Introduction ====
Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.

==== Prerequisites ====

* An API key provided by the High-Performance Computing (HPC) team.

==== Constants ====

* Base URL for the SLURM REST API: <code><nowiki>https://slurmtron.tau.ac.il</nowiki></code>
* Endpoint for token generation: <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code>

==== Python Example for creating a JWT token ====
<syntaxhighlight lang="python3">
import requests

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')
</syntaxhighlight>

==== Manual Token Generation (General Method) ====

# Send a POST request to <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code> with your username and API key.
# The request should be in the format of a JSON payload containing your credentials.
# On success, the server will return a JSON response in the format: <code>{ "SlurmJWT": "token" }</code>.
# Extract the <code>SlurmJWT</code> value from the response. This is your required JWT token.

==== Security and Best Practices ====

* Keep your API key and JWT token confidential.
* Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.

= Job Submission to SLURM REST API: =

=== Introduction ===
This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University

==== Prerequisites ====

* Access to the SLURM REST API.
* An API key and username, provided by the Tel Aviv University HPC team.
* A tool or library for making HTTP requests (e.g., <code>requests</code> in Python).

== Python Example: ==
<syntaxhighlight lang="python3">
#!/usr/bin/python3

import requests

# Base URL for authentication and token generation
base_url_auth = 'https://slurmtron.tau.ac.il'
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"
# Base URL for job submission
base_url = f"{base_url_auth}/slurmrestd"
# Job submission URL
job_url = f'{base_url}/slurm/v0.0.40/job/submit'

# User credentials
current_user = "user"
api_key = "token"

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Authorization headers with the obtained token
headers = {
'X-SLURM-USER-NAME': current_user,
'X-SLURM-USER-TOKEN': get_api_token(current_user, api_key)
}

# Job submission request
jobs_request = requests.post(
job_url,
headers=headers,
json={
# Example job script
"script": "#!/bin/bash\n\n"
"srun hostname\n"
"echo \"hello world444\"\n"
"sleep 30",
"job": {
"partition": "< queue/partition_name >",
"tasks": 1,
"name": "< job_name> ",
"account": "< account_name >",
"nodes": "1",
"cpus_per_task": < cpu_number >,
"memory_per_node": {
"number": <ram in MB >,
"set": True,
"infinite": False
},
# Full path to your error/output file.
"standard_output": "/path/to/your/output.txt",
"standard_error": "/path/to/your/error.txt",
"current_working_directory": "/tmp/",
# Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
"environment": [
"PATH=/bin:/usr/bin/:/usr/local/bin/",
"LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
],
},
}
)

# Processing the job submission result
jobs_result = jobs_request.json()['result']
for key, value in jobs_result.items():
print(key, value)

</syntaxhighlight>
==== Important Notes ====

* The <code>script</code> parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
* Use full and appropriate paths for <code>"standard_output"</code> and <code>"standard_error"</code>. Replace the placeholders with actual paths where you want the output and error files to be stored.
* Environment modules (<code>module load</code>) should not be used directly under the <code>script</code> parameter. Instead, set all necessary environment variables under the <code>environment</code> parameter.

==== Security and Best Practices ====

* Securely handle your API key and other sensitive information.
* Regularly review and update your scripts to align with updates in the SLURM REST API.

'''More Examples''':

https://docs.lxp.lu/cloud/slurmrestd/

Slurm API

2024-04-07T13:57:37Z

Levk: /* Python Example: */

This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.

For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html
= Authentication =

==== Introduction ====
Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.

==== Prerequisites ====

* An API key provided by the High-Performance Computing (HPC) team.

==== Constants ====

* Base URL for the SLURM REST API: <code><nowiki>https://slurmtron.tau.ac.il</nowiki></code>
* Endpoint for token generation: <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code>

==== Python Example for creating a JWT token ====
<syntaxhighlight lang="python3">
import requests

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')
</syntaxhighlight>

==== Manual Token Generation (General Method) ====

# Send a POST request to <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code> with your username and API key.
# The request should be in the format of a JSON payload containing your credentials.
# On success, the server will return a JSON response in the format: <code>{ "SlurmJWT": "token" }</code>.
# Extract the <code>SlurmJWT</code> value from the response. This is your required JWT token.

==== Security and Best Practices ====

* Keep your API key and JWT token confidential.
* Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.

= Job Submission to SLURM REST API: =

=== Introduction ===
This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University

==== Prerequisites ====

* Access to the SLURM REST API.
* An API key and username, provided by the Tel Aviv University HPC team.
* A tool or library for making HTTP requests (e.g., <code>requests</code> in Python).

== Python Example: ==
<syntaxhighlight lang="python3">
#!/usr/bin/python3

import requests

# Base URL for authentication and token generation
base_url_auth = 'https://slurmtron.tau.ac.il'
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"

# User credentials
current_user = "user"
api_key = "token"

import requests

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')

# Base URL for job submission
base_url = f"{base_url_auth}/slurmrestd"

# Job submission URL
job_url = f'{base_url}/slurm/v0.0.40/job/submit'

# Authorization headers with the obtained token
headers = {
'X-SLURM-USER-NAME': current_user,
'X-SLURM-USER-TOKEN': get_api_token(generate_token_url, current_user, api_key)['SlurmJWT']
}

# Job submission request
jobs_request = requests.post(
job_url,
headers=headers,
json={
# Example job script
"script": "#!/bin/bash\n\n"
"srun hostname\n"
"echo \"hello world444\"\n"
"sleep 30",
"job": {
"partition": "< queue/partition_name >",
"tasks": 1,
"name": "< job_name> ",
"account": "< account_name >",
"nodes": "1",
"cpus_per_task": < cpu_number >,
"memory_per_node": {
"number": <ram in MB >,
"set": True,
"infinite": False
},
# Full path to your error/output file.
"standard_output": "/path/to/your/output.txt",
"standard_error": "/path/to/your/error.txt",
"current_working_directory": "/tmp/",
# Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
"environment": [
"PATH=/bin:/usr/bin/:/usr/local/bin/",
"LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
],
},
}
)

# Processing the job submission result
jobs_result = jobs_request.json()['result']
for key, value in jobs_result.items():
print(key, value)

</syntaxhighlight>

==== Important Notes ====

* The <code>script</code> parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
* Use full and appropriate paths for <code>"standard_output"</code> and <code>"standard_error"</code>. Replace the placeholders with actual paths where you want the output and error files to be stored.
* Environment modules (<code>module load</code>) should not be used directly under the <code>script</code> parameter. Instead, set all necessary environment variables under the <code>environment</code> parameter.

==== Security and Best Practices ====

* Securely handle your API key and other sensitive information.
* Regularly review and update your scripts to align with updates in the SLURM REST API.

'''More Examples''':

https://docs.lxp.lu/cloud/slurmrestd/

Submitting a job to a slurm queue

2024-03-07T12:36:29Z

Levk: /* Finding your account and partition */

SLURM (Simple Linux Utility for Resource Management) is a job scheduler used on many high-performance computing systems. It manages and allocates resources such as compute nodes and controls job execution.

=== Accessing the System ===
To submit jobs to the SLURM scheduler at Tel Aviv University, you must access the system through one of the designated login nodes. These nodes act as the gateway for submitting and managing your SLURM jobs. The available login nodes are:

* <code>powerslurm-login.tau.ac.il</code>
* <code>powerslurm-login2.tau.ac.il</code>

==== Login Requirements: ====

# Membership in the "power" group: Ensure you are a part of the "power" group which grants the necessary permissions for accessing the HPC resources.
# University Credentials: Log in using your Tel Aviv University credentials. This ensures secure access and that your job submissions are appropriately accounted for under your user profile.

Remember, these login nodes are the initial point of contact for all your job management tasks, including job submission, monitoring, and other SLURM-related operations.

=== Basic Job Submission Commands ===
====Finding your account and partition====
In order to submit jobs to slurm, one needs to know the accounts and partitions she belongs to. Each account may belong to one or more partitions.

To see the account I belong to, please type:
<pre>
sacctmgr show associations where user=dvory format=Account%20
</pre>
If you know your partition, and would like to know which account you need to specify when using it, please do (on powerslurm-login)
<pre>
check_allowed_account -p <partition>
</pre>
For example:
<pre>
check_allowed_account -p power-general
</pre>

====Example for submitting jobs====
# sbatch: Submit a batch job script.
#* Example: <code>sbatch --ntasks=1 --time=10 -p power-general -A power-general-users pre_process.bash</code>
#* This submits <code>pre_process.bash</code> with 1 task for 10 minutes.
#* Example of chaining jobs: <code>sbatch --ntasks=128 --time=60 -p power-general -A power-general-users --depend=45001 do_work.bash</code>
#* Example with GPU: <code>sbatch --ntasks=1 --time=10 --gres=gpu:2 -p gpu-general -A gpu-general-users pre_process.bash</code> or <code>sbatch --gres=gpu:1 -p gpu-general -A gpu-general-users gpu_job.sh</code>
# salloc: Allocate resources for an interactive job but doesn't start it immediately.
#* Example: <code>salloc --ntasks=8 --time=10 -p power-general -A power-general-users bash</code>
# srun: Submit an interactive job with MPI (Message Passing Interface), often called a "job step."
#* Example: <code>srun --ntasks=2 -p power-general -A power-general-users --label hostname</code>
#* With MPI: <code>srun -intasks=2 -p power-general -A power-general-users--label hostname</code>
# sattach: Attach stdin/out/err to an existing job or job step.

=== Interactive Job Examples ===
* Opening a bash shell: <code>srun --ntasks=56 -p power-general -A power-general-users --pty bash</code>
* Specifying compute nodes: <code>srun --ntasks=56 -p power-general -A power-general-users --nodelist="compute-0-12" --pty bash</code>
* Using a GPU: <code>salloc --ntasks=8 --time=10 --gres=gpu:4 -p gpu-general -A gpu-general-users bash</code>

=== Script Examples: ===
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=my_job # Job name
#SBATCH --account=power-general-users # Account name for billing
#SBATCH --partition=power-general # Partition name
#SBATCH --time=02:00:00 # Time allotted for the job (hh:mm:ss)
#SBATCH --ntasks=4 # Number of tasks (processes)
#SBATCH --cpus-per-task=1 # Number of CPU cores per task
#SBATCH --mem-per-cpu=4G # Memory per CPU core
#SBATCH --output=my_job_%j.out # Standard output and error log (%j expands to jobId)
#SBATCH --error=my_job_%j.err # Separate file for standard error

# Load modules or software if required
# module load python/3.8

# Print some information about the job
echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Run your application, this could be anything from a custom script to standard applications
# ./my_program
# python my_script.py

# End of script
echo "Job completed"

</syntaxhighlight>

==== Script example with GPU ====
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=my_job # Job name
#SBATCH --account=my_account # Account name for billing
#SBATCH --partition=long # Partition name
#SBATCH --time=02:00:00 # Time allotted for the job (hh:mm:ss)
#SBATCH --ntasks=4 # Number of tasks (processes)
#SBATCH --cpus-per-task=1 # Number of CPU cores per task
#SBATCH --gres=gpu:NUMBER_OF_GPUS # number of GPU's to use in the job
#SBATCH --mem-per-cpu=4G # Memory per CPU core
#SBATCH --output=my_job_%j.out # Standard output and error log (%j expands to jobId)
#SBATCH --error=my_job_%j.err # Separate file for standard error

# Load modules or software if required
module load python/3.8

# Print some information about the job
echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Run your application, this could be anything from a custom script to standard applications
# ./my_program
# python my_script.py

# End of script
echo "Job completed"

</syntaxhighlight>

=== Error Handling ===
* On some clusters, specifying resources is necessary. Without them, the job may fail.
** Example error: <code>srun: error: Unable to allocate resources: No partition specified or system default partition</code>
** Correct usage: <code>srun --pty -c 1 --mem=2G -p power-yoren /bin/bash</code>
* Be aware that specifying GPU resources is crucial for jobs

=== SLURM Information Commands ===

* sinfo: View all queues (partitions).
* squeue: View all jobs.
* scontrol show partition: View all partitions.
* scontrol show job <job_number>: View a job's attributes.

=== Tips for Managing SLURM Jobs ===

* Chain jobs by using the <code>--depend</code> flag in <code>sbatch</code>.
* Use <code>salloc</code> for interactive jobs that require specific resources for a limited time.
* <code>srun</code> is versatile for both interactive and batch jobs, especially with MPI.
* Always specify necessary resources in clusters where defaults are not set.

Alphafold

2024-03-04T12:04:00Z

Levk:

==== '''Alphafold''' ====
AlphaFold is an artificial intelligence (AI) program developed by Alphabets's/Google's DeepMind which performs predictions of protein structure.

==== '''Databases:''' ====
Mounted on nodes with gpu, located at /alphafold_storage/alphafold_db.

===== Usage: =====
use run_alphafold.sh script located at /powerapps/share/centos7/alphafold/alphafold-2.3.1/run_alphafold.sh

Script reference:
<code>Required Parameters:
-d <data_dir> Path to directory of supporting data
-o <output_dir> Path to a directory that will store the results.
-f <fasta_paths> Path to FASTA files containing sequences. If a FASTA file contains multiple sequences, then it will be folded as a multimer. To fold more sequences one after another, write the files separated by a comma
-t <max_template_date> Maximum template release date to consider (ISO-8601 format - i.e. YYYY-MM-DD). Important if folding historical test sets
Optional Parameters:
-g <use_gpu> Enable NVIDIA runtime to run with GPUs (default: true)
-r <run_relax> Whether to run the final relaxation step on the predicted models. Turning relax off might result in predictions with distracting stereochemical violations but might help in case you are having issues with the relaxation stage (default: true)
-e <enable_gpu_relax> Run relax on GPU if GPU is enabled (default: true)
-n <openmm_threads> OpenMM threads (default: all available cores)
-a <gpu_devices> Comma separated list of devices to pass to 'CUDA_VISIBLE_DEVICES' (default: 0)
-m <model_preset> Choose preset model configuration - the monomer model, the monomer model with extra ensembling, monomer model with pTM head, or multimer model (default: 'monomer')
-c <db_preset> Choose preset MSA database configuration - smaller genetic database config (reduced_dbs) or full genetic database config (full_dbs) (default: 'full_dbs')
-p <use_precomputed_msas> Whether to read MSAs that have been written to disk. WARNING: This will not check if the sequence, database or configuration have changed (default: 'false')
-l <num_multimer_predictions_per_model> How many predictions (each with a different random seed) will be generated per model. E.g. if this is 2 and there are 5 models then there will be 10 predictions per input. Note: this FLAG only applies if model_preset=multimer (default: 5)
-b <benchmark> Run multiple JAX model evaluations to obtain a timing that excludes the compilation time, which should be more indicative of the time required for inferencing many proteins (default: 'false')</code>

===== Sample Qsub Script: =====
create folder for output in your home dir mkdir ~/alphafold_output then run the script

* you may download dummy_test folder from this github as well for the output

<nowiki>https://github.com/kalininalab/alphafold_non_docker</nowiki>

* /home/alphafold_folder/alphafold_multimer_non_docker/example/query.fasta = this is sample data, please point to the data you need to query.
* The lines '''export CUDA_VISIBLE_DEVICES=$(python3 /powerapps/scripts/check_avail_gpu.py)''' and the flag '''a $CUDA_VISIBLE_DEVICES''' make it so you can use the next free GPU on the server, please leave it as is.
* $ALPHAFOLD_SCRIPT_PATH = /powerapps/share/centos7/alphafold/alphafold-2.3.1/
* $ALPHAFOLD_DB_PATH = /alphafold_storage/alphafold_db
<syntaxhighlight lang="bash">
#!/bin/bash
#PBS -l select=1:ncpus=4:ngpus=1
##choose any gpu queue: gpu/gpu2
#PBS -q gpu2

# Description: AlphaFold-Multimer (Non-Docker) with auto-gpu selection

# load conda env
module load alphafold/alphafold_non_docker_2.3.1

# call to check_available_gpu python script
# returns the param for CUDA_VISIBLE_DEVICE which the run alphafold script uses

export CUDA_VISIBLE_DEVICES=$(python3 /powerapps/scripts/check_avail_gpu.py)
# echo "CUDA_VISIBLE_DEVICES: $CUDA_VISIBLE_DEVICES"
bash $ALPHAFOLD_SCRIPT_PATH/run_alphafold.sh -d $ALPHAFOLD_DB_PATH -o ~/output_dir -f $ALPHAFOLD_SCRIPT_PATH/examples/query.fasta -t 2020-05-14 -a $CUDA_VISIBLE_DEVICES
</syntaxhighlight>

===== Sample Slurm Script =====
<syntaxhighlight lang="bash">
#!/bin/bash
#SBATCH --job-name=AlphaFold-Multimer # Job name
#SBATCH --partition=gpu2 # Specify GPU partition
#SBATCH --nodes=1 # Number of nodes
#SBATCH --ntasks=1 # Number of tasks (processes)
#SBATCH --cpus-per-task=4 # Number of CPU cores per task
#SBATCH --gres=gpu:1 # Request 1 GPU
#SBATCH --output=alphafold_%j.out # Standard output (with job ID)
#SBATCH --error=alphafold_%j.err # Standard error (with job ID)

# Description: AlphaFold-Multimer (Non-Docker) with auto-gpu selection

# Load the required module/environment
module load alphafold/alphafold_non_docker_2.3.1

# Run the AlphaFold script
bash $ALPHAFOLD_SCRIPT_PATH/run_alphafold.sh -d $ALPHAFOLD_DB_PATH -o ~/output_dir -f $ALPHAFOLD_SCRIPT_PATH/examples/query.fasta -t 2020-05-14 -a $CUDA_VISIBLE_DEVICES

</syntaxhighlight>

Submitting a job to a slurm queue

2024-03-04T12:00:14Z

Levk:

SLURM (Simple Linux Utility for Resource Management) is a job scheduler used on many high-performance computing systems. It manages and allocates resources such as compute nodes and controls job execution.

=== Accessing the System ===
To submit jobs to the SLURM scheduler at Tel Aviv University, you must access the system through one of the designated login nodes. These nodes act as the gateway for submitting and managing your SLURM jobs. The available login nodes are:

* <code>powerslurm-login.tau.ac.il</code>
* <code>powerslurm-login2.tau.ac.il</code>

==== Login Requirements: ====

# Membership in the "power" group: Ensure you are a part of the "power" group which grants the necessary permissions for accessing the HPC resources.
# University Credentials: Log in using your Tel Aviv University credentials. This ensures secure access and that your job submissions are appropriately accounted for under your user profile.

Remember, these login nodes are the initial point of contact for all your job management tasks, including job submission, monitoring, and other SLURM-related operations.

=== Basic Job Submission Commands ===

# sbatch: Submit a batch job script.
#* Example: <code>sbatch --ntasks=1 --time=10 pre_process.bash</code>
#* This submits <code>pre_process.bash</code> with 1 task for 10 minutes.
#* Example of chaining jobs: <code>sbatch --ntasks=128 --time=60 --depend=45001 do_work.bash</code>
#* Example with GPU: <code>sbatch --ntasks=1 --time=10 --gres=gpu:2 pre_process.bash</code> or <code>sbatch --gres=gpu:1 gpu_job.sh</code>
# salloc: Allocate resources for an interactive job but doesn't start it immediately.
#* Example: <code>salloc --ntasks=8 --time=10 bash</code>
# srun: Submit an interactive job with MPI (Message Passing Interface), often called a "job step."
#* Example: <code>srun --ntasks=2 --label hostname</code>
#* With MPI: <code>srun -intasks=2 --label hostname</code>
# sattach: Attach stdin/out/err to an existing job or job step.

=== Interactive Job Examples ===
* Opening a bash shell: <code>srun --ntasks=56 --pty bash</code>
* Specifying compute nodes: <code>srun --ntasks=56 -p gcohen_2018 --nodelist="compute-0-12" --pty bash</code>
* Using a GPU: <code>salloc --ntasks=8 --time=10 --gres=gpu:4 bash</code>

=== Script Examples: ===
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=my_job # Job name
#SBATCH --account=my_account # Account name for billing
#SBATCH --partition=long # Partition name
#SBATCH --time=02:00:00 # Time allotted for the job (hh:mm:ss)
#SBATCH --ntasks=4 # Number of tasks (processes)
#SBATCH --cpus-per-task=1 # Number of CPU cores per task
#SBATCH --mem-per-cpu=4G # Memory per CPU core
#SBATCH --output=my_job_%j.out # Standard output and error log (%j expands to jobId)
#SBATCH --error=my_job_%j.err # Separate file for standard error

# Load modules or software if required
# module load python/3.8

# Print some information about the job
echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Run your application, this could be anything from a custom script to standard applications
# ./my_program
# python my_script.py

# End of script
echo "Job completed"

</syntaxhighlight>

==== Script example with GPU ====
<syntaxhighlight lang="bash">
#!/bin/bash

#SBATCH --job-name=my_job # Job name
#SBATCH --account=my_account # Account name for billing
#SBATCH --partition=long # Partition name
#SBATCH --time=02:00:00 # Time allotted for the job (hh:mm:ss)
#SBATCH --ntasks=4 # Number of tasks (processes)
#SBATCH --cpus-per-task=1 # Number of CPU cores per task
#SBATCH --gres=gpu:NUMBER_OF_GPUS # number of GPU's to use in the job
#SBATCH --mem-per-cpu=4G # Memory per CPU core
#SBATCH --output=my_job_%j.out # Standard output and error log (%j expands to jobId)
#SBATCH --error=my_job_%j.err # Separate file for standard error

# Load modules or software if required
module load python/3.8

# Print some information about the job
echo "Starting my SLURM job"
echo "Job ID: $SLURM_JOB_ID"
echo "Running on nodes: $SLURM_JOB_NODELIST"
echo "Allocated CPUs: $SLURM_JOB_CPUS_PER_NODE"

# Run your application, this could be anything from a custom script to standard applications
# ./my_program
# python my_script.py

# End of script
echo "Job completed"

</syntaxhighlight>

=== Error Handling ===
* On some clusters, specifying resources is necessary. Without them, the job may fail.
** Example error: <code>srun: error: Unable to allocate resources: No partition specified or system default partition</code>
** Correct usage: <code>srun --pty -c 1 --mem=2G -p power-yoren /bin/bash</code>
* Be aware that specifying GPU resources is crucial for jobs

=== SLURM Information Commands ===

* sinfo: View all queues (partitions).
* squeue: View all jobs.
* scontrol show partition: View all partitions.
* scontrol show job <job_number>: View a job's attributes.

=== Tips for Managing SLURM Jobs ===

* Chain jobs by using the <code>--depend</code> flag in <code>sbatch</code>.
* Use <code>salloc</code> for interactive jobs that require specific resources for a limited time.
* <code>srun</code> is versatile for both interactive and batch jobs, especially with MPI.
* Always specify necessary resources in clusters where defaults are not set.

Submitting a job to a slurm queue

2024-01-17T14:04:54Z

Levk:

Slurm API

2024-01-11T13:31:46Z

Levk:

This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.

For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html
= Authentication =

==== Introduction ====
Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.

==== Prerequisites ====

* An API key provided by the High-Performance Computing (HPC) team.

==== Constants ====

* Base URL for the SLURM REST API: <code><nowiki>https://slurmtron.tau.ac.il</nowiki></code>
* Endpoint for token generation: <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code>

==== Python Example for creating a JWT token ====
<syntaxhighlight lang="python3">
import requests

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')
</syntaxhighlight>

==== Manual Token Generation (General Method) ====

# Send a POST request to <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code> with your username and API key.
# The request should be in the format of a JSON payload containing your credentials.
# On success, the server will return a JSON response in the format: <code>{ "SlurmJWT": "token" }</code>.
# Extract the <code>SlurmJWT</code> value from the response. This is your required JWT token.

==== Security and Best Practices ====

* Keep your API key and JWT token confidential.
* Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.

= Job Submission to SLURM REST API: =

=== Introduction ===
This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University

==== Prerequisites ====

* Access to the SLURM REST API.
* An API key and username, provided by the Tel Aviv University HPC team.
* A tool or library for making HTTP requests (e.g., <code>requests</code> in Python).

== Python Example: ==
<syntaxhighlight lang="python3">
#!/usr/bin/python3

import requests

# Base URL for authentication and token generation
base_url_auth = 'https://slurmtron.tau.ac.il'
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"

# User credentials
current_user = "user"
api_key = "token"

def get_api_token(url, username, api_key):
"""
Retrieves an API token for SLURM REST API access.

Parameters:
url (str): The URL endpoint for obtaining the API token.
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""
payload = {
'username': username,
'api_key': api_key
}
response = requests.post(url, data=payload)

if response.status_code == 200:
return response.json() # Assuming the token is returned in JSON format
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Base URL for job submission
base_url = "https://slurmtron.tau.ac.il/slurmrestd"

# Job submission URL
job_url = f'{base_url}/slurm/v0.0.40/job/submit'

# Authorization headers with the obtained token
headers = {
'X-SLURM-USER-NAME': current_user,
'X-SLURM-USER-TOKEN': get_api_token(current_user, api_key)['SlurmJWT']
}

# Job submission request
jobs_request = requests.post(
job_url,
headers=headers,
json={
# Example job script
"script": "#!/bin/bash\n\n"
"srun hostname\n"
"echo \"hello world444\"\n"
"sleep 30",
"job": {
"partition": "< queue/partition_name >",
"tasks": 1,
"name": "< job_name> ",
"account": "< account_name >",
"nodes": "1",
"cpus_per_task": < cpu_number >,
"memory_per_node": {
"number": <ram in MB >,
"set": True,
"infinite": False
},
# Full path to your error/output file.
"standard_output": "/path/to/your/output.txt",
"standard_error": "/path/to/your/error.txt",
"current_working_directory": "/tmp/",
# Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
"environment": [
"PATH=/bin:/usr/bin/:/usr/local/bin/",
"LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
],
},
}
)

# Processing the job submission result
jobs_result = jobs_request.json()['result']
for key, value in jobs_result.items():
print(key, value)

</syntaxhighlight>

==== Important Notes ====

* The <code>script</code> parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
* Use full and appropriate paths for <code>"standard_output"</code> and <code>"standard_error"</code>. Replace the placeholders with actual paths where you want the output and error files to be stored.
* Environment modules (<code>module load</code>) should not be used directly under the <code>script</code> parameter. Instead, set all necessary environment variables under the <code>environment</code> parameter.

==== Security and Best Practices ====

* Securely handle your API key and other sensitive information.
* Regularly review and update your scripts to align with updates in the SLURM REST API.

'''More Examples''':

https://docs.lxp.lu/cloud/slurmrestd/

Slurm API

2024-01-09T11:47:28Z

Levk:

This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.

For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html
= Authentication =

==== Introduction ====
Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.

==== Prerequisites ====

* An API key provided by the High-Performance Computing (HPC) team.

==== Constants ====

* Base URL for the SLURM REST API: <code><nowiki>https://slurmtron.tau.ac.il</nowiki></code>
* Endpoint for token generation: <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code>

==== Python Example for creating a JWT token ====
<syntaxhighlight lang="python3">
import requests

def get_api_token(username, api_key):
"""
Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

Parameters:
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""

generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

payload = {
'username': username,
'api_key': api_key
}

response = requests.post(generate_token_url, data=payload)

if response.status_code == 200:
# Extracting the token from the JSON response
return response.json()['SlurmJWT']
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')
</syntaxhighlight>

==== Manual Token Generation (General Method) ====

# Send a POST request to <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code> with your username and API key.
# The request should be in the format of a JSON payload containing your credentials.
# On success, the server will return a JSON response in the format: <code>{ "SlurmJWT": "token" }</code>.
# Extract the <code>SlurmJWT</code> value from the response. This is your required JWT token.

==== Security and Best Practices ====

* Keep your API key and JWT token confidential.
* Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.

= Job Submission to SLURM REST API: =

=== Introduction ===
This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University

==== Prerequisites ====

* Access to the SLURM REST API.
* An API key and username, provided by the Tel Aviv University HPC team.
* A tool or library for making HTTP requests (e.g., <code>requests</code> in Python).

== Python Example: ==
<syntaxhighlight lang="python3">
#!/usr/bin/python3

import requests

# Base URL for authentication and token generation
base_url_auth = 'https://slurmtron.tau.ac.il'
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"

# User credentials
current_user = "user"
api_key = "token"

def get_api_token(url, username, api_key):
"""
Retrieves an API token for SLURM REST API access.

Parameters:
url (str): The URL endpoint for obtaining the API token.
username (str): The username of the user requesting the token.
api_key (str): The API key provided by the HPC team.

Returns:
str: The API token if the request is successful.

Raises:
Exception: If the request fails with a non-200 status code.
"""
payload = {
'username': username,
'api_key': api_key
}
response = requests.post(url, data=payload)

if response.status_code == 200:
return response.json() # Assuming the token is returned in JSON format
else:
raise Exception(f"Error: {response.status_code}, {response.text}")

# Base URL for job submission
base_url = "https://slurmtron.tau.ac.il/slurmrestd"

# Job submission URL
job_url = f'{base_url}/slurm/v0.0.40/job/submit'

# Authorization headers with the obtained token
headers = {
'X-SLURM-USER-NAME': current_user,
'X-SLURM-USER-TOKEN': get_api_token(generate_token_url, current_user, api_key)['SlurmJWT']
}

# Job submission request
jobs_request = requests.post(
job_url,
headers=headers,
json={
# Example job script
"script": "#!/bin/bash\n\n"
"srun hostname\n"
"echo \"hello world444\"\n"
"sleep 30",
"job": {
"partition": "< queue/partition_name >",
"tasks": 1,
"name": "< job_name> ",
"account": "< account_name >",
"nodes": "1",
"cpus_per_task": < cpu_number >,
"memory_per_node": {
"number": <ram in MB >,
"set": True,
"infinite": False
},
# Full path to your error/output file.
"standard_output": "/path/to/your/output.txt",
"standard_error": "/path/to/your/error.txt",
"current_working_directory": "/tmp/",
# Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
"environment": [
"PATH=/bin:/usr/bin/:/usr/local/bin/",
"LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
],
},
}
)

# Processing the job submission result
jobs_result = jobs_request.json()['result']
for key, value in jobs_result.items():
print(key, value)

</syntaxhighlight>

==== Important Notes ====

* The <code>script</code> parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
* Use full and appropriate paths for <code>"standard_output"</code> and <code>"standard_error"</code>. Replace the placeholders with actual paths where you want the output and error files to be stored.
* Environment modules (<code>module load</code>) should not be used directly under the <code>script</code> parameter. Instead, set all necessary environment variables under the <code>environment</code> parameter.

==== Security and Best Practices ====

* Securely handle your API key and other sensitive information.
* Regularly review and update your scripts to align with updates in the SLURM REST API.

'''More Examples''':

https://docs.lxp.lu/cloud/slurmrestd/

Slurm API

2023-11-30T09:24:06Z

Levk: Created page with "This page describes how to connect and use slurm API in order to submit a job in powerslurm cluster, Including a job originating from a web site Official Documentation: https..."

This page describes how to connect and use slurm API in order to submit a job in powerslurm cluster, Including a job originating from a web site

Official Documentation: https://slurm.schedmd.com/rest_api.html
= Authentication =
In-order to authenticate against the API, you need to use your TAU username and a JWT token.

Tokens can only be created on Login nodes, such as powerslurm-login.

Python example for token creation<syntaxhighlight lang="python3">
#!/usr/bin/env python3

import requests
from jwt import JWT
from jwt.jwa import HS256
from jwt.jwk import jwk_from_dict
from jwt.utils import b64encode
import time
import getpass

def generate_jwt_token(expiration_time=60):
# Get the currently logged in user
current_user = getpass.getuser()
with open("/var/spool/slurm/statesave/jwt_hs256.key", "rb") as f:
priv_key = f.read()
signing_key = jwk_from_dict({
'kty': 'oct',
'k': b64encode(priv_key)
})
message = {
"exp": int(time.time() + expiration_time),
"iat": int(time.time()),
"sun": current_user
}
jwt_instance = JWT()
compact_jws = jwt_instance.encode(message, signing_key, alg='HS256')
return compact_jws

# example output:
print(generate_jwt_token())
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOiAxNjk5NTMyNTkwLCAiaWF0IjogMTY5OTUzMjUzMCwgInN1biI6ICJsZXZrIn0.YCxJohapkovR16TQ75DsO3G9ODcisoSeOVbAYwA4Q7E
</syntaxhighlight>

= Submit Job: =
The base url for the API is: <nowiki>https://slurmtron.tau.ac.il</nowiki>

Job submissions url is: <nowiki>https://slurmtron.tau.ac.il/slurm/v0.0.39/job/submit</nowiki>

== Python Example: ==
<syntaxhighlight lang="python3">
#!/usr/bin/python3

import requests
from jwt import JWT
from jwt.jwa import HS256
from jwt.jwk import jwk_from_dict
from jwt.utils import b64encode
import time
import getpass

current_user = getpass.getuser()
def generate_jwt_token(expiration_time=60):
with open("/var/spool/slurm/statesave/jwt_hs256.key", "rb") as f:
priv_key = f.read()
signing_key = jwk_from_dict({
'kty': 'oct',
'k': b64encode(priv_key)
})
message = {
"exp": int(time.time() + expiration_time),
"iat": int(time.time()),
"sun": current_user
}
jwt_instance = JWT()
compact_jws = jwt_instance.encode(message, signing_key, alg='HS256')
return compact_jws

# api url
base_url = "https://slurmtron.tau.ac.il"
# auth token
jwt_token = generate_jwt_token()

# job submission url
job_url = f'{base_url}/slurm/v0.0.39/job/submit'
# Auth Headers
headers = {
'X-SLURM-USER-NAME': current_user,
'X-SLURM-USER-TOKEN': jwt_token
}

# the job request
jobs_request = requests.post(
job_url,
headers=headers,
json={
"script": "#!/bin/bash\n\n"
"srun hostname\n"
"echo \"hello world444\"\n"
"sleep 30",
"job": {

"partition": "Partition_Name",
"tasks": 1,
"name": "test",
"account": "Slurm_Account_Name",
"nodes": "1",
# how much CPU you need
"cpus_per_task": 2,
# How much Memory you need per node, in MB
"memory_per_node": {
"number": 2048,
"set": False,
"infinite": True
},
# List of nodes where the job must be allocated (uncomment the below 3 lines, and specify node name)
# "required_nodes": [
# "Node_Name"
# ],
"standard_input": "/dev/null",
"standard_output": "FULL_PATH_TO_OUTPUT_FILE",
"standard_error": "FULL_PATH_TO_INPUT_FILE",
"environment": [
"PATH=/bin:/usr/bin/:/usr/local/bin/",
"LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
],
},

})
jobs_result = jobs_request.json()
</syntaxhighlight>'''More Examples''':

https://docs.lxp.lu/cloud/slurmrestd/

2022-07-19T11:32:04Z

Levk:

Palo Alto VPN for linux

2022-07-19T11:30:28Z

Levk:

For security reason TelAviv University starts a VPN with double authentication standard.

In order to do that users have to check/fill in their mobile phone at myTAU page
(https://mytau.tau.ac.il/GetResource.php) and enroll to the service.
Then you need install GoogleAuthenticator on you mobile device and register it at TAU.

After that you may download and install PaloAlto GlobalProtect VPN client on your device (all
operation systems are supported: IOS, Android, Linux MAC and even Window)

The steps:
==Enrollment==
Go to https://mytau.tau.ac.il/GetResource.php

Choose the “1” then “2” :

Then you will receive SMS with 2-minute code and enter it immediately to the filed:
Then you will be redirected to the QR code for GoogleAuthenticator account setup:
Scan it using your mobile Google Authenticator app using “+” on bottom right corner of mobile device
and enter the generated code from mobile GoogleAuthenticator to the field and press the green button.

==Download==
Download and install VPN client, from the browser, go to:

If within vpn tunnel, download one of the below versions:
<pre>
http://hpc-tftp.tau.ac.il/public_files/PanGPLinux-5.3.1-c9.tgz
http://hpc-tftp.tau.ac.il/public_files/PanGPLinux-6.0.0-c18.tgz
</pre>
If without vpn tunnel, may download one of the below file
<pre>
https://www.tau.ac.il/~danny/vpn/PanGPLinux-5.3.1-c9.tgz
https://www.tau.ac.il/~danny/vpn/PanGPLinux-5.3.2-c3.tgz
https://www.tau.ac.il/~danny/vpn/PanGPLinux-6.0.0-c18.tgz
</pre>
Linux package should be extracted and installed appropriated version:

'''Debian/Ubuntu'''
<pre>
dpkg -i GlobalProtect_UI_deb-5.3.1.0-36.deb
</pre>
'''Redhat/Centos'''
<pre>
yum localinstall GlobalProtect_UI_rpm-5.3.1.0-36.rpm
</pre>

==Configure==

[[File:Paloalto3.PNG|thumb|right]]

Execute and configure VPN client on Linux (another OS are similar) :

Open client by pressing on the relevant icon ("1" as in the picture on the right)

And enter address '''vpn.tau.ac.il''' ("2" as in the picture on the right)

==Error==
===SSL Error===
On latest ubuntu version, ubuntu 22.04, after installing and configuring globalprotect VPN, you get this error:

[[File:Vpn ssl error.png|none|thumb]]

here is how to workaround it:

open <code>/usr/lib/ssl/openssl.cnf</code>

comment out this section:<syntaxhighlight lang="bash">
# [openssl_init]

# providers = provider_sect
</syntaxhighlight>add this new section under the commented one from earlier:<syntaxhighlight lang="bash">
[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation

</syntaxhighlight>reboot globalprotect app and the error should be fixed.

<blockquote>source:https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1960268</blockquote>

==TAU credentials==
[[File:Paloalto4.PNG|thumb|right]]
Fill in pop-upped windows with your TAU credentials:

Open your mobile GoogleAuthenticator and enter code from there

Congratulations: you are done!