Difference between revisions of "Slurm API"

From HPC Guide
Jump to navigation Jump to search
(Created page with "This page describes how to connect and use slurm API in order to submit a job in powerslurm cluster, Including a job originating from a web site Official Documentation: https...")
 
Line 1: Line 1:
This page describes how to connect and use slurm API in order to submit a job in powerslurm cluster, Including a job originating from a web site
+
This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.
  
Official Documentation: https://slurm.schedmd.com/rest_api.html
+
For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html
 
= Authentication =
 
= Authentication =
In-order to authenticate against the API, you need to use your TAU username and a JWT token.
 
  
Tokens can only be created on Login nodes, such as powerslurm-login.
+
==== Introduction ====
 +
Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.
  
Python example for token creation<syntaxhighlight lang="python3">
+
==== Prerequisites ====
#!/usr/bin/env python3
 
  
 +
* An API key provided by the High-Performance Computing (HPC) team.
 +
 +
==== Constants ====
 +
 +
* Base URL for the SLURM REST API: <code><nowiki>https://slurmtron.tau.ac.il</nowiki></code>
 +
* Endpoint for token generation: <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code>
 +
 +
==== Python Example for creating a JWT token ====
 +
<syntaxhighlight lang="python3">
 
import requests
 
import requests
from jwt import JWT
 
from jwt.jwa import HS256
 
from jwt.jwk import jwk_from_dict
 
from jwt.utils import b64encode
 
import time
 
import getpass
 
  
def generate_jwt_token(expiration_time=60):
+
def get_api_token(username, api_key):
     # Get the currently logged in user
+
     """
     current_user = getpass.getuser()
+
    Retrieves a JWT token for SLURM REST API access for powerslurm cluster.
     with open("/var/spool/slurm/statesave/jwt_hs256.key", "rb") as f:
+
 
        priv_key = f.read()
+
    Parameters:
     signing_key = jwk_from_dict({
+
    username (str): The username of the user requesting the token.
        'kty': 'oct',
+
     api_key (str): The API key provided by the HPC team.
        'k': b64encode(priv_key)
+
 
     })
+
     Returns:
    message = {
+
    str: The API token if the request is successful.
         "exp": int(time.time() + expiration_time),
+
 
         "iat": int(time.time()),
+
    Raises:
        "sun": current_user
+
    Exception: If the request fails with a non-200 status code.
 +
    """
 +
 
 +
     generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'
 +
 
 +
     payload = {
 +
         'username': username,
 +
         'api_key': api_key
 
     }
 
     }
     jwt_instance = JWT()
+
      
     compact_jws = jwt_instance.encode(message, signing_key, alg='HS256')
+
    response = requests.post(generate_token_url, data=payload)
    return compact_jws
+
 
 +
     if response.status_code == 200:
 +
        # Extracting the token from the JSON response
 +
        return response.json()['SlurmJWT']
 +
    else:
 +
        raise Exception(f"Error: {response.status_code}, {response.text}")
 +
 
 +
# Example usage
 +
# Replace 'your_username' and 'your_api_key' with actual values
 +
# token = get_api_token('your_username', 'your_api_key')
 +
</syntaxhighlight>
 +
 
 +
==== Manual Token Generation (General Method) ====
 +
 
 +
# Send a POST request to <code><nowiki>https://slurmtron.tau.ac.il/slurmapi/generate-token/</nowiki></code> with your username and API key.
 +
# The request should be in the format of a JSON payload containing your credentials.
 +
# On success, the server will return a JSON response in the format: <code>{ "SlurmJWT": "token" }</code>.
 +
# Extract the <code>SlurmJWT</code> value from the response. This is your required JWT token.
 +
 
 +
==== Security and Best Practices ====
  
 +
* Keep your API key and JWT token confidential.
 +
* Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.
  
 +
= Job Submission to SLURM REST API: =
  
# example output:
+
=== Introduction ===
print(generate_jwt_token())
+
This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOiAxNjk5NTMyNTkwLCAiaWF0IjogMTY5OTUzMjUzMCwgInN1biI6ICJsZXZrIn0.YCxJohapkovR16TQ75DsO3G9ODcisoSeOVbAYwA4Q7E
 
</syntaxhighlight>
 
  
= Submit Job: =
+
==== Prerequisites ====
The base url for the API is: <nowiki>https://slurmtron.tau.ac.il</nowiki>
 
  
Job submissions url is: <nowiki>https://slurmtron.tau.ac.il/slurm/v0.0.39/job/submit</nowiki>
+
* Access to the SLURM REST API.
 +
* An API key and username, provided by the Tel Aviv University HPC team.
 +
* A tool or library for making HTTP requests (e.g., <code>requests</code> in Python).
  
 
== Python Example: ==
 
== Python Example: ==
Line 53: Line 83:
  
 
import requests
 
import requests
from jwt import JWT
 
from jwt.jwa import HS256
 
from jwt.jwk import jwk_from_dict
 
from jwt.utils import b64encode
 
import time
 
import getpass
 
  
current_user = getpass.getuser()
+
# Base URL for authentication and token generation
def generate_jwt_token(expiration_time=60):
+
base_url_auth = 'https://slurmtron.tau.ac.il'
    with open("/var/spool/slurm/statesave/jwt_hs256.key", "rb") as f:
+
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"
        priv_key = f.read()
+
 
     signing_key = jwk_from_dict({
+
# User credentials
        'kty': 'oct',
+
current_user = "user"
        'k': b64encode(priv_key)
+
api_key = "token"
     })
+
 
     message = {
+
def get_api_token(url, username, api_key):
        "exp": int(time.time() + expiration_time),
+
    """
        "iat": int(time.time()),
+
    Retrieves an API token for SLURM REST API access.
         "sun": current_user
+
 
 +
    Parameters:
 +
    url (str): The URL endpoint for obtaining the API token.
 +
     username (str): The username of the user requesting the token.
 +
    api_key (str): The API key provided by the HPC team.
 +
 
 +
    Returns:
 +
     str: The API token if the request is successful.
 +
 
 +
     Raises:
 +
    Exception: If the request fails with a non-200 status code.
 +
    """
 +
    payload = {
 +
        'username': username,
 +
         'api_key': api_key
 
     }
 
     }
     jwt_instance = JWT()
+
     response = requests.post(url, data=payload)
     compact_jws = jwt_instance.encode(message, signing_key, alg='HS256')
+
 
     return compact_jws
+
     if response.status_code == 200:
 +
        return response.json() # Assuming the token is returned in JSON format
 +
     else:
 +
        raise Exception(f"Error: {response.status_code}, {response.text}")
  
 +
# Base URL for job submission
 +
base_url = "https://slurmtron.tau.ac.il/slurmrestd"
  
# api url
+
# Job submission URL
base_url = "https://slurmtron.tau.ac.il"
+
job_url = f'{base_url}/slurm/v0.0.40/job/submit'
# auth token
 
jwt_token = generate_jwt_token()
 
  
# job submission url
+
# Authorization headers with the obtained token
job_url = f'{base_url}/slurm/v0.0.39/job/submit'
 
# Auth Headers
 
 
headers = {
 
headers = {
 
     'X-SLURM-USER-NAME': current_user,
 
     'X-SLURM-USER-NAME': current_user,
     'X-SLURM-USER-TOKEN': jwt_token
+
     'X-SLURM-USER-TOKEN': get_api_token(generate_token_url, current_user, api_key)['SlurmJWT']
 
}
 
}
  
# the job request
+
# Job submission request
 
jobs_request = requests.post(
 
jobs_request = requests.post(
 
     job_url,
 
     job_url,
 
     headers=headers,
 
     headers=headers,
 
     json={
 
     json={
 +
        # Example job script
 
         "script": "#!/bin/bash\n\n"
 
         "script": "#!/bin/bash\n\n"
 
                   "srun hostname\n"
 
                   "srun hostname\n"
Line 101: Line 141:
 
                   "sleep 30",
 
                   "sleep 30",
 
         "job": {
 
         "job": {
 
+
             "partition": "< queue/partition_name >",
             "partition": "Partition_Name",
 
 
             "tasks": 1,
 
             "tasks": 1,
             "name": "test",
+
             "name": "< job_name> ",
             "account": "Slurm_Account_Name",
+
             "account": "< account_name >",
 
             "nodes": "1",
 
             "nodes": "1",
# how much CPU you need
+
             "cpus_per_task": < cpu_number >,
             "cpus_per_task": 2,
 
# How much Memory you need per node, in MB
 
 
             "memory_per_node": {
 
             "memory_per_node": {
                 "number": 2048,
+
                 "number": <ram in MB >,
                 "set": False,
+
                 "set": True,
                 "infinite": True
+
                 "infinite": False
              },
+
            },
# List of nodes where the job must be allocated (uncomment the below 3 lines, and specify node name)
+
            # Full path to your error/output file.
             # "required_nodes":
+
             "standard_output": "/path/to/your/output.txt",
            #    "Node_Name"
+
             "standard_error": "/path/to/your/error.txt",
            # ],
+
             "current_working_directory": "/tmp/",
             "standard_input": "/dev/null",
+
             # Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
             "standard_output": "FULL_PATH_TO_OUTPUT_FILE",
 
             "standard_error": "FULL_PATH_TO_INPUT_FILE",
 
 
             "environment": [
 
             "environment": [
 
                 "PATH=/bin:/usr/bin/:/usr/local/bin/",
 
                 "PATH=/bin:/usr/bin/:/usr/local/bin/",
Line 127: Line 162:
 
             ],
 
             ],
 
         },
 
         },
 +
    }
 +
)
 +
 +
# Processing the job submission result
 +
jobs_result = jobs_request.json()['result']
 +
for key, value in jobs_result.items():
 +
    print(key, value)
 +
 +
</syntaxhighlight>
 +
 +
==== Important Notes ====
 +
 +
* The <code>script</code> parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
 +
* Use full and appropriate paths for <code>"standard_output"</code> and <code>"standard_error"</code>. Replace the placeholders with actual paths where you want the output and error files to be stored.
 +
* Environment modules (<code>module load</code>) should not be used directly under the <code>script</code> parameter. Instead, set all necessary environment variables under the <code>environment</code> parameter.
 +
 +
==== Security and Best Practices ====
 +
 +
* Securely handle your API key and other sensitive information.
 +
* Regularly review and update your scripts to align with updates in the SLURM REST API.
 +
  
    })
+
'''More Examples''':
jobs_result = jobs_request.json()
 
</syntaxhighlight>'''More Examples''':
 
  
 
https://docs.lxp.lu/cloud/slurmrestd/
 
https://docs.lxp.lu/cloud/slurmrestd/

Revision as of 11:47, 9 January 2024

This documentation provides comprehensive guidance on interfacing with the SLURM API for job submission within the PowerSlurm cluster. These instructions can be used for submitting jobs that originate from a web interface.

For a detailed understanding of the API's capabilities and functionalities, refer to the official SLURM documentation at SLURM REST API Documentation. https://slurm.schedmd.com/rest_api.html

Authentication

Introduction

Secure access to the SLURM REST API is managed through JWT (JSON Web Tokens). This section provides a step-by-step guide on how to obtain a JWT token, which is essential for authenticating and authorizing API requests.

Prerequisites

  • An API key provided by the High-Performance Computing (HPC) team.

Constants

  • Base URL for the SLURM REST API: https://slurmtron.tau.ac.il
  • Endpoint for token generation: https://slurmtron.tau.ac.il/slurmapi/generate-token/

Python Example for creating a JWT token

import requests

def get_api_token(username, api_key):
    """
    Retrieves a JWT token for SLURM REST API access for powerslurm cluster.

    Parameters:
    username (str): The username of the user requesting the token.
    api_key (str): The API key provided by the HPC team.

    Returns:
    str: The API token if the request is successful.

    Raises:
    Exception: If the request fails with a non-200 status code.
    """

    generate_token_url = 'https://slurmtron.tau.ac.il/slurmapi/generate-token/'

    payload = {
        'username': username,
        'api_key': api_key
    }
    
    response = requests.post(generate_token_url, data=payload)

    if response.status_code == 200:
        # Extracting the token from the JSON response
        return response.json()['SlurmJWT']
    else:
        raise Exception(f"Error: {response.status_code}, {response.text}")

# Example usage
# Replace 'your_username' and 'your_api_key' with actual values
# token = get_api_token('your_username', 'your_api_key')

Manual Token Generation (General Method)

  1. Send a POST request to https://slurmtron.tau.ac.il/slurmapi/generate-token/ with your username and API key.
  2. The request should be in the format of a JSON payload containing your credentials.
  3. On success, the server will return a JSON response in the format: { "SlurmJWT": "token" }.
  4. Extract the SlurmJWT value from the response. This is your required JWT token.

Security and Best Practices

  • Keep your API key and JWT token confidential.
  • Use the JWT token in the header of your API requests for authorized access to the SLURM REST API.

Job Submission to SLURM REST API:

Introduction

This section covers the process of submitting a job to the SLURM REST API at Tel Aviv University

Prerequisites

  • Access to the SLURM REST API.
  • An API key and username, provided by the Tel Aviv University HPC team.
  • A tool or library for making HTTP requests (e.g., requests in Python).

Python Example:

#!/usr/bin/python3

import requests

# Base URL for authentication and token generation
base_url_auth = 'https://slurmtron.tau.ac.il'
generate_token_url = f"{base_url_auth}/slurmapi/generate-token/"

# User credentials
current_user = "user"
api_key = "token"

def get_api_token(url, username, api_key):
    """
    Retrieves an API token for SLURM REST API access.

    Parameters:
    url (str): The URL endpoint for obtaining the API token.
    username (str): The username of the user requesting the token.
    api_key (str): The API key provided by the HPC team.

    Returns:
    str: The API token if the request is successful.

    Raises:
    Exception: If the request fails with a non-200 status code.
    """
    payload = {
        'username': username,
        'api_key': api_key
    }
    response = requests.post(url, data=payload)

    if response.status_code == 200:
        return response.json()  # Assuming the token is returned in JSON format
    else:
        raise Exception(f"Error: {response.status_code}, {response.text}")

# Base URL for job submission
base_url = "https://slurmtron.tau.ac.il/slurmrestd"

# Job submission URL
job_url = f'{base_url}/slurm/v0.0.40/job/submit'

# Authorization headers with the obtained token
headers = {
    'X-SLURM-USER-NAME': current_user,
    'X-SLURM-USER-TOKEN': get_api_token(generate_token_url, current_user, api_key)['SlurmJWT']
}

# Job submission request
jobs_request = requests.post(
    job_url,
    headers=headers,
    json={
        # Example job script
        "script": "#!/bin/bash\n\n"
                  "srun hostname\n"
                  "echo \"hello world444\"\n"
                  "sleep 30",
        "job": {
            "partition": "< queue/partition_name >",
            "tasks": 1,
            "name": "< job_name> ",
            "account": "< account_name >",
            "nodes": "1",
            "cpus_per_task": < cpu_number >,
            "memory_per_node": {
                "number": <ram in MB >,
                "set": True,
                "infinite": False
            },
            # Full path to your error/output file.
            "standard_output": "/path/to/your/output.txt",
            "standard_error": "/path/to/your/error.txt",
            "current_working_directory": "/tmp/",
            # Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.
            "environment": [
                "PATH=/bin:/usr/bin/:/usr/local/bin/",
                "LD_LIBRARY_PATH=/lib/:/lib64/:/usr/local/lib"
            ],
        },
    }
)

# Processing the job submission result
jobs_result = jobs_request.json()['result']
for key, value in jobs_result.items():
    print(key, value)

Important Notes

  • The script parameter in the job submission request is an example. Customize this script to fit your specific job requirements.
  • Use full and appropriate paths for "standard_output" and "standard_error". Replace the placeholders with actual paths where you want the output and error files to be stored.
  • Environment modules (module load) should not be used directly under the script parameter. Instead, set all necessary environment variables under the environment parameter.

Security and Best Practices

  • Securely handle your API key and other sensitive information.
  • Regularly review and update your scripts to align with updates in the SLURM REST API.


More Examples:

https://docs.lxp.lu/cloud/slurmrestd/