Submit jobs with Slurm

1. Before you start

This guide is public and can be read without signing in. Running jobs still requires your own workstation account and the SSH connection details supplied by the administrator. Public access to this page does not grant access to the workstation.

This guide is for ENCELDREN042343 (RTX A6000). Your laptop → login node → Slurm allocates resources → your program runs on the compute node. Its login and compute nodes share your home directory; accounts and files do not automatically sync to other workstations.

New accounts: email Bo directly at bshang@umd.edu with your name, UMD email, preferred username, server ENCELDREN042343 and SSH public key. Do not Reply all. Wait for your account confirmation before logging in. CERRA website accounts and workstation accounts are separate.

Run this on your own computer. The server address is already filled in; replace YOUR_USERNAME with your assigned username:

ssh -p 2222 YOUR_USERNAME@10.229.55.207

Use the approved network route in your account confirmation. On first login, check the host fingerprint for this TCP 2222 entry; different ports may use different keys. Contact Bo if it differs.

The login node shares 2 virtual CPUs and 4 GiB of RAM among its users and has no GPU. Use it to edit files, transfer data and submit or check jobs. Small shell commands can run there, but all research computation belongs in Slurm. Starting Python with nohup or tmux on the login node does not request compute resources.

Gateway accounts cannot use an ordinary direct login to the compute host. Existing workstation accounts have not automatically been migrated to this gateway policy. The workflow below is for accounts using the login gateway.

Use sbatch for long or unattended jobs. For interactive work, start srun --pty on the login node and wait for the allocated compute-node shell before running Python. No second SSH connection is needed. Start with “Request resources” below; the later sections cover batch jobs and Python environment setup.

Within the same workstation, your home directory is shared between its login and compute nodes. Keep scripts, data and environments under $HOME, for example ~/project and ~/venvs/research. Each node has separate /tmp and system software; the other workstation does not automatically receive these files.

Both CPU-only and GPU jobs use the gpu partition. Request --gres=gpu:1 only for a job that needs the GPU. Except for SSH and file-transfer commands explicitly labeled for your laptop, start the commands below on the login node.

All jobs on this server share the following resources in the gpu partition. These totals are not a separate allowance for each user:

Specify --mem and --time explicitly. More CPUs help only when your program uses them; Slurm resource limits do not cap watts or disable CPU boost.

This page uses the workstation’s recorded configuration. Follow the network arrangements supplied by the administrator; VPN connectivity depends on the access rules. If a connection times out, send Bo the attempt time and exact error.

On the login node, use sinfo and scontrol show partition gpu to check current availability and limits.

2. Request an account and share your SSH public key

If you already have an SSH key, send its .pub file or the complete public-key line. Otherwise run this on the computer you will connect from, replacing the email address. Do not overwrite an existing key:

ssh-keygen -t ed25519 -C "yourname@umd.edu"

For the default filename, display the public key in a macOS/Linux terminal or Windows PowerShell:

cat ~/.ssh/id_ed25519.pub

Send only the public key directly to bshang@umd.edu. The file without .pub is your private key; keep it and its passphrase on your own computer. For a non-default key filename, add -i with its private-key path to SSH and scp; do not send that private key to the administrator.

Use the TCP 2222 host fingerprint supplied in your account confirmation. A verified fingerprint for this entry is not listed here; do not substitute a key from a different server or port.

3. Request resources, then run your program

Logging in gives you a shell on the login node; it does not reserve compute resources. For interactive work, follow the steps below. For batch jobs, the #SBATCH lines in your script are the resource request: submit with sbatch from the login node and Slurm will queue, allocate and run it automatically. You do not need an interactive allocation before sbatch.

Step 1 — Request a compute shell from the login node. After SSH login, run:

srun --partition=gpu --nodes=1 --ntasks=1 \
  --cpus-per-task=1 --mem=1G --time=00:05:00 --pty bash

This requests one CPU, 1 GiB of total system RAM and up to five minutes, with no GPU. --partition=gpu selects the queue; --cpus-per-task sets the CPU count, --mem sets system RAM, and --time sets the time limit. --pty bash starts an interactive shell on the allocated compute node. Adjust the amounts for your real workload.

Wait for the new shell prompt before entering the next commands. If resources are busy, Slurm may say the job is queued and waiting for resources. Keep this terminal open; a second login terminal can run squeue -u "$USER" to show the job and its wait reason. Ctrl+C cancels a request that is still waiting.

Step 2 — Check where you are, then run a small Python command inside the allocated shell:

hostname
echo "$SLURM_JOB_ID"
/usr/bin/python3 -c 'print("Hello from Slurm!")'

hostname should now identify the compute node rather than the login node, and SLURM_JOB_ID should print a job number. You should see Hello from Slurm! No second SSH connection is needed. On this system, salloc alone does not move your shell to the compute node; use the srun --pty command above.

Step 3 — Finish the interactive session and release its resources:

exit

You are back on the login node. Keep SSH open while doing interactive work, and use sbatch for a long or unattended program. The requested time limit also ends the allocation, so save your work before it expires.

Need a GPU? After exiting the CPU session, start a new allocation from the login node with --gres=gpu:1:

srun --partition=gpu --nodes=1 --ntasks=1 \
  --cpus-per-task=1 --mem=1G --gres=gpu:1 --time=00:05:00 --pty bash

Wait for the GPU compute shell, then check the allocated GPU and exit when finished:

nvidia-smi
exit

The partition name gpu does not itself allocate a GPU. --gres=gpu:1 is required, and --mem is system RAM, not GPU memory. This quick check needs no Python packages; prepare your project environment as described in “Interactive work and Python setup” before running your own program.

4. Your first CPU job — copy and run

This small job runs on the compute node using only system Python and its standard library. It needs no dataset, Conda environment or GPU. Create and submit it from the login node:

The #SBATCH lines below request this job’s CPU, RAM and time. sbatch makes the request and queues the script; srun inside the script starts Python within that same allocation. Submit this from the login node without first running srun --pty or salloc.

mkdir -p "$HOME/slurm-quickstart/logs"
cd "$HOME/slurm-quickstart"

cat > hello.sbatch <<'EOF'
#!/bin/bash
#SBATCH --job-name=hello-cpu
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=256M
#SBATCH --time=00:02:00
#SBATCH --output=logs/hello-%j.out
#SBATCH --error=logs/hello-%j.err

set -euo pipefail
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export MKL_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OPENBLAS_NUM_THREADS="$SLURM_CPUS_PER_TASK"

srun /usr/bin/python3 -u -c 'import os, socket; print("Host:", socket.gethostname()); print("Job:", os.environ["SLURM_JOB_ID"]); print("Visible CPU threads:", len(os.sched_getaffinity(0))); print("Result:", sum(i*i for i in range(10000)))'
EOF

sbatch hello.sbatch
squeue -u "$USER"

sbatch prints a job ID, such as Submitted batch job 123. Replace 123 below with your own ID:

cat logs/hello-123.out
cat logs/hello-123.err

A short job may finish before squeue displays it. Output includes the host, job ID, visible CPU-thread count and result. An empty error file is normal; once sbatch accepts a job, disconnecting SSH does not cancel it.

5. Run your own CPU program

On the login node, create a shared project directory:

mkdir -p "$HOME/project/logs"
cd "$HOME/project"

On your laptop, from the folder containing your project files, upload them with scp (uppercase -P specifies the port). Replace the example filenames and connection placeholders:

scp -P 2222 your_script.py input.csv YOUR_USERNAME@10.229.55.207:project/

Before using the template below, prepare ~/venvs/research inside a compute allocation as shown in the Python setup section and install the dependencies your project needs. That path is an example you create, not an environment already supplied with a new account.

Save the following as cpu.sbatch in ~/project. Replace your_script.py with your program and arguments. If you use an existing Conda or other environment, replace the Python executable in the final srun command with that environment's actual full path.

#!/bin/bash
#SBATCH --job-name=my-cpu-job
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=02:00:00
#SBATCH --output=logs/cpu-%j.out
#SBATCH --error=logs/cpu-%j.err

set -euo pipefail
cd "$SLURM_SUBMIT_DIR"

# First create ~/venvs/research on the compute node as shown in the Python setup section.

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export MKL_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OPENBLAS_NUM_THREADS="$SLURM_CPUS_PER_TASK"

# Replace your_script.py with your actual program and add its arguments.
srun "$HOME/venvs/research/bin/python" -u your_script.py

Back on the login node, submit from the shared project directory. Create the log directory before submission:

cd "$HOME/project"
mkdir -p logs
sbatch cpu.sbatch

Important details:

• --cpus-per-task=4 requests CPUs for one task. It does not automatically create four Python workers or make serial code parallel.
• --mem=16G means 16 GiB total system RAM for this single-node job, including its workers. It is not 16 GiB per CPU or GPU memory. Do not combine --mem and --mem-per-cpu in these templates.
• Set worker counts to fit your allocation. In Python, int(os.environ["SLURM_CPUS_PER_TASK"]) gives the requested count. Avoid setting workers to the whole machine's CPU count.
• For one process using threaded numerical libraries, the thread settings above are appropriate. If you instead launch multiple worker processes, usually set OMP_NUM_THREADS, MKL_NUM_THREADS and OPENBLAS_NUM_THREADS to 1 and keep the total worker count within the allocated CPUs. Otherwise each worker may start another full set of threads.
• #SBATCH lines must precede executable commands. They do not expand shell variables: do not write #SBATCH --output=$HOME/logs/.... Use paths such as logs/cpu-%j.out; %j is the job ID.
• Slurm does not create missing parent directories for log files. mkdir -p logs belongs before sbatch, not only inside the job script.
• Your program, data and environment must be accessible on the compute node. Keep them in your shared home directory; sbatch does not upload your laptop project.

6. GPU template

First prepare ~/venvs/research as shown in the Python setup section and ensure it contains the GPU packages your project needs, compatible with the workstation driver. Creating a venv does not install CUDA or PyTorch. Save this as gpu.sbatch in the shared project directory, adapt the environment and program, then run mkdir -p logs and sbatch gpu.sbatch on the login node.

#!/bin/bash
#SBATCH --job-name=my-gpu-job
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --gres=gpu:1
#SBATCH --time=04:00:00
#SBATCH --output=logs/gpu-%j.out
#SBATCH --error=logs/gpu-%j.err

set -euo pipefail
cd "$SLURM_SUBMIT_DIR"

# Prepare your research environment and compatible GPU dependencies first (the Python setup section).

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export MKL_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OPENBLAS_NUM_THREADS="$SLURM_CPUS_PER_TASK"

srun nvidia-smi
# Replace this with your GPU-enabled program and its arguments.
srun "$HOME/venvs/research/bin/python" -u your_gpu_script.py

Request a GPU only when your program uses it. Do not override CUDA_VISIBLE_DEVICES to reach an unallocated device. A CPU-only allocation is not allowed to use the GPU. Requesting more --mem does not increase GPU VRAM; a CUDA out-of-memory error may require a smaller batch size or model.

7. Interactive work and Python setup

On the login node, request a CPU allocation and an interactive shell:

srun --partition=gpu --nodes=1 --ntasks=1 --cpus-per-task=2 --mem=8G --time=00:30:00 --pty bash

Wait until Slurm starts the new shell on the compute node. hostname shows where you are running, and SLURM_JOB_ID identifies the allocation. No second SSH command is needed. salloc alone does not move your shell onto the compute node.

For a new account, create a Python environment once inside this allocated shell. This is also where you prepare dependencies for the CPU and GPU batch templates:

hostname
echo "$SLURM_JOB_ID"
mkdir -p "$HOME/venvs" "$HOME/project/logs"
/usr/bin/python3 -m venv "$HOME/venvs/research"
source "$HOME/venvs/research/bin/activate"
python --version

If your project supplies a reviewed requirements.txt, install it inside the allocated shell after uploading it to ~/project:

cd "$HOME/project"
python -m pip install -r requirements.txt

Ask the administrator which GPU package versions match the driver and your project when needed. Do not use sudo or change system Python. The environment path is shared, but the nodes have separate system libraries; build and run research software on the compute node. Request more time or memory for installation if required.

In later interactive allocations, activate the existing environment instead of creating it again. Set thread or worker counts for your allocation, run your program, then exit to return to the login node and release resources:

source "$HOME/venvs/research/bin/activate"
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export MKL_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OPENBLAS_NUM_THREADS="$SLURM_CPUS_PER_TASK"
cd "$HOME/project"
# Once your program and dependencies are ready, for example:
# python -u your_script.py
exit

If you need a GPU, request it when starting a new interactive allocation:

srun --partition=gpu --nodes=1 --ntasks=1 --cpus-per-task=2 --mem=8G --gres=gpu:1 --time=00:30:00 --pty bash

After that shell starts, activate your environment and set thread or worker counts again. Keep the SSH connection open for interactive work; use sbatch for long or unattended jobs. Closing an interactive shell releases its allocation.

8. Check, cancel and troubleshoot

Replace JOB_ID with your numeric job ID:

squeue -u "$USER"
scontrol show job JOB_ID
tail -f logs/cpu-JOB_ID.out
cat logs/cpu-JOB_ID.err
scancel JOB_ID

Use the log prefix for your script: hello-, cpu- or gpu-. Ctrl+C exits tail without cancelling the job. scancel cancels the selected job; check the ID first. scontrol can inspect active and recently completed jobs; older jobs may no longer be available there. Keep your log files and job IDs.

• SSH timeout or public-key authentication failure: check gateway port 2222, your account, SSH key and approved network connection. This is separate from a Slurm queue problem. Send the exact SSH error to the administrator, never your private key.
• PENDING / PD: the job is waiting, not yet running. Read the reason shown by squeue or scontrol. Resources often means CPUs, RAM or the GPU are busy; Priority means another job is ahead. A larger request can wait longer. Do not submit repeated copies to get ahead.
• RUNNING / R: the job has started. Output files are created when execution starts, so a pending job may not have logs yet.
• OUT_OF_MEMORY, oom-kill or Killed: a process may have exceeded its memory allocation. Read the error log. Reduce memory use or request a realistic larger --mem value, within the shared pool. A plain Killed message alone does not prove OOM.
• TIMEOUT: the requested runtime expired. Save checkpoints and request sufficient time. This server’s maximum is --time=7-00:00:00; larger requests can be rejected or prevented from starting.
• Empty or delayed output: use python -u, as in the templates, or flush output in your program. Also check the error file and job state.
• Python/package not found: configure the correct environment inside the script. Batch jobs do not necessarily initialize your interactive shell's Conda setup.
• Invalid resource request: check the gpu partition and request size. This server has 22 CPU slots (one thread per physical core), 160 GiB aggregate RAM and one GPU; current free capacity may be lower.
• Node DOWN/DRAIN or a persistent unexplained failure: send the administrator the job ID, submission time, sbatch script and relevant error-log excerpt. Remove passwords, tokens and private data before sharing.

Official reference (this server’s recorded Slurm version is 19.05; local man pages match its installed version):

• Quick start: https://slurm.schedmd.com/quickstart.html
• sbatch options: https://slurm.schedmd.com/sbatch.html
• srun options: https://slurm.schedmd.com/srun.html

On the workstation: man sbatch, man srun, man squeue.

9. Privacy: files, administrators and the AI assistant

Can other users read my code and data? This depends on the actual directory, file and ACL permissions. This page does not guarantee one permission policy for every account on this server. Check ls -ld "$HOME" and confirm with Bo: a home directory owned by you with mode 700 normally permits only you and administrators to access it; shared locations and /tmp need separate checks.

What about administrators? Administrators retain access to system files and processes for maintenance and troubleshooting. Slurm controls CPU, memory and GPU use; jobs do not receive individual private virtual machines. This storage does not keep your work secret from administrators.

What may others see? Job names, usernames, queue status and resource requests are visible through Slurm. Process names and command-line arguments on the compute node may also be visible to other local users. Do not put passwords, tokens or confidential details in job names, command arguments or logs.

How do I protect files? Keep private work in your home directory and preserve its permissions. Use umask 077 before creating sensitive files and chmod 600 for existing credential files. Share selected files deliberately instead of opening your whole home directory.

Optional AI assistant: your message and selected conversation history pass through CERRA to OpenAI. Remove private keys, tokens and confidential code or data first. Conversation state lives in page memory; refreshing or starting a new conversation clears the page, which does not establish deletion from provider systems. The assistant cannot SSH, read server files or submit jobs. Reading the guide and using templates does not require the assistant.

10. VS Code and graphical workflows

Edit in VS Code on your own computer, upload through a terminal or graphical SFTP client, and run through Slurm. The examples already use ENCELDREN042343 at 10.229.55.207; replace only your username and project filenames.

Step 1: prepare your_script.py and cpu.sbatch using the CPU template above, and create its Python environment as described in the environment section. In your local VS Code terminal, create the remote folder and upload:

ssh -p 2222 YOUR_USERNAME@10.229.55.207 'mkdir -p ~/project/logs'
scp -P 2222 your_script.py cpu.sbatch YOUR_USERNAME@10.229.55.207:project/
ssh -p 2222 YOUR_USERNAME@10.229.55.207

Step 2: after SSH connects, submit on the login node:

cd ~/project
sbatch cpu.sbatch
squeue -u "$USER"

For GPU work, use gpu.sbatch from the GPU template with --gres=gpu:1. For interactive work, follow “Request resources” with srun --pty, wait for the compute shell, then run your program and exit when finished.

Step 3: have your program save plots as PNG/PDF files, then download them in another local terminal and view them graphically. Replace result.png with an actual output filename:

scp -P 2222 YOUR_USERNAME@10.229.55.207:project/result.png ./

For Matplotlib, use the non-interactive Agg backend inside your job and savefig to save plots. In a graphical file client, select SFTP, port 2222, your username and local SSH key. File transfer does not request compute resources.

VS Code Run/Debug does not automatically request Slurm resources. Allocating one terminal does not move other terminals, extensions or notebook kernels into that allocation. Run compute commands inside the allocated terminal or submit with sbatch.

Remote-SSH, Jupyter and remote desktops: forwarding settings and GUI connections for this entry have not been verified here. Successful terminal SSH alone does not establish support for these modes. Use the local editing flow above; contact Bo to arrange and validate remote GUI access.

VS Code reference: https://code.visualstudio.com/docs/remote/ssh