Using Scratch

What is Scratch?

Scratch is temporary local storage on the compute/worker node where the job is running. Depending on the application’s input/output (I/O) patterns, this may be the fastest storage available to a running job. The scratch space is temporary and accessible only while the job is running, and it is discarded after the job finishes. Therefore, any important data from the scratch space should be moved to a permanent location on the cluster (such as $WORK|$HOME|$COMMON). The scratch space is not backed-up.

When to use Scratch?

Using the correct tool for task at hand is important (tool being worker node local scratch vs permanent storage locations), so know your applications I/O patterns to make the correct selection. Using scratch improves the performance for certain applications that cause load issues for network-attached storage (which $WORK|$HOME|$COMMON are), some problematic I/O patterns for network-attached storage include:

• perform many rapid I/O operations (directory/file creation, renaming or removal)
• interact with and modify many files
• non-sequential/random seeking over file contents
• rapid temporary file i/o patterns involving a mixture of the above

How to use Scratch?

Scratch is accessible on the compute node while the job is running and no additional permission or setup is needed for its access.

Scratch can be utilized efficiently by:

• copying all needed input data to the temporary scratch space at the beginning of a job to ensure fast reading
• writing job output to scratch using the proper output arguments from the used program
• copying needed output data/folder back to a permanent location on the cluster before the job finishes

These modifications are done in the submit SLURM script. To access the scratch storage, one can do that with using /scratch.

Below is an example SLURM submit script. This script assumes that the input data is in the current directory (please change that line if different), and the final output data is copied back to $WORK. my_program --output is used just as an example, and it should be replaced with the program/application you use and its respective output arguments.

#!/bin/bash
#SBATCH --job-name=example
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=01:00:00
#SBATCH --mem=5gb
#SBATCH --output=example.%J.out
#SBATCH --error=example.%J.err

# load necessary modules

# copy all needed input data to /scratch [input matches problematic I/O patterns]
cp -r input_data /scratch/

# if needed, change current working directory e.g., $WORK job path to /scratch
# pushd /scratch

# use your program of interest and write program output to /scratch
# using the proper output arguments from the used program, e.g.,
my_program --output /scratch/output

# return the batch script shell to where it was at when pushd was called
# and copy the data to that prior $WORK job path
# popd
# cp -r /scratch/output .

# copy needed output to $WORK
cp -r /scratch/output $WORK

Additional examples of SLURM submit scripts that use scratch and are used on Swan are provided for BLAST and Trinity.

Disadvantages of Scratch

• limited storage capacity
• capacity shared with other jobs that are running on the same compute/worker node
• job spanning across multiple compute nodes have its own unique scratch storage per compute node
• data stored in scratch on one compute node can not be directly accessed by a different compute node and the processes that run there
• temporary storage while the job is running
• if the job fails, no output is saved and checkpointing can not be used