Good HCC Practices

Swan, our high-performance cluster, is shared among all our users. Sometimes, some users’ activities may negatively impact the cluster and the users. To avoid this, we provide the following guidelines for good HCC practices.

Login Node

  • Be kind to the login node. The login node is shared among all users and it should be used only for light tasks, such as moving and editing files, compiling programs, and submitting and monitoring jobs. If a researcher runs a computationally intensive task on the login node, that will negatively impact the performance for other users. Moreover, the resources on the login node are limited, so any lengthy or intensive task will highly likely exceed these resources and be terminated. For any CPU or memory intensive operations, such as testing and running applications, one should use an interactive session, or submit a job to the batch queue.
  • Avoid launching multiple simultaneous processes on the login node. This may include using lots of threads for compiling applications, or checking the job status multiple times a minute.

File Systems

  • No POSIX file system performs well with an excessive number of files, as each file operation requires opening and closing, which is relatively expensive.
  • Moreover, network data transfer operations that involve frequent scanning (walking) of every file in a set for syncing operations (backups, automated copying) can become excessively taxing for network file systems, especially at scale.
  • Large numbers of files can take an inordinate amount of time to transfer in or out of network file systems during data migration operations.
  • Computing workflows can be negatively impacted by unnecessarily large numbers of file operations, including file transfers.
  • Some I/O intensive jobs may benefit from copying the data to the fast, temporary /scratch file system local to each worker nodes. The /scratch directories are unique per job, and are deleted when the job finishes. Thus, the last step of the batch script should copy the needed output files from /scratch to either /work or /common. Please see the Running BLAST Alignment page for an example.
  • /work has two quotas - one for file count and the second one for disk space. Reaching these quotas may additionally stress the file system. Therefore, please make sure you monitor these quotas regularly, and delete all the files that are not needed or copy them to more permanent location.
  • /work is intended to be temporary location for storing job outputs and files. After that, all the necessary files need to be either moved to a permanent storage, or deleted.
  • Avoid rapidly opening and closing many files, as well as frequently reading and writing to disk, in your program. This approach stresses the file system and may cause general issues. Instead, consider reading and writing large blocks of data in memory over time, or utilizing more advanced parallel I/O libraries, such as parallel hdf5 and parallel netcdf.

Internal and External Networks

  • Use archives to transfer large number of files. If you are performing file transfer of many small files, please put these files in an archive file format, such that the many files are replaced by a single file. We recommend using zip as the archive format as zip files keep an index of the files. Moreover, zip files can be quickly indexed by the various zip tools, and allow extraction of all files or a subset of files. The tar formats are stream oriented, and a full decompression is required for the tools to know if the requested files have been found.

Running Applications

  • Before you request multiple nodes and cores in your submit script, make sure that the application you are using supports that. MPI applications can utilize multiple nodes and cores, while threaded or OpenMP applications are limited to a single node. Misusing this information may harm the researcher’s waiting time in queue, as well as the application performance.
  • Threaded and OpenMP applications can utilize multiple cores within a node. However, most of the applications do not perform significantly better when more than 16 cores are used. On the other hand, requesting more cores increases the waiting time for resources in queue, so please make sure you request a reasonable number of cores.
  • If an application uses multiple threads or cores, that number needs to be specified with the ”–ntasks-per-node” or ”–ntasks” options of SLURM. If you use multiple threads or cores with your application, but you don’t specify the respective SLURM options, your application will use only 1 core by default.
  • Avoid submitting large number of short (less than half an hour of running time) SLURM jobs. The scheduler spends more time and memory in processing those jobs, which may cause problems and reduce the scheduler’s responsiveness for everyone. Instead, group the short tasks into jobs that will run longer.
  • The maximum running time on our cluster is 7 days. If your job needs more time than that, please consider improving the code, splitting the job into smaller tasks, or using checkpointing tools such as DMTCP.
  • Before submitting a job, it is recommended to make sure that you are executing the application correctly, you are passing the right arguments, and you don’t have typos. You can do this using an interactive session. Otherwise, your job may be waiting for resources to only immediately fail because of typo or missing argument.
  • If no memory, time, and core requirements are specified in your submit SLURM script, the default resources allocated are 1GB of RAM, 1 hour of running time, and a single CPU core respectively. Oftentimes, these resources are not enough. If the job is terminated, there is a high chance that the reason is exceeded resources, so please make sure you set the memory and time requirements appropriately.
  • The run time and memory usage depend heavily on the application and the data used. You can monitor your application’s needs with tools such as Allinea Performance Reports and mem_report. While these tools can not predict the needed resources, they can provide useful information the researcher can use the next time that particular application is run.
  • Before you request GPU in your submit script, make sure that the application and code you are using supports that. Only code that is written to support GPU can take advantage of GPU nodes. Please read the documentation of the application and code you are using to see if GPU can be used. Misusing this information may harm the researcher’s waiting time in queue and result in underused resources. It is very important to request GPUs only when your code and application can efficiently utilize them.
  • Before you request multiple GPUs in your submit script, make sure that the application and code you are using supports that. Only code that is written to support multiple GPUs can take advantage of multiple GPUs. Please read the documentation of the application and code you are using to see if multiple GPUs can be used. Misusing this information may harm the researcher’s waiting time in queue and result in underused resources. It is very important to request multiple GPUs only when your code and application can efficiently utilize them.

We strongly recommend you to read and follow this guidance. If you have any concerns about your workflows or need any assistance, please contact HCC Support at hcc-support@unl.edu.