• UNL
• HCC
• FAQ

Holland Computing Center

HCC

Credit Edit

1. How do I log into Firefly?
2. What is the Maui Scheduler?
3. What is module and how do I use it?
4. How do I create a script for a Serial job?
5. Where should I store my data?
6. How do I create a script for a parallel job?
7. How do I submit a job to the scheduler?
8. Now that I've submitted my job, how do I check the status?
9. How do I kill a job?
10. How do I compile an MPI program?
11. I still have questions about Maui or Firefly?
12. How do I check my group's disk usage on Firefly?
13. I need to run some quick tests on my program but the queue times are long. What can I do to get my test job started?
14. How do I use R software environment for statistical computing and graphics?
15. My job requires dual core (or quad core) nodes. How do I request dual/quad core nodes on Firefly?
16. My job requires me to use an entire node exclusively. How do I request this?
17. What are the complexity requirements when changing my password?

How do I log into Firefly?

Using any ssh client, ssh to ff.unl.edu.

What is the Maui Scheduler?

Maui is a cluster scheduler similiar to Condor, PBS, and SGE that are on Prairiefire and Merritt. Maui uses Torque (a variant of PBS) to execute jobs on remote nodes.

What is module and how do I use it?

Module is available for use on HCC machines. The module software simplifies the use of different compilers and versions by setting the environment for each with the use of a single command.

To see the list of available modules, run the command module avail.

To use a particular module, run module load modulename. For example, to use the 9.0-3 version of the PGI compiler suite, run module load pgi/9.0-3. To unload a module, run module unload modulename. To see the currently loaded module(s), run module list.

Switching modules may be done by either first unloading the old and then loading the new module, or running module switch oldmodule newmodule.

To see a complete list of module commands/options, run module help.

Please note that if you compile your application using a particular module, you must include the appropriate module load statement in your submit script.

How do I create a script for a Serial job?

The most common way to submit to Firefly is with a submit script.
A sample submit script would look like:

NOTICE: PBS/Torque directives start with a #PBS.
NOTICE: If you do not put in a realistic walltime (accurate to within 3 days), it can severely penalize your queue priority.

#!/bin/sh - This tells the computer that this is an executable script, and it should be executed with /bin/sh interpreter. This could be /bin/bash (or other shells), or even /usr/bin/python.

#PBS -N TestJob - This tells PBS that the name of this job is 'TestJob'

#PBS -l select=1 - All requests for resources start with the -l command. This tells Maui that we want 1 processor for the execution of this job.

#PBS -l walltime=00:01:00 - Again, this is a request for resources. This line says that we want 1 Minute of run time.

#PBS -o TestJob.stdout - This says that we want to redirect standard output to the file TestJob.stdout.

#PBS -e TestJob.stderr - Redirect standard error to TestJob.stderr.

sleep 10 - For this example, we only sleep for 10 seconds. But this could be a command to run an executable, or any regular command that can be executed.

Where should I store my data?

All HCC machines have two separate areas for every user to store data, each intended for a different purpose. 

Your home directory (i.e. /home/[group]/[username]) is meant for items that take up relatively small amounts of space.  For example:  source code, program binaries, configuration files, etc.  This space is quota-limited on a per-group basis.  The home directories are backed up for the purposes of best-effort disaster recovery.  This space is not intended as an area for I/O to active jobs.

Every user has a corresponding directory under /work using the same naming convention as /home (i.e. /work/[group]/[username]).  We encourage all users to use this space for I/O to running jobs.  This directory can also be used when larger amounts of space are temporarily needed.  It is not quota-limited; however space in /work is shared among all users.  It should be treated as short-term scratch space, and is not backed up.  HCC reserves the right to delete data from this area when space becomes low; whenever the situation allows, users will be notified before this occurs and asked to voluntarily clear space. 

If you have space requirements outside what is currently provided, please email hcc-support@unl.edu and we will gladly discuss alternatives.

How do I create a script for a parallel job?

Similiar to the serial job above, a parallel job is submitted using a script.

This time, lets look at the lines that we added/changed from the serial job above.

#PBS -l select=10 - This tells the scheduler to reserve 10 processors for execution of the job.

module load openmpi-1.3.3/gcc-4.1.2 - This loads up the module environment for your job (see above).

cd $PBS_O_WORKDIR - The $PBS_O_WORKDIR environement variable is set to the directory that you submitted the job from. Without this statement, you will start at your home directory, or ~/.

NPROCS=`wc -l < $PBS_NODEFILE` - This line counts the number of hosts in the nodefile to send to the mpirun exectuable.

mpirun -n $NPROCS -machinefile $PBS_NODEFILE ./a.out - This starts the mpijob 'a.out' running on $NPROCS from the machines listed in $PBS_NODEFILE.

How do I submit a job to the scheduler?

First, you must create a submission script. Then the command 'qsub <script_name> ' will submit the job.

Now that I've submitted my job, how do I check the status?

To check the status of your job, use the 'qstat' command.

How do I kill a job?

To kill a job, use 'qdel <JobID> '.

How do I compile an MPI program?

First, login and use the module command to select the version of OpenMPI you wish to use. For example, module load openmpi-1.3.3/gcc-4.1.2. (See the above section for more information on using module.)

The MPI binaries and libraries will now be available in your environment.
Please note that you will need to include the appropriate module load statement in your submit script.

I still have questions about Maui or Firefly?

If you have questions about Maui, there is documentation from Cluster Resources.

Questions about job submission, see PBS documentation.

Or contact us at hcc-support@unl.edu

How do I check my group's disk usage on Firefly?

diskusage

I need to run some quick tests on my program but the queue times are long. What can I do to get my test job started?

You can submit your job to a reserved pool of nodes for short jobs. Short jobs can run for up to 6 hours before the scheduler terminates the job. In order to submit a short job that will start almost immediately, your PBS script file needs to request nodes with the "short.0.0" advanced reservation. Add the following PBS directive to your submit script:

Alternatively, you may request this with the qsub command:

How do I use R software environment for statistical computing and graphics?

Using the module software, set the environment so that the R and its binaries etc. are pre-pended to the PATH, MANPATH, etc..
i.e.  run module load R/R-2.11.1

My job requires dual core (or quad core) nodes. How do I request dual/quad core nodes on Firefly?

Firefly consists of nodes with dual core and quad core processors. While most jobs are fine running on either processors, some jobs perform better on dual or quad core nodes.

To request dual core nodes:

To request quad core nodes:

where N is the number of CPU cores your job requires.

My job requires me to use an entire node exclusively. How do I request this?

There are times your jobs require exclusive access to nodes.

To request exclusive dual core nodes:

To request exclusive quad core nodes:

where N is the number of nodes your job requires.

What are the complexity requirements when changing my password?

New passwords must be at least 9 characters long, and contain at least one number, one capital letter, and one symbol or punctuation mark.