Skip Navigation

Holland Computing Center

Firefly - Frequently Asked Questions

Table of Contents
  1. How do I log into Firefly?
  2. What is the Moab Scheduler?
  3. What is module and how do I use it?
  4. How do I create a script for a Serial job?
  5. How do I compile a mpi program?
  6. How do I create a script for a Parallel job?
  7. How do I submit a job to Firefly.
  8. Now that I've submitted my job, how do I check the status?
  9. How do I kill a job?
  10. I still have questions about Moab or Firefly?

How do I log into Firefly?

Using any ssh client, ssh to ff.unl.edu.

What is the Moab Scheduler?

Moab is a cluster scheduler similiar to Condor, PBS, and SGE that are on Prairiefire and Merritt. Moab uses Torque (a variant of PBS) to execute jobs on remote nodes.

What is module and how do I use it?

Module is available for use on Firefly. The module software simplifies the use of different compilers and versions by setting the environment for each with the use of a single command.

To see the list of available modules, run the command module avail.

To use a particular module, run module load modulename. For example, to use the 9.0-3 version of the PGI compiler suite, run module load pgi/9.0-3. To unload a module, run module unload modulename. To see the currently loaded module(s), run module list.

Switching modules may be done by either first unloading the old and then loading the new module, or running module switch oldmodule newmodule.

To see a complete list of module commands/options, run module help.

Please note that if you compile your application using a particular module, you must include the appropriate module load statement in your submit script.

How do I create a script for a Serial job?

The most common way to submit to Firefly is with a submit script.
A sample submit script would look like:

#!/bin/sh
#PBS -N MoabTest
#PBS -l select=1
#PBS -l walltime=00:01:00,mem=10mb
#PBS -o MoabTest.stdout
#PBS -e MoabTest.stderr

sleep 10

NOTICE: PBS/Torque directives start with a #PBS.
NOTICE: If you do not put in a realistic walltime (accurate to within 3 days), it can severely penalize your queue priority.

#!/bin/sh - This tells the computer that this is an executable script, and it should be executed with /bin/sh interpreter. This could be /bin/bash (or other shells), or even /usr/bin/python.

#PBS -N MoabTest - This tells PBS that the name of this job is 'MoabTest'

#PBS -l select=1 - All requests for resources start with the -l command. This tells Moab that we want 1 processor for the execution of this job.

#PBS -l walltime=00:01:00,mem=10mb - Again, this is a request for resources. This line says that we want 1 Minute of run time, and 10MB of ram.

#PBS -o MoabTest.stdout - This says that we want to redirect standard output to the file MoabTest.stdout.

#PBS -e MoabTest.stderr - Redirect standard error to MoabTest.stderr.

sleep 10 - For this example, we only sleep for 10 seconds. But this could be a command to run an executable, or any regular command that can be executed.

Top

How do I compile a MPI program?

First login to ff.unl.edu, and use the module command to select the version of OpenMPI you wish to use. For example, module load openmpi-1.3.3/gcc-4.1.2. (See the above section for more information on using module.)

The MPI binaries and libraries will now be available in your environment.
Please note that you will need to include the appropriate module load statement in your submit script.

How do I create a script for a parallel job?

Similiar to the serial job above, a parallel job is submitted using a script.

#!/bin/sh
#PBS -N MPI.Job
#PBS -l select=10
#PBS -l walltime=00:01:00,mem=10mb
#PBS -o mpi.stdout
#PBS -e mpi.stderr
#PBS -V

module load openmpi-1.3.3/gcc-4.1.2
cd $PBS_O_WORKDIR
NPROCS=`wc -l < $PBS_NODEFILE`

mpirun -n $NPROCS -machinefile $PBS_NODEFILE ./a.out

This time, lets look at the lines that we added/changed from the serial job above.

#PBS -l select=10 - This tells Moab to reserve 10 processors for execution of the job.

module load openmpi-1.3.3/gcc-4.1.2 - This loads up the module environment for your job (see above).

cd $PBS_O_WORKDIR - The $PBS_O_WORKDIR environement variable is set to the directory that you submitted the job from. Without this statement, you will start at your home directory, or ~/.

NPROCS=`wc -l < $PBS_NODEFILE` - This line counts the number of hosts in the nodefile to send to the mpirun exectuable.

mpirun -n $NPROCS -machinefile $PBS_NODEFILE ./a.out - This starts the mpijob 'a.out' running on $NPROCS from the machines listed in $PBS_NODEFILE.

Top

How do I submit a job to Firefly?

First, you must create a submission script. Then the command 'msub <script_name>' or `qsub <script_name>' will submit the job.

Now that I've submitted my job, how do I check the status?

To check the status of your job, use the 'showq' command.

How do I kill a job?

To kill a job, use 'qdel <job_number>'

I still have questions about Moab or Firefly?

If you have questions about Moab, there is documentation from Cluster Resources.

Questions about job submission, see PBS documentation.

Or contact us at hcc-support@unl.edu.

Top