• About Us
  • Research
  • News
  • Contact Us
  • Location
• New Users
  • Request an Account
  • Request a Group
  • Our Technology
  • Virtual Tour
• Firefly
  • Firefly FAQ
• Merritt
  • FAQ
  • Hardware
  • Software
• PrairieFire
  • SGE FAQ
  • Condor FAQ
  • Hardware
  • Software
  • Policy
• Red
  • CMS
  • Open Science Grid
  • MyOSG
• Reports and Jobs
  • Ganglia
  • PrairieFire Jobs
  • Publications
  • Red
  • MyHCC

What is SGE?

SGE is short for Sun Grid Engine. Basically, it is used so that the user can simply submit their jobs to SGE and not have to worry about finding the right machines on a cluster to run on. It also allows for the administrators of a machine to make sure that a user will not over-use the machine as well. It may seem like a pain to have to submit jobs to the queue (SGE is essentially also a queueing software), but really, it will make your life easier. SGE will allow you to run jobs as smoothly as possible by giving you the proper resources while keeping all users from breaking policy as much as it can. You also do not need to worry at all about what nodes on the cluster are free or keeping track of which nodes your jobs went to.

How do I use SGE?

The following questions will show you how to create and submit a SGE job.

How do I create a SGE script for running jobs?

A SGE script is a script that is used to submit jobs to the queue. Here is a simple example of an SGE script for running a serial job on PrairieFire.

Note that the '#$' prefix is not a comment. These lines are SGE directives that request certain resources and modify the behavior for how the queue handles the job.

#$ -N jobname- This line tells SGE what the name of the job is. You should change 'jobname' to whatever will help you keep track of that job. SGE does not care what your jobname is as long as it only contains numerals and letters.

#$ -l h_rt=0:20:30 - This tells SGE how much time your program needs in hours:minutes:seconds. The sample script is asking for 20 minutes and 30 seconds.

#$ -S /bin/bash - This will tell SGE what shell to use. If your program requires bash, you should use bash. If it requires tcsh, use tcsh. By default, bash is a safe one to use if you aren't sure what you are doing. Note that when you use the particular shells, the 'rc' files (ie. tcshrc and bashrc) may need to be sourced manually.

#$ -q pf24h - This is the line where you tell SGE which queue you would like to use.

#$ -M johndoe@cse.unl.edu - You should use this line if you would like emails when your job does something. The next command allows you to control when you get emails. If you do not want emails sent to you to notify you of the status of your job, you may omit this and the next line.

#$ -o out.log - This line is the name of your output file. If you have a program that will display an output to the screen, it will get included into this output file.

Everything under the #$ prefixed commands are for you. You can put anything there you would like and it will run just as it would in a shell script. The shell you chose to use is the language your shell script should be in. Also, in this case, "# change directory ...' is a comment.

How do I create a SGE script for running parallel jobs?

Running an MPI or other parallel jobs requires an SGE script pretty much like the one for the serial job. Here is a sample SGE script for an MPI job.

As mentioned before, the script for a parallel job looks very similar to that of a serial job. The main difference here is the following line:

#$ -pe ib_a 4 - This line tells SGE to run in 'Parallel Environment' ib_a or ib_b and use four cores. You can change this number to how many numbers of processors you would like to run your job on, up to the maximum number allowed by the queue.

For a parallel job, a good thing to know is that you can get a list of the nodes you are running on by typing 'cat $TMPDIR/machine'. You can redirect this into a file and go from there if necessary.

These are parallel environments you may use:

How do I submit a job?

To submit a job to SGE, first create an SGE script (using examples above). Once your script is ready, submit it by typing 'qsub SGE_script' where SGE_script is the name of the script you created.

What queues are available to me?

There are three queues available for general use, pf24h, pf240h and smp. If you have special requirements, contact the system administrator and we may be able to negotiate getting a custom queue set up for your purposes.

Queues not advertised here may be available only for users with special arrangements. Please also know that if you publish, are a co-pi or a pi for a grant, or if you donate money, you will be allowed extra resources.

• 1 Publication = 1 unit
• 1 Co-PI = 4 units
• 1 PI = 6 units
• Donating money = negotiated

For more information on a the queues, type: qstat -g c. To view more information about a specific queue, qconf -sq queuename.

If you want to submit a parallel job, do not specify the queue in your SGE script (i.e. remove the #$ -q ... line from your script). SGE will select the best available queue for your job based on your requested parallel environment (#$ -pe ...).

Now that I've submitted my job, how do I check the status?

To check the status of your pending/running jobs in SGE, use the command, 'qstat'.

To check the status of all pending/running jobs in SGE, use the command, 'qstat -u "*"'.

How do I kill a job?

Run the command 'qdel jobid' where jobid is the SGE job ID obtained by running `qstat'.

My job was killed. What should I do if I don't know why?

Look in your error log. If you 'joined' it with the output and error files, it will be 'jobname.oXXXX, (XXX is your job ID number) or whatever you decided to name your output file as if using '#$ -o output.txt'. If you didn't join the output and error files, it will have the suffix, 'jobname.eXXXX'. Take a look at those files and see if you can figure anything out. If not, please email the errors to hcc-support@unl.edu for assistance.

Why shouldn't I just ask for the maximum amount of resources possible at all times?

It almost sounds like a good idea to hog up all the resources you can, but it's actually to your disadvantage to do so. SGE tries to load-balance the jobs, and so if it sees that it can run a smaller job before a larger job will be able to run, it actually will run the smaller job in order to use as much resources as possible. So, if your job only needs 10 hours of runtime on 6 CPU and you ask for 72 hours, SGE will run a job that will run a job that needs 4 CPU and 5 hours if 6 CPUs will not be available for 6 hours. If you see a job running after you have submitted a job to queue, this is also probably the reason why. Also, SGE uses a fair share scheme in deciding which jobs to deploy. If you've run a number of jobs recently, it's possible that SGE will put another person's job in front of yours (if they haven't used as much of the system as you). In general, asking only for the amount of resources as you need and not more helps everybody including you.

What do I need to run through SGE and what can I run on the head node?

In general, all jobs need to go through SGE. You can compile, compress/uncompress, untar/tar, edit, and look through directories on the headnode. Of course, you can also monitor jobs and do other everyday use commands on the head node as well. Other commands than what is listed above that may take up resources should be run through SGE. If you have any questions, ask support for assistance.

I keep getting "/root/.bashrc: Permission denied" errors ...

If you are using bash as your shell, ignore these errors (it seems to be a known issue with bash and SGE). In your submission script force your script to manually source your profile files by adding these lines to your submit script, just below the SGE directives.

I still have questions about SGE on PrairieFire ...

You can check out the SGE manual pages on PrairieFire or online. If you still have questions, email us at: hcc-support@unl.edu.