Module
Module is available for use on PrairieFire. The module software simplifies the use of different
compilers and versions by setting the environment for each with the use of a single command.
To see the list of available modules, run the command module avail.
To use a particular module, run module load modulename. For example, to use the 8.0-6 version
of the PGI compiler suite, run module load pgi/8.0-6. To unload a module, run module unload modulename.
To see the currently loaded module(s), run module list.
Switching modules may be done by either first unloading the old and then loading the new module,
or running module switch oldmodule newmodule.
To see a complete list of module commands/options, run module help.
Please note that if you compile your application using a particular module, you must
include the appropriate module load statement in your SGE submit script.
WRF
There are two ways to get up and running with WRF/WPS on Prairiefire: 1) Use the pre-compiled verisons provided. 2) Configure and compile WRF/WPS from source. These instructions are meant to be a quick start guide - a more comprehensive tutorial for compiling and running WPS/WRF is available on the WRF Users Page.
Method 1) The first, and quickest, is to utilize the pre-compiled versions of WPS and WRF.
These are configured to run WPS in serial, and WRF (for real cases) in parallel using OpenMPI.
Essentially, the directories simply need to be copied to your home directory, and the WRF module loaded
(see the above section for information about the module software). The software is located in
/home/rowe/shared/WRF/{version number}, with WRF and WPS as separate subdirectories.
As as example, to copy the 3.1.1 version to your home directory,
run 'cp -R /home/rowe/shared/WRF/3.1.1 ~'
The WRF module for V3.1.1 can then be loaded by running 'module load wrf/wrf-3.1.1'. This will
load the appropriate versions of the OpenMPI compiler, as well as set the path to the NetCDF, NCL/NCARG, NCO,
Jasper, and ncview packages/utilities. This load statement must be included in all SGE submit scripts.
Please note that global copies of the geographical data are kept in /home/rowe/shared/WRF/geog/{version number}
The pre-built version of WPS points to this by default. There is no need to download another copy to your home directory,
as this data is quite large. Global copies of test data are located in /home/rowe/shared/WRF/testdata/{version number}.
Once you are ready to try running WRF, please see the SGE FAQ for information
on how to submit jobs.
Method 2) If you wish, you can also configure and compile WRF and WPS from source. The source tar files
are located in /home/rowe/shared/WRF/tars. Please note that these tar files have modified versions of
WRF/WPS. If you download them from elsewhere, you will have to take extra steps to get them to configure/build
without issues.
These instructions will result in versions identical to the precompiled examples;
if you need to use different options modify them as necessary.
The first step is to copy and untar the source to a directory under your home directory.
cd /home/rowe/shared/WRF/tars
cp WPSV3.1.1_pgi_8.0-6_serial.tar.gz ~/wrf
cp WRFV3.1.1_openmpi.tar.gz ~/wrf
cd ~/wrf
tar -zxvf WPSV3.1.1_pgi_8.0-6_serial.tar.gz
tar -zxvf WRFV3.1.1_openmpi.tar.gz
The WRF and WPS directories should now exist under ~/wrf. Load the wrf module by running 'module load wrf/wrf-3.1.1'. This load statement must be included in all SGE submit scripts. Change to the WRF directory (WRF must be compiled before WPS) and run the configure script ('./configure'). Select Option 3 for the compiler/platform, then the default (Option 1) for basic nesting.
To compile WRF for real data cases, run './compile em_real | tee compile.log'. Ensure that the following files exist in the main/ directory:
main/nup.exe
main/real.exe
main/wrf.exe
Once WRF is compiled, change the WPS directory and run the configure script ('./configure'). Select Option 6 for the PGI compiler, serial, with GRIB2. Compile WPS by running './compile | tee compile.log'. Ensure that the executables geogrid.exe, metgrid.exe, and ungrib.exe are present. There should also be several executables in the util/ directory. WRF and WPS should now be compiled and ready to run - please see the SGE FAQ for information on submitting jobs.
Installed Software
| Kernel Version: | 2.6.18-128.7.1.el5 (CentOS 5.3) |
| GCC Compilers: |
/usr/bin/ (version 4.1.2) (version 4.3.2) - add "43" to compiler name, i.e. "gcc43" /util/comp/gcc/3.4.5/bin/ (version 3.4.5) /util/comp/gcc/4.1.1/bin/ (version 4.1.1) /util/comp/gcc/4.2.2/bin/ (version 4.2.2) *For the above compilers, use -m32 for 32-bit, and -m64 for 64-bit. |
| PGI Compilers (64-bit): |
Compiler Names: C = pgcc, C++ = pgcpp, F77 = pgf77, F90 = pgf90 /util/comp/pgi/linux86-64/6.1/bin /util/comp/pgi/linux86-64/7.0/bin /util/comp/pgi/linux86-64/7.0-3/bin /util/comp/pgi/linux86-64/7.1/bin /util/comp/pgi/linux86-64/7.1-2/bin /util/comp/pgi/linux86-64/7.1-6/bin /util/comp/pgi/linux86-64/7.2-5/bin /util/comp/pgi/linux86-64/8.0-1/bin /util/comp/pgi/linux86-64/8.0-6/bin |
| Intel compilers: |
Compiler Names: C = icc, C++ = icpc, F77 and F90 = ifort /util/comp/intel (various versions of the Intel compliers are in this directory) |
| OpenMPI Compilers: |
/util/opt/openmpi/1.3.2/gcc/4.3.2-7/64/bin (64-bit build using gcc 4.3.2) /util/opt/openmpi/1.3.2/pgi/7.2-5/64/bin (64-bit build using PGI 7.2-5) /util/opt/openmpi/1.3.2/pgi/7.2-5/32/bin (32-bit build using PGI 7.2-5) /util/opt/openmpi/1.3.2/intel/10.1.015/64/bin (64-bit build using Intel 10.1.015) /util/opt/openmpi/1.3.2/intel/10.1.015/64-no-threads/bin (64-bit w/o thread support build using Intel 10.1.015) /util/opt/openmpi (various other versions of OpenMPI for different compilers are in this directory as well) |
| Glib Version: | glibc-2.5.34.el5_3.1 |
| Job Scheduler: | Sun Grid Engine 6.2u3b |
| Interconnection Libraries: | OpenIB Infiniband Driver Stack 1.3.2 Atlas 3.4.1 (in /usr/local/lib and /usr/local/include ), LAPACK 3.0.2 with Atlas (in /usr/lib) (IMSL) |
MPI tutorials
- Tutorial on MPI: The Message-Passing Interface by William Gropp contains slides for a presentation (also available as Postscript for Tutorial on MPI: The Message-Passing Interface and Four-up Postscript for Tutorial on MPI: The Message-Passing Interface ).
A set of exercises for this tutorial.
- Tuning MPI Applications for Peak Performance
- A short introduction to MPI , an overview of MPI and MPI research in the MCS division at ANL.
- A somewhat longer introduction to MPI , with some simple examples.
- Norm Matloff's MPICH MPI Tutorial and LAM MPI Tutorial .
- A draft of a Tutorial/User's Guide for MPI by Peter Pacheco.
- MPI , a May '97 talk by Marc Snir of IBM.
- MPI: from Fundamentals to Applications by David Walker
- Designing and Building Parallel Programs by Ian Foster is an online book about parallel computing that contains a chapter on MPI.
