Compiling Jobs on Triton

Porting Existing Programs to Triton

If you have an existing serial or MPI-based parallel application program already running on a distributed-memory platform:

Copy your application source files to the Triton Resource, either to your $HOME directory or to the $WORK area associated with your user account:

/work/<username> where <username> is your Triton login name

If this directory does not exist, you can create it yourself.

Note that the /work area is purged regularly and is not backed up. Long-term file storage should be in your $HOME directory or on HPSS.

Compiling Serial and MPI Programs

Compiling MPI Programs

MPI source code should be recompiled for the Triton system with the following default compiler commands:

• mpicc [options] file.c (C and C++) myrinet/mx switch and the Portland Compiler
• mpif77 [options] file.f (Fortran 77 source code) myrinet/mx switch and the Portland Compiler
• mpif90 [options] file.f (free format code/dynamic memory allocation/object oriented Fortran source code) myrinet/mx switch and the Portland Compiler

Other MPI stack/compiler combinations may be obtained by choosing the appropriate modules. The choices include:

Myrinet

• Portland Group compilers + myrinet/mx
• Intel compilers + myrinet/mx
• Gnu compilers + myrinet/mx

Gigabit Ethernet

• Portland Group compilers + gigabit ethernet
• Intel compilers + gigabit ethernet
• Gnu compilers + gigabit ethernet

Compiling Serial Programs

Serial source code should be recompiled for the Triton system with the following compiler commands:

Portland Group Compilers

• pgcc [options] file.c (C and C++)
• pgf77 [options] file.f (fixed form Fortran source code)
• pgf90 [options] file.f90 (free format Fortran source code)

Intel Compilers

• icc [options] file.c (C and C++)
• ifort [options] file.f (fixed form Fortran source code)
• ifort [options] file.f90 (free format Fortran source code)

Gnu Compilers

• gcc [options] file.c (C and C++)
• g77 [options] file.f (fixed form Fortran source code)
• gfortran [options] file.f90 (free format Fortran source code)

Compatibility Options for Fortran

• -Mcppruns the Fortran preprocessor on source files prior to compilation
• -Dname[=value]specifies name as a definition to use with conditional compilation directives or the Fortran preprocessor (-Mcpp)
• -silent/-w/-Minform=severesuppress messages about use of non-standard Fortran
• -byteswapioswap bytes from big-endian to little-endian or vice-versa for unformatted files
• -i8set size of INTEGER and LOGICAL variables to 8 bytes
• -i4set size of INTEGER and LOGICAL variables to 4 bytes
• -i2set size of INTEGER variables to 2 bytes
• -r8treat REAL and CMPLX types as REAL*8 and DCMPLX
• -Msavesave all local variables between calls (static allocation)

Detecting Programming Errors

• -g produce symbolic debug information in object file (implies -O0); required for debugging with DDT
• -C/-Mboundsarray bounds checking
• -kTrap=(option, option...)specifies behavior on floating point exceptions (used only for main program):
  • dnorm trap on denormalized (very, very small) operands
  • divz trap on divide by zero
  • fp trap on floating point exceptions
  • inexact trap on inexact result
  • inv trap on invalid operands
  • none (default) disables all traps
  • ovf trap on floating point overflow
  • unf trap on floating point underflow
• -traceback add debug information

Optimization

Optimization levels of the Portland Group compilers are:

• -O0 No optimization
• -O1 (default) Task scheduling within extended basic blocks is performed. Some register allocation; no global optimizations.
• -O2 all level 1 optimizations and global scalar optimizations (optimization over all blocks)
• -O3/O4 all level 1 and 2 optimizations as well as more agressive optimizations
• -Mflushz flush very, very small values to zero.

Alternatively, the Portland Compiler User Guide recommends the use of the -fast flag. This option is host dependent and typically has the following effects:

• set the optimization level at -O2
• unroll loops (-Munroll=c:1)
• do not generate code to set up a stack frame pointer for every function (-Mnoframe)
• enable loop redundancy elimination

Numerical Libraries

The Portland Group compilers come with the Optimized ACML library (LAPACK/BLAS/FFT).

To link:

pg90/pgf77 myprog.f -llapack -lblas

Intel has developed Math Kernel Library (MKL) which contains many linear algebra, FFT and other useful numerical routines.

• Basic linear algebra subprograms (BLAS) with additional sparse routines
• Fast Fourier Transforms (FFT) in 1 and 2 dimensions, complex and real
• The linear algebra package, LAPACK
• A C interface to BLAS
• Vector Math Library (VML)
• Vector Statistical Library (VSL)
• Multi-dimensional Discrete Fourier Transforms (DFTs)

Documentation is available in HTML and PDF formats in ${MKL_HOME}/doc.

To link the MKL libraries dynamically:

For BLAS only:
-L${MKL_HOME}/lib/64 -lmkl -lguide -lpthread

For LAPACK and BLAS:
-L${MKL_HOME}/lib/64 -lmkl_lapack64 -lmkl -lguide -lpthread

To link the MKL libraries statically:

For BLAS only:
-L${MKL_HOME}/lib/64 -lmkl_ipf -lguide -lpthread

For LAPACK and BLAS:
-L${MKL_HOME}/lib/64 -lmkl_lapack -lmkl_ipf -lguide -lpthread

Notes:

• MKL routines can run in parallel (using OpenMP) if the environment variable OMP_NUM_THREADS is set to the number of threads. OMP_NUM_THREADS is not set by default.
• To force Intel MKL to serial mode, environment variable MKL_SERIAL should be set to YES. It works regardless of OMP_NUM_THREADS value. MKL_SERIAL is not set by default.
• If you get unresolved symbols when calling MKL routines from C/C++ programs with the non-default Intel 7.1 compilers, link with:
  -lPEPCF90 -lCEPCF90 -lF90 -lintrins.

All third-party applications can be found in /opt, or view the complete description of Triton software packages on the Triton Software page.