Running Jobs on Triton with Parallel MATLAB

The Triton Resource supports Parallel MATLAB, a development environment from The MathWorks. Below you will find instructions and examples for running jobs with the MATLAB Parallel Computing Toolbox on both desktop and Triton cluster environments.

How to Use MATLAB with Triton

There are two ways to use Parallel MATLAB on Triton:

• Use a personal copy of the MATLAB client that is installed on the user's desktop
• Use the Triton version of the client

Both methods are described here, and examples of each are provided.

MATLAB Architecture

Parallel MATLAB consists of two parts: the Parallel Computing Toolbox (PCT) and the MATLAB Distributed Computing Server (MDCS).

Features of the Parallel Computing Toolbox

The PCT, a module that runs on the MATLAB client, contains a number of useful capabilities, including:

1. Parallel for-Loops (parfor)
2. Distributed Arrays (arrays spread among several worker processes)
3. spmd blocks (single program, multiple data) which execute code in a manner similar to mpi runs; MATLAB code placed within an spmd block executes simultaneously on the pool of MATLAB processes that the user has allocated; each process may be identified by a labindex variable
4. The ability to define several independent tasks to run simultaneously within a single, embarrassingly parallel MATLAB job
5. Parallel integration of many toolboxes, e.g. bioinformatics, genetic algorithms and optimization
6. Several general parallel functions

Client-Server Environment

Users may run the PCT on their desktop and connect to the Triton server, or they may launch the MATLAB client on the Triton cluster. Users may run multiple worker processes on a desktop machine, or they may connect remotely to Triton and use the installed MDCS.

Using the MDCS allows users to run multiprocessor jobs on Triton via the batch queue system. The server may be accessed in either of two ways:

• from a user desktop with a MATLAB PCT installed
• from one of the MATLAB clients installed on Triton

The MATLAB PCT will automatically submit jobs to the MDCS (see below for details of this procedure).

There are four versions of the MDCS installed on Triton:

• /home/beta/matlab.2012a
• /home/beta/matlab.2011b
• /home/beta/matlab.2011a
• /home/beta/matlab_server_2010a

The default path /home/beta/matlab_server is linked to /home/beta/matlab.2012a.

Note that the MATLAB client version must match the MDCS version.

System Requirements

To use a desktop PCT with Triton for matlab2010a and earlier, you must have a secure shell program on UNIX systems or PuTTY if you are running Windows.

Starting with Matlab 2010b, no external SSH program is needed on any OS, but as with the earlier versions, you must have an ssh private/public key pair generated on the desktop with the public key installed on Triton to enable passwordless connections.

For instructions on using MDCS version 2010a and earlier, please refer to the MATLAB 2010 version of this page. For instructions on using MDCS version 2011a and 2011b, please refer to the MATLAB 2011 version of this page.

The following instructions and examples apply to MATLAB 2010b and newer for all architectures. The examples use MATLAB 2011a.

Linux and Mac OS X have built in key generating programs as part of their default system environments, while Windows does not. One option with Windows is to download PuTTY for use with generate the key pairs. See the page Generating SSH Public/Private Key Pairs on a Desktop for Use with MATLAB Distributed Computing Server, which describes how to generate key pairs on your desktop and install them on Triton. Triton also provides a PuTTY Installation Guide with screen shots.

Usage Examples

1. Using a Client on the User Desktop

For accessing MDCS via a desktop platform with MATLAB 2012a, download the test file archive containing the following files:

      communicatingSubmitFcn.m
      communicatingJobWrapper.sh
      createSubmitScript.m
      deleteJobFcn.m
      extractJobId.m
      getJobStateFcn.m
      getRemoteConnection.m
      getSubmitString.m
      independentJobWrapper.sh
      independentSubmitFcn.m

Copy these files to the toolbox/local directory of your local MATLAB installation. These are modified versions of the MATLAB files that come with the MATLAB release. These files allow you to specify several job parameters.

Using these modified files, you may also set

• number of processors per node
• account name
• queue name
• wall clock time

Other parameters that need to be set include

• name of the remote cluster (triton-login.sdsc.edu)
• directory where temporary data will be stored on Triton
• directory where data will be stored on the desktop
• path to the desktop ssh private key file
• path to MATLAB on Triton

+ Click here to see an example MATLAB function that creates a schedule object.

function [ cluster ] = getCluster(username,account,clusterHost,nodes,ppn,queue,time,DataLocation,RemoteDataLocation,keyfile,ClusterMatlabRoot)
    cluster = parcluster('genericconfig1');
    set(cluster,'HasSharedFilesystem',false);
    set(cluster,'JobStorageLocation',DataLocation);
    set(cluster,'OperatingSystem','unix');
    set(cluster,'ClusterMatlabRoot',ClusterMatlabRoot);
    set(cluster,'IndependentSubmitFcn',{@independentSubmitFcn,clusterHost,RemoteDataLocation,account,username,keyfile,time,queue});
    set(cluster,'CommunicatingSubmitFcn',{@communicatingSubmitFcn,clusterHost,RemoteDataLocation,account,username,keyfile,time,queue,ppn});
    set(cluster,'GetJobStateFcn',{@getJobStateFcn,username,keyfile});set(cluster,'DeleteJobFcn',{@deleteJobFcn,username,keyfile});

This function takes as its arguments all the parameters listed above, and returns a MATLAB cluster object (new for version 2012a) that will be used to create an MDCS job.

+ Click here to see a MATLAB function used in a simple MDCS example job.

function testparallel3()
    processors=16
    clusterHost='triton-login.sdsc.edu'
    ppn=8
    username='jgreenberg'
    account='jgreenberg'
    queue='batch'
    time='00:10:00'
    DataLocation='/Users/jpg/Documents/MATLAB/data'
    RemoteDataLocation='/home/jgreenberg/matlab/scratch'
    keyfile='/Users/jpg/.ssh/id_rsa'
    matlabRoot='/home/beta/matlab.2012a'
    cluster = getCluster(username,account,clusterHost,processors/ppn,ppn,queue,time,DataLocation,RemoteDataLocation,keyfile,matlabRoot);
    j = createCommunicatingJob(cluster);
    j.AttachedFiles={'testparfor2.m'};
    set(j,'NumWorkersRange',[1 processors]);
    set(j,'Name','Test');
    t = createTask(j,@testparfor2,1,{processors});
    submit(j);
    wait(j);
    pause(30);
    o=j.fetchOutputs;

In this example, a cluster object (cluster) is returned by getCluster(), which is passed to createCommunicatingJob(), which returns a job object. The files that are required on the cluster are defined, as well as the number of processors (16 in this case). A task is created that will call the function testparfor2() which has one output argument and one input argument with the value 16. The job is then submitted, and the output is stored in the MATLAB cell array.

+ Click here to see the function that is submitted to run.

function a = testParfor2(N)
      a = zeros(N,1);
      parfor(i=1:N)
      feature getpid
      a(i) = ans
 end

In this simple example, an array of dimension N is initialized with zeros, and the process ID is written to each array element. Since in this case we have passed the number 16 to N and we have asked for 16 processors, we might expect to get 16 different processes to run the tasks.

+ However, when we examine our output array (click to view),

ans =

        9496
        9495
        9494
        9493
        9492
        9491
        9490
       13269
       13268
       13266
       13264
       13263
       13262
       13261
       13260
        9491

we can see that only 15 worker processes were used to perform the job. Two of the loop passes were performed by the same process (9491). The reason is that MATLAB uses one worker to run the serial code, allocating the remaining workers to perform the parallel functions. Since this only leaves 15 available workers, one of them must handle two loop iterations.

2. Using the Triton Client

To access Parallel MATLAB from the Triton cluster, start the client: /home/beta/matlab.2012a/bin/matlab. When using Triton clients, different submission functions must be used than when running on remote desktop systems. Unlike the case for submitting a job remotely where a cluster object was used, when submitting a job on Triton the scheduler object is still used (as in previous MATLAB versions).

+ Click to view an example function for creating and submitting a scheduling object from Triton.

This function returns a scheduling object suitable for submitting a job from Triton:

    function [ sched ] = getSchedule(account, nodes, ppn, queue, time, DataLocation, ClusterMatlabRoot)
        sched = findResource('scheduler','Type','torque');
        set(sched,'HasSharedFilesystem',true);
        set(sched,'DataLocation',DataLocation);
        setupForParallelExecution(sched,'unix');
        set(sched,'SubmitArguments',sprintf('-A %s -q %s -lwalltime=%s',account,queue,time));
        set(sched,'RshCommand','ssh');
        set(sched,'ResourceTemplate',sprintf('-lnodes=%d:ppn=%d',nodes,ppn));
        set(sched,'ClusterMatlabRoot',ClusterMatlabRoot);

Note that if the job is submitted on Triton, no information on a hostname, username, remote paths, or ssh key file paths are required.

The sample function (testparallel3.m) above can be modified for submission from Triton as follows:

function testparallel3()
        processors=16
        ppn=8
        account='jgreenberg'
        queue='batch'
        time='00:10:00'
        DataLocation='/home/jgreenberg/matlab/scratch'
        matlabRoot='/home/beta/matlab.2012a'
        sched = getSchedule(account,processors/ppn,ppn,queue,time,DataLocation,matlabRoot)
        j = createMatlabPoolJob(sched);
        j.FileDependencies={'testParfor2.m'};
        set(j,'MinimumNumberofWorkers',processors);
        set(j,'MaximumNumberofWorkers',processors);
        t = createTask(j,@testParfor2,1,{processors})
        set(t,'CaptureCommandWindowOutput',true);
        submit(j)
        j.waitForState
        o=j.getAllOutputArguments
        o{:}