Interactive Jobs

Blue Waters batch system provide interactive mode for interactive debugging and/or optimization activities. Long running jobs or production runs should not be run from interactive sessions. To start an interactive session, use the following command

$> qsub -I -l nodes=1

The -I option indicates to the batch system to start an interactive job. The Blue Waters scheduler does not default interactive jobs to 1 node, so you must specify the number of nodes as shown in this example (this would be for a 1-node interactive job).  Other options accepted by qsub can be used in the interactive session.

For example, suppose you want access to 2 nodes for testing. Run the following command on a login node:

$> qsub -I -l nodes=2:ppn=32:xe -l walltime=01:00:00

where -I means interactive, and the other parameters define now many nodes and how long you want to run for.

The job will go into the queue. When the command returns, you'll be in a shell running on a "MOM" node. MOM node is a shared service resource that manages job execution. Use aprun command to send the job to compute nodes.

Following is an example of testing application performance by trying different aprun options. Once the interactive session begins, start your application a.out by typing in

$> aprun -N32 -n16 -d2 ./a.out job.inp > job16.out

where job.inp is the application input data, and job16.out is the application output.

After this job finishes, type in

$> aprun -N64 -n32 -d1 ./a.out  job.inp > job32.out

Also try other aprun options. See "man aprun" for details. Determine which options give the best performance and then use those in the production run. The benefit of trying several runs from the same interactive job is to get the job executed each time on the same nodeset thus to minimize the performance variability due to non-reproducible job placement. Note that such sequence of aprun commands can be programmed into the batch script. The advantage of interactive session is in flexibility of experimenting without the need to wait for new session to start.

After the requested time has passed (1 hour in this example), the connection will be closed.

If the intended use of the interactive session is to work directly on a compute node (for example: to access the GPU in an XK node), consider using CCM mode, which provides the opportunity to do computations interactively on a dedicated compute node rather than on a shared MOM node.