Signal Handling

You may now advise the batch environment that you would like a signal sent to your application at the walltime limit.  Use the -K flag to request a SIGTERM for your job and an integer number of seconds to be allowed beyond the walltime limit.  Because the default action for SIGTERM is to terminate a process, each process in your workflow should trap and handle the signal.  For starters, the batch script needs to trap it so that the shell keeps running your job script after SIGTERM.  aprun will deliver SIGTERM to your application and if you choose to handle it, you have the opportunity to run any code or routine you specify.  

One can use signal handling along with the job preemption  (#PBS -l flags=preemptee) or using a flexible wallclock start (#PBS -l minwclimit=[mintime]  -l walltime=[maxtime]) is an efficient way to get a charge discount (see https://bluewaters-archive.ncsa.illinois.edu/charge-factor-discounts) and minimize data loss. 

Here is the typical jobscript showing the signal trapping for the shell (bash ).

#!/bin/bash

#PBS -lnodes=2:ppn=2:xe,walltime=00:09:00

#PBS -e $PBS_JOBID.err

#PBS -o $PBS_JOBID.out

#PBS -K 135

#PBS -V

cd $PBS_O_WORKDIR

# trap SIGTERM in the batch script or it will exit and tear down the job

trap '{

  echo batch_script caught SIGTERM

}' TERM

aprun -n 4 -N 2 ./mpi_signalcatcher

Your application code may also be written to handle SIGTERM as well.

Icon

SIGTERM happens externally to your code. Execution jumps immediately to your handler routine specified in the signal() call.

This is a C code example.  For Fortran, we would suggest wrapping the signal() C system call named with a trailing underscore (_) and calling C from Fortran.  

#include <stdio.h>

#include <time.h>

#include <string.h>

#include <mpi.h>

#include <math.h>

#include <sched.h>

#include <unistd.h>

#include <signal.h>

 

// signal handler , execution jumps here on receipt of SIGTERM anytime after signal() has setup this function as the handler

void mysig(int sig) {

    // see: man 7 signal for signal-safe functions

    write(1,"rank 0: caught SIGTERM\n",23);

    // set a global variable as a checkpoint flag 

    // checkpoint_next_timestep=1;

}

 

// Fortran interfaces, callable from Fortran as signalrankzero and signalallothers

// void signalrankzero_(void) { signal(SIGTERM, mysig); }

// void signalallothers_(void) { signal(SIGTERM, SIG_IGN); }

 

int main(int argc, char *argv[])

{

    int     rank, size, len, core,x=0 ;

    int     timestep=0;

    char            name[MPI_MAX_PROCESSOR_NAME], mytime[255];

    int vecadd(void);

    double dosomething(double);

    time_t  time_raw_format;

    struct tm * ptr_time;

    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    MPI_Comm_size(MPI_COMM_WORLD, &size);

     

    MPI_Get_processor_name(name, &len);

    core= sched_getcpu();

    printf ("rank %d of %d on %s core %d\n", rank, size, name, core);

// setup signal handler(s)

     if (rank == 0)

        signal(SIGTERM, mysig);

     else  // non-zero ranks may ignore the signal

        signal(SIGTERM, SIG_IGN);

 

     while(1) // simulate main iteration loop

     {

        for (int i=0 ; i<10; i++) // simulate compute kernel work loop

        {

            dosomething(x++);

            if (rank == 0)

            {

                putc('.', stdout);

                fflush(stdout);

            }

        }

        if (rank==0)

        {

            time ( &time_raw_format);

            ptr_time = localtime ( &time_raw_format );

            strcpy(mytime, asctime(ptr_time) );

            mytime[strlen(mytime)]='\0';

            mytime[strlen(mytime)-1]=' ';

            printf ( "%s: ", mytime);

            printf("iteration: %d\n",++timestep);

        }

     }

     return 0;

     MPI_Finalize();

}

double dosomething(double x)

{

  int i;

  int nrepeat = 195000000;  /* Takes about 1 second to do this many sqrts */

                           /* providing no compiler optimisation is used */

  for (i=0; i < nrepeat; i++)

    {

      x = sqrt(x);

    }

  return(x);

}

Now upon receipt of SIGTERM from the batch environment at walltime limit, the application will run your mysig() function and continue beyond the walltime limit for -K seconds.  Possible actions of mysig() might include: attempt to execute the application checkpoint routine, close open files or similar cleanup activities.  The sample code displays a message to stdout and continues with the main loop iterations.

Icon
  • All of the application ranks will receive SIGTERM and each of them must handle (or ignore via SIG_IGN) it.
  • Each thread in active OpenMP code loops or regions will receive the signal.

..........Wed Jul 12 12:53:13 2017 : iteration: 40

..........Wed Jul 12 12:53:28 2017 : iteration: 41

........rank 0: caught SIGTERM

.Wed Jul 12 12:53:44 2017 : iteration: 42

..........Wed Jul 12 12:54:00 2017 : iteration: 43

..........Wed Jul 12 12:54:16 2017 : iteration: 44

..........Wed Jul 12 12:54:31 2017 : iteration: 45

..........Wed Jul 12 12:54:47 2017 : iteration: 46

references: