Checkpointing is taking a snapshot of the current state of a program in such a way that the program can be restarted from that state at a later time. Checkpointing gives the HTCondor scheduler the freedom to reconsider scheduling decisions through preemptive-resume scheduling. If the scheduler decides to no longer allocate a machine to a job (for example, when the owner of that machine returns), it can checkpoint the job and preempt it without losing the work the job has already accomplished. The job can be resumed later when the scheduler allocates it a new machine. Additionally, periodic checkpointing provides fault tolerance in HTCondor. Snapshots are taken periodically, and after an interruption in service the program can continue from the most recent snapshot.
HTCondor provides checkpointing services to single process jobs on a number of Unix platforms. To enable checkpointing, the user must link the program with the HTCondor system call library (libcondorsyscall.a), using the condor_compile command. This means that the user must have the object files or source code of the program to use HTCondor checkpointing. However, the checkpointing services provided by HTCondor are strictly optional. So, while there are some classes of jobs for which HTCondor does not provide checkpointing services, these jobs may still be submitted to HTCondor to take advantage of HTCondor's resource management functionality. (See section 2.4.1 on page for a description of the classes of jobs for which HTCondor does not provide checkpointing services.)
Process checkpointing is implemented in the HTCondor system call library as a signal handler. When HTCondor sends a checkpoint signal to a process linked with this library, the provided signal handler writes the state of the process out to a file or a network socket. This state includes the contents of the process stack and data segments, all shared library code and data mapped into the process's address space, the state of all open files, and any signal handlers and pending signals. On restart, the process reads this state from the file, restoring the stack, shared library and data segments, file state, signal handlers, and pending signals. The checkpoint signal handler then returns to user code, which continues from where it left off when the checkpoint signal arrived.
HTCondor processes for which checkpointing is enabled perform a checkpoint when preempted from a machine. When a suitable replacement execution machine is found (of the same architecture and operating system), the process is restored on this new machine from the checkpoint, and computation is resumed from where it left off. Jobs that can not be checkpointed are preempted and restarted from the beginning.
HTCondor's periodic checkpointing provides fault tolerance. HTCondor pools are each configured with the PERIODIC_CHECKPOINT expression which controls when and how often jobs which can be checkpointed do periodic checkpoints (examples: never, every three hours, etc.). When the time for a periodic checkpoint occurs, the job suspends processing, performs the checkpoint, and immediately continues from where it left off. There is also a condor_ckpt command which allows the user to request that an HTCondor job immediately perform a periodic checkpoint.
In all cases, HTCondor jobs continue execution from the most recent complete checkpoint. If service is interrupted while a checkpoint is being performed, causing that checkpoint to fail, the process will restart from the previous checkpoint. HTCondor uses a commit style algorithm for writing checkpoints: a previous checkpoint is deleted only after a new complete checkpoint has been written successfully.
In certain cases, checkpointing may be delayed until a more appropriate time. For example, an HTCondor job will defer a checkpoint request if it is communicating with another process over the network. When the network connection is closed, the checkpoint will occur.
The HTCondor checkpointing facility can also be used for any Unix process outside of the HTCondor batch environment. Standalone checkpointing is described in section 4.2.1.
HTCondor can produce and use compressed checkpoints. Configuration variables (detailed in section 3.3.12 control whether compression is used. The default is to not compress.
By default, a checkpoint is written to a file on the local disk of the machine where the job was submitted. An HTCondor pool can also be configured with a checkpoint server or servers that serve as a repository for checkpoints. (See section 3.8 on page .) When a host is configured to use a checkpoint server, jobs submitted on that machine write and read checkpoints to and from the server rather than the local disk of the submitting machine, taking the burden of storing checkpoint files off of the submitting machines and placing it instead on server machines (with disk space dedicated to the purpose of storing checkpoints).
Using the HTCondor checkpoint library without the remote system call functionality and outside of the HTCondor system is known as standalone mode checkpointing.
To prepare a program for standalone checkpointing, simply use the condor_compile utility as for a standard HTCondor job, but do not use condor_submit. Run the program from the command line. The checkpointing library will print a message to let you know that checkpointing is enabled and to inform you of the default name for the checkpoint image. The message is of the form:
HTCondor: Notice: Will checkpoint to program_name.ckpt HTCondor: Notice: Remote system calls disabled.
Platforms that use address space randomization will need a modified invocation of the program, as described in section 6.1.2 on page . The invocation disables the address space randomization.
To force the program to write a checkpoint image and stop, send it the SIGTSTP signal or press control-Z. To force the program to write a checkpoint image and continue executing, send it the SIGUSR2 signal.
To restart a program using a checkpoint, run the program with the argument -_condor_restart followed by the name of the checkpoint image file. As an example, if the program is called P1 and the checkpoint is called P1.ckpt, use
P1 -_condor_restart P1.ckptAgain, platforms that implement address space randomization will need a modified invocation, as described in section 6.1.2.
By default, the program will restart in the same directory in which it originally ran, and the program will fail if it can not change to that absolute path. To suppress this behavior, also pass the -_condor_relocatable argument to the program. Not all programs will continue to work. Doing this may simplify moving standalone checkpoints between machines. Continuing the example given above, the command would be
P1 -_condor_restart P1.ckpt -_condor_relocatable
Some programs have fundamental limitations that make them unsafe for checkpointing. For example, a program that both reads and writes a single file may enter an unexpected state. Here is an example of how this might happen.
In this example, the program would re-read data from the file, but instead of finding the original data, would see data created in the future, and yield unexpected results.
To prevent this sort of accident, HTCondor displays a warning if a file is used for both reading and writing. You can ignore or disable these warnings if you choose (see section 4.2.3,) but please understand that your program may compute incorrect results.
HTCondor has warning messages in the case unexpected behaviors in your program. For example, if file x is opened for reading and writing, you will see:
HTCondor: Warning: READWRITE: File '/tmp/x' used for both reading and writing.
You may control how these messages are displayed with the
-_condor_warning command-line argument. This argument
accepts a warning category and a mode. The category describes a certain
class of messages, such as READWRITE or ALL. The mode describes what
to do with the category. It may be ON, OFF, or ONCE.
If a category is ON, it is always displayed.
If a category is OFF, it is never displayed.
If a category is ONCE, it is displayed only once.
To show all the available categories and modes, just use
-_condor_warning with no arguments.
For example, to limit read/write warnings to one instance:
-_condor_warning READWRITE ONCE
To turn all ordinary notices off:
-_condor_warning NOTICE OFF
The same effect can be accomplished within a program by using the function
_condor_warning_config, described in section 4.2.4.
A program need not be rewritten to take advantage of checkpointing. However, the checkpointing library provides several C entry points that allow for a program to control its own checkpointing behavior if needed.