Computing On Demand (COD) extends HTCondor's high throughput computing abilities to include a method for running short-term jobs on instantly-available resources.
The motivation for COD extends HTCondor's job management to include interactive, compute-intensive jobs, giving these jobs immediate access to the compute power they need over a relatively short period of time. COD provides computing power on demand, switching predefined resources from working on HTCondor jobs to working on the COD jobs. These COD jobs (applications) cannot use the batch scheduling functionality of HTCondor, since the COD jobs require interactive response-time. Many of the applications that are well-suited to HTCondor's COD capabilities involve a cycle: application blocked on user input, computation burst to compute results, block again on user input, computation burst, etc. When the resources are not being used for the bursts of computation to service the application, they should continue to execute long-running batch jobs.
Here are examples of applications that may benefit from COD capability:
recalculate button,
the predefined HTCondor resources (nodes)
work on the computation and send the results
back to the master application providing the user interface and
displaying the data.
Ideally, while the user is entering new data or modifying formulas,
these nodes work on non-COD jobs.
The way HTCondor helps these kinds of applications is to provide an infrastructure to use HTCondor batch resources for the types of compute nodes described above. HTCondor does NOT provide tools to parallelize existing GUI applications. The COD functionality is an interface to allow these compute nodes to interact with long-running HTCondor batch jobs. The user provides both the compute node applications and the interactive master application that controls them. HTCondor only provides a mechanism to allow these interactive (and often parallelized) applications to seamlessly interact with the HTCondor batch system.
The resources of a HTCondor pool (nodes) run jobs. When a high-priority COD job appears at a node, the lower-priority (currently running) batch job is suspended. The COD job runs immediately, while the batch job remains suspended. When the COD job completes, the batch job instantly resumes execution.
Administratively, an interactive COD application puts claims on nodes. While the COD application does not need the nodes (to run the COD jobs), the claims are suspended, allowing batch jobs to run.
Claims on nodes are assigned to users. A user with a claim on a resource can then suspend and resume a COD job at will. This gives the user a great deal of power on the claimed resource, even if it is owned by another user. Because of this, it is essential that users allowed to claim COD resources can be trusted not to abuse this power. Users are authorized to have access to the privilege of creating and using a COD claim on a machine. This privilege is granted when the HTCondor administrator places a given user name in the VALID_COD_USERS list in the HTCondor configuration for the machine (usually in a local configuration file).
In addition, the tools to request and manage COD claims
require that the user issuing the commands be authenticated.
Use one of the strong authentication methods described
in section 3.6.1 ``Security Configuration'' on
page ![[*]](crossref.png)
.
If one of these methods cannot be used,
then file system authentication may be used
when directly logging in to that machine (to be claimed)
and issuing the command locally.
To run an application on a claimed COD resource, an authorized user defines characteristics of the application. Examples of characteristics are the executable or script to use, the directory to run the application in, command-line arguments, and files to use for standard input and output. COD users specify a ClassAd that describes these characteristics for their application. There are two ways for a user to define a COD application's ClassAd:
These two methods for defining the ClassAd can be used together. For example, the user can define some attributes in the configuration file, and only provide a few dynamically defined attributes with the condor_cod tool.
Regardless of how the COD application's ClassAd is defined, the application's executable and input data must be pre-staged at the node. This is a current limitation of HTCondor's support for COD that will eventually go away. For now, there is no mechanism to transfer files for a COD application, and all I/O must be performed locally or onto a network file system that is accessible by a node.
The following three sections detail defining the attributes. The first lists the attributes that can be used to define a COD application. The second describes how to define these attributes in an HTCondor configuration file. The third explains how to define these attributes using the condor_cod tool.
Attributes for a COD application are either required or optional. The following attributes are required:
").
There is no default.
").
The following list of attributes are optional:
").
If the IWD is not specified, the temporary execution sandbox
created by the condor_starter will be used as the initial working
directory.
").
").
NAME=value.
Multiple variables are delimited with a semicolon.
An example: Env = "PATH=/usr/local/bin:/usr/bin;TERM=vt100"
It is a string attribute, and must therefore be enclosed in
quotation marks (").
").
").
Multiple file names may be delimited with either commas or white space
characters, and
therefore, file names can not contain spaces.
KillSig = "SIGQUIT"), or as an integer
(KillSig = 3)
The default is to use SIGTERM.
").
TRUE.
The default if not specified is FALSE.
NOTE: If any path attribute (Cmd, In, Out,Err, StarterUserLog) is not a full path name, HTCondor automatically prepends the value of IWD.
The final set of attributes define an identification for a COD application. The job ID is made up of both the ClusterId and ProcId attributes (as described below). This job ID is similar to the job ID that is created whenever a regular HTCondor batch job is submitted. For regular HTCondor batch jobs, the job ID is assigned automatically by the condor_schedd whenever a new job is submitted into the persistent job queue. However, since there is no persistent job queue for COD, the usual mechanism to identify the jobs does not exist. Moreover, commands that require the job ID for batch jobs such as condor_q and condor_rm do not exist for COD. Instead, the claim ID is the unique identifier for COD jobs and COD-related commands.
When using COD, the job ID is only used to identify the job in various
log messages and in the COD-specific output of condor_status.
The COD job ID is part of the information included in all
events written to the StarterUserLog
regarding a given job.
The COD job ID is also used in the HTCondor debugging logs described in
section 3.3.4 on
page
For example, in the condor_starter daemon's log file for COD jobs
(called StarterLog.cod by default) or in the condor_startd
daemon's log
file (called StartLog by default).
These COD IDs are optional. The job ID is useful to define where it helps a user with accounting or debugging of their own application. In this case, it is the user's responsibility to ensure uniqueness, if so desired.
NOTE: The cluster and proc identifiers can also be specified as command-line arguments to the condor_cod_activate tool when spawning a given COD application. See section 4.3.4 below for details on using condor_cod_activate.
To define COD attributes in the HTCondor configuration file for a given application, the user selects a keyword to uniquely name ClassAd attributes of the application. This case-insensitive keyword is used as a prefix for the various configuration file attribute names. When a user wishes to spawn a given application, the keyword is given as an argument to the condor_cod tool and the keyword is used at the remote COD resource to find attributes which define the application.
Any of the ClassAd attributes described in the previous section can be
specified in the configuration file with the keyword prefix followed
by an underscore character ("_").
For example, if the user's keyword for a given fractal generation application is ``FractGen'', the resulting entries in the HTCondor configuration file may appear as:
FractGen_Cmd = "/usr/local/bin/fractgen" FractGen_Iwd = "/tmp/cod-fractgen" FractGen_Out = "/tmp/cod-fractgen/output" FractGen_Err = "/tmp/cod-fractgen/error" FractGen_Args = "mandelbrot -0.65865,-0.56254 -0.45865,-0.71254"
In this example, the executable may create other files. The Out and Err attributes specified in the configuration file are only for standard output and standard error redirection.
When the user wishes to spawn an instance of this application, they use the -keyword option of FractGen in the command-line of the condor_cod_activate command.
NOTE: If a user is defining all attributes of their COD application in the HTCondor configuration files, and the condor_startd daemon on the COD resource they are using is running as root, the user must also define Owner to be the user that the COD application should run as (see section 4.3.3 above).
COD users may define attributes dynamically (at the time they spawn a COD application). In this case, the user writes the ClassAd attributes into a file, and the file name is passed to the condor_cod_activate tool using the -jobad command-line option. These attributes are read by the condor_cod tool and passed through the system onto the condor_starter daemon which spawns the COD application. If the file name given is -, the condor_cod tool will read from standard input (stdin).
Users should not add a keyword prefix when defining attributes with the condor_cod_activate tool. The attribute names can be used in the file directly.
WARNING: The current syntax for this file is not the same as the syntax in the file used with condor_submit.
NOTE: Users should not define the Owner attribute when using condor_cod_activate on the command line, since HTCondor will automatically insert the correct value based on what user runs the condor_cod_activate command and how that user authenticates to the COD resource. If a user defines an attribute that does not match the authenticated identity, HTCondor treats this case as an error, and it will fail to launch the application.
Separate commands are provided by HTCondor to manage COD claims on batch resources. Once created, each COD claim has a unique identifying string, called the claim ID. Most commands require a claim ID to specify which claim you wish to act on. These commands are the means by which COD applications interact with the rest of the HTCondor system. They should be issued by the controller application to manage its compute nodes. Here is a list of the commands:
To issue these commands, a user or application invokes the condor_cod tool. A command may be specified as the first argument to this tool, as
condor_cod request -name c02.cs.wisc.eduor the condor_cod tool can be installed in such a way that the same binary is used for a set of names, as
condor_cod_request -name c02.cs.wisc.edu
Other than the command name itself (which must be included in full) additional options supported by each tool can be abbreviated to the shortest unambiguous value. For example, -name can also be specified as -n. However, for a command like condor_cod_activate that supports both -classad and -cluster, the user must use at least -cla or -clu. If the user specifies an ambiguous option, the condor_cod tool will exit with an error message.
In addition, there is now a -cod option to condor_status.
The following sections describe each option in greater detail.
A user must be granted authorization to create COD claims on a specific machine. In addition, when the user uses these COD claims, the application binary or script they wish to run (and any input data) must be pre-staged on the machine. Therefore, a user cannot simply request a COD claim at random.
The user specifies the resource on which to make a COD claim. This is accomplished by specifying the name of the condor_startd daemon desired by invoking condor_cod_request with the -name option and the resource name (usually the host name). For example:
condor_cod_request -name c02.cs.wisc.edu
If the condor_startd daemon desired belongs to a different HTCondor pool than the one where executing the COD commands, use the -pool option to provide the name of the central manager machine of the other pool. For example:
condor_cod_request -name c02.cs.wisc.edu -pool condor.cs.wisc.edu
An alternative is to provide the IP address and port number where the condor_startd daemon is listening with the -addr option. This information can be found in the condor_startd ClassAd as the attribute StartdIpAddr or by reading the log file when the condor_startd first starts up. For example:
condor_cod_request -addr "<128.105.146.102:40967>"
If neither -name or -addr are specified, condor_cod_request attempts to connect to the condor_startd daemon running on the local machine (where the request command was issued).
If the condor_startd daemon to be used for the COD claim is an SMP machine and has multiple slots, specify which resource on the machine to use for COD by providing the full name of the resource, not just the host name. For example:
condor_cod_request -name slot2@c02.cs.wisc.edu
A constraint on what slot is desired may be provided, instead of specifying it by name. For example, to run on machine c02.cs.wisc.edu, not caring which slot is used, so long as it the machine is not currently running a job, use something like:
condor_cod_request -name c02.cs.wisc.edu -requirements 'State!="Claimed"'
In general, be careful with shell quoting issues, so that your shell is not confused by the ClassAd expression syntax (in particular if the expression includes a string). The safest method is to enclose any requirement expression within single quote marks (as shown above).
Once a given condor_startd daemon has been contacted to request a new COD claim, the condor_startd daemon checks for proper authorization of the user issuing the command. If the user has the authority, and the condor_startd daemon finds a resource that matches any given requirements, the condor_startd daemon creates a new COD claim and gives it a unique identifier, the claim ID. This ID is used to identify COD claims when using other commands. If condor_cod_request succeeds, the claim ID for the new claim is printed out to the screen. All other commands to manage this claim require the claim ID to be provided as a command-line option.
When the condor_startd daemon assigns a COD claim,
the ClassAd describing the resource is returned to the user that
requested the claim.
This ClassAd is a snap-shot of
the output of condor_status -long for the given machine.
If condor_cod_request is invoked with the -classad option
(which takes a file name as an argument), this ClassAd will be written
out to the given file.
Otherwise, the ClassAd is printed to the screen.
The only essential piece of information in this ClassAd is the Claim
ID, so that is printed to the screen, even if the whole ClassAd is
also being written to a file.
The claim ID as given after listing the machine ClassAd appears as this example:
ID of new claim is: "<128.105.121.21:49973>#1073352104#4"When using this claim ID in further commands, include the quote marks as well as all the characters in between the quote marks.
NOTE: Once a COD claim is created, there is no persistent record of it kept by the condor_startd daemon. So, if the condor_startd daemon is restarted for any reason, all existing COD claims will be destroyed and the new condor_startd daemon will not recognize any attempts to use the previous claims.
Also note that it is your responsibility to ensure that the claim is eventually removed (see section 4.3.4). Failure to remove the COD claim will result in the condor_startd continuing to hold a record of the claim for as long as condor_startd continues running. If a very large number of such claims are accumulated by the condor_startd, this can impact its performance. Even worse: if a COD claim is unintentionally left in an activated state, this results in the suspension of any batch job running on the same resource for as long as the claim remains activated. For this reason, an optional -lease argument is supported by condor_cod_request. This tells the condor_startd to automatically release the COD claim after the specified number of seconds unless the lease is renewed with condor_cod_renew. The default lease is infinitely long.
Once a user has created a valid COD claim and has the claim ID, the
next step is to spawn a COD job using the claim.
The way to do this is to activate the claim, using the
condor_cod_activate command.
Once a COD application is active on a COD claim, the COD claim will
move into the Running state, and any batch HTCondor job
on the same resource will be suspended.
Whenever the COD application is inactive (either suspended, removed
from the machine, or if it exits on its own), the state of the COD
claim changes. The new state depends on why the application
became inactive.
The batch HTCondor job then resumes.
To activate a COD claim, first define attributes about the job to be run in either the local configuration of the COD resource, or in a separate file as described in this manual section. Invoke the condor_cod_activate command to launch a specific instance of the job on a given COD claim ID. The options given to condor_cod_activate vary depending on if the job attributes are defined in the configuration file or are passed via a file to the condor_cod_activate tool itself. However, the -id option is always required by condor_cod_activate, and this option should be followed by a COD claim ID that the user acquired via condor_cod_request.
If the application is defined in the configuration files for the COD
resource, the user provides the keyword (described in
section 4.3.3) that uniquely identifies the
application's configuration attributes.
To continue the example from that section, the user would spawn their
job by specifying -keyword FractGen, for example:
condor_cod_activate -id "<claim_id>" -keyword FractGenSubstitute the
<claim_id> with the valid Cod Claim Id.
Using the same example as given above, this example would be:
condor_cod_activate -id "<128.105.121.21:49973>#1073352104#4" -keyword FractGen
If the job attributes are placed into a file to be passed to the condor_cod_activate tool, the user must provide the name of the file using the -jobad option. For example, if the job attributes were defined in a file named cod-fractgen.txt, the user spawns the job using the command:
condor_cod_activate -id "<claim_id>" -jobad cod-fractgen.txtAlternatively, if the filename specified with -jobad is -, the condor_cod_activate tool reads the job ClassAd from standard input (stdin).
Regardless of how the job attributes are defined, there are
other options that condor_cod_activate accepts.
These options specify the job ID for the application to be run.
The job ID can either be specified in the job's ClassAd,
or it can be
specified on the command line to condor_cod_activate.
These options are -cluster and -proc.
For example, to launch a COD job with keyword foo
as cluster 23, proc 5, or 23.5,
the user invokes:
condor_cod_activate -id "<claim_id>" -key foo -cluster 23 -proc 5The -cluster and -proc arguments are optional, since the job ID is not required for COD. If not specified, the job ID defaults to
1.0.
Once a COD application has been activated with condor_cod_activate
and is running on a COD resource, it may be temporarily suspended
using condor_cod_suspend.
In this case, the claim state becomes Suspended.
Once a given COD job is suspended, if there are no other running COD
jobs on the resource, an HTCondor batch job can use the resource.
By suspending the COD application, the batch job is allowed to run.
If a resource is idle when a COD application is first spawned,
suspension of the COD job makes the batch resource available
for use in the HTCondor system.
Therefore, whenever a COD application has no work to perform, it should be
suspended to prevent the resource from being wasted.
The interface of condor_cod_suspend supports the single option -id, to specify the COD claim ID to be suspended. For example:
condor_cod_suspend -id "<claim_id>"
If the user attempts to suspend a COD job that is not running, condor_cod_suspend exits with an error message. The COD job may not be running because it is already suspended or because the job was never spawned on the given COD claim in the first place.
This command tells the condor_startd to renew the lease on the COD claim for the amount of lease time specified when the claim was created. See section 4.3.4 for more information on using leases.
The condor_cod_renew tool supports only the -id option to specify the COD claim ID the user wishes to renew. For example:
condor_cod_renew -id "<claim_id>"
If the user attempts to renew a COD job that no longer exists, condor_cod_renew exits with an error message.
Once a COD application has been suspended with condor_cod_suspend,
it can be resumed using condor_cod_resume.
In this case, the claim state returns to Running.
If there is a regular batch job running on the same resource, it will
automatically be suspended if a COD application is resumed.
The condor_cod_resume tool supports only the -id option to specify the COD claim ID the user wishes to resume. For example:
condor_cod_resume -id "<claim_id>"
If the user attempts to resume a COD job that is not suspended, condor_cod_resume exits with an error message.
If a given COD application does not exit on its own and needs to be removed manually, invoke the condor_cod_deactivate command to kill the job, but leave the COD claim ID valid for future COD jobs. The user must specify the claim ID they wish to deactivate using the -id option. For example:
condor_cod_deactivate -id "<claim_id>"
By default, condor_cod_deactivate attempts to gracefully cleanup
the COD application and give it time to exit.
In this case the COD claim goes into the Vacating state and the
condor_starter process controlling the job will send it the
KillSig defined for the job (SIGTERM by default).
This allows the COD job to catch the signal and do whatever final work
is required to exit cleanly.
However, if the program is stuck or if the user does not want to give
the application time to clean itself up, the user may use the
-fast option to tell the condor_starter to quickly kill the
job and all its descendants using SIGKILL.
In this case the COD claim goes into the Killing state.
For example:
condor_cod_deactivate -id "<claim_id>" -fast
In either case, once the COD job has finally exited, the COD claim
will go into the Idle state and will be available for future
COD applications.
If there are no other active COD jobs on the same resource, the
resource would become available for batch HTCondor jobs.
Whenever the user wishes to spawn another COD application, they can
reuse this idle COD claim by using the same claim ID, without having
to go through the process of running condor_cod_request.
If the user attempts a condor_cod_deactivate request on a COD claim
that is neither Running nor Suspended, the condor_cod
tool exits with an error message.
If users no longer wish to use a given COD claim, they can release the claim with the condor_cod_release command. If there is a COD job running on the claim, the job will first be shut down (as if condor_cod_deactivate was used), and then the claim itself is removed from the resource and the claim ID is destroyed. Further attempts to use the claim ID for any COD commands will fail.
The condor_cod_release command always prints out the state the COD claim was in when the request was received. This way, users can know what state a given COD application was in when the claim was destroyed.
Like most COD commands, condor_cod_release requires the claim ID to
be specified using -id.
In addition, condor_cod_release supports the -fast option
(described above in the section about condor_cod_deactivate).
If there is a job running or suspended on the claim when it is
released with condor_cod_release -fast, the job will be
immediately killed.
If -fast is not specified, the default behavior is to use a
graceful shutdown, sending whatever signal is specified in the
KillSig attribute for the job (SIGTERM by default).
In some cases, a user will want to delegate a copy of their user
credentials (in the form of an x509 proxy) to the machine where one of
their COD jobs will run.
For example, sites wishing to spawn the condor_starter using glexec
will need a copy of this credential before the claim can be activated.
Therefore, beginning with HTCondor version 6.9.2, COD users have access
to a the command delegate_proxy.
If users do not specifically require this proxy delegation, this
command should not be used and the rest of this section can be skipped.
The delegate_proxy command optionally takes a -x509proxy
argument to specify the path to the proxy file to use.
Otherwise, it uses the same discovery logic that condor_submit uses
to find the user's currently active proxy.
Just like every other COD command (except request), this
command requires a valid COD claim id (specified with -id) to
indicate what COD claim you wish to delegate the credentials to.
This command can only be sent to idle COD claims, so it should be done
before activate is run for the first time.
However, once a proxy has been delegated, it can be reused by
successive claim activations, so normally this step only has to happen
once, not before every activate.
If a proxy is going to expire, and a new one should be sent, this
should only happen after the existing COD claim has been deactivated.
HTCondor's support for COD has a few limitations.
The following items are all limitations we plan to remove in future releases of HTCondor:
None of the above items are fundamentally difficult to add and we hope to address them relatively quickly. If you run into one of these limitations, and it is a barrier to using COD, please contact condor-admin@cs.wisc.edu with the subject ``COD limitation'' to gain quick help.
The following list are more fundamental limitations that we do not plan to address: