Examples for Submit Description Files
This page provides many examples to introduce submitting a job to HTCondor. It is assumed that HTCondor is both installed and configured. The first job presents the simplest example, and it is fairly complete. Further examples elaborate on modifications that utilize more sophistication.
A First Job
A first HTCondor job starts with writing, compiling, and submitting the simple "hello, world" program. The C language source code is written:
#include <stdio.h>
int main(void)
{
printf("hello, HTCondor\n");
return 0;
}
This code is compiled (using gcc) to produce object files:
gcc -c hello.c -o hello.o
The standard universe will be used when submitting this program
as a job to run under HTCondor.
Therefore, the program must be linked with the HTCondor libraries
to provide the remote system call technology.
This I/O support will be needed to send the program's
output to a file.
Link in the HTCondor libraries (again using gcc):
condor_compile gcc hello.o -o hello
A very simple submit description file goes with this example.
It queues up program hello.out to run one time
under HTCondor.
The submit description file is called submit.hello for this
example.
######################## # Submit description file for hello program ######################## Executable = hello Universe = standard Output = hello.out Log = hello.log Queue
This submit description file is placed in the same directory
as the executable (hello).
It contains four instructions about the submission of the program.
The executable is specified, as it must be for every job submission.
A file where output it to be sent is specified for this program.
No input or error
commands are given in the submit description file, so
there will be no input, and any error messages
will be thrown away.
(Files stdin, and stderr refer to
/dev/null on Unix by default).
A log file, hello.log, will be produced that contains events
the job had during its lifetime inside of HTCondor.
When the job finishes, its exit conditions will be noted in the log file.
It is recommended that you always have a log file to keep track
of what happened to your jobs.
No platform (architecture and operating system) is specified,
so HTCondor will use its default,
which is to run the job on a machine with the same
platform as the machine from which it was submitted.
The command Queue tells HTCondor to submit the job for
execution one time.
The program is submitted for execution under HTCondor using the
program condor_submit.
condor_submit submit.hello
Example 2 - Vanilla Universe
A second example submits a run of the program
mathematica as a vanilla universe job.
If a program cannot be re-compiled using condor_compile,
it can still be run under HTCondor.
This will be necessary where the source
and/or object code to a program is not available.
The execution of the job under HTCondor may be less efficient
under the vanilla universe, but it can be run.
########################
# Submit description file for mathematica
########################
executable = mathematica
universe = vanilla
input = test.data
output = test.out
error = test.error
log = test.log
queue
Note that the capitalization within the submit description file
is unimportant for the commands, but it is preserved for file
names.
Example 3 - command line arguments
Command line arguments for the executable can be given in the submit description file. This third example shows the use of arguments.
#################### # Show use of command line arguments. #################### executable = fib universe = standard arguments = 40 output = fib.out error = fib.error log = fib.log queueThis submit description file submits program
fib
one time for execution under HTCondor.
The standard universe is explicitly specified.
In those cases where the universe is not explicitly
specified, a configuration variable may specify
a universe to use.
Where the configuration variable is not present,
the standard universe is assumed.
When program fib is executed, it is given as
command line arguments the string from the arguments
command.
Example 4 - multiple submission
A fourth example submits the program mathematica twice.
Two directories are used to hold the input and output from the
executions of the program.
The first submission uses files in directory run_1
and the second submission uses files in directory run_2.
The same file names are used, and the separate directories
serve to identify the two separate submissions of the program.
For both submissions,
stdin will be test.data,
stdout will be test.out. and
stderr will be test.error.
This is a convenient way to organize data when multiple submissions
of the same program are to be run with different input data sets.
####################
# Two submissions of mathematica
####################
Executable = mathematica
Universe = vanilla
input = test.data
output = test.out
error = test.error
log = test.log
Initialdir = run_1
Queue
Initialdir = run_2
Queue
Example 5 - more on multiple submission
The submit description file for this fifth example queues 150
runs of program foo which has been compiled and linked
using condor_compile, and therefore uses the standard universe.
It shows the use of a macro to separate the input and output
files for 150 executions of the program.
#################### # Show use of pre-defined macros. #################### Executable = foo Universe = standard Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150Placing the integer after the
Queue command tells
HTCondor to run the program, not the default of one time,
but a number of times, as specified by the integer.
Each job submission using condor_submit is given a
a unique number, called the cluster number.
And, each queued execution of the program is given a own unique
number, called a process number.
Together, these numbers uniquely identify each execution of
a program under HTCondor.
They are listed as Cluster.Process when
given, for example the ID number from condor_q.
The numbers are available for use within the submit description
file as macros, Cluster and Process.
This example uses the Process macro to name the
input, output, and error files of the 150 executions of program foo.
Process numbers start at 0 for each job submission.
Therefore, the first execution of the program within the 150
will use
in.0 for input, out.0 for output, and err.0 for
errors.
in.1, out.1,
and err.1 will be used for the second execution of the program.
A single log file containing entries
about when and where HTCondor runs, checkpoints, and migrates processes for
the 150 queued programs
will be written into file foo.log.
Example 6 - one more on multiple submission
A common situation has one executable that is executed many times, each time with a different input set. If the program wants the input in a file with a fixed name, then the solution of choice runs each queued job in its own directory. This example submit description file is similar to Example 4, but it uses a macro (like Example 5) to describe the directory for each job to be queued.
#################### # Multiple jobs queued, each in its own directory #################### universe = standard executable = a_fortran_job output = job_output error = job_error log = job_log initialdir = job.$(Process) queue 10Assume there is one input file for each of the 10 jobs to be queued. The program uses a fixed name for this input file, so
input is not specified in the submit description file.
This input file is prestaged in a directory before submitting
the job.
The directories must be named job.0,
job.1,
job.2, . . .,
job.9 for the ten directories needed.
In addition to the input file within its own directory,
each directory will receive its own output in a file called
job_output,
its own error messages will go into job_error, and
HTCondor will log each job's progress within its directory in
the file called job_log.
Example 7 - on requirements and rank
The submit description file for this seventh example expands upon
the fifth example.
It queues 150 runs of program foo which has been compiled and linked
using condor_compile specifically for
Silicon Graphics workstations running IRIX 6.5.
#################### # Use both requirements and rank #################### Executable = foo Requirements = Memory >= 32 && OpSys == "IRIX65" && Arch =="SGI" Rank = Memory >= 64 Image_Size = 28 Meg Error = err.$(Process) Input = in.$(Process) Output = out.$(Process) Log = foo.log Queue 150This job requires HTCondor to run the program on machines which have greater than 32 Megabytes of physical memory, with the specified platform. It expresses a preference to run the program on machines with more than 64 Megabytes of physical memory, if such machines are available. The
Image_size command advises HTCondor that the program
will use up to 28 megabytes of memory when running.
Example 8 - Executables for more than one platform
An program is compiled and assembled to execute on a specific platform (architecture and operating system combination). Therefore, HTCondor must run an executable program on a specific platform. This limits the choice of machines that can run the program, where more than one platform exists in the pool of machines. This difficulty can be mollified by producing executables for more than one platform, and using the submit description file to inform HTCondor of the situation.
####################
# Use both requirements and rank
####################
Universe = vanilla
Executable = mathematica.$$(OpSys).$$(Arch)
Requirements = (OpSys == "IRIX65" && Arch =="SGI") ||
(OpSys == "LINUX" && Arch =="INTEL")
Error = mathematica.err
Input = mathematica.in
Output = mathematica.out
Log = mathematica.log
Queue
In this example, program mathematica has
executables for two platforms.
The executables or links to the executables are in the
directory with the submit description file.
One of the executables is for an SGI machine running Irix 6.5,
and the other is for an Intel machine running Linux.
The requirements command requires that program
be run on one of these two platforms.
This executable is chosen after HTCondor allocates
a machine for running the program.
A specialized macro is used in the executable command
to specify the name of the executable.
After macro substitution, the name of the executable
with be one of the two:
mathematica.IRIX65.SGI mathematica.LINUX.INTEL