UW-Madison CS Dept. HTCondor Pool Information
Our local HTCondor pool is maintained by the HTCondor Team, not the CSL. Please send all HTCondor-related requests to firstname.lastname@example.org. HTCondor utilities are installed in /unsup/condor/bin.
We encourage you to read about the policies currently in effect for our pool. When setting these policies, we strive to run jobs efficiently while minimizing the impact HTCondor makes on workstation owners. Please contact us with comments about our policies, especially if you believe that HTCondor is not effectively enforcing the policies. Please also contact us if you would like to customize the HTCondor configuration of your workstation.
The HTCondor Manual contains an overview of the HTCondor system and instructions for submitting jobs to HTCondor.
UWCS HTCondor announcements mailing list
There is a mailing list for announcements about the UWCS HTCondor Pool. The HTCondor Team will try and give as much notice to this list as possible when we update the version of HTCondor installed on the pool or take downtime on critical machines. It will be an announcements-only, low-volume (1-2 messages a week on a very busy week) list. We would encourage all existing pool users to subscribe. To subscribe, please go to the mailing list signup page.
You will then need to respond to the confirmation email. Simply replying will be enough for most mail clients.
As always, the HTCondor pool is maintained by the HTCondor Team, and our preferred contact email address is email@example.com Please do not send HTCondor issues to firstname.lastname@example.org.
Compiling on CS Workstations
If you use the standard universe, use the compilers in /usr/bin.
The UW Madison CS HTCondor pool has been configured to check for a MemoryRequirements parameter in all HTCondor jobs. This parameter specifies, in megabytes, how much physical memory your job needs to run efficiently. If this parameter is not specified, HTCondor will assume a default value of 128 MB. HTCondor will only run jobs on machines with enough available physical memory to satisfy the jobs' memory requirements.
To specify this parameter, please add the following to your job description file(s):
+MemoryRequirements = 90replacing 90 with the actual memory requirements of your job in megabytes.
We encourage you to continue to specify your job's virtual memory requirements with the image_size command in your job description file.
We encourage you to read about the policies currently in effect for our pool.
First, do not use /tmp. Use the directory HTCondor starts your job in. HTCondor automatically gives your job a temporary directory that will be automatically cleaned. In many cases, HTCondor will give you more than is available in /tmp.
Second, if you have an estimate of how much space your job will require, specify it with "+DiskUsage" in your submit file. +DiskUsage is an integer measued in kilobytes. This will ensure that HTCondor doesn't try to run your job on a computer with not enough space. For example, if your job needs 750 MB of space, you might add the following to your submit file before the "queue" line:
File Permissions and AFS
HTCondor runs without AFS credentials. By default, when a job starts up on another computer, it will not have access to your home directory. You have two options:
1. Use local disk and file transfer:
You'll need to store your input files, collect your output files, and generally interact with HTCondor from a local disk. Most CSL machines have a /scratch directory well suited to the purpose. We recommend creating a subdirectory in /scratch for your work.
You'll also need to ask HTCondor to copy all of your input over to the execute machine and all of your output back. Your executable, standard input, standard output, and standard error are transferred automatically. Any other input files must be listed in transfer_input_files. You also need to set "should_transfer_files=yes" and "when_to_transfer_output=on_exit" . You can use on_exit_or_evict instead of on_exit if partial results are useful to you. Any new files created will be automatically retrieved, but you can instead use transfer_output_files to retrieve only a subset.
A typical submit file might look like this:
universe = vanilla executable = my-graphing-program input = input.txt output = standard-output.txt error = standard-error.txt arguments = file1.csv file2.csv --output=output.jpg transfer_input_files = file1.csv, file2.csv when_to_transfer_output = on_exit should_transfer_files = yes queue
You can see more documentation on file transfer in the HTCondor manual.
2. Grant everyone read and write access to your work:
You can grant net:cs access to all HTCondor job input, output, and log files stored in AFS directories. This is insecure, as anyone in the CS department can read and potentially write into this directory.
To find if a directory is in AFS, run pwd in that directory. If the output of the pwd command begins with /afs, the directory is on AFS and these instructions apply.
To do open a directory up in this way, run "fs setacl path net:cs read" or "fs setacl path net:cs write" for each directory path accessed by your HTCondor jobs.
For more information about AFS, please consult the CSL web pages (search for "AFS").