This is an outdated version of the HTCondor Manual. You can find current documentation at http://htcondor.org/manual.

10.3 Development Release Series 8.3

This is the development release series of HTCondor. The details of each version are described below.

Version 8.3.5

Release Notes:

• HTCondor version 8.3.5 released on April 20, 2015.

New Features:

• The RPM packages have been restructured to allow running a 32-bit static shadow on Red Hat Enterprise Linux 6. The new condor-all RPM is used to install all of the RPMs for a typical HTCondor installation. Since the binary distribution of HTCondor for Red Hat Enterprise Linux 6 and 7 consists of more that a handful of RPMs, the RPMs are only available from our YUM repository. (Ticket #4621).

• New features increase the power of job specification in the submit description file. Submit description files are now parsed the same as configuration files.
  • The queue submit command may be used in flexible and powerful new ways to specify job submissions. See section 2.5.2 for details. (Ticket #4819).

  • New macro functions are supported, and may be used in submit description files as well as in configuration. (Ticket #4944).

  • condor_submit has new command line options -queue and -dry-run, to provide flexible and powerful new ways to specify job submissions, as well as to test what job would be submitted without submitting. (Ticket #4933).

  • condor_submit now supports assignment of ClassAd attributes on the command line. (Ticket #4983).

  • condor_submit accepts if and include statements in the same way that configuration files do. (Ticket #4913).

• The machine ClassAd attribute VirtualMemory is now set correctly for dynamic slots running under a partitionable slot. (Ticket #4608).

• An EC2 grid universe job now advertises its access key ID in its job ClassAd. (Ticket #4903).

• For an executing job, HTCondor now sets the environment variable OMP_NUM_THREADS to the number of cores of the slot it is running in. This prevents OpenMP-linked jobs (including Matlab) from attempting to use more cores than have been provisioned. (Ticket #4884).

• Improved the performance of writing to a user's job event log and the event log. Disabling locking when writing to these files, as controlled by configuration variables ENABLE_USERLOG_LOCKING and EVENT_LOG_LOCKING, is now safe. The default location of the lock file for rotating the event log that is defined by configuration variable EVENT_LOG_ROTATION_LOCK has been changed to $(LOCK)/EventLogLock. (Ticket #4908).

• HTCondor is now more likely to be compatible with Windows systems that have a Winsock Layered Service Provider (LSP) installed. (Ticket #304).

• The new configuration variables SUBMIT_REQUIREMENT_NAMES, SUBMIT_REQUIREMENT_<Name>, and SUBMIT_REQUIREMENT_<Name>_REASON support the ability to configure HTCondor to reject the submission of jobs that do not meet specified criteria. (Ticket #2640).

• A new DAGMan feature implements the retry of a PRE or POST script after a specified delay, where a retry attempt is based on the exit code from the initial execution of the script. (Ticket #4488).

• condor_dagman no longer supports Stork jobs. (Ticket #4550).

• condor_dagman no longer has the capability to read individual per- job log files. This means that recovery mode will no longer work on a DAG originally submitted with version 7.9.1 or earlier. (Ticket #4528).

• An experimental new feature allows PanDA monitoring of jobs, as documented with a link from the HTCondor wiki page, https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=ExperimentalFeatures. (Ticket #4270).

• The condor_schedd now generates a report about its attempts to reconnect to the condor_startd daemons of previously running jobs on start up. The report is written to the location specified by the new configuration variable SCHEDD_RESTART_REPORT. Once all reconnect attempts are complete, a copy of the report is also emailed to HTCondor administrator account. (Ticket #4783).

• The new ClassAd functions envV1ToV2() and mergeEnvironment() are useful for manipulating the environment variable lists in job ClassAds. (Ticket #4574).

• The new configuration variable CURB_MATCHMAKING can be used to configure the condor_schedd daemon to cease requesting more machine resources from the central manager during overload situations. (Ticket #4905).

• Some HTCondor-specific environment variables that are set in the environment of a batch job, such as _CONDOR_JOB_AD, are now also set for condor_ssh_to_job and interactive job sessions. (Ticket #4943).

• condor_q with the -constraint option will now display a summary line. (Ticket #4992).

• The new directQuery() Python binding queries a daemon directly for its ClassAd, instead of querying the condor_collector. (Ticket #4843).

• The new send_alive() Python binding sends keep alive messages to the condor_master daemon. This allows the Python bindings to be used in a daemon managed by the condor_master. (Ticket #4875).

• A new Python binding sets the HTCondor subsystem name and type. This allows the bindings to initialize logging or configuration as if they are a particular HTCondor daemon. (Ticket #4876).

• The new log() Python binding provides the ability to log messages via the HTCondor logging subsystem. (Ticket #4883).

Bugs Fixed:

• The condor_schedd will no longer request more resources from a central manager when the number of jobs running has exceeded MAX_JOBS_RUNNING. (Ticket #4905).

• Fixed a bug in condor_chirp in which generic events logged to the job event log with the ulog command where written in the time zone of the execute machine, not the submit machine. (Ticket #4951).

• Fixed a bug in which a condor_collector daemon would have been unable to update its own ClassAd. (Ticket #4850).

• Fixed a bug introduced in HTCondor version 8.3.2 that caused the -absent option of condor_status to print duplicated information about each ClassAd. (Ticket #4853).

• The python bindings now accept ClassAd expressions for the constraint of a condor_schedd daemon or a condor_collector daemon query, matching the behavior documented in this reference manual. (Ticket #4907).

• Fixed a bug introduced in HTCondor version 8.3.3 that caused local universe jobs to execute repeatedly on Windows platforms. (Ticket #4909).

• Fixed a bug that caused messages similar to

  warning: setting UserUid to 11021, was 6004 previously
  

  to be logged in the condor_schedd daemon log. (Ticket #4889).

• Fixed a bug in the Python bindings in which multi-threaded Python programs would deadlock when a Schedd.query() returned more than one result. Also fixed a bug in Schedd.xquery() in which the constraint parameter did not default to True as documented, but had to be explicitly defined. (Ticket #5000).

Version 8.3.4

Release Notes:

• HTCondor version 8.3.4 released on March 5, 2015.

New Features:

• None.

Bugs Fixed:

• Fixed a condor_schedd daemon bug that could have prevented jobs from matching resources without an apparent reason, when the condor_schedd was flocking and configuration variable SIGNIFICANT_ATTRIBUTES was not set. (Ticket #4919).

Known Issue:

• If the condor_schedd daemon is configured to enable both IPv4 and IPv6 communication, it will not be able to start any jobs which use a condor_shadow. In effect, mixed mode IPv4 and IPv6 does not work. (Ticket #4911).

Version 8.3.3

Release Notes:

• HTCondor version 8.3.3 released on Feb. 19, 2015.

New Features:

• Configuration variable ENCRYPT_EXECUTE_DIRECTORY is now honored on Linux platforms. The Linux platform must have the ecryptfs-utils package installed and the Linux kernel must be version 2.6.29 or a more recent version. The new submit command encrypt_execute_directory allows the user to specify directory encryption on a per-job basis. (Ticket #4856).

• By default, HTCondor will no longer have access to Linux system credentials, such as OpenAFS tokens or eCryptFS keys. This new behavior ensures these credentials cannot be unintentionally obtained by user jobs. For more information, see new configuration variable DISCARD_SESSION_KEYRING_ON_STARTUP in section 3.3.8. (Ticket #4856).

• condor_q now offers the new command line option -autocluster, which causes it to output condor_schedd daemon auto cluster information. The information is an ID number and the number of jobs in each auto cluster. (Ticket #4792).

• EC2 grid universe jobs may now specify an IAM (instance) profile. (Ticket #4810).

• EC2 jobs may now specify security group IDs instead of names. This allows the use of VPC instances with non-default security groups. (Ticket #4796).

• EC2 jobs may now specify additional parameters to use when starting the corresponding instance. (Ticket #4795).

• Improved the preliminary support for IPv4 and IPv6 dual-protocol submit nodes, by allowing them to work with the condor_shared_port daemon. (Ticket #4712).

• The throughput of queries to the condor_collector has been improved, as the condor_collector now never forks to handle queries about condor_collector, condor_negotiator, and condor_schedd ClassAds. (Ticket #4821).

• The new configuration variable ADD_SIGNIFICANT_ATTRIBUTES lists job attributes to be added to the condor_negotiator-determined list when considering auto clustering. The new configuration variable REMOVE_SIGNIFICANT_ATTRIBUTES lists job attributes to be removed from the condor_negotiator-determined list when considering auto clustering. (Ticket #4797).

• For all tools that support the -format or -autoformat option, the new %r conversion specifier causes values to be displayed in their unevaluated, or raw form. (Ticket #4814).

• condor_dagman now prints the hold reason to the dagman.out file when node jobs go on hold. (Ticket #4766).

• New python bindings can be used to query and set the configuration variables of running daemons. (Ticket #4778).

• Python functions can be invoked directly from ClassAd expressions within HTCondor daemons. A system administrator must set the new configuration variable CLASSAD_USER_PYTHON_MODULES to specify the python modules that are accessible from within ClassAds. (Ticket #4773).

• The new Python LogReader class permits the reading and access of individual daemon log events. (Ticket #4588).

• The default value for configuration variable DAGMAN_MAX_JOBS_IDLE has changed from 0, which imposes no limits, to 1000. (Ticket #4748).

• Configuration variable DAGMAN_USE_OLD_DAG_READER is no longer supported. Setting it to True will result in a warning, and the setting will have no effect on how a DAG input file is read. (Ticket #4186).

• ClassAd attributes written by the condor_schedd that count the number of jobs in various states now include all jobs, not only jobs that need to be matched by the condor_negotiator daemon. These attributes include TotalRunningJobs, TotalIdleJobs, TotalHeldJobs, and TotalRemovedJobs. (Ticket #4683).

• condor_q and condor_history offer the new command line option -limit, which limits number of results returned. (Ticket #4806).

• A new, more efficient query protocol has been added as the default for condor_q, when querying a condor_schedd daemon that is version 8.3.3 or later. To disable this new protocol, set configuration variable CONDOR_Q_USE_V3_PROTOCOL to False. (Ticket #4828).

Bugs Fixed:

• Fixed a bug that prevented CCB servers from running on Windows platforms. (Ticket #4713).

• Configuration variables ENABLE_IPV4 or ENABLE_IPV6 may now be safely set in any configuration file. Previously, setting them in any file other than the first configuration file parsed could have led to unpredictable behavior. (Ticket #4829).

• Fixed a bug in the condor_startd daemon introduced in HTCondor version 8.3.2. The condor_startd slots could get stuck forever in the Preempting/Killing state when they were claimed by an HTCondor version 8.3.1 or older condor_schedd. (Ticket #4807).

• Fixed an issue introduced in HTCondor version 8.3.0 that could cause the condor_schedd to become unresponsive when a client set configuration variable CONDOR_Q_USE_V3_PROTOCOL to its non default value of True. (Ticket #4805).

• Configuration variable GRACEFULLY_REMOVE_JOBS now controls how a running job is killed for all cases, including when a job policy expression causes the job to be held or removed. Previously, this configuration variable was consulted only when the user ran condor_hold or condor_rm. (Ticket #4679).

Known Bugs:

• The ability to transparently encrypt execute directories is not supported by execute hosts using RHEL 7 and derivative distributions, as these distributions no longer contain the eCryptfs kernel module.

• The submit commands on_exit_hold and on_exit_remove do not do what they are supposed to do for local universe jobs on Windows machines. (Ticket #4902).

Version 8.3.2

Release Notes:

• HTCondor version 8.3.2 released on December 23, 2014. This version contains all bug fixes from HTCondor version 8.2.6.

New Features:

• It is now possible run a dual-protocol (IPv4 and IPv6) submit node, submitting to single-protocol execute nodes. This is preliminary work. (Ticket #4494).

• The port used by the condor_shared_port daemon is now 9618 by default. (Ticket #3813).

• Improved the handling when vm universe jobs failure to start. Failures which do not appear to be the fault of the job now cause the job to be rescheduled and the machine stops advertising the ability to run vm universe jobs. The new condor_condor_update_machine_ad tool facilitates changing the machine ClassAd. (Ticket #4375).

• The memory footprint of the condor_shadow has been reduced when Kerberos or SSL authentication methods are not used, as these libraries are now loaded on demand at run time. (Ticket #374).

• The responsiveness of a busy condor_schedd daemon to queries has been improved. (Ticket #4735). (Ticket #4594).

• Added the ability to specify the block device mapping for EC2 jobs. (Ticket #4657).

• The new python binding register() has been added to allow python functions be registered with the ClassAds library. This allows python functions to be invoked from within ClassAds. (Ticket #4598).

• The new python bindings externalRefs() and internalRefs() have been added to allow the ClassAd object to determine internal and external references from an expression. (Ticket #4681).

• When the condor_startd has a live condor_starter, claim keep alives are sent by the existing TCP connection between the condor_starter and condor_schedd, rather than creating a new connection to the condor_schedd from the condor_startd. (Ticket #4491).

• Added the DAGMan feature of ALWAYS-UPDATE for updates of a DAGMan node status file. Specifying this causes the node status file to be overwritten, even if no nodes have changed status since the file was last written. (Ticket #4607).

• Configuration variable MAX_JOBS_RUNNING has been modified such that it only applies to job universes that require a condor_shadow process. Scheduler and local universe jobs are no longer affected by this variable. The number of running scheduler and local universe jobs can be controlled with configuration variables START_SCHEDULER_UNIVERSE and START_LOCAL_UNIVERSE, respectively. (Ticket #4589).

• The specific versions of Globus GSI libraries to be loaded at run time are determined at compile time. (Ticket #4585).

• HTCondor now sets environment variable _CONDOR_JOB_AD for scheduler universe jobs. Its value will be the path to a file which contains the job ClassAd as it was when the job was started. This feature already exists for vanilla, parallel, java, and local universe jobs. (Ticket #4558).

• The new -debug option to condor_userprio sends debug output to stderr. (Ticket #4636).

• HTCondor daemons now support a whitelist of statistics attributes to publish from their ClassAd to the condor_collector. This is intended to ease configuration on systems that use ganglia for monitoring. (Ticket #4645).

• New statistics have been added to the condor_schedd to monitor runtime spent doing DNS queries, using fsync, and rebuilding the priority list for negotiation. Also additional attributes for average, maximum and minimum have been added to runtime statistics for command handlers for all HTCondor daemons. These changes are intended to help direct future scalability work. (Ticket #4593). (Ticket #4595).

• The new daemon logging level, D_SUB_SECOND, enables millisecond resolution timestamps in daemon logs. (Ticket #4660).

Bugs Fixed:

• Fixed a bug introduced in HTCondor version 8.3.1 that caused daemons to be unreachable if they were configured to use the condor_shared_port daemon, but the condor_master was not. (Ticket #4603).

• Updated the CREAM client library used in the cream_gahp. This fixes the delegation of RFC format proxies, in addition to other bug fixes. (Ticket #4596).

• Fixed a bug that could cause a segmentation fault of condor_dagman for some DAG input file syntax errors, rather than printing an appropriate error message. (Ticket #4616).

• Fixed a bug that could cause the condor_shared_port daemon to fail on Mac OS X platforms, if configuration variable LOCK was not explicitly set in a configuration file. (Ticket #4678).

• Fixed a bug that caused both condor_dagman and the condor_schedd daemon to generate commands to remove condor_dagman's node jobs when the condor_dagman job is the target of condor_rm. Now, only the condor_schedd generates the command, avoiding the extra load of running two identical commands. (Ticket #4618).

• Fixed a bug that caused the DAGMan node status file, as detailed in section 2.10.12, to not reflect the final status of a DAG when the DAG is removed by issuing a condor_rm command, or when the DAG is aborted due to an ABORT-DAG-ON specification in the DAG input file. (Ticket #4686).

Version 8.3.1

Release Notes:

• HTCondor version 8.3.1 released on September 11, 2014.

New Features:

• If cgroups are enabled on Linux platforms, the amount of swap space used by a job is now limited to the size specified by the machine ClassAd attribute VirtualMemory for the slot that the job is running on. (Ticket #4417).

• The new configuration variable COLLECTOR_PORT specifies the default port used by the condor_collector daemon and command line tools. The default value is 9618. This default is the same port as has been used in previous HTCondor versions. (Ticket #4432).

• The condor_shared_port daemon will now work if the default location given by configuration variable DAEMON_SOCKET_DIR, which is $(LOCK)/daemon_sock, is longer than 90 characters in length. On Linux platforms, abstract sockets are now the primary method for condor_shared_port to forward an incoming connection to the intended daemon. (Ticket #4465).

• Improvements to CCB increase performance. (Ticket #4453).

• The use of a single log file to write events and enforce the dependencies of a DAG represented by a condor_dagman instance is mandatory. To implement this, the -dont_use_default_node_log command-line option to condor_submit_dag is disabled, and an attempt to set configuration variable DAGMAN_ALWAYS_USE_NODE_LOG to False will generate an error. (Ticket #3994).

• The new condor_dagman configuration variable DAGMAN_SUPPRESS_JOB_LOGS allows users to prevent DAG node jobs from writing to the log file specified in their submit description file. See section 3.3.24 for details. (Ticket #4353).

• New special variables @(OWNER) and @(NODE_NAME) are available when defining configuration variable DAGMAN_DEFAULT_NODE_LOG. These values make it easier to avoid log file name collisions. (Ticket #4334).

• condor_submit will no longer insert an OpSys requirement for a job when one of OpSysAndVer, OpSysLongName, OpSysName, or OpSysShortName is already specified by the user in the Requirements expression of the submit description file. (Ticket #4519).

• The configuration file $(HOME)/.condor/condor_config is no longer considered for the single, initial, global configuration file. Instead, a user-specific configuration file has been added as the last file parsed. The new configuration variable USER_CONFIG_FILE may change the default file name or disable this feature. Section 3.3.1 describes the ordering in which configuration files are parsed. (Ticket #3158).

• Daemons now authenticate many client network connections in parallel, rather than one at a time. This improves the scalability of daemons that receive many client connections, like the condor_schedd and condor_collector. The improvement is most noticeable when using the FS and GSI authentication methods. (Ticket #4137).

• The GSI security libraries are now loaded into memory only when GSI authentication is required. This reduces memory usage when GSI authentication is not used. The memory reduction will be most noticeable when there are many condor_shadow processes running. (Ticket #4483).

• Implemented fine-grained locking in the HTCondor python module to allow other python threads to run during HTCondor calls. (Ticket #4507).

Bugs Fixed:

• None.

Version 8.3.0

Release Notes:

• HTCondor version 8.3.0 released on August 12, 2014. This release contains all improvements and bug fixes from HTCondor version 8.2.2.

New Features:

• When a daemon creates a child daemon process, it also creates a security session shared with the child daemon. This makes the initial communication between the daemons more efficient. (Ticket #4405).

• Negotiation cycle performance has been improved, especially over a wide-area network, by reducing network traffic and latency between a submit machine and a central manager. The new configuration variable NEGOTIATOR_RESOURCE_REQUEST_LIST_SIZE does performance tuning, as defined in section 3.3.16. (Ticket #4460).

• The synchronization of the job event log was improved by only using fsync() where necessary and fdatasync() where sufficient. This should provide a small reduction in disk I/O to the condor_schedd daemon. (Ticket #4283).

• CPU usage by the condor_collector has been reduced when handling normal queries from condor_status, and CPU usage by the condor_schedd has been reduced when handling normal queries from condor_q. (Ticket #4448).

• HTCondor can now internally cache the result of Globus authorization callouts. The caching behavior is enabled by setting configuration variable GSS_ASSIST_GRIDMAP_CACHE_EXPIRATION to a non-zero value. This feature will be useful for sites that use the Globus authorization callouts based only on DN and VOMS FQAN, and for sites that have performance issues. (Ticket #4138).

• The job ClassAd attribute DAG_Status is included in the dagman.out file. (Ticket #4381).

• The new -DoRecovery command line option for condor_dagman and condor_submit_dag causes condor_dagman to run in recovery mode. (Ticket #2218).

• The new -ads option to condor_status permits a set of ClassAds to be read from a file, processing the ClassAds as if they came from the condor_collector. (Ticket #4414).

• Daemon ClassAd hooks implementing Startd Cron functionality can now return multiple ClassAds, and the hooks can specify which ClassAds their output should merge into. (Ticket #4398).

• Two new condor_schedd ClassAd statistics attributes are available: JobsRunning and JobsAccumExceptionalBadputTime. (Ticket #4409).

Bugs Fixed:

• Fixed a bug that caused condor_dagman to unnecessarily attempt to read node job submit description files, which could cause spurious warnings when in recovery mode. Strictly speaking, the bug is fixed only for the default case in which DAGMAN_ALWAYS_USE_NODE_LOG is set to True. (Ticket #3843).

• Fixed a bug in the condor_schedd daemon that caused the values of the ClassAd attributes JobsRunningSizes and JobsRunningRuntimes to be much larger than they should have been. (Ticket #4409).