This is an outdated version of the HTCondor Manual. You can find current documentation at http://htcondor.org/manual.

10.3 Upgrading from the 8.2 series to the 8.4 series of HTCondor

Upgrading from the 8.2 series of HTCondor to the 8.4 series will bring new features introduced in the 8.3 series of HTCondor. These new features include:

• Implemented numerous scalability changes (such as reducing memory footprint, using fewer system resources, and streamlined algorithms) to handle 200,000 simultaneously running HTCondor jobs in a single pool.

• Added a Docker Universe to run a Docker container as an HTCondor job.

• New features increase the power of job specification in the submit description file.
  • Submit description files are now parsed the same as configuration files.

  • The queue submit command may be used in flexible and powerful new ways to specify job submissions. See section 2.5.2 for details. (Ticket #4819).

  • New macro functions are supported, and may be used in submit description files as well as in configuration. (Ticket #4944).

  • condor_submit has new command line options -queue and -dry-run, to provide flexible and powerful new ways to specify job submissions, as well as to test what job would be submitted without submitting. (Ticket #4933).

  • condor_submit now supports assignment of ClassAd attributes on the command line. (Ticket #4983).

  • condor_submit accepts if and include statements in the same way that configuration files do. (Ticket #4913).

• HTCondor pools can use IPv4 and IPv6 simultaneously.

• Execute Directories can be encrypted upon user or administrator request.

• Vanilla Universe jobs can utilize periodic application-level checkpoints.

• The administrator can establish requirements that must be satisfied in order for a job to be queued.

Upgrading from the 8.2 series of HTCondor to the 8.4 series will also introduce changes that administrators of sites running from an older HTCondor version should be aware of when planning an upgrade. Here is a list of items that administrators should be aware of.

• New configuration and changes to existing configuration:
  • The RPM packages have been restructured to allow running a 32-bit static shadow on Red Hat Enterprise Linux 6. The new condor-all RPM is used to install all of the RPMs for a typical HTCondor installation. Since the binary distribution of HTCondor for Red Hat Enterprise Linux 6 and 7 consists of more that a handful of RPMs, the RPMs are only available from our YUM repository. (Ticket #4621).

  • If enabled, the condor_shared_port daemon will now use port 9618 instead of the previous default, which was to randomly select a port from the allowed range (from LOWPORT to HIGHPORT; see section see section 3.9.1). To restore condor_shared_port's previous behavior, set SHARED_PORT_PORT to 0. (Ticket #4752).

  • Configuration variable MAX_JOBS_RUNNING has been modified such that it only applies to job universes that require a condor_shadow process. Scheduler and local universe jobs are no longer affected by this variable. The number of running scheduler and local universe jobs can be controlled with configuration variables START_SCHEDULER_UNIVERSE and START_LOCAL_UNIVERSE, respectively. (Ticket #4589).

  • ClassAd attributes written by the condor_schedd that count the number of jobs in various states now include all jobs, not only jobs that need to be matched by the condor_negotiator daemon. These attributes include TotalRunningJobs, TotalIdleJobs, TotalHeldJobs, and TotalRemovedJobs. (Ticket #4683).

  • On Linux platforms, the condor_master daemon now runs a script when it starts up. This script tunes several Linux kernel parameters to the values we suggest for better scalability. New configuration variables ENABLE_KERNEL_TUNING, KERNEL_TUNING_LOG, and LINUX_KERNEL_TUNING_SCRIPT enable the use of the script and specify file locations. (Ticket #5126).

  • The default values of the configuration variables and ClassAd attributes listed in Table 10.2 have changed, such that the default now represents the commonly configured value.
    
    

    Table 10.1: Changes to defaults in HTCondor 8.3.7

    Variable Name	Previous Default	New Default
    NEGOTIATOR_INFORM_STARTD	True	False
    MAX_JOBS_PER_SUBMISSION	largest positive integer	20000
    MAX_JOBS_PER_OWNER	largest positive integer	100000
    MAX_JOBS_RUNNING (on Windows)	200	2000
    PRIVATE_NETWORK_NAME	no default	$(FULL_HOSTNAME)
    JobLeaseDuration	20 minutes	40 minutes
    WANT_VACATE	False	True
    SCHEDD_SEND_VACATE_VIA_TCP	False	True

    
    

• None.