⇐ ↙ ↓ ⇑ ⇒ Contents Index
11.2 Development Release Series 8.7
This is the development release series of HTCondor. The details of each version are described below.
- HTCondor version 8.7.8 released on May 10, 2018.
- condor_annex may now be setup in multiple regions simultaneously. Use the -aws-region flag with
-setup to add new regions. Use the -aws-region flag with other condor_annex commands to choose
which region to operate in. You may change the default region by setting ANNEX_DEFAULT_AWS_REGION.
- Added default AMIs for all four US regions to simplify using condor_annex in those regions. (Ticket
- HTCondor will no longer mangle CUDA_VISIBLE_DEVICES or GPU_DEVICE_ORDINAL if those environment
variables are set when it starts up. As a result, HTCondor will report GPU usage with the original
device index (rather than starting over at 0). (Ticket #6584).
- When reporting GPUsUsage, HTCondor now also reports GPUsMemoryUsage. This is like MemoryUsage,
except it is the peak amount of GPU memory used by the job. This feature only works for nVidia
GPUs. (Ticket #6544).
- Improved error messages when delegation of an X.509 proxy fails. (Ticket #6575).
- condor_q will no longer limit the width of the output to 80 columns when it outputs to a file or pipe.
- Submission of jobs via the Python bindings Submit class will now attempt to put all jobs submitted
in a single transaction under the same ClusterId. (Ticket #6649).
- Added support for condor_schedd option in the Python bindings. (Ticket #6619).
- Eliminated SOAP support. (Ticket #6648).
- Fixed a problem where, when starting enough condor_annex instances simultaneously, some
(approximately 1 in 100) instances would neither join the pool nor terminate themselves. (Ticket
- When running in a HAD setup, there is a configuration parameter, COLLECTOR_HOST_FOR_NEGOTIATOR
which tells the active negotiator which collector to prefer. Previously, this parameter had no default, so
the negotiator might arbitrarily chose a far-away collector. Now this knob defaults to the local collector
in a HAD setup. (Ticket #6616).
- Fixed a bug when running in a configuration with more than one condor_collector, the
condor_negotiator would only send the accounting ads to one of them. The result of this bug is that
the condor_userprio tool would show now results about half of the time it was run. (Ticket #6615).
- Fixed a bug where condor_annex would fail with a malformed authorization header when using AWS
resources in a region other than us-east-1. (Ticket #6629).
- Fixed a bug that prevented Docker universe jobs with no executable listed in the submit file from
running. (Ticket #6612).
- Fixed a bug where the condor_starter would fail with an error after a docker job exits. (Ticket #6623).
- Fixed a bug where condor_userprio would always show zero resources in use when
NEGOTIATOR_CONSIDER_PREEMPTION=false was set. (Ticket #6621).
- Fixed a bug where “.update.ad” was not being updated atomically. (Ticket #6591).
- Fixed a bug that could cause a machine slot to become stuck in the Claimed/Busy state after a job
completes. (Ticket #6597).
- HTCondor version 8.7.7 released on March 13, 2018.
- condor_ssh_to_job now works with Docker Universe, the interactive shell is started inside the
container. This assume that there is a shell executable inside the container, but not necessarily an sshd.
- Improved error messages in the job log for Docker universe jobs that do not start. (Ticket #6567).
- Release a 32-bit condor_shadow for Enterprise Linux 7 platforms. (Ticket #6495).
- HTCondor now reports, in the job ad and user log, which custom machine resources were assigned to
the slot in which the job ran. (Ticket #6549).
- HTCondor now reports CPUsUsage for each job. This attribute is like MemoryUsage and DiskUsage,
except it is the average number of CPUs used by the job. (Ticket #6477).
- The use feature: GPUs metaknob now causes HTCondor to report GPUsUsage for each job. This is
like CPUsUsage, except it is the average number of GPUs used by the job. This feature only works for
nVIDIA GPUs. (Ticket #6477).
- Administrators may now, for each custom machine resource, define a custom resource monitor. Such a
script reports the usage(s) of each instance of the corresponding machine resource since the last time
it reported; HTCondor aggregates these reports between resource instances and over time to produce
a <Resource>Usage attribute, which is like GPUsUsage, except for the custom machine resource in
question. (Ticket #6477).
- The condor_startd now periodically writes a file to each job’s sandbox named “.update.ad”. This file
is a copy of the slot’s machine ad, but unlike “.machine.ad”, it is regularly updated. Jobs may read this
file to observe their own usage attributes. (Ticket #6477).
- A new option -unmatchable was added to condor_q that causes condor_q to show only jobs that
will not match any of the available slots. (Ticket #6529).
- OpenMPI jobs launched in the parallel universe via openmpiscript now work with shared file systems
(again). (Ticket #6556).
- Allow a parallel universe job with parallel scheduling group to select a new parallel scheduling group
when held and released. (Ticket #6516).
- Allow p-slot preemption to work with parallel universe. (Ticket #6517).
- Added the ability in condor_dagman to specify submit files with spaces in their path names. Paths
that include spaces must be wrapped in quotes (i.e. JOB A "/path to/job.sub"). (Ticket #6389).
- Added the ability in condor_submit to specify executable, error and output files with spaces in their
paths. Previously, adding whitespace to these fields would result in an error claiming certain attributes
could only take exactly one argument. Now, whitespace is treated as part of the path. (Ticket #6389).
- An IPv6 address can now be specified in the configuration file either with or without square brackets
in most cases. If specifying a port number in the same value, the square brackets are required. If using
a wild card to specify a range of possible addresses, square brackets are not allowed. (Ticket #5697).
- Improved support for IPv6 link-local addresses, in particular using the correct scope id. Using a wild
card or device name in NETWORK_INTERFACE now works properly when NO_DNS is set to True. (Ticket
- Python bindings installed via pip on a system without a HTCondor install (i.e. without a
condor_config present) will use a “null” config and print a warning. (Ticket #6515).
- The new configuration parameter NEGOTIATOR_JOB_CONSTRAINT defines an expression which constrains
which job ads are considered for matchmaking by the condor_negotiator. (Ticket #6250).
- The condor_startd will now keep trying to delete a job sandbox until it succeeds. The retries are
attempted with an exponential back off in frequency. (Ticket #6500).
- condor_q will no longer batch jobs with different cluster ids together unless they have the same
JobBatchName attribute or are in the same DAG. (Ticket #6532).
- condor_q will now sort jobs by job id when the -long argument is used. (Ticket #6287).
- Improve the performance of reading and writing ClassAds to the network. The performance of reading
ClassAds from UDP is particularly improved, up to 20% faster than previously. (Ticket #6555). (Ticket
- Several minor performance improvements. (Ticket #6550). (Ticket #6551). (Ticket #6565). (Ticket
- Removed configuration parameters ENABLE_ADDRESS_REWRITING and
SHARED_PORT_ADDRESS_REWRITING. (Ticket #6525).
- Removed the deprecated AvailStats attribute from the machine ad. This was being computing
incorrectly, and apparently never used. (Ticket #6526).
- Added basic support for a "Credential Management" subsystem which will eventually be used to
support interaction with OAuth services (like SciTokens, Box.com, Google Drive, DropBox, etc.). Still
in preliminary phases and not really ready for public use. (Ticket #6513).
- Fixed a bug where Docker universe jobs that exited via a signal did not properly report the signal.
- Fixed a bug where HTCondor would misreport the number of custom machine resources (GPUs)
allocated to a job in certain cases. (Ticket #6549).
- IPv4 addresses are now ignored when resolving a hostname and ENABLE_IPV4 is set to False. (Ticket
- Fixed a race condition in the condor_startd that could result in skipping the code that makes sure
that a job sandbox was deleted in the event that the condor_starter did not delete it. (Ticket #6524).
- Fixed a bug in condor_q when both the -tot and -global options were used, that would result in no
output when querying a condor_schedd running version 8.7 or later. (Ticket #6494).
- Fixed a bug that could prevent grid universe batch jobs from working properly on Debian and Ubuntu.
- HTCondor version 8.7.6 released on January 4, 2018.
- Changed the default value of configuration parameter IS_OWNER to False. The previous default value
is now set as part of the use POLICY : Desktop configuration template. (Ticket #6463).
- You may now use SCHEDD and JOB instead of MY and TARGET in SUBMIT_REQUIREMENTS
expressions. (Ticket #4818).
- Added cmake build option WANT_PYTHON_WHEELS and make target pypi_staging to build the
framework for Python wheels. This option and target are not enabled by default and are not likely to
work outside of Linux environments with a single Python installation. (Ticket #6486).
- Added new job attributes BatchProject and BatchRuntime for grid-type batch jobs. They specify the
project/allocation name and maximum runtime in seconds for the job that’s submited to the underlying
batch system. (Ticket #6451).
- HTCondor now respects ATTR_JOB_SUCCESS_EXIT_CODE when sending job notifications. (Ticket
- Added some graph metrics (height, width, etc.) to DAGMan’s metrics file output. (Ticket #6470).
- Removed Quill from HTCondor codebase. (Ticket #6496).
- HTCondor now reports all submit warnings, not just the first one. (Ticket #6446).
- The job log will no longer contain empty submit warnings. (Ticket #6465).
- DAGMan previously connected to condor_schedd every time it detected an update in its internal
state. This is too aggressive for rapidly changing DAGs, so we’ve changed the connection to happen in
time intervals defined by DAGMAN_QUEUE_UPDATE_INTERVAL, by default once every five minutes. (Ticket
- DAGMan now enforces the DAGMAN_MAX_JOB_HOLDS limit by the number of held jobs in a cluster at
the same time. Previously it counted all holds over the lifetime of a cluster, even if only a small number
of them are active at the same time. (Ticket #6492).
- Fixed a bug where on rare occasions the ShadowLog would become owned by root. (Ticket #6485).
- Fixed a bug where using condor_qedit to change any of the concurrency limits of a job would have no
effect. (Ticket #6448).
- When copy_to_spool is set to True, condor_submit now attempts to transfer the job exectuable only
once per job cluster, instead of once per job. (Ticket #6459).
- Fixed a bug that could result in an incorrect total reported by condor_rm when the -totals option is
used. (Ticket #6450).
- HTCondor version 8.7.5 released on November 14, 2017.
- HTCondor version 8.7.4 released on October 31, 2017.
- Added support for late materialization into condor_dagman. DAGs that include late materialized jobs
now work correctly in both normal and recovery conditions. (Ticket #6274).
- We now produce run time statistics in condor_dagman, tracking how much time DAGMan spends
idle, how much time it spends submitting jobs and processing log files. This information could be used
to determine why a DAG is submitting jobs slowly and how to optimize it. These statistics currently
get dumped into the .dagman.out file at the end of a DAGs execution. (Ticket #6411).
- Added a new knob to condor_dagman, DAGMAN_AGGRESSIVE_SUBMIT. When set to True,
this tells DAGMan to ignore the interval time limit for submitting jobs (defined by
DAGMAN_USER_LOG_SCAN_INTERVAL) and to continuously submit jobs until no more are ready, or until
it hits a different limit. (Ticket #6386).
- Added status command to condor_annex. This command invokes condor_status to display information
about annex instances that have reported to the collector. It also gathers information about annex
instances from EC2 and forwards that data to condor_status to detect instances which the collector
does not yet or any longer know about. The annex instance ads fabricated for this purpose are not real
slot ads, so some options you may know from condor_status do not apply to the status command of
condor_annex. See section 6 for details. (Ticket #6321).
- Added a “merge” mode to condor_status. When invoked with the [-merge <file>] option, ads will
be read from file, which can be - to indicate standard in, and compared to the ads selected by the
query specified as usual by the remainder of the command-line. Ads will be compared on the basis
of the sort key (which you can change with [-sort <key>]). condor_status will print three tables
based on that comparison: the first table will be generated from those ads whose key was in the query
but not in file; the second table will be generated from those ads whose key was appeared in both the
query and in file, and the third table will be generated from those ads whose key appeared only in file.
- Added off command to condor_annex. This command invokes condor_off -annex appropriately.
- Updated condor_annex -check-setup to check collector security as well as connectivity. (Ticket #6322).
- Added submit warnings. See section 3.7.2. (Ticket #5971).
- openmpiscript now uses condor_chirp to run Open MPI’s execute daemons (orted) directly under the
condor_starter (instead of using SSH). openmpiscript now also puts information about the number
of CPUs in the hostfile given to mpirun and now includes an option for jobs that intend to use hybrid
Open MPI+OpenMP. (Ticket #6403).
- The High Availability condor_replication daemon now works on machines using mixed IPV6/IPV4
addressing or using the condor_shared_port daemon. (Ticket #6413).
- When Docker universe starts a job, it no longer uses the docker run command line to do so. Now, it
first creates a container with docker create, then starts it with docker start. This allows HTCondor to
better isolate errors at container creation time, but should not result in any user visible changes at run
time. The StarterLog will now always print the docker command line for the start and create, and
not the run that it used to. (Ticket #6377).
- When docker universe reports memory usage, it now reports the RSS (Resident Set Size) of the
container, previously it reported RSS + page cache size (Ticket #6430).
- Added support for both user and daemon authentication using the MUNGE service. (Ticket #6404).
- Added a new -macro argument to condor_config_val. This argument causes condor_config_val to
show the results of doing $() expansion of its arguments as if they were the result of a look up rather
than the names of configuration variables to look up. (Ticket #6416).
- CErequirements for the BLAHP can now be expressed in a simple form such as a string or nested
ClassAd. (Ticket #6133).
- Fixed a bug introduced in 8.7.0 where the job attributes RemoteUserCpu and RemoteSysCpu where
never updated in the history file, or in condor_q output. The user log would show the correct values.
- The new behavior of the -expand command line argument of condor_config_val was breaking some
scripts, so that functionality has been moved and -expand reverted to the pre 8.7.2 behavior. (Ticket
- Grid type boinc jobs are now considered running when they are reported as IN_PROGRESS. (Ticket
- HTCondor version 8.7.3 released on September 12, 2017.
- Our current implementation of late materialization is incompatible with condor_dagman and will cause
unexpected behavior, including failing without warning. This is a top-priority issue which aim to resolve
in an upcoming release. (Ticket #6292).
- Changed condor_top tool to monitor the condor_schedd by default, to show more useful columns in
the default view, to better format output when redirected or piped, and to optionally take input of
two ClassAd files. (Ticket #6352).
- Changed how auto works for ENABLE_IPV4 and ENABLE_IPV6. HTCondor now ignores addresses that
are likely to be useless (loopback or link-local) unless no address is likely to be useful (private or public).
- Added support for Public Input Files in HTCondor jobs. This allows users to transfer input files over
a publicly-available HTTP web service, which can benefit from caching proxies, load balancers, and
other tools to improve file transfer performance. (Ticket #6356).
- Added -grid:ec2 to condor_q to avoid truncating AWS’ new, longer, instance IDs. Replaced useless
(given the instance ID) instance host name with the CMD column, to help distinguish EC2 jobs from
each other. (Ticket #5478).
- Added statistical output for job input files transferred from web servers using the curl_plugin tool.
Statistics are stored in ClassAd format, saved by default to a transfer_history file in the local logs
folder. (Ticket #6229).
- Fixed some small memory leaks in the HTCondor daemons. (Ticket #6361).
- Fixed a bug that would prevent dollar-dollar expansion from working correctly for parallel universe
jobs running on partitionable slots. (Ticket #6370).
- HTCondor version 8.7.2 released on June 22, 2017.
- Our current implementation of late materialization is incompatible with condor_dagman and will cause
unexpected behavior, including failing without warning. This is a top-priority issue which aim to resolve
in an upcoming release. (Ticket #6292).
- Improved the performance of the condor_schedd by setting the default for the knob
SUBMIT_SKIP_FILECHECKS to true. This prevents the condor_schedd from checking the readability of
all input files, and skips the creation of the output files on the submit side at submit time. Output
files are now created either at transfer time, when file transfer is on, or by the job itself, if a shared
filesystem is used. As a result of this change, it is possible that a job will run to completion, and only
then is put on hold because the output file on the submit machine cannot be written. (Ticket #6220).
- Changed condor_submit to not create empty stdout and stderr files before submitting jobs by
default. This caused confusion for users, and slowed down the submission process. The older behavior,
where condor_submit would fail if it could not create this files, is available when the parameter
SUBMIT_SKIP_FILECHECKS is set to false. The default is now true. (Ticket #6220).
- condor_q will now show expanded totals when querying a condor_schedd that is version 8.7.1 or later.
The totals for the current user and for all users are provided by the condor_schedd. To get the old
totals display set the configuration parameter CONDOR_Q_SHOW_OLD_SUMMARY to true. (Ticket #6254).
- The condor_annex tool now logs to the user configuration directory. Added an audit log of
condor_annex commands and their results. (Ticket #6267).
- Changed condor_off so that the -annex flag implies the -master flag, since this is more likely to be
the right thing. (Ticket #6266).
- Added -status flag to condor_annex, which reports on instances which are running but not in the
pool. (Ticket #6257).
- If invoked with an annex name and duration (but not an instance or slot count), condor_annex will
now adjust the duration of the named annex. (Ticket #6161).
- Job input files which are downloaded from http:// web addresses now have mechanisms to recover from
transfer failures. This should increase the reliability of using web-based input files, especially under
slow and/or unstable network conditions. (Ticket #5886).
- Reduced load on the condor_collector by optimizing queries performed when an HTCondor daemon
needs to look up the address of another daemon. (Ticket #6223).
- Reduced load on the condor_collector by optimizing queries performed when using condor_q with
several different command-line options such as -submitter and -global. (Ticket #6222).
- Added the condor_top tool, an automated version of the now-defunct condor_top.pl which uses the
python bindings to monitor the status of daemons. (Ticket #6205).
- Added a new option -cron to condor_gpu_discovery that allows it to be used directly as an executable
of a condor_startd cron job. (Ticket #6012).
- The configuration variable MAX_RUNNING_SCHEDULER_JOBS_PER_OWNER was set to default to 100. It
formerly had no default value. (Ticket #6260).
- Added a parameter DEDICATED_SCHEDULER_USE_SERIAL_CLAIMS which defaults to false. When true,
allows the dedicated schedule to use claimed/idle slots that the serial scheduler has claimed. (Ticket
- The condor_advertise tool now assumes an update command if one is not specified on the command-line
and attempts to determine exact command by inspecting the first ad to be advertised. (Ticket #6296).
- Improved support for running several condor_negotiators in a single pool. NEGOTIATOR_NAME now
works like MASTER_NAME. condor_userprio has a -name option to select a specific condor_negotiator.
Accounting ads from multiple condor_negotiators can co-exist in the condor_collector. (Ticket #5717).
- Package EC2 Annex components in the condor-annex-ec2 sub RPM. (Ticket #6202).
- Added configuration parameter ALTERNATE_JOB_SPOOL, an expression evaluated against the job ad,
which specifies an alternate spool directory to use for files related to that job. (Ticket #6221).
- With an empty configuration file, HTCondor would behave as if ALLOW_ADMINISTRATOR were *.
Changed the default to $(CONDOR_HOST), which is much less insecure. (Ticket #6230).
- Fixed a bug in the condor_schedd where it did not account for the initial state of late materialize
jobs when calculating the running totals of jobs by state. This bug resulted in condor_q displaying
incorrect totals when CONDOR_Q_SHOW_OLD_SUMMARY was set to false. (Ticket #6272).
- Fixed a bug where the condor_schedd would incorrectly try to check the validity of output files
and directories for late materialize jobs. The condor_schedd will now always skip file checks for late
materialize jobs. (Ticket #6246).
- Changed the output of the condor_status command so that the Load Average field now displays the
load average of just the condor job running in that slot. Previously, load associated from outside of
condor was proportionately distributed into the condor slots, resulting in much confusion. (Ticket
- Illegal chars (’+’, ’.’) are now prohibited in DAGMan node names. (Ticket #5966).
- Improve audit log messages by including the connection ID and properly filtering out shadow and
gridmanager modifications to the job queue log. (Ticket #6289).
- condor_root_switchboard has been removed from the release, since PrivSep is no longer supported.
- HTCondor version 8.7.1 released on April 24, 2017.
- Previously, when the number of forked children processing Collector queries surpassed the maximum
set by the configuration knob COLLECTOR_QUERY_WORKERS, the Collector handled all new incoming
queries in-processes (i.e. without forking). As processing a query and sending out the result
to the network could take a long time, the result of servicing such queries in-process in the
Collector is likely to drop a lot of updates. So now in v8.7.1, instead of servicing such queries
in-process, they are queued up for servicing as soon as query worker child processes become
available. The configuration knob COLLECTOR_QUERY_WORKERS_PENDING was introduced; see sectionefparam:CollectorQueryWorkersPending. (Ticket #6192).
- Default value for COLLECTOR_QUERY_WORKERS changed from 2 to 4. (Ticket #6192).
- Introduced configuration macro COLLECTOR_QUERY_WORKERS_RESERVE_FOR_HIGH_PRIO so that the
collector prioritizes queries that are important for the operation of the pool (such as queries from the
negotiator) ahead of servicing user invocations of condor_status. (Ticket #6192).
- Introduced configuration macro COLLECTOR_QUERY_MAX_WORKTIME to define the maximum amount
of time the collector may service a query from a client like condor_status. See sectionefparam:CollectorQueryMaxWorktime. (Ticket #6192).
- Added several new statistics on collector query performance into the Collector ClassAd,
including ActiveQueryWorkers, ActiveQueryWorkersPeak, PendingQueries, PendingQueriesPeak,
DroppedQueries, and RecentDroppedQueries. See sectionefsec:Collector-ClassAd-Attributes.
- Further refinement and initial documentation of the HTCondor Annex. (Ticket #6147). (Ticket
#6149). (Ticket #6150). (Ticket #6155). (Ticket #6157). (Ticket #6184). (Ticket #6196). (Ticket
#6216). (Ticket #6218).
- Docker universe jobs can now use condor_chirp command (if it is in the image). (Ticket #6162).
- In the Job Router, when a candidate job matches multiple routes, the first route is now always
selected. The old behavior of spreading jobs across all matching routes round-robin style can be enabled
by setting the new configuration parameter JOB_ROUTER_ROUND_ROBIN_SELECTION to True. (Ticket
- The condor_schedd now keeps a count of jobs by state for each owner and submitter and will
report them to condor_q. Condorq will display these totals unless the new configuration parameter
CONDOR_Q_SHOW_OLD_SUMMARY is set to true. In 8.7.1 this parameter defaults to true. (Ticket #6160).
- Milestone 1 for late materialization in the condor_schedd was completed. This milestone adds the
undocumented option -factory to condor_q that can be used to submit a late materializing job
cluster to the condor_schedd. The condor_schedd will refuse the submission unless the configuration
parameter SCHEDD_ALLOW_LATE_MATERIALIZATION is set to true. (Ticket #6212).
- Increased the default value for configuration parameter NEGOTIATOR_SOCKET_CACHE_SIZE to 500.
- Added new DaemonCore statistics UdpQueueDepth to measure the number of bytes in the UDP receive
queue for daemons with a UDP command port. (Ticket #6183).
- Improved speed of handling queries to the collector by caching the the configuration knob
SHARED_PORT_ADDRESS_REWRITING. (Ticket #6187).
- The condor_collector on Linux now handles some queries in process and some by forking a child
process. This allows it to avoid the overhead of forking to handle queries that will take little time. The
policy for deciding which queries to handle in process is controlled by a new configuration parameter
HANDLE_QUERY_IN_PROC_POLICY. (Ticket #6191).
- Added -limit option to condor_status and changed the condor_collector to honor it. (Ticket #6198).
- condor_submit was changed to use the same utility library that the submit python bindings use. This
should help insure that submit via python bindings will give the same results as using condor_submit.
- HTCondor version 8.7.0 released on March 2, 2017.
- Optimized the code that reads reads ClassAds off the wire making the maximum possible update rate
for the Collector about 1.7 times higher than it was before. (Ticket #6105). (Ticket #6130).
- New statistics have been added to the Collector ad to show time spent handling queries. (Ticket #6123).
- Changed the formatting of the printing of ClassAd expressions with parentheses. Now there is no space
character after every open parenthesis, or before every close parenthesis This looks more natural, is
somewhat faster for the condor to parse, and saves space. That is, an expression that used to print like
( ( ( foo ) ) )
now will print like this
- Technology preview of the HTCondor Annex. The HTCondor Annex allows one to extend their HTCondor pool into
the cloud. https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToUseCondorAnnexWithOnDemandInstances
- Added -annex option to condor_status and condor_off . Requires an argument; the request is constrained to
match machines whose AnnexName ClassAd attribute matches the argument. (Ticket #6116). (Ticket
- A refreshed X.509 proxy is now forwarded to the remote cluster in Bosco. (Ticket #5841).
- Added several new statistics to the Negotiator ad, mainly detailing how time is spent in the negotiation cycle.
- Removed redundant updates to the job queue by the Job Router. (Ticket #6102).
⇐ ↙ ↑ ⇑ ⇒ Contents Index