The Python module provides bindings to the client-side APIs for HTCondor and the ClassAd language.
These Python bindings depend on loading the HTCondor shared libraries; this means the same code is used here as the HTCondor client tools. It is more efficient in terms of memory and CPU to utilize these bindings than to parse the output of the HTCondor client tools when writing applications in Python.
Currently, the Python bindings are only available on Linux platforms.
htcondor module functions:
| platform( )
Returns the platform of HTCondor this module is running on. |
| version( )
Returns the version of HTCondor this module is linked against. |
| reload_config( )
Reload the HTCondor configuration from disk. |
| send_command( ad, (DaemonCommands)dc, (str)target = None)
Send a command to an HTCondor daemon specified by a location ClassAd. ad is a ClassAd specifying the location of the daemon; typically, found by using Collector.locate(...). dc is a command type; must be a member of the enum DaemonCommands. target is an optional parameter, representing an additional command to send to a daemon. Some commands require additional arguments; for example, sending DaemonOff to a condor_master requires one to specify which subsystem to turn off. |
| read_events( file_obj, is_xml = True )
Read and parse an HTCondor event log file. Returns a python iterator of ClassAds. Parameter file_obj is a file object corresponding to an HTCondor event log. The optional parameter is_xml specifies whether the event log is XML-formatted. |
| lock( file_obj, lock_type )
Take a lock on a file object using the HTCondor locking protocol, which is distinct from typical POSIX locks. Returns a context manager object; the lock is released as this context manager object is destroyed. Parameter file_obj is a file object corresponding to the file which should be locked. Parameter lock_type specifies the string "ReadLock" if the lock should be for reads or "WriteLock" if the lock should be for writes. |
| enable_debug( )
Enable debugging output from HTCondor, where output is sent to stderr. The logging level is controlled by TOOL_DEBUG. |
| enable_log( )
Enable debugging output from HTCondor, where output is sent to a file. The log level is controlled by TOOL_DEBUG, and the file used is controlled by TOOL_LOG. |
The module object, param, is a dictionary-like object providing access to the configuration variables in the current HTCondor configuration.
The Schedd class:
| __init__( classad )
Create an instance of the Schedd class. Optional parameter classad describes the location of the remote condor_schedd daemon. If the parameter is omitted, the local condor_schedd daemon is used. |
| transaction( flags = 0, continue_txn = False )
Start a transaction with the condor_schedd. Returns a transaction context manager. Starting a new transaction while one is ongoing is an error. The optional parameter flags defaults to 0. Transaction flags are from the the enum htcondor.TransactionFlags, and the three flags are NonDurable, SetDirty, or ShouldLog. NonDurable is used for performance, as it eliminates extra fsync() calls. If the condor_schedd crashes before the transaction is written to disk, the transaction will be retried on restart of the condor_schedd. SetDirty marks the changed ClassAds as dirty, so an update notification is sent to the condor_shadow and the condor_gridmanager. ShouldLog causes changes to the job queue to be logged in the job event log file. The optional parameter continue_txn defaults to false; set the value to true to extend an ongoing transaction. |
| act( (JobAction)action, (object)job_spec )
Change status of job(s) in the condor_schedd daemon. The integer return value is a ClassAd object describing the number of jobs changed. Parameter action is the action to perform; must be of the enum JobAction. Parameter job_spec is the job specification. It can either be a list of job IDs or a string specifying a constraint to match jobs. |
| edit( (object)job_spec, (str)attr, (object)value )
Edit one or more jobs in the queue. Parameter job_spec is either a list of jobs, with each given as ClusterId.ProcId or a string containing a constraint to match jobs against. Parameter attr is the attribute name of the attribute to edit. Parameter value is the new value of the job attribute. It should be a string, which will be converted to a ClassAd expression, or an ExprTree object. |
| query( constraint = true, attr_list = [] )
Query the condor_schedd daemon for jobs. Returns a list of ClassAds representing the matching jobs, containing at least the requested attributes requested by the second parameter. The optional parameter constraint provides a constraint for filtering out jobs. It defaults to True. Parameter attr_list is a list of attributes for the condor_schedd daemon to project along. It defaults to having the condor_schedd daemon return all attributes. |
| xquery( constraint = true, attr_list = [] )
Query the condor_schedd daemon for jobs. Returns an iterator of ClassAds representing the matching jobs containing at least the list of attributes requested by the second parameter. The optional parameter constraint provides a constraint for filtering out jobs. It defaults to True. Parameter attr_list is a list of attributes for the condor_schedd daemon to project along. It defaults to having the condor_schedd daemon return all attributes. |
| history( (object) requirements, (list) projection, (int) match )
Request history records from the condor_schedd daemon. Returns an iterator to a set of ClassAds representing completed jobs. Parameter requirements is either an ExprTree or a string that can be parsed as an expression. The expression represents the requirements that all returned jobs should match. Parameter projection is a list of all the ClassAd attributes that are to be included for each job. The empty list causes all attributes to be included. Parameter match is an integer cap on the number of jobs to include. |
| submit( ad, count = 1, spool = false, ad_results = None )
Submit one or more jobs to the condor_schedd daemon. Returns the newly created cluster ID. This method requires the invoker to provide a ClassAd for the new job cluster; such a ClassAd contains attributes with different names than the commands in a submit description file. As an example, the stdout file is referred to as output in the submit description file, but Out in the ClassAd. To generate an example ClassAd, take a sample submit description file and invoke condor_submit -dump <filename> [cmdfile] Then, load the resulting contents of <filename> into Python. Parameter ad is the ClassAd describing the job cluster. Parameter count is the number of jobs to submit to the cluster. Defaults to 1. Parameter spool inserts the necessary attributes into the job for it to have the input files spooled to a remote condor_schedd daemon. This parameter is necessary for jobs submitted to a remote condor_schedd. Parameter ad_results, if set to a list, will contain the job ClassAds resulting from the job submission. These are useful for interacting with the job spool at a later time. |
| spool( ad_list )
Spools the files specified in a list of job ClassAds to the condor_schedd. Throws a RuntimeError exception if there are any errors. Parameter ad_list is a list of ClassAds containing job descriptions; typically, this is the list filled by the ad_results argument of the submit method call. |
| retrieve( job_spec )
Retrieve the output sandbox from one or more jobs. Parameter job_spec is an expression string matching the list of job output sandboxes to retrieve. |
| refreshGSIProxy(cluster, proc, filename, lifetime)
Refresh the GSI proxy of a job with job identifier given by parameters cluster and proc. This will refresh the remote proxy with the contents of the file identified by parameter filename. Parameter lifetime indicates the desired lifetime (in seconds) of the delegated proxy. A value of 0 specifies to not shorten the proxy lifetime. A value of -1 specifies to use the value of configuration variable DELEGATE_JOB_GSI_CREDENTIALS_LIFETIME. Note that, depending on the lifetime of the proxy in filename, the resulting lifetime may be shorter than the desired lifetime. |
The Collector class:
| __init__( pool = None )
Create an instance of the Collector class. Optional parameter pool is a string with host:port pair specified or a list of pairs. If omitted, the value of configuration variable COLLECTOR_HOST is used. |
| locate( (DaemonTypes)daemon_type, (str)name )
Query the condor_collector for a particular daemon. Returns the ClassAd of the requested daemon. Parameter daemon_type is the type of daemon; must be of the enum DaemonTypes. Optional parameter name is the name of daemon to locate. If not specified, it searches for the local daemon. |
| locateAll( (DaemonTypes)daemon_type )
Query the condor_collector daemon for all ClassAds of a particular type. Returns a list of matching ClassAds. Parameter daemon_type is the type of daemon; must be of the enum DaemonTypes. |
| query( (AdTypes)ad_type, constraint=True, attrs=[] )
Query the contents of a condor_collector daemon. Returns a list of ClassAds that match the constraint parameter. Optional parameter ad_type is the type of ClassAd to return, where the types are from the enum AdTypes. If not specified, the type will be ANY_AD. Optional parameter constraint is a constraint for the ClassAd query. It defaults to True. Optional parameter attrs is a list of attributes. If specified, the returned ClassAds will be projected along these attributes. |
| advertise( ad_list, command=UPDATE_AD_GENERIC, use_tcp = True )
Advertise a list of ClassAds into the condor_collector. Parameter ad_list is the list of ClassAds to advertise. Optional parameter command is a command for the condor_collector. It defaults to UPDATE_AD_GENERIC. Other commands, such as UPDATE_STARTD_AD, may require reduced authorization levels. Optional parameter use_tcp causes updates to be sent via TCP. Defaults to True. |
The Negotiator class:
| __init__( (ClassAd)ad = None )
Create an instance of the Negotiator class. Optional parameter ad is a ClassAd containing the location of the condor_negotiator daemon. If omitted, uses the local pool. |
| deleteUser( (str)user )
Delete a user from the accounting. user is a fully-qualified user name, "USER@DOMAIN". |
| getPriorities( [(bool)rollup = False ] )
Retrieve the pool accounting information. Returns a list of accounting ClassAds. Optional parameter rollup identifies if accounting information, as applied to hierarchical group quotas, should be summed for groups and subgroups (True) or not (False, the default). |
| getResourceUsage( (str)user )
Get the resource usage for a specified user. Returns a list of ClassAd attributes. Parameter user is a fully-qualified user name, "USER@DOMAIN". |
| resetAllUsage( )
Reset all usage accounting. |
| resetUsage( (str)user )
Reset all usage accounting of the specified user. Parameter user is a fully-qualified user name, "USER@DOMAIN"; resets the usage of only this user. |
| setBeginUsage( (str)user, (time_t)value )
Initialize the time that a user begins using the pool. Parameter user is a fully-qualified user name, "USER@DOMAIN". Parameter value is the time of initial usage. |
| setLastUsage( (str)user, (time_t)value )
Set the time that a user last began using the pool. Parameter user is a fully-qualified user name, "USER@DOMAIN". Parameter value is the time of last usage. |
| setFactor( (str)user, (float)factor )
Set the priority factor of a specified user. Parameter user is a fully-qualified user name, "USER@DOMAIN". Parameter factor is the priority factor to be set for the user; must be greater than or equal to 1.0. |
| setPriority( (str)user, (float)prio )
Set the real priority of a specified user. Parameter user is a fully-qualified user name, "USER@DOMAIN". Parameter prio is the priority to be set for the user; must be greater than 0.0. |
| setUsage( (str)user, (float)usage )
Set the accumulated usage of a specified user. Parameter user is a fully-qualified user name, "USER@DOMAIN". Parameter usage is the usage to be set for the user. |
The SecMan class accesses the internal security object. This class allows access to the security layer of HTCondor.
Currently, this is limited to resetting security sessions and doing test authorizations against remote daemons.
If a security session becomes invalid, for example, because the remote daemon restarts, reuses the same port, and the client continues to use the session, then all future commands will fail with strange connection errors. This is the only mechanism to invalidate in-memory sessions.
| __init__( )
Create a SecMan object. |
| invalidateAllSessions( )
Invalidate all security sessions. Any future connections to a daemon will cause a new security session to be created. |
| ping ( (ClassAd)ad, (str)command )
or ping ( (string)sinful, (str)command ) Perform a test authorization against a remote daemon for a given command. Returns the ClassAd of the security session. Parameter ad is the ClassAd of the daemon as returned by Collector.locate; alternately, the sinful string can be given directly as the first parameter. Optional parameter command is the DaemonCore command to try; if not given, DC_NOP will be used. |
Module enums:
| AdTypes
A list of types used as values for the MyType ClassAd attribute. These types are only used by the HTCondor system, not the ClassAd language. Typically, these specify different kinds of daemons. |
| DaemonCommands
A list of commands which can be sent to a remote daemon. |
| DaemonTypes
A list of types of known HTCondor daemons. |
| JobAction
A list of actions that can be performed on a job in a condor_schedd. |
$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import htcondor
>>> import classad
>>> coll = htcondor.Collector("red-condor.unl.edu")
>>> results = coll.query(htcondor.AdTypes.Startd, "true", ["Name"])
>>> len(results)
3812
>>> results[0]
[ Name = "slot1@red-d20n35"; MyType = "Machine"; TargetType = "Job"; CurrentTime = time() ]
>>> scheddAd = coll.locate(htcondor.DaemonTypes.Schedd, "red-gw1.unl.edu")
>>> scheddAd["ScheddIpAddr"]
'<129.93.239.132:53020>'
>>> schedd = htcondor.Schedd(scheddAd)
>>> results = schedd.query('Owner =?= "cmsprod088"', ["ClusterId", "ProcId"])
>>> len(results)
63
>>> results[0]
[ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356722353; ClusterId = 674143; ProcId = 0; CurrentTime = time() ]
>>> htcondor.param["COLLECTOR_HOST"]
'hcc-briantest.unl.edu'
>>> schedd = htcondor.Schedd() # Defaults to the local schedd.
>>> results = schedd.query()
>>> results[0]["RequestMemory"]
ifthenelse(MemoryUsage isnt undefined,MemoryUsage,( ImageSize + 1023 ) / 1024)
>>> results[0]["RequestMemory"].eval()
1L
>>> ad=classad.parse(open("test.submit.ad"))
>>> print schedd.submit(ad, 2) # Submits two jobs in the cluster; edit test.submit.ad to preference.
110
>>> print schedd.act(htcondor.JobAction.Remove, ["111.0", "110.0"])'
[
TotalNotFound = 0;
TotalPermissionDenied = 0;
TotalAlreadyDone = 0;
TotalJobAds = 2;
TotalSuccess = 2;
TotalChangedAds = 1;
TotalBadStatus = 0;
TotalError = 0
]
>>> print schedd.act(htcondor.JobAction.Hold, "Owner =?= \"bbockelm\"")'
[
TotalNotFound = 0;
TotalPermissionDenied = 0;
TotalAlreadyDone = 0;
TotalJobAds = 2;
TotalSuccess = 2;
TotalChangedAds = 1;
TotalBadStatus = 0;
TotalError = 0
]
>>> schedd.edit('Owner =?= "bbockelm"', "Foo", classad.ExprTree('"baz"'))
>>> schedd.edit(["110.0"], "Foo", '"bar"')
>>> coll = htcondor.Collector()
>>> master_ad = coll.locate(htcondor.DaemonTypes.Master)
>>> htcondor.send_command(master_ad, htcondor.DaemonCommands.Reconfig) # Reconfigures the local master and all children
>>> htcondor.version()
'$CondorVersion: 7.9.4 Jan 02 2013 PRE-RELEASE-UWCS $'
>>> htcondor.platform()
'$CondorPlatform: X86_64-ScientificLinux_6.3 $'
The bindings can use a dictionary where a ClassAd is expected. Here is an example that uses the ClassAd:
htcondor.Schedd().submit(classad.ClassAd({"Cmd": "/bin/echo"}))
This same example, using a dictionary instead of constructing a ClassAd:
htcondor.Schedd().submit({"Cmd": "/bin/echo"})
The classad module class provides a dictionary-like mechanism for interacting with the ClassAd language. classad objects implement the iterator interface to iterate through the classad's attributes. The constructor can take a dictionary, and the object can take lists, dictionaries, and ClassAds as values.
classad module functions:
| parse( input )
Parse input into a ClassAd. Returns a ClassAd object. Parameter input is a string-like object or a file pointer. |
| parseOld( )
Parse old ClassAd format input into a ClassAd. |
| version( )
Return the version of the linked ClassAd library. |
| Attribute( name )
Given the string name, return an ExprTree object which is a reference to an attribute of that name. The ClassAd expression foo == 1 can be constructed by the python Attribute("foo") == 1. |
| Function( name, arg1, arg2, ... )
Given function name name, and zero-or-more arguments, construct an ExprTree which is a function call expression. The function is not evaluated. The ClassAd expression strcat("hello ", "world") can be constructed by the python Function("strcat", "hello ", "world"). |
| Literal( obj )
Given python object obj, convert it to a ClassAd literal. Python strings, floats, integers, and booleans have equivalent literals. |
Standard Python object methods for the ClassAd class:
| __init__( str )
Create a ClassAd object from string, str, passed as a parameter. The string must be formatted in the new ClassAd format. |
| __len__( )
Returns the number of attributes in the ClassAd; allows len(object) semantics for ClassAds. |
| __str__( )
Converts the ClassAd to a string and returns the string; the formatting style is new ClassAd, with square brackets and semicolons. For example, [ Foo = "bar"; ] may be returned. |
The classad object has the following dictionary-like methods:
| items( )
Returns an iterator of tuples. Each item returned by the iterator is a tuple representing a pair (attribute,value) in the ClassAd object. |
| values( )
Returns an iterator of objects. Each item returned by the iterator is a value in the ClassAd. If the value is a literal, it will be cast to a native Python object, so a ClassAd string will be returned as a Python string. |
| keys( )
Returns an iterator of strings. Each item returned by the iterator is an attribute string in the ClassAd. |
| get( attr, value )
Behaves like the corresponding Python dictionary method. Given the attr as key, returns either the value of that key, or if the key is not in the object, returns None or the optional second parameter when specified. |
| __getitem__( attr )
Returns (as an object) the value corresponding to the attribute attr passed as a parameter. ClassAd values will be returned as Python objects; ClassAd expressions will be returned as ExprTree objects. |
| __setitem__( attr, value )
Sets the ClassAd attribute attr to the value. ClassAd values will be returned as Python objects; ClassAd expressions will be returned as ExprTree objects. |
| setdefault( attr, value )
Behaves like the corresponding Python dictionary method. If called with an attribute, attr, that is not set, it will set the attribute to the specified value. It returns the value of the attribute. If called with an attribute that is already set, it does not change the object. |
| update( object )
Behaves like the corresponding Python dictionary method. Updates the ClassAd with the key/value pairs of the given object. Returns nothing. |
Additional methods:
| eval( attr )
Evaluate the value given a ClassAd attribute attr. Throws ValueError if unable to evaluate the object. Returns the Python object corresponding to the evaluated ClassAd attribute. |
| lookup( attr )
Look up the ExprTree object associated with attribute attr. No attempt will be made to convert to a Python object. Returns an ExprTree object. |
| printOld( )
Print the ClassAd in the old ClassAd format. Returns a string. |
| quote( str )
Converts the Python string, str, into a ClassAd string literal. Returns the string literal. |
| unquote( str )
Converts the Python string, str, escaped as a ClassAd string back to a Python string. Returns the Python string. |
| parseAds( input )
Given input of a string or file, return an iterator of ClassAds where the ClassAds are in the New ClassAd format. Returns an iterator. |
| parseOldAds( input )
Given input of a string or file, return an iterator of ClassAds where the ClassAds are in the Old ClassAd format. Returns an iterator. |
| flatten( expression )
Given ExprTree object expression, perform a partial evaluation. All the attributes in expression and defined in this object are evaluated and expanded. Any constant expressions, such as 1 + 2, are evaluated. Returns a new ExprTree object. |
| matches( ad )
Given ClassAd object ad, check to see if this object matches the Requirements attribute of ad. Returns true if it does. |
| symmetricMatch( ad )
Returns true if the given ad matches this and this matches ad. Equivalent to self.matches(ad) and ad.matches(self). |
The ExprTree class object represents an expression in the ClassAd language. The python operators for ExprTree have been overloaded so, if e1 and e2 are ExprTree objects, then e1 + e2 is also a ExprTree object. Lazy-evaluation is used, so an expression "foo" + 1 does not produce an error until it is evaluated with a call to bool() or the .eval() class member.
ExprTree class methods:
| __init__( str )
Parse the string str to create an ExprTree. |
| __str__( )
Represent and return the ClassAd expression as a string. |
| eval( )
Evaluate the expression and return as a ClassAd value, typically a Python object. |
$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import classad
>>> ad = classad.ClassAd()
>>> expr = classad.ExprTree("2+2")
>>> ad["foo"] = expr
>>> print ad["foo"].eval()
4
>>> ad["bar"] = 2.1
>>> ad["baz"] = classad.ExprTree("time() + 4")
>>> print list(ad)
['bar', 'foo', 'baz']
>>> print dict(ad.items())
{'baz': time() + 4, 'foo': 2 + 2, 'bar': 2.100000000000000E+00}
>>> print ad
[
bar = 2.100000000000000E+00;
foo = 2 + 2;
baz = time() + 4
]
>>> ad2=classad.parse(open("test_ad", "r"));
>>> ad2["error"] = classad.Value.Error
>>> ad2["undefined"] = classad.Value.Undefined
>>> print ad2
[
error = error;
bar = 2.100000000000000E+00;
foo = 2 + 2;
undefined = undefined;
baz = time() + 4
]
>>> ad2["undefined"]
classad.Value.Undefined
Here is an example that illustrates the dictionary properties of the constructor.
>>> classad.ClassAd({"foo": "bar"})
[ foo = "bar" ]
>>> ad = classad.ClassAd({"foo": [1, 2, 3]})
>>> ad
[ foo = { 1,2,3 } ]
>>> ad["foo"][2]
3L
>>> ad = classad.ClassAd({"foo": {"bar": 1}})
>>> ad
[ foo = [ bar = 1 ] ]
>>> ad["foo"]["bar"]
1L
Here are examples that illustrate the get method.
>>> ad = classad.ClassAd({"foo": "bar"})
>>> ad
[ foo = "bar" ]
>>> ad["foo"]
'bar'
>>> ad.get("foo")
'bar'
>>> ad.get("foo", 2)
'bar'
>>> ad.get("baz", 2)
2
>>> ad.get("baz")
>>>
Here are examples that illustrate the setdefault method.
>>> ad = classad.ClassAd()
>>> ad
[ ]
>>> ad["foo"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'foo'
>>> ad.setdefault("foo", 1)
1
>>> ad
[ foo = 1 ]
>>> ad.setdefault("foo", 2)
1L
>>> ad
[ foo = 1 ]
Here is an example that illustrates the use of the iterator parseOldAds method on a history log.
>>> import classad
>>> import os
>>> fd = os.popen("condor_history -l -match 4")
>>> ads = classad.parseOldAds(fd)
>>> print [ad["ClusterId"] for ad in ads]
[23389L, 23388L, 23386L, 23387L]
>>>