next up previous contents index
Next: 7. Platform-Specific Information Up: 6. Application Programming Interfaces Previous: 6.7 The HTCondor Perl   Contents   Index

Subsections


6.8 Python Bindings

The Python module provides bindings to the client-side APIs for HTCondor and the ClassAd language.

These Python bindings depend on loading the HTCondor shared libraries; this means the same code is used here as the HTCondor client tools. It is more efficient in terms of memory and CPU to utilize these bindings than to parse the output of the HTCondor client tools when writing applications in Python.

Currently, the Python bindings are only available on Linux platforms.


6.8.1 htcondor Module

The htcondor module provides a client interface to the various HTCondor daemons. It tries to provide functionality similar to the HTCondor command line tools.

htcondor module functions:

platform( )

Returns the platform of HTCondor this module is running on.

version( )

Returns the version of HTCondor this module is linked against.

reload_config( )

Reload the HTCondor configuration from disk.

send_command( ad, (DaemonCommands)dc, (str)target = None)

Send a command to an HTCondor daemon specified by a location ClassAd.

ad is a ClassAd specifying the location of the daemon; typically, found by using Collector.locate(...).

dc is a command type; must be a member of the enum DaemonCommands.

target is an optional parameter, representing an additional command to send to a daemon. Some commands require additional arguments; for example, sending DaemonOff to a condor_master requires one to specify which subsystem to turn off.

The module object, param, is a dictionary-like object providing access to the configuration variables in the current HTCondor configuration.

The condor_schedd class:

__init__( classad )

Create an instance of the Schedd class.

Optional parameter classad describes the location of the remote condor_schedd daemon. If the parameter is omitted, the local condor_schedd daemon is used.

act( (JobAction)action, (object)job_spec )

Change status of job(s) in the condor_schedd daemon. The integer return value is a ClassAd object describing the number of jobs changed.

Parameter action is the action to perform; must be of the enum JobAction.

Parameter job_spec is the job specification. It can either be a list of job IDs or a string specifying a constraint to match jobs.

edit( (object)job_spec, (str)attr, (object)value )

Edit one or more jobs in the queue.

Parameter job_spec is either a list of jobs, with each given as ClusterId.ProcId or a string containing a constraint to match jobs against.

Parameter attr is the attribute name of the attribute to edit.

Parameter value is the new value of the job attribute. It should be a string, which will be converted to a ClassAd expression, or an ExprTree object.

query( constraint = true, attr_list = [] )

Query the condor_schedd daemon for jobs. Returns a list of ClassAds representing the matching jobs, containing at least the requested attributes requested by the second parameter.

The optional parameter constraint provides a constraint for filtering out jobs. It defaults to True.

Parameter attr_list is a list of attributes for the condor_schedd daemon to project along. It defaults to having the condor_schedd daemon return all attributes.

submit( ad, count = 1, spool = false, ad_results = None )

Submit one or more jobs to the condor_schedd daemon. Returns the newly created cluster ID.

This method requires the invoker to provide a ClassAd for the new job cluster; such a ClassAd contains attributes with different names than the commands in a submit description file. As an example, the stdout file is referred to as output in the submit description file, but Out in the ClassAd. To generate an example ClassAd, take a sample submit description file and invoke

condor_submit -dump <filename> [cmdfile]

Then, load the resulting contents of <filename> into Python.

Parameter ad is the ClassAd describing the job cluster.

Parameter count is the number of jobs to submit to the cluster. Defaults to 1.

Parameter spool inserts the necessary attributes into the job for it to have the input files spooled to a remote condor_schedd daemon. This parameter is necessary for jobs submitted to a remote condor_schedd.

Parameter ad_results, if set to a list, will contain the job ClassAds resulting from the job submission. These are useful for interacting with the job spool at a later time.

spool( ad_list )

Spools the files specified in a list of job ClassAds to the condor_schedd. Throws a RuntimeError exception if there are any errors.

Parameter ad_list is a list of ClassAds containing job descriptions; typically, this is the list filled by the ad_results argument of the submit method call.

retrieve( job_spec )

Retrieve the output sandbox from one or more jobs.

Parameter job_spec is an expression string matching the list of job output sandboxes to retrieve.

The Collector class:

__init__( pool = None )

Create an instance of the Collector class.

Optional parameter pool is a string with host:port pair specified. If omitted, the local condor_schedd daemon is used.

locate( (DaemonTypes)daemon_type, (str)name )

Query the condor_collector for a particular daemon. Returns the ClassAd of the requested daemon.

Parameter daemon_type is the type of daemon; must be of the enum DaemonTypes.

Optional parameter name is the name of daemon to locate. If not specified, it searches for the local daemon.

locateAll( (DaemonTypes)daemon_type )

Query the condor_collector daemon for all ClassAds of a particular type. Returns a list of matching ClassAds.

Parameter daemon_type is the type of daemon; must be of the enum DaemonTypes.

query( (AdTypes)ad_type, constraint=True, attrs=[] )

Query the contents of a condor_collector daemon. Returns a list of ClassAds that match the constraint parameter.

Optional parameter ad_type is the type of ClassAd to return, where the types are from the enum AdTypes. If not specified, the type will be ANY_AD.

Optional parameter constraint is a constraint for the ClassAd query. It defaults to True.

Optional parameter attrs is a list of attributes. If specified, the returned ClassAds will be projected along these attributes.

advertise( ad_list, command=UPDATE_AD_GENERIC, use_tcp = True )

Advertise a list of ClassAds into the condor_collector.

Parameter ad_list is the list of ClassAds to advertise.

Optional parameter command is a command for the condor_collector. It defaults to UPDATE_AD_GENERIC. Other commands, such as UPDATE_STARTD_AD, may require reduced authorization levels.

Optional parameter use_tcp causes updates to be sent via TCP. Defaults to True.

The SecMan class accesses the internal security object. Currently, the class only allows the developer to reset security sessions. If a security session becomes invalid, for example if the remote daemon restarts and reuses the same port and the client continues to use the session, then all future commands will fail with strange connection errors. This is the only mechanism to invalidate in-memory sessions.

__init__( )

Create a SecMan object.

invalidateAllSessions( )

Invalidate all security sessions. Any future connections to a daemon will cause a new security session to be created.

Module enums:

AdTypes

A list of types used as values for the MyType ClassAd attribute. These types are only used by the HTCondor system, not the ClassAd language. Typically, these specify different kinds of daemons.

DaemonCommands

A list of commands which can be sent to a remote daemon.

DaemonTypes

A list of types of known HTCondor daemons.

JobAction

A list of actions that can be performed on a job in a condor_schedd.


6.8.2 Sample Code using the htcondor Python Module

This sample code illustrates interactions with the htcondor Python Module.

$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import condor
>>> import classad
>>> coll = condor.Collector("red-condor.unl.edu")
>>> results = coll.query(condor.AdTypes.Startd, "true", ["Name"])
>>> len(results)
3812
>>> results[0]
[ Name = "slot1@red-d20n35"; MyType = "Machine"; TargetType = "Job"; CurrentTime = time() ]
>>> scheddAd = coll.locate(condor.DaemonTypes.Schedd, "red-gw1.unl.edu")
>>> scheddAd["ScheddIpAddr"]
'<129.93.239.132:53020>'
>>> schedd = condor.Schedd(scheddAd)
>>> results = schedd.query('Owner =?= "cmsprod088"', ["ClusterId", "ProcId"])
>>> len(results)
63
>>> results[0]
[ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356722353; ClusterId = 674143; ProcId = 0; CurrentTime = time() ]
>>> condor.param["COLLECTOR_HOST"]
'hcc-briantest.unl.edu'
>>> schedd = condor.Schedd() # Defaults to the local schedd.
>>> results = schedd.query()
>>> results[0]["RequestMemory"]
ifthenelse(MemoryUsage isnt undefined,MemoryUsage,( ImageSize + 1023 ) / 1024)
>>> results[0]["RequestMemory"].eval()
1L
>>> ad=classad.parse(open("test.submit.ad"))
>>> print schedd.submit(ad, 2) # Submits two jobs in the cluster; edit test.submit.ad to preference.
110
>>> print schedd.act(condor.JobAction.Remove, ["111.0", "110.0"])'
    [
        TotalNotFound = 0; 
        TotalPermissionDenied = 0; 
        TotalAlreadyDone = 0; 
        TotalJobAds = 2; 
        TotalSuccess = 2; 
        TotalChangedAds = 1; 
        TotalBadStatus = 0; 
        TotalError = 0
    ]
>>> print schedd.act(condor.JobAction.Hold, "Owner =?= \"bbockelm\"")'
    [
        TotalNotFound = 0; 
        TotalPermissionDenied = 0; 
        TotalAlreadyDone = 0; 
        TotalJobAds = 2; 
        TotalSuccess = 2; 
        TotalChangedAds = 1; 
        TotalBadStatus = 0; 
        TotalError = 0
    ]
>>> schedd.edit('Owner =?= "bbockelm"', "Foo", classad.ExprTree('"baz"'))
>>> schedd.edit(["110.0"], "Foo", '"bar"')
>>> coll = condor.Collector()
>>> master_ad = coll.locate(condor.DaemonTypes.Master)
>>> condor.send_command(master_ad, condor.DaemonCommands.Reconfig) # Reconfigures the local master and all children
>>> condor.version()
'$CondorVersion: 7.9.4 Jan 02 2013 PRE-RELEASE-UWCS $'
>>> condor.platform()
'$CondorPlatform: X86_64-ScientificLinux_6.3 $'


6.8.3 ClassAd Module

The classad module class provides a dictionary-like mechanism for interacting with the ClassAd language. classad objects implement the iterator interface to iterate through the classad's attributes. The constructor can take a dictionary, and the object can take lists, dictionaries, and ClassAds as values.

classad module functions:

parse( input )

Parse input into a ClassAd. Returns a ClassAd object.

Parameter input is a string-like object or a file pointer.

parseOld( )

Parse old ClassAd format input into a ClassAd.

version( )

Return the version of the linked ClassAd library.

Standard Python object methods for the ClassAd class:

__init__( str )

Create a ClassAd object from string, str, passed as a parameter. The string must be formatted in the new ClassAd format.

__len__( )

Returns the number of attributes in the ClassAd; allows len(object) semantics for ClassAds.

__str__( )

Converts the ClassAd to a string and returns the string; the formatting style is new ClassAd, with square brackets and semicolons. For example, [ Foo = "bar"; ] may be returned.

The classad object has the following dictionary-like methods:

items( )

Returns an iterator of tuples. Each item returned by the iterator is a tuple representing a pair (attribute,value) in the ClassAd object.

values( )

Returns an iterator of objects. Each item returned by the iterator is a value in the ClassAd.

If the value is a literal, it will be cast to a native Python object, so a ClassAd string will be returned as a Python string.

keys( )

Returns an iterator of strings. Each item returned by the iterator is an attribute string in the ClassAd.

get( attr, value )

Behaves like the corresponding Python dictionary method. Given the attr as key, returns either the value of that key, or if the key is not in the object, returns None or the optional second parameter when specified.

__getitem__( attr )

Returns (as an object) the value corresponding to the attribute attr passed as a parameter.

ClassAd values will be returned as Python objects; ClassAd expressions will be returned as ExprTree objects.

__setitem__( attr, value )

Sets the ClassAd attribute attr to the value.

ClassAd values will be returned as Python objects; ClassAd expressions will be returned as ExprTree objects.

setdefault( attr, value )

Behaves like the corresponding Python dictionary method. If called with an attribute, attr, that is not set, it will set the attribute to the specified value. It returns the value of the attribute. If called with an attribute that is already set, it does not change the object.

Additional methods:

eval( attr )

Evaluate the value given a ClassAd attribute attr. Throws ValueError if unable to evaluate the object.

Returns the Python object corresponding to the evaluated ClassAd attribute.

lookup( attr )

Look up the ExprTree object associated with attribute attr. No attempt will be made to convert to a Python object.

Returns an ExprTree object.

printOld( )

Print the ClassAd in the old ClassAd format.

Returns a string.

The ExprTree class object represents an expression in the ClassAd language.

ExprTree class methods:

__init__( str )

Parse the string str to create an ExprTree.

__str__( )

Represent and return the ClassAd expression as a string.

eval( )

Evaluate the expression and return as a ClassAd value, typically a Python object.


6.8.4 Sample Code using the classad Module

This sample Python code illustrates interactions with the classad module.

$ python
Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import classad
>>> ad = classad.ClassAd()
>>> expr = classad.ExprTree("2+2")
>>> ad["foo"] = expr
>>> print ad["foo"].eval()
4
>>> ad["bar"] = 2.1
>>> ad["baz"] = classad.ExprTree("time() + 4")
>>> print list(ad)
['bar', 'foo', 'baz']
>>> print dict(ad.items())
{'baz': time() + 4, 'foo': 2 + 2, 'bar': 2.100000000000000E+00}
>>> print ad
    [
        bar = 2.100000000000000E+00; 
        foo = 2 + 2; 
        baz = time() + 4
    ]
>>> ad2=classad.parse(open("test_ad", "r"));
>>> ad2["error"] = classad.Value.Error
>>> ad2["undefined"] = classad.Value.Undefined
>>> print ad2
    [
        error = error; 
        bar = 2.100000000000000E+00; 
        foo = 2 + 2; 
        undefined = undefined; 
        baz = time() + 4
    ]
>>> ad2["undefined"]
classad.Value.Undefined

Here is an example that illustrates the dictionary properties of the constructor.

>>> classad.ClassAd({"foo": "bar"})
[ foo = "bar" ]
>>> ad = classad.ClassAd({"foo": [1, 2, 3]})
>>> ad
[ foo = { 1,2,3 } ]
>>> ad["foo"][2]
3L
>>> ad = classad.ClassAd({"foo": {"bar": 1}})
>>> ad
[ foo = [ bar = 1 ] ]
>>> ad["foo"]["bar"]
1L

Here are examples that illustrate the get method.

>>> ad = classad.ClassAd({"foo": "bar"})
>>> ad
[ foo = "bar" ]
>>> ad["foo"]
'bar'
>>> ad.get("foo")
'bar'
>>> ad.get("foo", 2)
'bar'
>>> ad.get("baz", 2)
2
>>> ad.get("baz")
>>>

Here are examples that illustrate the setdefault method.

>>> ad = classad.ClassAd()
>>> ad
[  ]
>>> ad["foo"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'foo'
>>> ad.setdefault("foo", 1)
1
>>> ad
[ foo = 1 ]
>>> ad.setdefault("foo", 2)
1L
>>> ad
[ foo = 1 ]


next up previous contents index
Next: 7. Platform-Specific Information Up: 6. Application Programming Interfaces Previous: 6.7 The HTCondor Perl   Contents   Index
htcondor-admin@cs.wisc.edu