Tjs Ad Transform Language

condor_transform_ads language

Transform rules files consist of lines containing key=value pairs or transform commands such as SET , RENAME , etc. Transform commands execute as they are read and can make use of values set up until that point using the $(key) macro substitution commands that HTCondor configuration files and condor_submit files use.

Most constructs that work in these files will also work in rules files such as if/else. Macro substitution will fetch attributes of the ClassAd to be transformed when $(MY.attr) is used.

The transform commands are:

SET <attr> <expr> Set <attr> to <expr>
EVALSET <attr> <expr> Evaluate <expr> and then set <attr> to the result
DEFAULT <attr> <expr> Set <attr> to <expr> if <attr> is undefined or missing
COPY <attr> <newattr> Copy the value of <attr> to <newattr>
COPY /<regex>/ <newattrs> Copy the values of attributes whose names match <regex> to <newattrs>
RENAME <attr> <newattr> Rename <attr> to <newattr>
RENAME /<regex>/ <newattrs> Rename attributes matching <regex> to <newattrs>
DELETE <attr> <newattr> Delete <attr>
DELETE /<regex>/ Delete attributes matching <regex>
EVALMACRO <key> <expr> Evaluate <expr> and then insert it as a transform macro value

In the above commands <attr> must be a valid attribute name and <expr> a valid ClassAd expression or literal. The various $() macros will be expanded in <expr> , <newattr> or <newattrs> before they are parsed as ClassAd expressions or attribute names.

When a COPY , RENAME , or DELETE with <regex> is used, regex capture groups are substituted in <newattrs> after $() expansion. \0 will expand to the entire match, \1 to the first capture, etc.

Optionally, a transform rule set can end with an iteration command

TRANSFORM [<N>] [<vars>] [in <list> | from <file> | matching <pattern>]

A TRANSFORM command must be the last command in the rules file. It takes the same options as the QUEUE statement from a HTCONDOR submit file. There is an implicit TRANSFORM 1 at the end of a rules file that has no explicit TRANSFORM command.

OSGs default route expressed in the TJ's proposed new transform language.

lines starting with the keywords set , delete , copy , rename , name are commands to the transform engine and are executed as they are read. lines that are of the form key = value set temporary variables that can be referenced in transform commmands using the various $() and $function() macro expansions that are common to config and submit files. 

new transform language
NAME OSG CE Default route

MaxIdleJobs = 2000
MaxJobs = 10000

# by default, accept all jobs
Requirements = True

# these triggers control IF statements later in the transform
tmp.ExpireJob = False
tmp.RemoveIfIdle = False


# modify routed job attributes
#
DELETE CondorCE
SET    RoutedJob True

# remove routed job if the client disappears for 48 hours or it is idle for 6
#
IF $(tmp.RemoveIfIdle)
  SET    PeriodicRemove (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60)
ELSE
  DELETE PeriodicRemove
ENDIF

# insert HOME and OSG_* into environment
#
tmp.osg_env = OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE'
tmp.osg_env = $(tmp.osg_env) OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'
SET osg_environment "$(tmp.osg_env)"

tmp.user_home_expr = userHome(Owner, "/")
tmp.user_home = HOME=$EVAL(tmp.user_home_expr)

COPY Environment orig_environment
SET  Environment "$(tmp.user_home) $(MY.orig_environment) $(MY.osg_environment)"

# pick up GlobusRSL settings, we will use those later in the transform
# NOTE: is it a bug to leave this attribute behind?
#
# set InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));
# or possibly this
IF DEFINED MY.GlobusRSL
  SET InputRSL eval_rsl(GlobusRSL)
ELSE
  SET InputRSL []
ENDIF

# Set new requirements
IF $(tmp.ExpireJob)
  SET Requirements (LastClientContact - time()) < 30*60
ELSE
  SET Requirements True
ENDIF

# pass attributes (maxMemory,xcount,jobtype,queue)
# via gWMS Factory described within ClassAd if undefined via RSL
# Note default memory request of 2GB
#
IF DEFINED MY.InputRSL.MaxMemory
  SET RequestMemory $(MY.InputRSL.MaxMemory)
ELIF $(MY.MaxMemory)
  SET RequestMemory MaxMemory
ELSE
  SET RequestMemory $(MY.default_maxMemory:2000)
ENDIF

IF DEFINED MY.InputRSL.Queue
  SET remote_queue "$(MY.InputRSL.Queue)"
ELIF DEFINED MY.Queue
  SET remote_queue Queue
ELSE
  SET remote_queue "$(MY.default_queue)"
ENDIF

# Figure out the number of cores. HTCondor uses RequestCpus
# blahp uses SMPGranularity and NodeNumber.  Default is 1 core.
#
IF DEFINED MY.InputRSL.xcount
  tmp.cpus = $(MY.InputRSL.xcount)
ELIF $(MY.xcount)
  tmp.cpus = $(MY.xcount)
ELSE
  tmp.cpus = $(MY.default_xcount:1)
ENDIF
SET RequestCpus $(tmp.cpus)
SET SMPGranularity $(tmp.cpus)
SET NodeNumber $(tmp.cpus)

# If remote_cerequirements is a string, BLAH will parse it as an expression before examining it
#
SET remote_cerequirements "CONDOR_CE == 1"

# add a walltime to the remote_cerequirements expression if one is given.
tmp.MaxWalltime_expr = 60 * (InputRSL.MaxWalltime ?: MaxWalltime ?: default_MaxWalltime ?: 0)
IF $INT(tmp.MaxWalltime_expr)
  SET remote_cerequirements "Walltime == $INT(tmp.MaxWalltime_expr) && CondorCE == 1"
ENDIF

For reference here is the same route expressed in the current new classad syntax 

current job router language
[ MaxIdleJobs = 2000;
 MaxJobs = 10000;
 /* by default, accept all jobs */
 Requirements = True;

 /* now modify routed job attributes */

 /* remove routed job if the client disappears for 48 hours or it is idle for 6 */
 /*set_PeriodicRemove = (LastClientContact - time() > 48*60*60) || (JobStatus == 1 && (time() - QDate) > 6*60); */
 delete_PeriodicRemove = true;
 delete_CondorCE = true;
 set_RoutedJob = true;
 copy_environment = "orig_environment";
 set_osg_environment = "OSG_GRID='/etc/osg/wn-client/' OSG_SQUID_LOCATION='fermicloud133.fnal.gov:3128' OSG_SITE_READ='None' OSG_APP='/share/osg/app' OSG_GLEXEC_LOCATION='None' OSG_DATA='UNAVAILABLE' OSG_HOSTNAME='fermicloud136.fnal.gov' OSG_STORAGE_ELEMENT='False' OSG_SITE_NAME='herp' GLOBUS_LOCATION='/usr' OSG_WN_TMP='None' OSG_DEFAULT_SE='None' OSG_SITE_WRITE='None'";
 eval_set_environment = debug(strcat("HOME=",
                              userHome(Owner, "/"),
                              " ",
                              ifThenElse(orig_environment is undefined,
                                         osg_environment,
                                         strcat(osg_environment, " ", orig_environment) )));

 /* Set new requirements */
 /* set_requirements = LastClientContact - time() < 30*60;*/
 set_requirements = True;

 set_InputRSL = ifThenElse(GlobusRSL is null, [], eval_rsl(GlobusRSL));

 /* Note default memory request of 2GB */
 /* Note yet another nested condition allow pass attributes (maxMemory,xcount,jobtype,queue) via gWMS Factory described within ClassAd if undefined via RSL */
 eval_set_RequestMemory = ifThenElse(InputRSL.maxMemory isnt null,
                                     InputRSL.maxMemory,
                                     ifThenElse(maxMemory isnt null,
                                                maxMemory,
                                                ifThenElse(default_maxMemory isnt null,
                                                           default_maxMemory, 2000)));
 eval_set_remote_queue = ifThenElse(InputRSL.queue isnt null,
                                    InputRSL.queue,
                                    ifThenElse(queue isnt null,
                                               queue,
                                               ifThenElse(default_queue isnt null, default_queue, "")));

 /* HTCondor uses RequestCpus;
 blahp uses SMPGranularity and NodeNumber.  Default is 1 core. */
 eval_set_RequestCpus = ifThenElse(InputRSL.xcount isnt null,
                                   InputRSL.xcount,
                                   ifThenElse(xcount isnt null,
                                              xcount,
                                              ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_SMPGranularity = ifThenElse(InputRSL.xcount isnt null,
                                             InputRSL.xcount,
                                             ifThenElse(xcount isnt null,
                                                        xcount,
                                                        ifThenElse(default_xcount isnt null, default_xcount, 1)));
 eval_set_remote_NodeNumber = ifThenElse(InputRSL.xcount isnt null,
                                         InputRSL.xcount,
                                         ifThenElse(xcount isnt null,
                                                    xcount,
                                                    ifThenElse(default_xcount isnt null, default_xcount, 1)));

 /* If remote_cerequirements is a string, BLAH will parse it as an expression before examining it */
 eval_set_remote_cerequirements = ifThenElse(InputRSL.maxWalTlime isnt null,
                                             strcat("Walltime == ",string(60*InputRSL.maxWallTime)," && CondorCE == 1"),
                                             ifThenElse(maxWallTime isnt null,
                                                        strcat("Walltime == ",string(60*maxWallTime)," && CondorCE == 1"),
                                                        ifThenElse(default_maxWallTime isnt null,
                                                                   strcat("Walltime == ", string(60*default_maxWallTime), " && CondorCE == 1"),
                                                                   "CondorCE == 1")));
]