Earlier today we learned about some grid technologies including Condor-G and DAGMan. In this session, we will install and submit some simple jobs using these components.
First, create a scratch directory to do your work. In the example below, we use /tmp. This is a good choice for the purposes of this tutorial.
Be sure to create a unique name for your directory in /tmp, the directory may be shared with other users. Where the example specifies "username", use your username, your real name, or something else unique.
Lets get going! First make the subdirectories:
$ cd /tmp $ mkdir username-condor-g-dagman-tutorial $ cd username-condor-g-dagman-tutorial $ mkdir submit
Next create a short lived proxy for this tutorial. (The default proxy length is 12 hours. For a long lived task, you might create proxies with lifespans of 24 hours, several days, or even several months.) (The "-verify" option is not required, but can is useful for debugging. -verify will warn you if an expected Certificate Authority Certificate is missing.)
$ grid-proxy-info -all ERROR: Couldn't find a valid proxy. Use -debug for further information. $ grid-proxy-init -hours 4 -verify Your identity: /C=US/O=Globus/O=University of Wisconsin/OU=Computer Sciences Department/CN=Alan De Smet Enter GRID pass phrase for this identity: Your pass phrase Creating proxy ........................................... Done Proxy Verify OK Your proxy is valid until Thu Jul 10 16:06:13 2003 $ grid-proxy-info -all subject : /C=US/O=Globus/O=University of Wisconsin/OU=Computer Sciences Department/CN=Alan De Smet/CN=proxy issuer : /C=US/O=Globus/O=University of Wisconsin/OU=Computer Sciences Department/CN=Alan De Smet type : full strength : 512 bits timeleft : 3:59:57
Do a quick test with globus-job-run to ensure that everything is working. Note that the below will submit to the fork jobmanager on a server at NeSC. Instead, you will probably be instructed to submit to a server local to your location -- ask your instructor:
$ globus-job-run lab-ctrl.nesc.ed.ac.uk /bin/date Wed Jul 9 17:57:49 CDT 2003
Our basic Globus setup appears to be functioning. To start up the Condor services if they are not already running, use the condor_on command. We'll then do a ps command to observe which processes are started.
$ condor_on Sent "Spawn-All-Daemons" command to local master $ ps -efwwwww | grep condor_ | grep -v grep condor 1051 1 0 14:33 ? 00:00:01 /opt/vdt/condor/sbin/condor_master condor 3287 1051 0 16:22 ? 00:00:00 condor_collector -f condor 3288 1051 0 16:22 ? 00:00:00 condor_negotiator -f condor 3289 1051 1 16:22 ? 00:00:05 condor_startd -f condor 3290 1051 0 16:22 ? 00:00:00 condor_schedd -f
Now we are ready to submit our first job with Condor-G. The basic procedure is to create a Condor job submit description file. This file can tell Condor what executable to run, what resources to use, how to handle failures, where to store the job's output, and many other characteristics of the job submission. Then this file is give to condor_submit.
First, move to our scratch submission location:
$ pwd /tmp/username-condor-g-dagman-tutorial $ cd submit
Create a Condor submit file. As you can see from the condor_submit manual page, there are many options that can be specified in a Condor-G submit description file. We will start out with just a few. We'll be sending the job to the computer "lab-ctrl.nesc.ed.ac.uk" and running under the "jobmanager-fork" job manager. We're setting notification to never to avoid getting email messages about the completion of our job, and redirecting the stdout/err of the job back to the submission computer.
(Feel free to use your favorite editor, but we will demonstrate with 'cat' in the example below. When using cat to create files, press Ctrl-D to close the file -- don't actually type "Ctrl-D" into the file. Whenever you create a file using cat, we suggest you use cat to display the file and confirm that it contains the expected text.)
Create the submit file, then verify that it was entered correctly:
$ cat > myjob.submit executable=myscript.sh arguments=TestJob 10 output=results.output error=results.error log=results.log notification=never universe=globus globusscheduler=lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork queue Ctrl-D $ cat myjob.submit executable=myscript.sh arguments=TestJob 10 output=results.output error=results.error log=results.log notification=never universe=globus globusscheduler=lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork queue
Create a little program to run on the grid.
$ cat > myscript.sh #! /bin/sh echo "I'm process id $$ on" `hostname` echo "This is sent to standard error" 1>&2 date echo "Running as binary $0" "$@" echo "My name (argument 1) is $1" echo "My sleep duration (argument 2) is $2" sleep $2 echo "Sleep of $2 seconds finished. Exiting" echo "RESULT: 0 SUCCESS" Ctrl-D $ cat myscript.sh #! /bin/sh echo "I'm process id $$ on" `hostname` echo "This is sent to standard error" 1>&2 date echo "Running as binary $0" "$@" echo "My name (argument 1) is $1" echo "My sleep duration (argument 2) is $2" sleep $2 echo "Sleep of $2 seconds finished. Exiting" echo "RESULT: 0 SUCCESS"
Make the program executable and test it.
$ chmod a+x myscript.sh $ ./myscript.sh TEST 1 I'm process id 3428 on puffin.cs.wisc.edu This is sent to standard error Thu Jul 10 12:21:11 CDT 2003 Running as binary ./myscript.sh TEST 1 My name (argument 1) is TEST My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting RESULT: 0 SUCCESS
Submit your test job to Condor-G.
$ condor_submit myjob.submit Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 1.
Occasionally run condor_q to watch the progress of your job. You may also want to occasionally run "condor_q -globus" which presents Globus specific status information. (Additional documentation on condor_q)
$ condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 adesmet 7/10 17:28 0+00:00:00 I 0 0.0 myscript.sh TestJo 1 jobs; 1 idle, 0 running, 0 held $ condor_q -globus -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 1.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond $ condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 adesmet 7/10 17:28 0+00:00:27 R 0 0.0 myscript.sh TestJo 1 jobs; 0 idle, 1 running, 0 held $ condor_q -globus -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 1.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond $ condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 1.0 adesmet 7/10 17:28 0+00:00:40 C 0 0.0 myscript.sh 0 jobs; 0 idle, 0 running, 0 held $ condor_q -globus -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 1.0 adesmet DONE fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u $ condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held
In another window you can run "tail -f" to watch the log file for your job to monitor its progress. For the remainder of this tutorial, we suggest you re-run this command when you submit one or more jobs. This will allow you to see monitor how typical Condor-G jobs progress. Use "Ctrl-C" to stop watching the file.
In a second window:
$ cd /tmp/username-condor-g-dagman-tutorial/submit
$ tail -f --lines=500 results.log
000 (001.000.000) 07/10 17:28:48 Job submitted from host: <128.105.185.14:35688>
...
017 (001.000.000) 07/10 17:29:01 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2321/696/1057876132/
Can-Restart-JM: 1
...
001 (001.000.000) 07/10 17:29:01 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (001.000.000) 07/10 17:30:08 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
When the job is no longer listed in condor_q or when the log file reports "Job terminated," you can see the results in condor_history.
$ condor_history ID OWNER SUBMITTED RUN_TIME ST COMPLETED CMD 1.0 adesmet 7/10 10:28 0+00:00:00 C ??? /afs/cs.wisc.ed
When the job completes, verify that the output is as expected. (The binary name is different from what you created because of how Globus and Condor-G cooperate to stage your file to execute computer.)
$ ls myjob.submit myscript.sh* results.error results.log results.output $ cat results.error This is sent to standard error $ cat results.output $I'm process id 733 on lab-ctrl Thu Jul 10 17:28:57 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/28/fcae5001dbcd99cc476984b4151284/md5/af/355c4959dc83a74b18b7c03eb27201/data TestJob 10 My name (argument 1) is TestJob My sleep duration (argument 2) is 10 Sleep of 10 seconds finished. Exiting RESULT: 0 SUCCESS
If you didn't watch the results.log file with tail -f above, you will want to examine the information logged now:
$ cat results.log
Clean up the results:
$ rm results.*
When an problem occurs in the middleware, Condor-G will place your job on "Hold". Held jobs remain in the queue, but wait for user intervention. When you resolve the problem, you can use condor_release to free job to continue.
You can places jobs on hold yourself, perhaps if you want to delay your run using condor_hold
For this example, we'll make the output file non-writable. The job will be unable to copy the results back and will be placed on hold.
Submit the job again, but this time immediately after submitting it, mark the output file as read-only:
$ condor_submit myjob.submit ; chmod a-w results.output Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 2.
Watch the job with tail. When the job goes on hold, use Ctrl-C to exit tail. Note that condor_q reports that the job is in the "H" or Held state.
$ tail -f --lines=500 results.log
000 (003.000.000) 07/12 22:35:44 Job submitted from host: <192.167.1.23:32864>
...
017 (003.000.000) 07/12 22:35:57 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:33178/12497/1058042148/
Can-Restart-JM: 1
...
001 (003.000.000) 07/12 22:35:57 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
012 (003.000.000) 07/12 22:36:52 Job was held.
Globus error 129: the standard output/error size is different
Code 2 Subcode 129
...
Ctrl-C
$ condor_q
-- Submitter: pc-23.gs.unina.it : <192.167.1.23:32864> : pc-23.gs.unina.it
ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD
2.0 adesmet 7/12 22:35 0+00:00:55 H 0 0.0 myscript.sh TestJo
1 jobs; 0 idle, 0 running, 1 held
Fix the problem (make the file writable again), then release the job. You can specifiy the job's ID, or just use "-all" to release all held jobs.
$ chmod u+w results.output $ condor_release -all All jobs released.
Again, watch the log until the job finishes:
$ tail -f --lines=500 results.log
000 (003.000.000) 07/12 22:35:44 Job submitted from host: <192.167.1.23:32864>
...
017 (003.000.000) 07/12 22:35:57 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:33178/12497/1058042148/
Can-Restart-JM: 1
...
001 (003.000.000) 07/12 22:35:57 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
012 (003.000.000) 07/12 22:36:52 Job was held.
Globus error 129: the standard output/error size is different
...
013 (003.000.000) 07/12 22:44:33 Job was released.
via condor_release (by user Todd)
...
001 (003.000.000) 07/12 22:44:46 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (003.000.000) 07/12 22:44:51 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
Ctrl-C
Your job finished, the results have been retreived successfully:
$ cat results.output I'm process id 12528 on lab-ctrl.nesc.ed.ac.uk Sat Jul 12 22:35:53 CEST 2003 Running as binary /home/home45/Aland/.globus/.gass_cache/local/md5/6d/217f3f7926c06a529143f6129bf269/md5/a7/2af94ba728c69c588e523a99baaefd/data TestJob 10 My name (argument 1) is TestJob My sleep duration (argument 2) is 10 Sleep of 10 seconds finished. Exiting RESULT: 0 SUCCESS
Before continuing, clean up the results:
$ rm results.*
Since it will be handy for the rest of the tutorial, create a little shell script to monitor the Condor-G queue:
$ cat > watch_condor_q
#! /bin/sh
while true; do
condor_q
condor_q -globus
sleep 10
done
Ctrl-D
$ cat watch_condor_q
#! /bin/sh
while true; do
condor_q
condor_q -globus
sleep 10
done
$ chmod a+x watch_condor_q
Create a minimal DAG for DAGMan. This DAG will have a single node.
$ cat > mydag.dag Job HelloWorld myjob.submit Ctrl-D $ cat mydag.dag Job HelloWorld myjob.submit
Submit it with condor_submit_dag, then watch the run. Notice that condor_dagman is running as a job and that condor_dagman submits your real job without your direct intervention. You might happen to catch the "C" (completed) state as your job finishes, but that often goes by too quickly to notice.
Again, in another window you may want to run "tail -f --lines=500 results.log" in a second window to watch the job log file as your job runs. You might also want to watch DAGMan's log file with "tail -f --lines=500 mydag.dag.dagman.out" in a third window. (mydag.dag.dagman.out) in the same way in a third window. For the remainder of this tutorial, we suggest you re-run this command when you submit a DAG. This will allow you to see how typical DAGs progress. Use "Ctrl-C" to stop watching the file.
Third window:
$ cd /tmp/username-condor-g-dagman-tutorial/submit $ tail -f --lines=500 mydag.dag.dagman.out 7/10 10:36:43 ****************************************************** 7/10 10:36:43 ** condor_scheduniv_exec.6.0 (CONDOR_DAGMAN) STARTING UP 7/10 10:36:43 ** $CondorVersion: 6.5.1 Apr 22 2003 $ 7/10 10:36:43 ** $CondorPlatform: INTEL-LINUX-GLIBC22 $ 7/10 10:36:43 ** PID = 26844 7/10 10:36:43 ****************************************************** 7/10 10:36:44 DaemonCore: Command Socket at <128.105.185.14:34571> 7/10 10:36:44 argv[0] == "condor_scheduniv_exec.6.0" 7/10 10:36:44 argv[1] == "-Debug" 7/10 10:36:44 argv[2] == "3" 7/10 10:36:44 argv[3] == "-Lockfile" 7/10 10:36:44 argv[4] == "mydag.dag.lock" 7/10 10:36:44 argv[5] == "-Condorlog" 7/10 10:36:44 argv[6] == "results.log" 7/10 10:36:44 argv[7] == "-Dag" 7/10 10:36:44 argv[8] == "mydag.dag" 7/10 10:36:44 argv[9] == "-Rescue" 7/10 10:36:44 argv[10] == "mydag.dag.rescue" 7/10 10:36:44 Condor log will be written to results.log 7/10 10:36:44 DAG Lockfile will be written to mydag.dag.lock 7/10 10:36:44 DAG Input file is mydag.dag 7/10 10:36:44 Rescue DAG will be written to mydag.dag.rescue 7/10 10:36:44 Parsing mydag.dag ... 7/10 10:36:44 Dag contains 1 total jobs 7/10 10:36:44 Bootstrapping... 7/10 10:36:44 Number of pre-completed jobs: 0 7/10 10:36:44 Submitting Job HelloWorld ... 7/10 10:36:44 assigned Condor ID (7.0.0) 7/10 10:36:45 Event: ULOG_SUBMIT for Job HelloWorld (7.0.0) 7/10 10:36:45 0/1 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 10:37:05 Event: ULOG_GLOBUS_SUBMIT for Job HelloWorld (7.0.0) 7/10 10:37:05 Event: ULOG_EXECUTE for Job HelloWorld (7.0.0) 7/10 10:38:10 Event: ULOG_JOB_TERMINATED for Job HelloWorld (7.0.0) 7/10 10:38:10 Job HelloWorld completed successfully. 7/10 10:38:10 1/1 done, 0 failed, 0 submitted, 0 ready, 0 pre, 0 post 7/10 10:38:10 All jobs Completed! 7/10 10:38:10 **** condor_scheduniv_exec.6.0 (condor_DAGMAN) EXITING WITH STATUS 0
First window:
$ condor_submit_dag mydag.dag Checking your DAG input file and all submit files it references. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : mydag.dag.condor.sub Log of DAGMan debugging messages : mydag.dag.dagman.out Log of Condor library debug messages : mydag.dag.lib.out Log of the life of condor_dagman itself : mydag.dag.dagman.log Condor Log file for all jobs of this DAG : results.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 2. ----------------------------------------------------------------------- $ ./watch_condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 2.0 adesmet 7/10 17:33 0+00:00:03 R 0 2.6 condor_dagman -f - 3.0 adesmet 7/10 17:33 0+00:00:00 I 0 0.0 myscript.sh TestJo 2 jobs; 1 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 3.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 2.0 adesmet 7/10 17:33 0+00:00:33 R 0 2.6 condor_dagman -f - 3.0 adesmet 7/10 17:33 0+00:00:15 R 0 0.0 myscript.sh TestJo 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 3.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 2.0 adesmet 7/10 17:33 0+00:01:03 R 0 2.6 condor_dagman -f - 3.0 adesmet 7/10 17:33 0+00:00:45 R 0 0.0 myscript.sh TestJo 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 3.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE Ctrl-C
Verify your results:
$ ls -l total 12 -rw-r--r-- 1 adesmet adesmet 28 Jul 10 10:35 mydag.dag -rw-r--r-- 1 adesmet adesmet 523 Jul 10 10:36 mydag.dag.condor.sub -rw-r--r-- 1 adesmet adesmet 608 Jul 10 10:38 mydag.dag.dagman.log -rw-r--r-- 1 adesmet adesmet 1860 Jul 10 10:38 mydag.dag.dagman.out -rw-r--r-- 1 adesmet adesmet 29 Jul 10 10:38 mydag.dag.lib.out -rw------- 1 adesmet adesmet 0 Jul 10 10:36 mydag.dag.lock -rw-r--r-- 1 adesmet adesmet 175 Jul 9 18:13 myjob.submit -rwxr-xr-x 1 adesmet adesmet 194 Jul 10 10:36 myscript.sh -rw-r--r-- 1 adesmet adesmet 31 Jul 10 10:37 results.error -rw------- 1 adesmet adesmet 833 Jul 10 10:38 results.log -rw-r--r-- 1 adesmet adesmet 261 Jul 10 10:37 results.output -rwxr-xr-x 1 adesmet adesmet 81 Jul 10 10:35 watch_condor_q $ cat results.error This is sent to standard error $ cat results.output I'm process id 29149 on lab-ctrl This is sent to standard error Thu Jul 10 10:38:44 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/aa/ceb9e04077256aaa2acf4dff670897/md5/27/2f50da149fc049d07b1c27f30b67df/data TEST 1 My name (argument 1) is TEST My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting RESULT: 0 SUCCESS
Looking at DAGMan's various files, we see that DAGMan itself ran as a Condor job (specifically, a scheduler universe job):
$ ls mydag.dag mydag.dag.dagman.log mydag.dag.lib.out myjob.submit results.error results.output mydag.dag.condor.sub mydag.dag.dagman.out mydag.dag.lock myscript.sh results.log watch_condor_q $ cat mydag.dag.condor.sub # Filename: mydag.dag.condor.sub # Generated by condor_submit_dag mydag.dag universe = scheduler executable = /afs/cs.wisc.edu/u/a/d/adesmet/miron-condor-g-dagman-talk/vdt/condor/bin/condor_dagman getenv = True output = mydag.dag.lib.out error = mydag.dag.lib.out log = mydag.dag.dagman.log remove_kill_sig = SIGUSR1 arguments = -f -l . -Debug 3 -Lockfile mydag.dag.lock -Condorlog results.log -Dag mydag.dag -Rescue mydag.dag.rescue environment = _CONDOR_DAGMAN_LOG=mydag.dag.dagman.out;_CONDOR_MAX_DAGMAN_LOG=0 queue $ cat mydag.dag.dagman.log 000 (006.000.000) 07/10 10:36:43 Job submitted from host: <128.105.185.14:33785> ... 001 (006.000.000) 07/10 10:36:44 Job executing on host: <128.105.185.14:33785> ... 005 (006.000.000) 07/10 10:38:10 Job terminated. (1) Normal termination (return value 0) Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage 0 - Run Bytes Sent By Job 0 - Run Bytes Received By Job 0 - Total Bytes Sent By Job 0 - Total Bytes Received By Job ...
If you weren't watching the DAGMan output file with tail -f, you can examine the file with the following command:
$ cat mydag.dag.dagman.out
Clean up your results. Be careful when deleting the mydag.dag.* files, you do not want to delete the mydag.dag file, just mydag.dag.* .
$ rm mydag.dag.* results.*
Typically each node in a DAG will have its own Condor submit file. Create some more submit files by copying our existing file. For simplicity during this tutorial, we'll keep the submit files very similar, notably using the same executable, but your submit files and executables can differ in real-world use.
$ cp myjob.submit job.setup.submit $ cp myjob.submit job.work1.submit $ cp myjob.submit job.work2.submit $ cp myjob.submit job.workfinal.submit $ cp myjob.submit job.finalize.submit
Edit the various submit files. Change the output and error entries to point to results.NODE.output and results.NODE.error files where NODE is actually the middle word in the submit file (job.NODE.submit). So job.finalize.error would include:
output=results.finalize.output error=results.finalize.error
Here is one possible set of settings for the output entries:
$ grep '^output=' job.*.submit job.finalize.submit:output=results.finalize.output job.setup.submit:output=results.setup.output job.work1.submit:output=results.work1.output job.work2.submit:output=results.work2.output job.workfinal.submit:output=results.workfinal.output
This is important so that the various nodes don't overwrite each others output.
Leave the log entries alone. DAGMan allows for all nodes to output their logs in the same location. Condor will ensure that the different jobs will not overwrite each other's entries in the log.
log=results.log
Also change the arguments entries so that the first argument is something unique to each node (perhaps the NODE name).
For node work2, change the second argument to 120 so that it looks something like:
arguments=MyWorkerNode2 120
Add the new nodes to your DAG:
$ cat mydag.dag Job HelloWorld myjob.submit $ cat >> mydag.dag Job Setup job.setup.submit Job WorkerNode_1 job.work1.submit Job WorkerNode_Two job.work2.submit Job CollectResults job.workfinal.submit Job LastNode job.finalize.submit PARENT Setup CHILD WorkerNode_1 WorkerNode_Two PARENT WorkerNode_1 WorkerNode_Two CHILD CollectResults PARENT CollectResults CHILD LastNode Ctrl-D $ cat mydag.dag Job HelloWorld myjob.submit Job Setup job.setup.submit Job WorkerNode_1 job.work1.submit Job WorkerNode_Two job.work2.submit Job CollectResults job.workfinal.submit Job LastNode job.finalize.submit PARENT Setup CHILD WorkerNode_1 WorkerNode_Two PARENT WorkerNode_1 WorkerNode_Two CHILD CollectResults PARENT CollectResults CHILD LastNode Ctrl-D
condor_q -dag will organize jobs into their associated DAGs. Change watch_condor_q to use this:
$ rm watch_condor_q
$ cat > watch_condor_q
#! /bin/sh
while true; do
echo ....
echo .... Output from condor_q
echo ....
condor_q
echo ....
echo .... Output from condor_q -globus
echo ....
condor_q -globus
echo ....
echo .... Output from condor_q -dag
echo ....
condor_q -dag
sleep 10
done
Ctrl-D
$ cat watch_condor_q
#! /bin/sh
while true; do
echo ....
echo .... Output from condor_q
echo ....
condor_q
echo ....
echo .... Output from condor_q -globus
echo ....
condor_q -globus
echo ....
echo .... Output from condor_q -dag
echo ....
condor_q -dag
sleep 10
done
$ chmod a+x watch_condor_q
Submit your new DAG and monitor it.
Again, in separate windows you may want to run "tail -f --lines=500 results.log" and "tail -f --lines=500 mydag.dag.dagman.out" to monitor the job's progress.
$ condor_submit_dag mydag.dag Checking your DAG input file and all submit files it references. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : mydag.dag.condor.sub Log of DAGMan debugging messages : mydag.dag.dagman.out Log of Condor library debug messages : mydag.dag.lib.out Log of the life of condor_dagman itself : mydag.dag.dagman.log Condor Log file for all jobs of this DAG : results.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 8. ----------------------------------------------------------------------- $ ./watch_condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:08 R 0 2.6 condor_dagman -f - 5.0 adesmet 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh TestJo 6.0 adesmet 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 5.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 6.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:08 R 0 2.6 condor_dagman -f - 5.0 |-HelloWorld 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh TestJo 6.0 |-Setup 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:12 R 0 2.6 condor_dagman -f - 5.0 adesmet 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh TestJo 6.0 adesmet 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 5.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 6.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:12 R 0 2.6 condor_dagman -f - 5.0 |-HelloWorld 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh TestJo 6.0 |-Setup 7/10 17:45 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:42 R 0 2.6 condor_dagman -f - 5.0 adesmet 7/10 17:45 0+00:00:24 R 0 0.0 myscript.sh TestJo 6.0 adesmet 7/10 17:45 0+00:00:24 R 0 0.0 myscript.sh Setup 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 5.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 6.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:00:42 R 0 2.6 condor_dagman -f - 5.0 |-HelloWorld 7/10 17:45 0+00:00:24 R 0 0.0 myscript.sh TestJo 6.0 |-Setup 7/10 17:45 0+00:00:24 R 0 0.0 myscript.sh Setup 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:01:12 R 0 2.6 condor_dagman -f - 5.0 adesmet 7/10 17:45 0+00:00:54 R 0 0.0 myscript.sh TestJo 6.0 adesmet 7/10 17:45 0+00:00:54 R 0 0.0 myscript.sh Setup 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 5.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 6.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:01:12 R 0 2.6 condor_dagman -f - 5.0 |-HelloWorld 7/10 17:45 0+00:00:54 R 0 0.0 myscript.sh TestJo 6.0 |-Setup 7/10 17:45 0+00:00:54 R 0 0.0 myscript.sh Setup 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:01:42 R 0 2.6 condor_dagman -f - 7.0 adesmet 7/10 17:46 0+00:00:00 I 0 0.0 myscript.sh work1 8.0 adesmet 7/10 17:46 0+00:00:00 I 0 0.0 myscript.sh Worker 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 7.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 8.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:01:42 R 0 2.6 condor_dagman -f - 7.0 |-WorkerNode_ 7/10 17:46 0+00:00:00 I 0 0.0 myscript.sh work1 8.0 |-WorkerNode_ 7/10 17:46 0+00:00:00 I 0 0.0 myscript.sh Worker 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:02:12 R 0 2.6 condor_dagman -f - 7.0 adesmet 7/10 17:46 0+00:00:27 R 0 0.0 myscript.sh work1 8.0 adesmet 7/10 17:46 0+00:00:27 R 0 0.0 myscript.sh Worker 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 7.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 8.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:02:12 R 0 2.6 condor_dagman -f - 7.0 |-WorkerNode_ 7/10 17:46 0+00:00:27 R 0 0.0 myscript.sh work1 8.0 |-WorkerNode_ 7/10 17:46 0+00:00:27 R 0 0.0 myscript.sh Worker 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:02:42 R 0 2.6 condor_dagman -f - 7.0 adesmet 7/10 17:46 0+00:00:57 R 0 0.0 myscript.sh work1 8.0 adesmet 7/10 17:46 0+00:00:57 R 0 0.0 myscript.sh Worker 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 7.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond 8.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:02:43 R 0 2.6 condor_dagman -f - 7.0 |-WorkerNode_ 7/10 17:46 0+00:00:58 R 0 0.0 myscript.sh work1 8.0 |-WorkerNode_ 7/10 17:46 0+00:00:58 R 0 0.0 myscript.sh Worker 3 jobs; 0 idle, 3 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:03:13 R 0 2.6 condor_dagman -f - 8.0 adesmet 7/10 17:46 0+00:01:28 R 0 0.0 myscript.sh Worker 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 8.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:03:13 R 0 2.6 condor_dagman -f - 8.0 |-WorkerNode_ 7/10 17:46 0+00:01:28 R 0 0.0 myscript.sh Worker 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:03:43 R 0 2.6 condor_dagman -f - 8.0 adesmet 7/10 17:46 0+00:01:58 R 0 0.0 myscript.sh Worker 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 8.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:03:43 R 0 2.6 condor_dagman -f - 8.0 |-WorkerNode_ 7/10 17:46 0+00:01:58 R 0 0.0 myscript.sh Worker 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:04:13 R 0 2.6 condor_dagman -f - 9.0 adesmet 7/10 17:49 0+00:00:02 R 0 0.0 myscript.sh workfi 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 9.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:04:13 R 0 2.6 condor_dagman -f - 9.0 |-CollectResu 7/10 17:49 0+00:00:02 R 0 0.0 myscript.sh workfi 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:04:43 R 0 2.6 condor_dagman -f - 9.0 adesmet 7/10 17:49 0+00:00:32 R 0 0.0 myscript.sh workfi 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 9.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:04:43 R 0 2.6 condor_dagman -f - 9.0 |-CollectResu 7/10 17:49 0+00:00:32 R 0 0.0 myscript.sh workfi 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:05:13 R 0 2.6 condor_dagman -f - 9.0 adesmet 7/10 17:49 0+00:01:02 R 0 0.0 myscript.sh workfi 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 9.0 adesmet DONE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:05:13 R 0 2.6 condor_dagman -f - 9.0 |-CollectResu 7/10 17:49 0+00:01:02 C 0 0.0 myscript.sh workfi 1 jobs; 0 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:05:43 R 0 2.6 condor_dagman -f - 10.0 adesmet 7/10 17:50 0+00:00:13 R 0 0.0 myscript.sh Final 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 10.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:05:44 R 0 2.6 condor_dagman -f - 10.0 |-LastNode 7/10 17:50 0+00:00:13 R 0 0.0 myscript.sh Final 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:06:14 R 0 2.6 condor_dagman -f - 10.0 adesmet 7/10 17:50 0+00:00:43 R 0 0.0 myscript.sh Final 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 10.0 adesmet ACTIVE fork lab-ctrl.nesc.ed.ac.uk /tmp/username-cond -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 4.0 adesmet 7/10 17:45 0+00:06:14 R 0 2.6 condor_dagman -f - 10.0 |-LastNode 7/10 17:50 0+00:00:43 R 0 0.0 myscript.sh Final 2 jobs; 0 idle, 2 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:35688> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held Ctrl-C
Watching the logs or the condor_q output, you'll note that the CollectResults node ("workfinal") wasn't run until both of the WorkerNode nodes ("work1" and "work2") finished.
Examine your results.
$ ls job.finalize.submit mydag.dag.condor.sub myscript.sh results.setup.error results.workfinal.error job.setup.submit mydag.dag.dagman.log results.error results.setup.output results.workfinal.output job.work1.submit mydag.dag.dagman.out results.finalize.error results.work1.error watch_condor_q job.work2.submit mydag.dag.lib.out results.finalize.output results.work1.output job.workfinal.submit mydag.dag.lock results.log results.work2.error mydag.dag myjob.submit results.output results.work2.output $ tail --lines=500 results.*.error ==> results.finalize.error <== This is sent to standard error ==> results.setup.error <== This is sent to standard error ==> results.work1.error <== This is sent to standard error ==> results.work2.error <== This is sent to standard error ==> results.workfinal.error <== This is sent to standard error $ tail --lines=500 results.*.output ==> results.finalize.output <== I'm process id 29614 on lab-ctrl Thu Jul 10 10:53:58 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/0d/7c60aa10b34817d3ffe467dd116816/md5/de/03c3eb8a20852948a2af53438bbce1/data Finalize 1 My name (argument 1) is Finalize My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting ==> results.setup.output <== I'm process id 29337 on lab-ctrl Thu Jul 10 10:50:31 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/a5/fab7b658db65dbfec3ecf0a5414e1c/md5/f4/e9a04ae03bff43f00a10c78ebd60fd/data Setup 1 My name (argument 1) is Setup My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting ==> results.work1.output <== I'm process id 29444 on lab-ctrl Thu Jul 10 10:51:04 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/2e/17db42df4e113f813cea7add42e03e/md5/f6/f1bd82a2fec9a3a372a44c009a63ca/data WorkerNode1 1 My name (argument 1) is WorkerNode1 My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting ==> results.work2.output <== I'm process id 29432 on lab-ctrl Thu Jul 10 10:51:03 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/ea/9a3c8d16346b2fea808cda4b5969fa/md5/f6/f1bd82a2fec9a3a372a44c009a63ca/data WorkerNode2 120 My name (argument 1) is WorkerNode2 My sleep duration (argument 2) is 120 Sleep of 120 seconds finished. Exiting ==> results.workfinal.output <== I'm process id 29554 on lab-ctrl Thu Jul 10 10:53:27 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/c9/7ba5d43acad3d9ebdfa633839e75c3/md5/11/cd84efa75305d54100f0f451b46b35/data WorkFinal 1 My name (argument 1) is WorkFinal My sleep duration (argument 2) is 1 Sleep of 1 seconds finished. Exiting
Examine your log
$ cat results.log
000 (005.000.000) 07/10 17:45:24 Job submitted from host: <128.105.185.14:35688>
DAG Node: HelloWorld
...
000 (006.000.000) 07/10 17:45:24 Job submitted from host: <128.105.185.14:35688>
DAG Node: Setup
...
017 (006.000.000) 07/10 17:45:42 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2349/914/1057877133/
Can-Restart-JM: 1
...
001 (006.000.000) 07/10 17:45:42 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
017 (005.000.000) 07/10 17:45:42 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2348/915/1057877133/
Can-Restart-JM: 1
...
001 (005.000.000) 07/10 17:45:42 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (005.000.000) 07/10 17:46:50 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
005 (006.000.000) 07/10 17:46:50 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
000 (007.000.000) 07/10 17:46:55 Job submitted from host: <128.105.185.14:35688>
DAG Node: WorkerNode_1
...
000 (008.000.000) 07/10 17:46:56 Job submitted from host: <128.105.185.14:35688>
DAG Node: WorkerNode_Two
...
017 (008.000.000) 07/10 17:47:09 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2364/1037/1057877219/
Can-Restart-JM: 1
...
001 (008.000.000) 07/10 17:47:09 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
017 (007.000.000) 07/10 17:47:09 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2367/1040/1057877220/
Can-Restart-JM: 1
...
001 (007.000.000) 07/10 17:47:09 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (007.000.000) 07/10 17:48:17 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
005 (008.000.000) 07/10 17:49:18 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
000 (009.000.000) 07/10 17:49:22 Job submitted from host: <128.105.185.14:35688>
DAG Node: CollectResults
...
017 (009.000.000) 07/10 17:49:35 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2383/1185/1057877366/
Can-Restart-JM: 1
...
001 (009.000.000) 07/10 17:49:35 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (009.000.000) 07/10 17:50:42 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
000 (010.000.000) 07/10 17:50:42 Job submitted from host: <128.105.185.14:35688>
DAG Node: LastNode
...
017 (010.000.000) 07/10 17:50:55 Job submitted to Globus
RM-Contact: lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork
JM-Contact: https://lab-ctrl.nesc.ed.ac.uk:2392/1247/1057877446/
Can-Restart-JM: 1
...
001 (010.000.000) 07/10 17:50:55 Job executing on host: lab-ctrl.nesc.ed.ac.uk
...
005 (010.000.000) 07/10 17:52:02 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:00:00, Sys 0 00:00:00 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
...
Examine the DAGMan log
$ cat mydag.dag.dagman.out 7/10 17:45:24 ****************************************************** 7/10 17:45:24 ** condor_scheduniv_exec.4.0 (CONDOR_DAGMAN) STARTING UP 7/10 17:45:24 ** $CondorVersion: 6.5.1 Apr 22 2003 $ 7/10 17:45:24 ** $CondorPlatform: INTEL-LINUX-GLIBC22 $ 7/10 17:45:24 ** PID = 18826 7/10 17:45:24 ****************************************************** 7/10 17:45:24 DaemonCore: Command Socket at <128.105.185.14:35774> 7/10 17:45:24 argv[0] == "condor_scheduniv_exec.4.0" 7/10 17:45:24 argv[1] == "-Debug" 7/10 17:45:24 argv[2] == "3" 7/10 17:45:24 argv[3] == "-Lockfile" 7/10 17:45:24 argv[4] == "mydag.dag.lock" 7/10 17:45:24 argv[5] == "-Condorlog" 7/10 17:45:24 argv[6] == "results.log" 7/10 17:45:24 argv[7] == "-Dag" 7/10 17:45:24 argv[8] == "mydag.dag" 7/10 17:45:24 argv[9] == "-Rescue" 7/10 17:45:24 argv[10] == "mydag.dag.rescue" 7/10 17:45:24 Condor log will be written to results.log 7/10 17:45:24 DAG Lockfile will be written to mydag.dag.lock 7/10 17:45:24 DAG Input file is mydag.dag 7/10 17:45:24 Rescue DAG will be written to mydag.dag.rescue 7/10 17:45:24 Parsing mydag.dag ... 7/10 17:45:24 Dag contains 6 total jobs 7/10 17:45:24 Bootstrapping... 7/10 17:45:24 Number of pre-completed jobs: 0 7/10 17:45:24 Submitting Job HelloWorld ... 7/10 17:45:24 assigned Condor ID (5.0.0) 7/10 17:45:24 Submitting Job Setup ... 7/10 17:45:24 assigned Condor ID (6.0.0) 7/10 17:45:25 Event: ULOG_SUBMIT for Job HelloWorld (5.0.0) 7/10 17:45:25 Event: ULOG_SUBMIT for Job Setup (6.0.0) 7/10 17:45:25 0/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 17:45:45 Event: ULOG_GLOBUS_SUBMIT for Job Setup (6.0.0) 7/10 17:45:45 Event: ULOG_EXECUTE for Job Setup (6.0.0) 7/10 17:45:45 Event: ULOG_GLOBUS_SUBMIT for Job HelloWorld (5.0.0) 7/10 17:45:45 Event: ULOG_EXECUTE for Job HelloWorld (5.0.0) 7/10 17:46:55 Event: ULOG_JOB_TERMINATED for Job HelloWorld (5.0.0) 7/10 17:46:55 Job HelloWorld completed successfully. 7/10 17:46:55 Event: ULOG_JOB_TERMINATED for Job Setup (6.0.0) 7/10 17:46:55 Job Setup completed successfully. 7/10 17:46:55 Submitting Job WorkerNode_1 ... 7/10 17:46:55 assigned Condor ID (7.0.0) 7/10 17:46:55 Submitting Job WorkerNode_Two ... 7/10 17:46:56 assigned Condor ID (8.0.0) 7/10 17:46:56 Event: ULOG_SUBMIT for Job WorkerNode_1 (7.0.0) 7/10 17:46:56 Event: ULOG_SUBMIT for Job WorkerNode_Two (8.0.0) 7/10 17:46:56 2/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 17:47:11 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_Two (8.0.0) 7/10 17:47:11 Event: ULOG_EXECUTE for Job WorkerNode_Two (8.0.0) 7/10 17:47:11 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_1 (7.0.0) 7/10 17:47:11 Event: ULOG_EXECUTE for Job WorkerNode_1 (7.0.0) 7/10 17:48:21 Event: ULOG_JOB_TERMINATED for Job WorkerNode_1 (7.0.0) 7/10 17:48:21 Job WorkerNode_1 completed successfully. 7/10 17:48:21 3/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 17:49:21 Event: ULOG_JOB_TERMINATED for Job WorkerNode_Two (8.0.0) 7/10 17:49:21 Job WorkerNode_Two completed successfully. 7/10 17:49:21 Submitting Job CollectResults ... 7/10 17:49:22 assigned Condor ID (9.0.0) 7/10 17:49:22 Event: ULOG_SUBMIT for Job CollectResults (9.0.0) 7/10 17:49:22 4/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 17:49:37 Event: ULOG_GLOBUS_SUBMIT for Job CollectResults (9.0.0) 7/10 17:49:37 Event: ULOG_EXECUTE for Job CollectResults (9.0.0) 7/10 17:50:42 Event: ULOG_JOB_TERMINATED for Job CollectResults (9.0.0) 7/10 17:50:42 Job CollectResults completed successfully. 7/10 17:50:42 Submitting Job LastNode ... 7/10 17:50:42 assigned Condor ID (10.0.0) 7/10 17:50:42 Event: ULOG_SUBMIT for Job LastNode (10.0.0) 7/10 17:50:42 5/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 17:50:57 Event: ULOG_GLOBUS_SUBMIT for Job LastNode (10.0.0) 7/10 17:50:57 Event: ULOG_EXECUTE for Job LastNode (10.0.0) 7/10 17:52:02 Event: ULOG_JOB_TERMINATED for Job LastNode (10.0.0) 7/10 17:52:02 Job LastNode completed successfully. 7/10 17:52:02 6/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 0 post 7/10 17:52:02 All jobs Completed! 7/10 17:52:02 **** condor_scheduniv_exec.4.0 (condor_DAGMAN) EXITING WITH STATUS 0
Clean up your results. Be careful about deleting the mydag.dag.* files, you do not want to delete the mydag.dag file, just mydag.dag.* .
$ rm mydag.dag.* results.*A single DAG can send jobs to a variety of sites. Condor-G is capable of managing jobs being distributed to many different sites simultaneously.
DAGMan can handle a situation where some of the nodes in a DAG fails. DAGMan will run as many nodes as possible, then create a rescue DAG making it easy to continue when the problem is fixed.
Let's create a script that will fail so we can see this:
$ cat > myscript2.sh #! /bin/sh echo "I'm process id $$ on" `hostname` echo "This is sent to standard error" 1>&2 date echo "Running as binary $0" "$@" echo "My name (argument 1) is $1" echo "My sleep duration (argument 2) is $2" sleep $2 echo "Sleep of $2 seconds finished. Exiting" echo "RESULT: 1 FAILURE" exit 1 Ctrl-D $ cat myscript2.sh #! /bin/sh echo "I'm process id $$ on" `hostname` echo "This is sent to standard error" 1>&2 date echo "Running as binary $0" "$@" echo "My name (argument 1) is $1" echo "My sleep duration (argument 2) is $2" sleep $2 echo "Sleep of $2 seconds finished. Exiting" echo "RESULT: 1 FAILURE" exit 1 $ chmod a+x myscript2.sh
Modify job.work2.submit to run myscript2.sh instead of myscript.sh:
$ rm job.work2.submit $ cat > job.work2.submit executable=myscript2.sh output=results.work2.output error=results.work2.error log=results.log notification=never universe=globus globusscheduler=lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork arguments=WorkerNode2 60 queue Ctrl-D $ cat job.work2.submit executable=myscript2.sh output=results.work2.output error=results.work2.error log=results.log notification=never universe=globus globusscheduler=lab-ctrl.nesc.ed.ac.uk:/jobmanager-fork arguments=WorkerNode2 60 queue
Submit the dag again.
$ condor_submit_dag mydag.dag Checking your DAG input file and all submit files it references. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : mydag.dag.condor.sub Log of DAGMan debugging messages : mydag.dag.dagman.out Log of Condor library debug messages : mydag.dag.lib.out Log of the life of condor_dagman itself : mydag.dag.dagman.log Condor Log file for all jobs of this DAG : results.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 15. -----------------------------------------------------------------------
Use watch_condor_q to watch the jobs until they finish.
In separate windows run "tail -f --lines=500 results.log" and "tail -f --lines=500 mydag.dag.dagman.out" to monitor the job's progress.
$ ./watch_condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 15.0 adesmet 7/10 11:11 0+00:00:04 R 0 2.6 condor_dagman -f - 16.0 adesmet 7/10 11:11 0+00:00:00 I 0 0.0 myscript.sh 17.0 adesmet 7/10 11:11 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 16.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u 17.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 15.0 adesmet 7/10 11:11 0+00:00:04 R 0 2.6 condor_dagman -f - 16.0 |-HelloWorld 7/10 11:11 0+00:00:00 I 0 0.0 myscript.sh 17.0 |-Setup 7/10 11:11 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held Output of watch_condor_q truncated -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held Ctrl-C
Check your results:
$ ls job.finalize.submit mydag.dag.condor.sub myscript.sh results.output results.work2.output job.setup.submit mydag.dag.dagman.log myscript2.sh results.setup.error results.workfinal.error job.work1.submit mydag.dag.dagman.out results.error results.setup.output results.workfinal.output job.work2.submit mydag.dag.lib.out results.finalize.error results.work1.error watch_condor_q job.workfinal.submit mydag.dag.lock results.finalize.output results.work1.output mydag.dag myjob.submit results.log results.work2.error $ cat results.work2.output I'm process id 29921 on lab-ctrl Thu Jul 10 11:12:42 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/87/459c159766cefb36f0d75023de0e35/md5/70/5d82b930ec61460d9c9ca65cbe5a8a/data WorkerNode2 60 My name (argument 1) is WorkerNode2 My sleep duration (argument 2) is 60 Sleep of 60 seconds finished. Exiting RESULT: 1 FAILURE $ cat mydag.dag.dagman.out 7/10 11:11:55 ****************************************************** 7/10 11:11:55 ** condor_scheduniv_exec.15.0 (CONDOR_DAGMAN) STARTING UP 7/10 11:11:55 ** $CondorVersion: 6.5.1 Apr 22 2003 $ 7/10 11:11:55 ** $CondorPlatform: INTEL-LINUX-GLIBC22 $ 7/10 11:11:55 ** PID = 27126 7/10 11:11:55 ****************************************************** 7/10 11:11:55 DaemonCore: Command Socket at <128.105.185.14:34769> 7/10 11:11:55 argv[0] == "condor_scheduniv_exec.15.0" 7/10 11:11:55 argv[1] == "-Debug" 7/10 11:11:55 argv[2] == "3" 7/10 11:11:55 argv[3] == "-Lockfile" 7/10 11:11:55 argv[4] == "mydag.dag.lock" 7/10 11:11:55 argv[5] == "-Condorlog" 7/10 11:11:55 argv[6] == "results.log" 7/10 11:11:55 argv[7] == "-Dag" 7/10 11:11:55 argv[8] == "mydag.dag" 7/10 11:11:55 argv[9] == "-Rescue" 7/10 11:11:55 argv[10] == "mydag.dag.rescue" 7/10 11:11:55 Condor log will be written to results.log 7/10 11:11:55 DAG Lockfile will be written to mydag.dag.lock 7/10 11:11:55 DAG Input file is mydag.dag 7/10 11:11:55 Rescue DAG will be written to mydag.dag.rescue 7/10 11:11:55 Parsing mydag.dag ... 7/10 11:11:55 Dag contains 6 total jobs 7/10 11:11:55 Bootstrapping... 7/10 11:11:55 Number of pre-completed jobs: 0 7/10 11:11:55 Submitting Job HelloWorld ... 7/10 11:11:55 assigned Condor ID (16.0.0) 7/10 11:11:55 Submitting Job Setup ... 7/10 11:11:55 assigned Condor ID (17.0.0) 7/10 11:11:56 Event: ULOG_SUBMIT for Job HelloWorld (16.0.0) 7/10 11:11:56 Event: ULOG_SUBMIT for Job Setup (17.0.0) 7/10 11:11:56 0/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 11:12:16 Event: ULOG_GLOBUS_SUBMIT for Job HelloWorld (16.0.0) 7/10 11:12:16 Event: ULOG_EXECUTE for Job HelloWorld (16.0.0) 7/10 11:12:16 Event: ULOG_GLOBUS_SUBMIT for Job Setup (17.0.0) 7/10 11:12:16 Event: ULOG_EXECUTE for Job Setup (17.0.0) 7/10 11:12:21 Event: ULOG_JOB_TERMINATED for Job HelloWorld (16.0.0) 7/10 11:12:21 Job HelloWorld completed successfully. 7/10 11:12:21 1/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:12:31 Event: ULOG_JOB_TERMINATED for Job Setup (17.0.0) 7/10 11:12:31 Job Setup completed successfully. 7/10 11:12:31 Submitting Job WorkerNode_1 ... 7/10 11:12:32 assigned Condor ID (18.0.0) 7/10 11:12:32 Submitting Job WorkerNode_Two ... 7/10 11:12:32 assigned Condor ID (19.0.0) 7/10 11:12:32 Event: ULOG_SUBMIT for Job WorkerNode_1 (18.0.0) 7/10 11:12:32 Event: ULOG_SUBMIT for Job WorkerNode_Two (19.0.0) 7/10 11:12:32 2/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 11:12:47 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_Two (19.0.0) 7/10 11:12:47 Event: ULOG_EXECUTE for Job WorkerNode_Two (19.0.0) 7/10 11:12:47 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_1 (18.0.0) 7/10 11:12:47 Event: ULOG_EXECUTE for Job WorkerNode_1 (18.0.0) 7/10 11:13:07 Event: ULOG_JOB_TERMINATED for Job WorkerNode_1 (18.0.0) 7/10 11:13:07 Job WorkerNode_1 completed successfully. 7/10 11:13:07 3/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:13:57 Event: ULOG_JOB_TERMINATED for Job WorkerNode_Two (19.0.0) 7/10 11:13:57 Job WorkerNode_Two completed successfully. 7/10 11:13:57 Submitting Job CollectResults ... 7/10 11:13:57 assigned Condor ID (20.0.0) 7/10 11:13:57 Event: ULOG_SUBMIT for Job CollectResults (20.0.0) 7/10 11:13:57 4/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:14:12 Event: ULOG_GLOBUS_SUBMIT for Job CollectResults (20.0.0) 7/10 11:14:12 Event: ULOG_EXECUTE for Job CollectResults (20.0.0) 7/10 11:14:32 Event: ULOG_JOB_TERMINATED for Job CollectResults (20.0.0) 7/10 11:14:32 Job CollectResults completed successfully. 7/10 11:14:32 Submitting Job LastNode ... 7/10 11:14:32 assigned Condor ID (21.0.0) 7/10 11:14:32 Event: ULOG_SUBMIT for Job LastNode (21.0.0) 7/10 11:14:32 5/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:14:47 Event: ULOG_GLOBUS_SUBMIT for Job LastNode (21.0.0) 7/10 11:14:47 Event: ULOG_EXECUTE for Job LastNode (21.0.0) 7/10 11:15:02 Event: ULOG_JOB_TERMINATED for Job LastNode (21.0.0) 7/10 11:15:02 Job LastNode completed successfully. 7/10 11:15:02 6/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 0 post 7/10 11:15:02 All jobs Completed! 7/10 11:15:02 **** condor_scheduniv_exec.15.0 (condor_DAGMAN) EXITING WITH STATUS 0
Uh oh, DAGMan ran that remaining nodes based on bad data from node work2. Normally DAGMan checks the return code and considers non-zero a failure. We did modify myscript2.sh to return non-zero. That would normally work, but we're using Condor-G, not normal Condor. Condor-G relies on Globus and Globus doesn't return error codes.
If you're interested in having DAGMan notice a failed job and stopping the DAG at that point, you'll need to use a POST script to detect the problem. One solution is to wrap your executable in a script that will output the executable's return code to stdout and have the POST script scan the stdout for the status. Of perhaps your executable's normal output contains enough information to make the decision.
In this case, our executable is emitting a well known message. Let's add a POST script.
First, clean up your results. Be careful about deleting the mydag.dag.* files, you do not want to delete the mydag.dag file, just mydag.dag.* .
$ rm mydag.dag.* results.*
Now create a script to check the output.
$ cat > postscript_checker #! /bin/sh grep 'RESULT: 0 SUCCESS' $1 > /dev/null 2>/dev/null Ctrl-D $ cat postscript_checker #! /bin/sh grep 'RESULT: 0 SUCCESS' $1 > /dev/null 2>/dev/null $ chmod a+x postscript_checker
modify your mydag.dag to use the new script for the nodes.
$ cat >>mydag.dag Script POST Setup postscript_checker results.setup.output Script POST WorkerNode_1 postscript_checker results.work1.output Script POST WorkerNode_Two postscript_checker results.work2.output Script POST CollectResults postscript_checker results.workfinal.output Script POST LastNode postscript_checker results.finalize.output Ctrl-D $ cat mydag.dag Job HelloWorld myjob.submit Job Setup job.setup.submit Job WorkerNode_1 job.work1.submit Job WorkerNode_Two job.work2.submit Job CollectResults job.workfinal.submit Job LastNode job.finalize.submit PARENT Setup CHILD WorkerNode_1 WorkerNode_Two PARENT WorkerNode_1 WorkerNode_Two CHILD CollectResults PARENT CollectResults CHILD LastNode Script POST Setup postscript_checker results.setup.output Script POST WorkerNode_1 postscript_checker results.work1.output Script POST WorkerNode_Two postscript_checker results.work2.output Script POST CollectResults postscript_checker results.workfinal.output Script POST LastNode postscript_checker results.finalize.output $ ls job.finalize.submit job.work1.submit job.workfinal.submit myjob.submit myscript2.sh watch_condor_q job.setup.submit job.work2.submit mydag.dag myscript.sh postscript_checker
Submit the dag again with the new POST scripts in place.
$ condor_submit_dag mydag.dag Checking your DAG input file and all submit files it references. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : mydag.dag.condor.sub Log of DAGMan debugging messages : mydag.dag.dagman.out Log of Condor library debug messages : mydag.dag.lib.out Log of the life of condor_dagman itself : mydag.dag.dagman.log Condor Log file for all jobs of this DAG : results.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 22. -----------------------------------------------------------------------
Again, watch the job with watch_condor_q. In separate windows run "tail -f --lines=500 results.log" and "tail -f --lines=500 mydag.dag.dagman.out" to monitor the job's progress.
$ ./watch_condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 22.0 adesmet 7/10 11:25 0+00:00:03 R 0 2.6 condor_dagman -f - 23.0 adesmet 7/10 11:25 0+00:00:00 I 0 0.0 myscript.sh 24.0 adesmet 7/10 11:25 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 23.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u 24.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 22.0 adesmet 7/10 11:25 0+00:00:03 R 0 2.6 condor_dagman -f - 23.0 |-HelloWorld 7/10 11:25 0+00:00:00 I 0 0.0 myscript.sh 24.0 |-Setup 7/10 11:25 0+00:00:00 I 0 0.0 myscript.sh Setup 3 jobs; 2 idle, 1 running, 0 held Output of watch_condor_q truncated -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held Ctrl-C
Check your results:
$ ls job.finalize.submit mydag.dag mydag.dag.rescue results.error results.work1.error job.setup.submit mydag.dag.condor.sub myjob.submit results.log results.work1.output job.work1.submit mydag.dag.dagman.log myscript.sh results.output results.work2.error job.work2.submit mydag.dag.dagman.out myscript2.sh results.setup.error results.work2.output job.workfinal.submit mydag.dag.lib.out postscript_checker results.setup.output watch_condor_q $ cat mydag.dag.dagman.out 7/10 11:25:35 ****************************************************** 7/10 11:25:35 ** condor_scheduniv_exec.22.0 (CONDOR_DAGMAN) STARTING UP 7/10 11:25:35 ** $CondorVersion: 6.5.1 Apr 22 2003 $ 7/10 11:25:35 ** $CondorPlatform: INTEL-LINUX-GLIBC22 $ 7/10 11:25:35 ** PID = 27251 7/10 11:25:35 ****************************************************** 7/10 11:25:35 DaemonCore: Command Socket at <128.105.185.14:34913> 7/10 11:25:35 argv[0] == "condor_scheduniv_exec.22.0" 7/10 11:25:35 argv[1] == "-Debug" 7/10 11:25:35 argv[2] == "3" 7/10 11:25:35 argv[3] == "-Lockfile" 7/10 11:25:35 argv[4] == "mydag.dag.lock" 7/10 11:25:35 argv[5] == "-Condorlog" 7/10 11:25:35 argv[6] == "results.log" 7/10 11:25:35 argv[7] == "-Dag" 7/10 11:25:35 argv[8] == "mydag.dag" 7/10 11:25:35 argv[9] == "-Rescue" 7/10 11:25:35 argv[10] == "mydag.dag.rescue" 7/10 11:25:35 Condor log will be written to results.log 7/10 11:25:35 DAG Lockfile will be written to mydag.dag.lock 7/10 11:25:35 DAG Input file is mydag.dag 7/10 11:25:35 Rescue DAG will be written to mydag.dag.rescue 7/10 11:25:35 Parsing mydag.dag ... 7/10 11:25:35 jobName: Setup 7/10 11:25:35 jobName: WorkerNode_1 7/10 11:25:35 jobName: WorkerNode_Two 7/10 11:25:35 jobName: CollectResults 7/10 11:25:35 jobName: LastNode 7/10 11:25:35 Dag contains 6 total jobs 7/10 11:25:35 Bootstrapping... 7/10 11:25:35 Number of pre-completed jobs: 0 7/10 11:25:35 Submitting Job HelloWorld ... 7/10 11:25:35 assigned Condor ID (23.0.0) 7/10 11:25:35 Submitting Job Setup ... 7/10 11:25:35 assigned Condor ID (24.0.0) 7/10 11:25:36 Event: ULOG_SUBMIT for Job HelloWorld (23.0.0) 7/10 11:25:36 Event: ULOG_SUBMIT for Job Setup (24.0.0) 7/10 11:25:36 0/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 11:25:56 Event: ULOG_GLOBUS_SUBMIT for Job HelloWorld (23.0.0) 7/10 11:25:56 Event: ULOG_EXECUTE for Job HelloWorld (23.0.0) 7/10 11:25:56 Event: ULOG_GLOBUS_SUBMIT for Job Setup (24.0.0) 7/10 11:25:56 Event: ULOG_EXECUTE for Job Setup (24.0.0) 7/10 11:26:01 Event: ULOG_JOB_TERMINATED for Job HelloWorld (23.0.0) 7/10 11:26:01 Job HelloWorld completed successfully. 7/10 11:26:01 1/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:26:11 Event: ULOG_JOB_TERMINATED for Job Setup (24.0.0) 7/10 11:26:11 Job Setup completed successfully. 7/10 11:26:11 Running POST script of Job Setup... 7/10 11:26:11 1/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 1 post 7/10 11:26:16 Event: ULOG_POST_SCRIPT_TERMINATED for Job Setup (24.0.0) 7/10 11:26:16 POST Script of Job Setup completed successfully. 7/10 11:26:16 Submitting Job WorkerNode_1 ... 7/10 11:26:16 assigned Condor ID (25.0.0) 7/10 11:26:16 Submitting Job WorkerNode_Two ... 7/10 11:26:17 assigned Condor ID (26.0.0) 7/10 11:26:17 Event: ULOG_SUBMIT for Job WorkerNode_1 (25.0.0) 7/10 11:26:17 Event: ULOG_SUBMIT for Job WorkerNode_Two (26.0.0) 7/10 11:26:17 2/6 done, 0 failed, 2 submitted, 0 ready, 0 pre, 0 post 7/10 11:26:32 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_1 (25.0.0) 7/10 11:26:32 Event: ULOG_EXECUTE for Job WorkerNode_1 (25.0.0) 7/10 11:26:32 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_Two (26.0.0) 7/10 11:26:32 Event: ULOG_EXECUTE for Job WorkerNode_Two (26.0.0) 7/10 11:26:52 Event: ULOG_JOB_TERMINATED for Job WorkerNode_1 (25.0.0) 7/10 11:26:52 Job WorkerNode_1 completed successfully. 7/10 11:26:52 Running POST script of Job WorkerNode_1... 7/10 11:26:52 2/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 1 post 7/10 11:26:57 Event: ULOG_POST_SCRIPT_TERMINATED for Job WorkerNode_1 (25.0.0) 7/10 11:26:57 POST Script of Job WorkerNode_1 completed successfully. 7/10 11:26:57 3/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:27:42 Event: ULOG_JOB_TERMINATED for Job WorkerNode_Two (26.0.0) 7/10 11:27:42 Job WorkerNode_Two completed successfully. 7/10 11:27:42 Running POST script of Job WorkerNode_Two... 7/10 11:27:42 3/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 1 post 7/10 11:27:47 Event: ULOG_POST_SCRIPT_TERMINATED for Job WorkerNode_Two (26.0.0) 7/10 11:27:47 POST Script of Job WorkerNode_Two failed with status 1 7/10 11:27:47 3/6 done, 1 failed, 0 submitted, 0 ready, 0 pre, 0 post 7/10 11:27:47 ERROR: the following job(s) failed: 7/10 11:27:47 ---------------------- Job ---------------------- 7/10 11:27:47 Node Name: WorkerNode_Two 7/10 11:27:47 NodeID: 3 7/10 11:27:47 Node Status: STATUS_ERROR 7/10 11:27:47 Error: POST Script failed with status 1 7/10 11:27:47 Job Submit File: job.work2.submit 7/10 11:27:47 POST Script: postscript_checker results.work2.output 7/10 11:27:47 Condor Job ID: (26.0.0) 7/10 11:27:47 Q_PARENTS: 1, <END> 7/10 11:27:47 Q_WAITING: <END> 7/10 11:27:47 Q_CHILDREN: 4, <END> 7/10 11:27:47 --------------------------------------- <END> 7/10 11:27:47 Writing Rescue DAG file... 7/10 11:27:47 **** condor_scheduniv_exec.22.0 (condor_DAGMAN) EXITING WITH STATUS 1
DAGMan notices that one of the jobs failed. DAGMan ran as much of the DAG as possible and logged enough information to continue the run when the situation is resolved.
Look at the rescue DAG. It's the same structurally as your original DAG, but notes that finished are marked DONE. (DAGMan also reorganized the file.) When you submit the rescue DAG, DONE nodes will be skipped.
$ cat mydag.dag.rescue # Rescue DAG file, created after running # the mydag.dag DAG file # # Total number of Nodes: 6 # Nodes premarked DONE: 3 # Nodes that failed: 1 # WorkerNode_Two,<ENDLIST> JOB HelloWorld myjob.submit DONE JOB Setup job.setup.submit DONE SCRIPT POST Setup postscript_checker results.setup.output JOB WorkerNode_1 job.work1.submit DONE SCRIPT POST WorkerNode_1 postscript_checker results.work1.output JOB WorkerNode_Two job.work2.submit SCRIPT POST WorkerNode_Two postscript_checker results.work2.output JOB CollectResults job.workfinal.submit SCRIPT POST CollectResults postscript_checker results.workfinal.output JOB LastNode job.finalize.submit SCRIPT POST LastNode postscript_checker results.finalize.output PARENT Setup CHILD WorkerNode_1 WorkerNode_Two PARENT WorkerNode_1 CHILD CollectResults PARENT WorkerNode_Two CHILD CollectResults PARENT CollectResults CHILD LastNode
So we know there is a problem with the work2 step. Let's "fix" it.
$ rm myscript2.sh $ cp myscript.sh myscript2.sh
Now we can submit our rescue DAG. (If you didn't fix the problem, DAGMan would have generated another rescue DAG, this time "mydag.dag.rescue.rescue".) In separate windows run "tail -f --lines=500 results.log" and "tail -f --lines=500 mydag.dag.dagman.out" to monitor the job's progress.
$ condor_submit_dag mydag.dag.rescue Checking your DAG input file and all submit files it references. This might take a while... Done. ----------------------------------------------------------------------- File for submitting this DAG to Condor : mydag.dag.rescue.condor.sub Log of DAGMan debugging messages : mydag.dag.rescue.dagman.out Log of Condor library debug messages : mydag.dag.rescue.lib.out Log of the life of condor_dagman itself : mydag.dag.rescue.dagman.log Condor Log file for all jobs of this DAG : results.log Submitting job(s). Logging submit event(s). 1 job(s) submitted to cluster 27. ----------------------------------------------------------------------- $ ./watch_condor_q -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 27.0 adesmet 7/10 11:34 0+00:00:01 R 0 2.6 condor_dagman -f - 28.0 adesmet 7/10 11:34 0+00:00:00 I 0 0.0 myscript2.sh Worke 2 jobs; 1 idle, 1 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE 28.0 adesmet UNSUBMITTED fork lab-ctrl.nesc.ed.ac.uk /afs/cs.wisc.edu/u -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 27.0 adesmet 7/10 11:34 0+00:00:01 R 0 2.6 condor_dagman -f - 28.0 |-WorkerNode_ 7/10 11:34 0+00:00:00 I 0 0.0 myscript2.sh Worke 2 jobs; 1 idle, 1 running, 0 held Output of watch_condor_q truncated -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER STATUS MANAGER HOST EXECUTABLE -- Submitter: puffin.cs.wisc.edu : <128.105.185.14:33785> : puffin.cs.wisc.edu ID OWNER/NODENAME SUBMITTED RUN_TIME ST PRI SIZE CMD 0 jobs; 0 idle, 0 running, 0 held Ctrl-C
Check your results.
$ ls job.finalize.submit mydag.dag.lib.out myscript2.sh results.work1.error job.setup.submit mydag.dag.rescue postscript_checker results.work1.output job.work1.submit mydag.dag.rescue.condor.sub results.error results.work2.error job.work2.submit mydag.dag.rescue.dagman.log results.finalize.error results.work2.output job.workfinal.submit mydag.dag.rescue.dagman.out results.finalize.output results.workfinal.error mydag.dag mydag.dag.rescue.lib.out results.log results.workfinal.output mydag.dag.condor.sub mydag.dag.rescue.lock results.output watch_condor_q mydag.dag.dagman.log myjob.submit results.setup.error mydag.dag.dagman.out myscript.sh results.setup.output $ cat mydag.dag.rescue.dagman.out 7/10 11:34:33 ****************************************************** 7/10 11:34:33 ** condor_scheduniv_exec.27.0 (CONDOR_DAGMAN) STARTING UP 7/10 11:34:33 ** $CondorVersion: 6.5.1 Apr 22 2003 $ 7/10 11:34:33 ** $CondorPlatform: INTEL-LINUX-GLIBC22 $ 7/10 11:34:33 ** PID = 27317 7/10 11:34:33 ****************************************************** 7/10 11:34:33 DaemonCore: Command Socket at <128.105.185.14:35032> 7/10 11:34:33 argv[0] == "condor_scheduniv_exec.27.0" 7/10 11:34:33 argv[1] == "-Debug" 7/10 11:34:33 argv[2] == "3" 7/10 11:34:33 argv[3] == "-Lockfile" 7/10 11:34:33 argv[4] == "mydag.dag.rescue.lock" 7/10 11:34:33 argv[5] == "-Condorlog" 7/10 11:34:33 argv[6] == "results.log" 7/10 11:34:33 argv[7] == "-Dag" 7/10 11:34:33 argv[8] == "mydag.dag.rescue" 7/10 11:34:33 argv[9] == "-Rescue" 7/10 11:34:33 argv[10] == "mydag.dag.rescue.rescue" 7/10 11:34:33 Condor log will be written to results.log 7/10 11:34:33 DAG Lockfile will be written to mydag.dag.rescue.lock 7/10 11:34:33 DAG Input file is mydag.dag.rescue 7/10 11:34:33 Rescue DAG will be written to mydag.dag.rescue.rescue 7/10 11:34:33 Parsing mydag.dag.rescue ... 7/10 11:34:33 jobName: Setup 7/10 11:34:33 jobName: WorkerNode_1 7/10 11:34:33 jobName: WorkerNode_Two 7/10 11:34:33 jobName: CollectResults 7/10 11:34:33 jobName: LastNode 7/10 11:34:33 Dag contains 6 total jobs 7/10 11:34:33 Deleting older version of results.log 7/10 11:34:33 Bootstrapping... 7/10 11:34:33 Number of pre-completed jobs: 3 7/10 11:34:33 Submitting Job WorkerNode_Two ... 7/10 11:34:33 assigned Condor ID (28.0.0) 7/10 11:34:34 Event: ULOG_SUBMIT for Job WorkerNode_Two (28.0.0) 7/10 11:34:34 3/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:34:54 Event: ULOG_GLOBUS_SUBMIT for Job WorkerNode_Two (28.0.0) 7/10 11:34:54 Event: ULOG_EXECUTE for Job WorkerNode_Two (28.0.0) 7/10 11:35:59 Event: ULOG_JOB_TERMINATED for Job WorkerNode_Two (28.0.0) 7/10 11:35:59 Job WorkerNode_Two completed successfully. 7/10 11:35:59 Running POST script of Job WorkerNode_Two... 7/10 11:35:59 3/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 1 post 7/10 11:36:04 Event: ULOG_POST_SCRIPT_TERMINATED for Job WorkerNode_Two (28.0.0) 7/10 11:36:04 POST Script of Job WorkerNode_Two completed successfully. 7/10 11:36:04 Submitting Job CollectResults ... 7/10 11:36:04 assigned Condor ID (29.0.0) 7/10 11:36:04 Event: ULOG_SUBMIT for Job CollectResults (29.0.0) 7/10 11:36:04 4/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:36:19 Event: ULOG_GLOBUS_SUBMIT for Job CollectResults (29.0.0) 7/10 11:36:19 Event: ULOG_EXECUTE for Job CollectResults (29.0.0) 7/10 11:36:34 Event: ULOG_JOB_TERMINATED for Job CollectResults (29.0.0) 7/10 11:36:34 Job CollectResults completed successfully. 7/10 11:36:34 Running POST script of Job CollectResults... 7/10 11:36:34 4/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 1 post 7/10 11:36:39 Event: ULOG_POST_SCRIPT_TERMINATED for Job CollectResults (29.0.0) 7/10 11:36:39 POST Script of Job CollectResults completed successfully. 7/10 11:36:39 Submitting Job LastNode ... 7/10 11:36:39 assigned Condor ID (30.0.0) 7/10 11:36:39 Event: ULOG_SUBMIT for Job LastNode (30.0.0) 7/10 11:36:39 5/6 done, 0 failed, 1 submitted, 0 ready, 0 pre, 0 post 7/10 11:36:54 Event: ULOG_GLOBUS_SUBMIT for Job LastNode (30.0.0) 7/10 11:36:54 Event: ULOG_EXECUTE for Job LastNode (30.0.0) 7/10 11:37:09 Event: ULOG_JOB_TERMINATED for Job LastNode (30.0.0) 7/10 11:37:09 Job LastNode completed successfully. 7/10 11:37:09 Running POST script of Job LastNode... 7/10 11:37:09 5/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 1 post 7/10 11:37:14 Event: ULOG_POST_SCRIPT_TERMINATED for Job LastNode (30.0.0) 7/10 11:37:14 POST Script of Job LastNode completed successfully. 7/10 11:37:14 6/6 done, 0 failed, 0 submitted, 0 ready, 0 pre, 0 post 7/10 11:37:14 All jobs Completed! 7/10 11:37:14 **** condor_scheduniv_exec.27.0 (condor_DAGMAN) EXITING WITH STATUS 0 $ cat results.work2.output I'm process id 30478 on lab-ctrl Thu Jul 10 11:34:46 CDT 2003 Running as binary /n/uscms_share/home/adesmet2/.globus/.gass_cache/local/md5/23/61b50cd9b278330cac68107dd390d6/md5/5e/004f7216b8b846d548357da00985f4/data WorkerNode2 60 My name (argument 1) is WorkerNode2 My sleep duration (argument 2) is 60 Sleep of 60 seconds finished. Exiting RESULT: 0 SUCCESS $ exit
$ condor_off Sent "Kill-All-Daemons" command to local master
That's it. There is a lot more you can do with Condor-G and DAGMan, but this basic introduction is all you need to know to get started. Good luck!