Next Generation Build System

Contents

1. Desired Features
2. Useful Terminology
3. Semantics
   Submit-side Semantics
   Remote-side Semantics
4. Description of parameters in NMI submit file
5. Environment Variables available to the users
6. Using Marcos in the NMI submit file
7. Submitting your build/test
8. Seeing the results of your build/test
9. Local Builder
   Submitting your local build/test
   Seeing the results of your local build/test
10. Examples
    Gsissh
    Condor
    Additional examples

Desired Features

The aim of the new world order build/test framework is to separate the functionality of the  frame work and the component related build/test scripts. This is achieved by having a clear semantics on how the component related scripts will be invoked.

It should have following features –

1. Ease of use.

2. Ease of adding new components without modifying the build system code.

3. Ease of adding new platforms preferable without modifying the build system code.

4. Building multiple components at the same time if they are independent and follow the dependencies when required..

5. Log the build/test related information in the database.

6. Should transfer the required externals for a particular component to the remote build machine. This can even include externals like Java. Flexibility on this feature is debatable.

7. Framework should be smart enough to handle the operating system signals and should report failure if any of the scripts is killed.

8. Web interface for component builders/testers.

Useful Terminology

• Submit Machine
  Machine from where you actually submit your build/test job.
   

• Remote Machine(s)
  Machine where the platform specific tasks of your build/test executes. If you have submitted a job that involves more than one platform, there will be more than one remote machine (one per platform) where your jobs get executed.
   

• Glue scripts
  Set of scripts written by the user to build/test the component. Glue-Scripts bind the two entities (Framework scripts and component related scripts) together.
   

• Framework/Framework scripts
  Set of scripts that control the execution of the Glue scripts. User does not need to know much about these scripts other than the useful interfaces provided by the framework. Build system framework expects specific code of conduct from the Glue-Scripts but is not concerned with how they achieve their tasks.
   

• pre_all/platform_pre/platform_job/platform_post/post_all
  Subtasks of user's build/test that will be executed on the submit machine. More about them here.

• remote_pre_declare/remote_declare/remote_pre/remote_task/remote_post
  Subtasks of user's build/test that will be executed on the remote machine. More about them here.

Semantics

The goal for having the framework is to make it simple to do simple builds, without having to understand the complexity available for more complex builds/tests.

For example, one can build a product foo by fetching it's sources from the web, untarring the sources and running make (e.g., "ncftp foo.edu:/foo.tar.gz; tar zxvf foo.tar.gz; configure; make") doesn't need to do anything special for different platforms. So this can behave just as it would on a single system. You write each of those commands in a script, specify them for the corresponding build steps in your submit file, add a remote_post to tar up whatever build output you want to save, and you're done.

Thus the build system functionality is divided into two

1. Tasks done at the submit machine
2. Task done at the remote machine

These semantics are shown below. Lines written in "blue" are the functionalities provided by the build system. Lines written in "black" are the "Glue-Scripts" provided by the component builder (user). Lines in "green" are remarks or more information available to the user.

Submit-side Semantics

Submit side of the framework is mainly concerned with doing the preliminary tasks done before the component build/test is actually dispatched to the remote machine(s) and doing the post build tasks after the component build/test results arrive back from the remote machine(s).

User initiates the build/test by passing the NMI submit file to the framework. This NMI submit file has all the information that the framework requires to successfully build/test the component. A sample NMI submit file is shown below in examples. It contains the information about the component and where the glue scripts are and what glue script to invoke at a given time. Once this information is parsed by the framework it creates an initial database record for this build/test run and gets a runid. It also creates a working directory where the build/test related information is stored. It creates the required sub-directories based on the platform list. Once these general tasks are done framework creates DAG of condor jobs for the requested build/test run. Some of these jobs are executed in the submit machine and some on the remote machine(s). Once the stage is set framework fetches component sources as instructed in the "sources" option of the NMI submit file and places them in a known location. It then invokes the pre_all script for the component if declared. Framework also invokes the platform_pre and platform_post scripts if declared at appropriate time.

Once the condor job running on the remote machine(s) completes (successfully or unsuccessfully) framework invokes the post_all script.

Framework also updates the database about the results of each subtasks for your builds/test.

Remote-side Semantics (per platform)

Once the condor job (job to build the component on a targeted platform) comes to the remote machine, the very first script to be invoked is the framework wrapper script that actually executes the glue scripts. User can split the build/test into several subtasks of his/her choice in the remote_declare. If the user needs to take any action before declaring the remote tasklist he/she can do it in remote_pre_declare. Similarly if there are any tasks to be executed before the first build/test task but after the remote_declare, this can be done in remote_pre.

Then for every subtask declared by the user the build system runs them saves the statistics and results for the subtasks and and streams the output back to the submit machine. When the component is built/tested and all the subtasks are completed remote_post script is run. This essentially serves for the tasks like packaging the component or correlating the test results in a particular way that is not a part of the build/test but is required for the component distribution. The framework expects that the glue scripts create a single tar file called "results.tar.gz" that contains everything that is to be transferred back to the submit side. Stdout and Stderr for all the subtasks are transferred back by default. Any other user log files created should be included in results.tar.gz by the user.

Description of parameters used in the NMI submit file

Refer to examples in case of doubts

Environment Variables available to the users

Framework makes some of the information available in the environment. This information can be used in the glue scripts. Available environment variables are -

Using Macros in the NMI submit file

Framework allows basic macro substitution in the NMI submit file. If you define an environmnet variable $_NMI_FOO you can use macro $(FOO).

There is a default macro available for use by the system

Submitting your build/test

Setting your path

Assuming that your NMI submit file is named "cmdfile" file to build the component run -

Seeing the results of your build/test

A very basic web interface is available for the users to see the results of their build/test run. It is available at http://grandcentral.cs.wisc.edu/build/. We are working on making it more user friendly by providing more filters for easy access.

For every run that user submits the framework creates record in the "Run" table and assigns a runid to it. You can click the link corresponding to Run table to see information related to this record. Framework also creates a record for every task that is run like fetching the sources, pre_all script, platform_pre, tasks defined in the tasklist.nmi, etc. Each task has its own unique taskid. To get the information for the tasks related to a particular Run just look at the tasks which have the runid. The web interface also provides means to access the logs, output and error over the web.

Local Builder

Local Builder is a stripped down version of the framework. It enables the user to build/test their components on the same system where it is run from. This also means that users cannot build/test the binaries for platforms other than the local system. Conceptually it works on similar principles as the framework i.e. it uses the same glue scripts however it does not rely on jobs being run by condor. All the glue and framework related scripts are invoked locally. This enables the users to leverage on the work done by the NMI group and build/test the binaries on their own system. So all the discussion mentioned above for the framework also applies to the local builder unless otherwise stated.

Submitting your local build/test

Seeing the results of your local build/test

Since the local builder does not rely on the database being present, information for the local builds/tests are not logged into the database. nmi_run_local creates a local working directory (hence forth called $workdir) in the format <username>_<machine name>_<epoche secs>_$$. This $workdir is created in the current directory where the build/test is fired from. The results for your run can be found in $workdir/userdir/<platform>/results.tar.gz

Examples

Examples below show the command description files for GsiOpenssh and condor. The scripts used to build the products are also linked from the description file.

1. GsiOpenssh

2. Condor

3. Additional examples

You can do a CVS checkout of the glue scripts for NMI components by -

[$prompt] export CVSROOT=":ext:<username>@<cvs server>:/p/condor/repository/nmi"
[$prompt] export CVS_RSH="ssh"
[$prompt] export CVS_SERVER="/p/condor/public/bin/auth-cvs"
[$prompt] cvs co nwo/glue

Please replace the username with your login name and "cvs server" with the CSL machine name where you ran the "stashticket" command. You can find more details about this here