The latest blah blah about BLAHP.

a brief update.

The BLAHP is like the GAHP, but it's used to locally transfer a job to a 3^rd party batch system and control it.

Obviously stands for Batch Local ASCII Helper (Protocol)
Also a term with rich semantics.
Epitomized the long, strenuous, vacuous discussions that were taking places in not-to-be-named, copiously funded projects.
Or, as the dictionary goes: a word commonly used to describe an emotional state in which the person feels a sense of having no hope...
Others leveraged the idea:
Still, this name seems to be keeping us safely clear of marketing and trademark struggles: no need for HTBLAH(P) (!) yet.

So, when we met almost exactly 6 years ago we were talking about the gLite CE:

In the meanwhile:

Contributors within the EGEE and EMI projects provided SGE and SLURM support scripts.
The CREAM CE, that is also relying on the BLAH(P) for local job submission, was rolled into production at WLCG sites, including CERN with its O(100000) pending LSF jobs.
Like many other projects and components, BLAH was moved to github. Cmake build scripts are now available alongside EU-grid-project-flavoured autotools.
Added and tested IPv6 support where applicable in BLAH... Now we'll see how seriously the batch system providers will play with IPv6.
Other applications, new and old, such as BOSCO and the HTCondor-CE appeared.

BLAH could in principle be transparent and stateless. With moderate amounts of jobs it can indeed be deployed as such.
The least-scalable function of all batch systems we have been working with (including Condor - which led to the development of Quill a few years back) has always been getting back the status of submitted jobs.
Some canonical extra-layer-of-indirection had to be added...

Use of the BLAH registry is triggerable by configuration and tries to be as transparent as possibile.

The dilemma: telling a fast, completed job from a lost, disappeared job.
'History' commands are heavy for all batch systems, and there can be disrupting surprises.
Most batch systems can be configured to keep terminated jobs in their active job database for a while.
Still, for reasons that would deserve some investigation by Platform, LSF's command line tools (e.g. bjobs) can be up to four times slower than obtaining the same information via the lsbatch API!
Ulrich Schwickerath, a friend at CERN, maintains a few API-based tools for LSF that can be used in conjunction with BLAH. in the cernops/info.dynamic-scheduler-lsf github package.
But one needs to know: this knowledge still needs to be kept together in one place.

With just a fraction of three people at hand, not much is boiling in terms of new features:

A high availability mode, where updates to the job registry are shared (optionally via multicast) among a pool of BLAH servers (so that all can service requests on the same set of jobs) is in the code. Hasn't seen thorough testing yet.
Better, configurable logging (other than what is exchanged in the command line and/or logged to stdout and stderr).

We could not keep it as simple and dumb as we hoped, but the BLAH(P) is still around, now pushing around the majority of WLCG jobs, and possibly a few Higgs bosons here and there.
Simple semantics provided (as usual) the ability to compose new tools based on any real need at hand.
However, pushing the scale factor up to WLCG center production needs required measures that are normally not enabled when BLAH(P) is used from within HTCondor.
These can be enabled if needed, but knowledge and documentation are still not gathered in a single place, as it mostly comes from people with firsthand expertise at managing the various batch systems.
- Now that we don't depend on decisions and priorities set by upper-tier projects we should perhaps start populating the github Wiki.

BLAH doesn't have much of a roadmap ahead, but we are as usual open to suggestions, requests (feature or support), and to directly provide info beyond the scarce documentation:

blah@mi.infn.it
Thank you for your time. Hope it wasn't too BLAH.