Gozal
Technion's Condor enhancements

High availability, Usability and Manageability

Mark Silberstein, Assaf Schuster

Technion – Israel Institute of Technology

Paradyn/Condor Week 2004 UW-Madison

Introduction

Gozal (in Hebrew) - nestling

Started in 2002 as Condor deployment support project in the Distributed Systems Lab , headed by Prof Assaf Schuster

Today: 15 undergraduates, 4 graduate students, and 2 staff members

Our Goals

Adding functionality to Condor

Using Condor for teaching real distributed systems

Using Condor for research

1.Highly Available
Matchmaker

The problem we want to solve

Matchmaker is a single-point-of-failure

Our solution - Highly Available Matchmaker

Automatic failure detection

Transparent failover (no global configuration change) for the pool entities

Negligible throughput degradation in case of failure

2. Efficient invocation of
short data-intensive jobs

The problem we want to solve

Optimize Condor performance for large batches of short jobs

Our solution

Allows significant execution time speedup compared to regular Condor

Designed as add-on module

Condor native executables/configuration file are unmodified

Completely transparent

Only change – use our_condor_submit

Performance evaluation

BLAST invocation with short searches (1-3 sec)

Input data=20MB

14 CPUs

100Mb LAN

10000 Jobs

~12 times faster than regular Condor

Even better results expected over WAN

3. Resource protection - “sandbox”

The problem we want to solve

Malicious job can render a resource unusable or obtain private information

Condor can be exploited for distributed attacks on other systems

Solution – resource protection sandbox

Auditing of file system and network access, OS resources

Integration with Condor

4. Centralized policy enforcement

Problem we want to solve

Changing pool-wide policy requires global change of configuration of all resources

How to change resource or queue policy when there is no access to its configuration file

Pool-wide policy is difficult to enforce

How to force a user or an owner to specify new requirements or attributes?

Centralized policy enforcement – our solution

Negotiator is able to filter “wrong classads”

We added “wrong classads” criteria

Matchmaking

Resource

Memory=30MB

Job

Policy: allow only resources

with Memory>256MB

Policy: Reject user MARK

User=MARK

Negotiator can “AND” our expression with the Requirements expressions

Example: All computers from my lab must require jobs to explicitly specify memory use

Negotiator can alter specified classads

Example: If a (Job Owner is X or Y or Z), add GroupID=1

Centralized policy
enforcement – our solution

5. Centralized
configuration framework

“The down side of Condor flexibility is its complexity”

Lack of centralized pool configuration

Editing multiple copies of configuration files is error-prone

System administrator should learn Condor internals to configure Condor pool

Centralized configuration framework (cont)

Centralized DB for pool configurations

Resources pull the updates

Unified GUI for pool management

Configuration templates abstraction (define once, use many)

Functional change to Condor configuration, i.e. AllowUserOnResource(UserName),

Resource configuration groups

Resource can be a member of one or more groups

Resource inherits configuration from the groups of which it is a member

Projects

Highly available Condor Matchmaker

Efficient invocation of short data intensive jobs

Resource protection - “sandbox”

Centralized policy enforcement

Centralized configuration and management framework

Summary

We are developing addons for improved usability and manageability of Condor

We work in tight collaboration with the Condor team

Some of the projects already have the deliverables available in beta

The work is in progress by staff members to make releases stable

Do you want to try it?

More information at http://dsl.cs.technion.ac.il/projects/gozal

This work would not be possible without

Prof Schuster, Eran Issler, Noam Palatin, Gabi Kliot and all the undergraduate students who worked on these projects

and of course the Condor team and Prof Miron Livni

Contact: marks@tx.technion.ac.il

THANK YOU

In case I have time

Technical zoom on the projects

How it works
HA matchmaker 1

Collector

Collector

Collector

Resource

Queue

Negotiator

(backup-idle)

Resource

Resource

Queue

Queue

Negotiator

(Primary-active)

“I'm alive”

Negotiator

(backup-idle)

How it works
HA matchmaker 2

Collector

Collector

Collector

Resource

Queue

Negotiator

Resource

Resource

Queue

Queue

Negotiator

Negotiator

Leader election

How it works
HA matchmaker 3

Collector

Collector

Resource

Queue

Resource

Resource

Queue

Queue

Negotiator

“I'm alive”

Negotiator

idle

How it works
Efficient data-intensive short jobs

Submit machine

Condor

Condor

Executable cache

I/O pipeline

and optimizer

Resource

Condor

Condor

Jobs

How it works
Efficient data-intensive short jobs

Submit machine

Job queue

controller

Condor

Condor

Executable cache

I/O pipeline

and optimizer

Resource

Agent

Condor

Condor

Agent

submission

Agents

Jobs

How it works
Efficient data-intensive short jobs

Submit machine

Job queue

controller

Condor

Condor

Executable cache

I/O pipeline

and optimizer

Resource

Agent

Condor

Executable cache

I/O pipeline

and optimizer

Job

Condor

Next job parameters and input data

Output

Agents

Jobs

Matchmaker is the heart of Condor system. When it fails, newly submitted jobs will not be matched and resources will stay idle. Significant losses of CPU time, especially in large pools!

Highly available matchmaker runs on several computers in the pool-one primary and several backup. When one of them fails another will take over.

The failure is absolutely transparent for schedds and startds, as detailed in the following. No global communication is done.

Throughput degradation due to matchmaker switch is negligible even when the load is very high, and is absent at all with the regular load.

“Short” jobs - invocation/communication time is higher than computation time, i.e. Invocation overhead is too high. This can be due to such factors as remote connection, input and executable transfer, output transfer, etc.

What we usually do in such case is a preprocessing - aggregating jobs in larger jobs to minimize the per-job overhead. However it does not provides for maximum performance, as it is done statically, regardless of heterogeneous resources and variable job execution time.

Our module does not require any change of the usual submission procedure except of using special condor_submit script

Executable and the job input is transferred only once to the same resource,

Condor pools usually consist of NOT only “transient” resources

Administration of a pool with mixed computer labs resources and personal computers is difficult

Template - a functional change of Condor behavior: Example: AllowUserOnResource, SetNewMatchmakerHost. (Define Once, Use Many)

Each resource is a part of one or more configuration groups. Resource automatically acquires the group's configuration.

2004-04-04T22:22:35