Security Recommendation for sites using Condor

on the Open Science Grid (OSG)

 

Todd Tannenbaum, OSG Condor Liaison

Original Version: May 31, 2007

Last edited: Sept 24, 2007

 

1 Introduction

Many sites on OSG have Condor configured to use host-based authentication to establish trust amongst Condor services running on submit, execute, and central manager nodes.  Like any service configured to use host-based authentication, this means the site must trust all users that can launch a process on a trusted machine.  Furthermore, host-based authentication could become vulnerable to IP spoofing attacks. For these reasons, OSG strongly recommends that sites relying on host-based authentication refer to the recommendations below in Section 2 and switch to using credential-based authentication.  Failure to do so could permit a malicious user who can launch a process on an execute node to impersonate a Condor daemon. 

 

Condor version 6.8.x and above supports several credential-based authentication options, such as GSI, X509, Kerberos, and a shared secret (a password).  However, because it is easier and faster to setup, and because it is the default in Condor’s installation program, many OSG sites are still using host-based authentication.  To help remedy this situation, this document identifies best practices along with specific configuration recipes recommended for all Condor installations on the OSG.

 

2 Recommendations

OSG recommends that sites reconfigure Condor to use a credential-based authentication method as outlined in the Condor Manual ( see http://tinyurl.com/29wkm7 ).  To assist in doing so, some quick step-by-step directions are provided in Section 2.1.  For sites that cannot immediately switch over to credential-based authentication, please at least follow the best practices for Condor host-based recommendations outlined in Section 2.2.

 

2.1 Credential-based Authentication

Detailed below are instructions for two credential-based security methods that should be well-suited for OSG sites.  Section 2.1.1 describes how to secure your Condor pool by use of a shared secret, whereas section 2.1.2 describes how to secure your pool by leveraging pre-existing GSI host certificates.

 

2.1.1 Configuring Condor to use a shared secret (pool password)

Changing a Condor pool to use a password credential is a straightforward process.  Condor’s password security method provides mutual authentication through the use of a shared secret. This is often a good choice when strong security is desired, but an existing Kerberos, GSI, or X.509 infrastructure is not in place.  A Condor “pool password” is selected by the administrator and stored in a file on each machine in the Condor pool.  Access to this file is only permitted by the uid (user id) that is used to launch the Condor daemons; typically, this means the file should be only readable by root.   Condor daemons in the pool will only trust other Condor daemons on the network that can prove knowledge of the pool password.   

 

1) Generate the pool password.
 

The version of the condor_store_cred tool distributed with Condor 6.8.5 has an option that allows it to generate a file that can be used for PASSWORD authentication on UNIX Condor installations. To generate the password, run:

 
$ condor_store_cred -f pool_password
Enter password: <enter password here>

 

The file pool_password now contains the file needed by Condor to do PASSWORD

Authentication – skip to step 2 below.

 

If you are using a version of Condor between version 6.8.0 and 6.8.4 (inclusive), the condor_store_cred command does not have a “-f” option to easily create the pool password file.  You can either download the v6.8.5 condor_store_cred tool from the Condor web site, or you can perform the following instructions to create the pool password file on any v6.8.x version of Condor:

a)      login as root on your central manager

b)      place the following in your condor_config:

             ALLOW_CONFIG = root@mydomain/$(CONDOR_HOST)

     SEC_PASSWORD_FILE = /tmp/pool_password

   c) enter “condor_reconfig -full”.

   d) enter “condor_store_cred –c add” and answer the prompt.

   e) remove the ALLOW_CONFIG and SEC_PASSWORD_FILE settings from your

config file, and enter “condor_reconfig -full”.

 

The file /tmp/pool_password now contains the file needed by Condor to do PASSWORD

Authentication.

 
2) Distribute the pool password.
 
Now this file needs to be distributed to each machine in the Condor pool.
The location of the pool password file will then be configured as
SEC_PASSWORD_FILE in condor_config. A good place for this might be Condor's
LOCK directory. For example, you could copy the pool password file to
$(LOCK)/pool_password on each one of your machines running Condor. This can
be done using SSH or any other way that you can securely distribute the
file.
 
The pool password file should have a mode of 0600 (which is how the mode will
be set when it's generated) and must be owned by the real UID which the
Condor daemons run as.
 
3) Alter your configuration to use PASSWORD authentication
 
Once the pool password file has been distributed to all machines in your
pool, you can configure your daemons to authenticate to each other. The following example illustrates:
 
SEC_PASSWORD_FILE = $(LOCK)/pool_password
SEC_DAEMON_AUTHENTICATION = REQUIRED
SEC_DAEMON_INTEGRITY = REQUIRED
SEC_DAEMON_AUTHENTICATION_METHODS = PASSWORD
SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED
SEC_NEGOTIATOR_INTEGRITY = REQUIRED
SEC_NEGOTIATOR_AUTHENTICATION_METHODS = PASSWORD
SEC_CLIENT_AUTHENTICATION_METHODS = FS, PASSWORD, KERBEROS, GSI
 
You'll then need to reconfig your daemons for these settings to take effect.

 

2.1.2 Configuring Condor to authenticate using GSI host certificates

If you have already deployed GSI credentials for your hosts, Condor can use those to secure access to the Condor pool.  If not, the first step of course is to acquire host certificates for each host in your pool.  By convention, the host certificate and private key are stored in /etc/grid-security/hostcert.pem and /etc/grid-security/hostkey.pem respectively.  The hostkey.pem file should be readable only by root.

 

Note:  It is possible to use one certificate for the entire pool if desired.  However, this weakens the security somewhat and can lead to confusion because a given host will be presenting a credential claiming to be a different host.  It is best practice to get a unique and properly named certificate for each host.

 

After acquiring and distributing host certificates, Condor must be told to use them by adding settings to the Condor configuration file (condor_config) requiring GSI authentication for all daemon-to-daemon communication:

 

SEC_DAEMON_AUTHENTICATION = REQUIRED
SEC_DAEMON_INTEGRITY = REQUIRED
SEC_DAEMON_AUTHENTICATION_METHODS = GSI
SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED
SEC_NEGOTIATOR_INTEGRITY = REQUIRED
SEC_NEGOTIATOR_AUTHENTICATION_METHODS = GSI
 

It is not sufficient to simply require authentication, because users also have GSI credentials which they can use.  It must be further specified exactly which authenticated users are authorized to perform these communications.  The first step in this is mapping the authenticated names, which typically look like this:

  /DC=org/DC=doegrids/OU=Services/CN=vdt-rhas3-ia32.cs.wisc.edu

 

To map the authenticated name, you must place an entry in your grid map file, which typically is found in /etc/grid-security/grid-mapfile, for each subject name and map it

to the ‘condor’ user.  You must do this for each host certificate.  An example (substitute your site’s domain for ‘cs.wisc.edu’):

 

  “/DC=org/DC=doegrids/OU=Services/CN=vdt-rhas3-ia32.cs.wisc.edu” condor@cs.wisc.edu

 

Then you must specify which mapped names are authorized for daemon-to-daemon communication.  This is done by adding a condor_config setting:

 
ALLOW_DAEMON = condor@cs.wisc.edu
  ALLOW_NEGOTIATOR = condor@cs.wisc.edu

 

The value on the right-hand side of the equals sign must match whatever you put on the right-hand side in the grid-mapfile.

 

Finally, GSI authentication also requires that the issuer of the certificates is known and trusted.  This is accomplished by having the Certificate Authority (CA) public certificate stored locally along with a signing policy file.  OSG sites typically store these public certificates and policy files in $GLOBUS_LOCATION/share/certificates.  You must tell Condor the fully expanded path.  I.e., do not write ‘$GLOBUS_LOCATION’ but instead use the real path similar to ‘/software/vdt/globus’.  Add the following setting to your condor_config file (of course substituting the actual path for your site):

 

  GSI_DAEMON_TRUSTED_CA_DIR = /software/vdt/globus/share/certificates

 

After updating your grid-mapfile and condor_config, you need to restart your Condor daemons, including the condor_master.

 

2.2 Best Practices when using Host-Based Authentication

If you cannot switch your Condor installation over to some form of credential-based authentication immediately, please at least consider one or more of the following short-term recommendations to help secure your host-based authentication setup.  You can safely skip the recommendations in this section (2.2) if you are using a form of credential-based authentication as described above.

 

2.2.1 Execution nodes should only trust a small number of submit nodes

The Condor central manager and submit nodes need write authorization to all machines in the pool.  However, execute-only nodes (i.e. nodes only running a condor_master and a condor_startd) only need to give write authorization to submit nodes and the negotiator service on the central manager.  There is no need for an execute node to allow write access from another execute node.  So for example, instead of the following typical entries in condor_config :

            HOSTALLOW_WRITE = *.mycluster.whatever.gov

     HOSTALLOW_WRITE_STARTD = $(HOSTALLOW_WRITE), \

$(FLOCK_FROM)

we recommend the following:

            HOSTALLOW_WRITE = *.mycluster.whatever.gov

     SUBMIT_NODES = frontend1.mycluster.whatever.gov, \

                   submit.mycluster.whatever.gov

HOSTALLOW_WRITE_STARTD = $(SUBMIT_NODES), \

                        $(HOSTALLOW_NEGOTIATOR), \

$(FLOCK_FROM)

     HOSTALLOW_WRITE_STARTER = $(HOSTALLOW_WRITE_STARTD)

The above settings restrict write access to the startd and starter execute services to only to only submit nodes, trusted flocking sites, and the negotiator (matchmaker).

 

2.2.1 Only allow machines to join the pool from a well know IP and port

Using Condor version 6.8.5 and above, a new condor_collector configuration macro COLLECTOR_REQUIREMENTS has been added.  This macro is a ClassAd expression which is applied against every incoming ClassAd.  Only if the expression evaluates to True is the incoming ad trusted and subsequently added into the collector.    Using this mechanism, it is possible to prevent Condor daemons from joining a pool unless they are running on a specific well-known IP port.  If you are launching your Condor Daemons as root, you can specify that Condor bind to a privileged port, i.e. a port below 1024.  In this manner, you are able to confirm that a given Condor daemon at a given IP address is running as root, not as a normal user.  [Note: On Windows machines, normal users can bind to ports below 1024].  Even if you do not use privileged ports for Condor (perhaps because firewall policy would block it), using well-known ports above 1024 would still help as a non-authentic daemon likely could not successfully bind to it (because the authentic Condor daemon already is using the port).  Below is an example configuration you could place into your pool-wide condor_config file(s):

 

# Well-known ports for the trusted daemons

# Use the below ports if launching the condor_master

# as root; else, pick 3 ports above 1024.

MASTER_PORT = 890

SCHEDD_PORT = 891

STARTD_PORT = 892

 

MASTER_ARGS = -p $(MASTER_PORT)

SCHEDD_ARGS = -p $(SCHEDD_PORT)

STARTD_ARGS = -p $(STARTD_PORT)

 

COLLECTOR_REQUIREMENTS = \

( MyType =?= "Machine" && \

  regexp( "<[0-9.]*:$(STARTD_PORT)>" , MyAddress ) ) || \

( MyType =?= "Scheduler" && \

  regexp( "<[0-9.]*:$(SCHEDD_PORT)>" , MyAddress ) ) || \

( MyType =?= "DaemonMaster" && \

  regexp( "<[0-9.]*:$(MASTER_PORT)>" , MyAddress ) ) || \

( MyType =!= "Machine" && MyType =!= "Scheduler" && \

  MyType =!= "DaemonMaster" )

 

2.2.1 Use Linux iptables to restrict communication with the central manager

An alternative to the well-known port approach described in the section above would be to configure Linux’s firewall iptables service to only allow blessed/trusted user ids (such as root, and the “condor” user) to communicate with the Condor central manager. 

 

To do this, you must setup iptables on all machines listed in the HOSTALLOW_WRITE macro in the condor_config on the central manager.  Typically this means on your OSG CE node(s) as well as worker nodes.  Then, for example, on a Red Hat based system, you would enter the following into the /etc/sysconfig/iptables script assuming that 192.168.1.18 is the IP address of the central manager:

 

-A OUTPUT -o eth0 -p tcp -m owner --uid-owner root --destination 192.168.1.18/32 --dport 9618 -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m owner --uid-owner condor --destination 192.168.1.18/32 --dport 9618 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m owner --uid-owner root --destination 192.168.1.18/32 --dport 9618 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m owner --uid-owner condor --destination 192.168.1.18/32 --dport 9618 -j ACCEPT
-A OUTPUT -p tcp --destination 192.168.1.18/32  --dport 9618 -j REJECT
-A OUTPUT -p udp --destination 192.168.1.18/32  --dport 9618 -j REJECT

 

Note that the above will disable the condor_status command from working for any non root or condor user on the machines where the rules are installed.

 

2.3 Secure the Condor central manager

It is not recommended to run the Condor central manager on your OSG CE front-end, or on a machine running a schedd or startd.  This is especially true if you are still using host-based authentication.  Ideally, run your Condor central manager on a machine with no access by users.

 

2.4 Run the latest stable series release

Keep your Condor pool updated with the latest stable version release.  All Condor releases within a stable series are guaranteed to be backwards and forwards compatible over the network with other Condor releases in the same series.  This makes it easy for administrators to update a portion of their pool at a time without incompatibility concerns.  News of new Condor releases are monitored by the VDT and can also be found on the RSS feed:

      http://www.cs.wisc.edu/condor/index.rss

Further information about Condor’s release process can be found at:

      http://www.cs.wisc.edu/condor/version_number.html