Figure 1. How the MyProxy server handles a generic operation from a client.
This report presents the results of the insider, independent, first principles vulnerability assessment of MyProxy. Overall, relatively few vulnerabilities were found in MyProxy and those that were found did not compromise the certificates and their passphrases managed by MyProxy.
No design flaws were found in MyProxy. Several implementation bugs were found including some that could lead to vulnerabilities including:
The rest of this paper is divided into three sections. The first section discusses the vulnerabilities in more details. The second section presents details of the vulnerability assessment showing the artifacts of the architectural, resource, privilege and component analyses. The final section provides some insight into factors that prevented problems in MyProxy.
There are different types of vulnerabilities in MyProxy that can be used in a denial of service attack. These can make the system unaccessible to legitimate users and in the worst-case make the machine hosting the MyProxy server require a reboot.
When an incoming network connection is made to the MyProxy server, the server forks a copy of the server to handle the request. If the forked server is delayed (potentially indefinitely), then some limited system resource will be completely consumed and the operating system will start killing processes. If the wrong process is killed MyProxy and/or the host can become unusable.
The first type of vulnerability is due to a lack of a time-out when reading or writing data between the client and the server. If a client opens a connection and does not send all the data the forked server will wait indefinitely for the data to arrive or for the client to close the connection.
The second type of vulnerability is due to a lack of a limit on the amount of data the forked server is willing to read or write from the client. All data from the client is encrypted as part of a TLS stream (the successor of SSL). TLS is a stream based protocol that is transmitted as a sequence of packets. Each packet consists of a fixed size header containing the size of the payload (with a maximum of 16,384), and the encrypted payload. Since the size of the entire message is not transmitted in advance, the MyProxy server reads the header, the number of bytes in the payload and concatenates the payload to a memory buffer. It then immediately checks if there is more data available on the socket, and if so repeats the process. When no more data is immediately available it decrypts all the data, requiring another buffer of equal size to store the decrypted text, and then processes the request. A malicious client can send 100's of GB of data to client, which consumes memory on the host and 10's of minutes of CPU time to decrypt all this data. Sending several of these at once can overwhelm a server.
The third type of vulnerability is due to the way that MyProxy communicates
with other processes. MyProxy can be configured to call out to external
programs to implement certain policy decisions. It does this using a function
similar to popen. The replacement function does not suffer from
potential command line injections, but it can deadlock in the way that it is
used within MyProxy. The standard popen function lets the caller
decide to read or write the process's stdout or stdin. The replacement
function returns all three standard file descriptors connected to the process
and the process id. The idiom used in MyProxy is to write to the stdin file
descriptor, close it, wait for the process to exit, then read all the data from
stdout, and finally all the data from stderr. This can lead to deadlock if the
the process writes more data than can fit in the pipe as the helper process
will not be blocked writing until MyProxy reads, but MyProxy will not read
until the helper process exits. If an attacker can trigger the process to
write a large amount of data, such as a 10KB error message the forked MyProxy
server and the spawned process will deadlock.
The fourth potential problem is that MyProxy stores all the Proxy certificates in a single flat directory. A user can store an unlimited number of proxy certificates in MyProxy. Some file systems store the files in a directory in a list which must be linearly searched when looking up a name or creating a new entry in the directory. Even on file systems that use b-tree to store the directory, some operations on the MyProxy require reading the entire directory to locate names matching a particular pattern.
safe_is_path_trusted function from the safefile library
(http://www.cs.wisc.edu/~kupsch/safefile) or something similar to prevent this
problem. This is a low risk problem since the MyProxy host should limit access
and applications on the MyProxy host, and the path configured by the
administrator should be and most likely would be a trusted path.
Some of the client executables invoke helper applications using the
system function to perform some work. The string passed to
system uses user supplied data to create a string without special
quoting to eliminate shell injections. This can lead to arbitrary code being
run as the user on the host where the client is being run. This is low risk as
the user would not normally enter an unusual string that would cause them harm,
but they could be tricked into using the parameters in a social engineering
attack, or if the command is used in the portal or menu-based system it could
allow more access than intended for the user.
MyProxy consists of a single server executable, several client executables, and some administrative executables (these run on the server host and directly access the resources and were not considered in this analysis). The primary goal of the MyProxy server is store and manage PKI X.509 proxy certificates. It can also store and manage non-proxy certificates, and act as a certificate authority (CA) issuing newly created certificates to clients
Due to the sensitive information passing through a MyProxy server, it is intended to be run on highly secured host with limited local user access and other services. This should reduce the chances of an attack from another account on the machine.
All interactions between the MyProxy server and a client follows a pattern shown in Figure 1. The server uses a forking server model to handle all requests and only a single operation is performed in each forked server. The forked server model provides isolation from faults, and contamination of the process that could affect future requests. A TLS encrypted and integrity checked channel is established between the server and the client, and all further communications proceeds over this secure channel. This secure channel provides protection of sensitive information from being exposed or modified by an untrusted network.
Figure 1. How the MyProxy server handles a generic operation from a client.
The MyProxy server uses a configuration file that specifies what operations are allowed by a particular user. Users may specify a more restrictive policy for certificates they own. In most cases user's are identified and policy decisions are made using the distinguished name (DN) in the certificate they used when negotiating the TLS connection. For certain operations the MyProxy server can be configured to allows an anonymous client connection for operations like retrieving a certificate in the case where the client does not already have a certificate. In this case another means of authentication is used such as Kerberos, PAM, SASL or presenting the passphrase of the encrypted key of the stored certificate.
The MyProxy server supports the following operations:
The GET and PUT operations use the certificate signing request (CSR) protocol, so the keys of the original certificate and the derived proxy certificate are never known by the other party. In this way the original certificate's key cannot be compromised even if the other party has been compromised. This implies that the original certificate is not compromised in the case of derived proxy being compromised. A data flow diagram of the PUT and GET operations are shown in Figures 2 and 3 respectively. The STORE and RETRIEVE operations directly upload or download a certificate to and from the server, so if a certificate that was retrieved is compromised, so is the version on the server.
When storing a certificate a policy can be setup to only allow a GET to be performed only if the client is renewing the certificate (they can prove that they have a non-expired proxy credential with the same DN). This allows Service or Worker nodes to renew a short lived proxy for a particular user if they have a valid service certificate, know the passphrase for the stored proxy, and are in possession of a valid proxy already. This prevents a worker node that knows the passphrase alone from getting a proxy certificate.
MyProxy can also act as a CA. In this mode of operation proxy certificates are not stored on the server, but are created on demand. The connecting client authenticates to MyProxy using some authentication mechanism such as PAM (user has a local account and can prove identity with a passphrase), Kerberos, or SASL. If the user is able to authenticate, the local username is mapped to a DN using a grid-mapfile or an external program. This DN is then used to create a certificate using a CSR protocol with the client and the openssl CA functionality. In this mode of operation MyProxy must have access to the CA key, certificate, and other CA related data.
Figure 2. How a proxy certificate is created and stored in the MyProxy server from a user's certificate on the client host.
Figure 3. How a proxy certificate is created at the client from the certificate stored in the MyProxy server.
Besides physical resources such as CPU and network bandwidth, MyProxy also requires access to file system objects some for read-only access and one or two directories that it has write access (and which no others should be able to read as sensitive data is stored in them). The resources used are shown in Figure 4.
The main resource that the MyProxy server manages is the set of uploaded credentials. All credentials are stored in the same directory and consist of two files each. The first file stores the credential in PEM format (encrypted private key, signed certificate and the chain of certificates proving trust). The second file stores metadata about the certificate in a simple name=value line oriented format. This file stores the username and name of the certificate along with additional policy information about the certificates use.
MyProxy also needs at least read access to the host credential (certificate and key), the set of trusted root certificates, and the MyProxy configuration file that determines policy and behavior. If MyProxy uses helper applications as determined by the configuration file, the helper applications are also a resource and need to be readable. If MyProxy is configured to act as a CA, then it indirectly requires read access to the CA credential, and write access to the files required by the openssl CA. Not shown are log files that MyProxy may write.
Figure 4. Resources used and managed by the MyProxy server.
MyProxy runs as an unprivileged user. It needs read access to all the file system resources described above, and exclusive read and write access to the directories used to store certificates or by the openssl CA data. This is achieved using the file system permissions.
Internally the configuration file and the metadata for each stored certificate determine access control. The mechanism used includes one or more of the following:
Since the private key for a credential is normally stored encrypted if the certificate store directory is compromised a brute force attack could be launched on the credential stored there. This exposure is somewhat muted because proxy certificates should have a short life and the credential used to generate the proxy is not at risk.
On the other hand, if the MyProxy process is hijacked, more damage is possible as the attacker can collect passphrase while they are in memory temporarily for authentication and/or fulfilling the CSR protocol.
If the server is operating as a CA, then any compromise of the account will compromise the entire CA credential and all identities derived from it can not be trusted. This can be mitigated by using a tamper resistant hardware device for storing keys and signing certificates.
This section describes some of the techniques used to perform the component analysis where vulnerabilities are searched for using the prior analysis and looking at implementation details for bugs. The vulnerabilities discovered were described above.
Reviewed the design of MyProxy. The design is simple and no vulnerabilities were discovered in the design.
Checked how C-style strings and buffers were handled. No vulnerabilities related to C-style strings were found. The code used the following techniques to prevent problems:
Checked for the use of functions that commonly result in security problems. Two vulnerabilities were uncovered, while other uses of dangerous functions were correctly coded to prevent problems.
The first vulnerability was with problematic uses of system,
but it was only in client code, so its risk is low.
The second vulnerability involved a exec and fork
combination were used to create a popen-style function. This
replacement function prevents command injection that is normally associated
with popen, but introduces a potential deadlock with the child
process.
Checked how the code forms paths for use in the file system. The code combines the certificate storage directory with a file name. All files accessed are expected to be located in the certificate storage directory. The code transforms names that have unsafe characters into safe values, so this type of vulnerability is prevented.
open
Checked that flag O_EXCL is always used with O_CREAT
when creating a file so symbolic links are followed to create the file
in an arbitrary location. MyProxy did not have this problem.
Checked for common TOCTOU (Time Of Check Time Of Use) patterns. A few of these patterns exist in MyProxy but are coded correctly to prevent a vulnerability.
Checked if the path to the certificate storage directory was verified to be trusted. The directory itself was verified to be trusted, but its ancestor directories were not. This leads to a TOCTOU on the directory that is checked versus the directory that is actually used.
Looked at MyProxy's use of regular expressions. MyProxy uses POSIX Extended Regular expressions. Regular expressions can be exponential in time or space depending upon their implementation. The current POSIX regular expression library seems to be coded to prevent problems with common problematic patterns. I was only able to exhibit exponential time behavior when using back references (only available in POSIX Basic Regular Expressions).
MyProxy only uses -Wall when compiled with gcc.
There are many useful compiler warnings that are not enabled by
-Wall. Turning on additional compiler warnings and using the
latest version of gcc, 4.3.0, produced warnings about questionable
code and pointed to a couple of correctness problems, but did not reveal
anything exploitable.
Coverity Prevent found 61 defects in MyProxy. None of the defects found would likely lead to a security problem. They included resource leaks (that do not matter since the process is going to exit soon), complaints about unnecessary defensive checks (since Prevent deduces they always evaluate the same way), 1 byte buffer overflows that are not exploitable, falsely reported some TOCTOU and temporary file problems (they are correctly handled with other code eliminating the potential problem), and several others that pointed to correctness bugs that were not exploitable.
This section describes some thoughts on why so few vulnerabilities and no catastrophic ones were found.
The code base is only 26,000 lines of code, and other than the first year has had the same sole maintainer and gatekeeper for the smaller number of externally submitted changes.
There are only a small number of operations that a client can request. Only a single operation per connection is allowed and a simple text-based messaging protocol is used with a small fixed set of request-response style messages per operation.
Once a connection is made authentication is performed, the request is parsed and then authorization is performed. The authentication and authorization function called are same for each operation, so common functionality is exploited and updating this code is more likely to be done correctly.
MyProxy can be run as an ordinary user so when run as an ordinary user, it does not have security problems nor does it have the exposure on compromise associated with daemons that run as root or are setuid.
When a network connection for MyProxy arrives a copy of the MyProxy server is forked to handle the request and then exits after handling the single request. This eliminates problems of concurrency and simplifies I/O handling. It also eliminates problems caused by resource leaks along certain paths as they are localized to a single process that is short-lived.
For the most part, the code uses dynamically allocated strings, and a set of routines that automatically expand the buffer as needed. Where standard library functions that could overflow a buffer are used, checks are in place to verify that the buffer will not overflow before the call is made. In a few places where a string could a copied without its terminating null an unterminated string is prevented by the code defensively zeroing the buffer before use.
All communication to and from the MyProxy server is encrypted using the TLS protocol and the client always authenticates the server. Generally, The server also authenticates the client. This prevents exposure of sensitive data
In the normal mode of operation, all private keys stored on the server are encrypted by a user selected passphrase that the server retains only during the period required; the passphrase is never stored so it is only exposed if an attacker controls the server at the time a user makes a request using the passphrase.
Also when proxy certificates are used, the private key of the certificate is never transmitted over the TLS connection. A new proxy certificate is created and the CSR protocol is used so the side of the connection with the parent certificate never has access to the private key of the child and vice versa.