A Friday Afternoon Troubleshooting Ruby OpenSSL... it's a trap!

Last Friday I was trying out some new code that one of my colleagues wrote to help automate some of the work involved in releasing new versions of the Trust Keeper Scan engine. One of the many things the code did was send emails. I hate writing boilerplate emails, so I was excited to put it to use and save myself some precious time. Unfortunately, when I ran the code for the first time, it crashed with the following error when trying to connect to our Exchange Server:

Now, this error is pretty self-explanatory, and having spent time working with other Ruby libraries that utilize OpenSSL, this basically means that we're failing to verify the certificate of the server we're connecting to. The interesting part to me, was that when I visited this URL with Chrome and other web browsers, they successfully verified the certificate provided. Weird huh?

In this blog post, I'll explain some of the diggings around I had to do to get to the bottom of this issue and some other interesting bits I found along the way.

A Gem, Inside a Gem, Inside a Gem, Inside a Gem

First of all, Ruby Gems are pretty cool because you can use them as building blocks to build something bigger, badder, and meaner. One of the tricky aspects of having such a structure like this is tracking down who's responsible for an error when you run into problems.

In our case, we were using the Ruby Viewpoint gem. The Viewpoint gem provides a thin layer on top of Microsoft Exchange Web Service (EWS) and lets you do all kinds of fun things with Exchange, including sending emails. After getting the above error, I was able to track the failure down through the gem dependency chain down to it's source, which turned out to be just a couple gems deep.

• Layer 1: Viewpoint Gem - A light layer for talking Exchange Web Services
• Layer 2: Handsoap Gem - A library for creating SOAP clients in Ruby
• Layer 3: httpclient - A library for HTTP protocol communication
• Layer 4: OpenSSL - A library that interfaces with native OpenSSL

Certificate Verification Nuances in Ruby OpenSSL

So to get right down to it, we're basically trying to establish an SSL-wrapped socket with the target service. We can do this quite easily using Ruby OpenSSL.

Once we execute this code, we'd expect to see the same error as we saw above, right? Wrong, the code executes just fine and we've successfully connected to the SSL-wrapped socket. I'll show you in an IRB session to prove it…

The reason the connection doesn't pop the verify exception is because by default verify mode on SSL Context objects is set to nil. This effectively means they are set to "OpenSSL::SSL::VERIFY_NONE" or more plainly no verification. This also means that any SSL connection building using the default SSL context is susceptible to MiTM (Man in the Middle Attacks), but anyway, ONWARD!

So if we explicitly set the Ruby OpenSSL verification mode to be more strict, we can generate the same error that I saw when initially trying out that automated email code like so.

Now this is sort of good and sort of bad. If you remember earlier in this post, I noted that Ruby OpenSSL was failing to verify the certificate, but then my browser (Chrome in my case) was actually verifying the certificate. So I'm still stuck.

As it turns out Ruby OpenSSL does not populate its cert store by default. In fact, much like its default verify mode, the default cert store is set to nil.

This means that if we want to verify a certificate, we need to tell OpenSSL where to go for this information. This also means that we need to create a certificate store and populate it with sufficient information from the system's certificate store to verify the target server's provided certificate. This can be done like so.

Now that we've determined what could be the source of our problem, we can now start working our way back up the gem dependency stack to solve this issue for ourselves.

Working Our Way Back Up the Stack

Now that we've got a solid understanding of why SSL was complaining, let's start with HTTP client and its usage of OpenSSL. If we open up the source code for HTTP client's HTTP Client implementation we can also see the SSL Config implementation, which creates an OpenSSL::X509::Store object on instantiation but does not populate the store using the systems cert store using the #set_default_paths method. Unfortunately enough, the library also sets its default verification level to SSL::VERIFY_PEER, which is a good default, but when mixed with an empty cert store, it's going to be a common error case for casual SSL users like I've shown above.

However, there is a silver lining. If we look deeper into the source file, we can see that the author provided a method off of the SSL Config implementation #set_default_paths, which allows us to configure OpenSSL such that it will verify peers using the systems certificate store.

Our next stop is the hand soap gem. The hand soap gem implements an HTTP client from within its Handsoap::Http::Drivers::Http Client Driver class. The SSL configuration there is very minimal and is carried by the SSL configuration options set on the request objects that are passed into this class. The options available are to set a trust_ca_file, a client_cert_key_file, and/or the SSL verification mode. A configuration option does not exist from within the handsoap interface to tell the HTTP client that he should use the system cert store.

Our current state leaves most users with a similar incentive, as we saw with some of the HTTP client SSL defaults, which would be to disable the SSL verification. Fortunately for us, there is an alternative since we're in the wonderful land of Ruby and it's called Monkey Patching.

Monkey Patching Our Way Out of A Jam

So, we're in a use case where the existing library (handsoap in our case) doesn't provide the ability to configure HTTP client's SSL configuration options based on what we need. Well, in Ruby there is a concept called Monkey Patching that gives us a way out of this jam.

I'm sure there are much more formal descriptions and explanations of Monkey Patching, but my layman's description is that we can override the existing implementation's class dynamically to add the features that we're looking for.

In the case of our Viewpoint usage, I can copy the handsoap's HttpClientDriver class implementation into a separate file, make the changes I want, and make sure it gets loaded after the Viewpoint gem. This effectively reloads the HttpClientDriver class and causes the underlying library to use it rather than its implementation of the same class. Here are the modifications to the file we'll call http_client_driver_path.rb:

All we need to do is get the load order such that we load the vanilla implementation and then load our custom Monkey Patched version of the class like so.

The way Monkey Patching works, in this case, is sort of like whoever gets the last word in on the definition of an Object wins. So by loading our patch after the fact, we effectively can trick the handsoap gem being used by Viewpoint into using our implementation rather than theirs. This allows us to send emails with ease, continue to use SSL verification and not have to deal with those pesky exceptions that put a wrinkle in my Friday Afternoon.

This is also a really useful technique for exposing interfaces on Ruby implementations for security testing and such, but we'll save that discussion for another day.

Parting Thoughts

I realize this was sort of a random run-down half rant of some of my troubleshooting activities from last Friday, but I still think there were some good lessons learned in the process and I'll try to summarize them here:

• RubyOpenSSL SSLContext objects have a default verification setting of nil, which appears to be equivalent to VERIFY_NONE (no verification of SSL certificates).
• RubyOpenSSL SSLContext objects have a default cert store setting of nil, which appears to be equivalent to an empty store (certificate validation will fail ifverify mode is set to VERIFY_PEER).
• Rubyhttpclient is using VERIFY_PEER as its default verification mode, but has an empty cert store by default.
• Rubyhandsoap has minimal cert_store configuration control causing many casual SSL users to disable verification when they run into the above SSL certificate failure problem.
• MonkeyPatching, although sometimes controversial, can be an effective way of working around restrictive interfaces that incentivize casual SSL users to disable verification.

I also did a little bit of unscientific digging around on GitHub to see how many projects are disabling certificate verification for Ruby OpenSSL using VERIFY_NONE. As it turns out, it's pretty common and I got about 27,000 hits in my query. I've provided the query here in case you're interested in seeing which projects have certificate verification disabled.

GitHub Search for Ruby OpenSSL VERIFY_NONE projects

Additionally, I'd like to point out that if you are using an application that defaults to Ruby OpenSSL (or just OpenSSL in general) VERIFY_NONE, you run the risk of making the application susceptible to MiTM (Man in the Middle) attacks. If you're using Ruby-based projects, check out their source code and see how your code or the code you use deals with certificate verification and certificate stores. You might be surprised.

Thanks for reading. See you next time!