Two Out of Three Ain't Bad

For the past 6 years at Websolr (One More Cloud), I've advised dozens (maybe hundreds) of customers how to improve their Solr performance. 90% of the time, the customer is using a tool like NewRelic to benchmark their queries. When they see hundreds of ms per request, they reach out to ask why Solr is so slow.

The thing is, Solr is a battle-tested beast of a search engine, and is blazing fast. What NewRelic is actually measuring here, more than Solr, is the network transit time. It’s not uncommon for a round-trip, over-the-wire request to spend 95% of its time in transit, and 5% of its time in Solr.

In these cases, my advice is really just an affirmation of a triad of boring old best practices: "use HTTP keep-alive, compress over the wire, and load balance your reads and writes."

But, embarrassingly, I've never actually benchmarked this advice.

I did some research and noted that a majority of customers who had latency issues with Solr were running some kind of Ruby app (typically Rails) and were using the Sunspot gem. So that’s where I started.

Sunspot Under The Hood🔗

If you have a Ruby app using Solr, there is a good chance that you're relying on the RSolr library. Even if you're using the Sunspot gem, it's still using RSolr under the hood. RSolr is responsible for managing the connection to Solr, via the Faraday HTTP client library.

Faraday provides a common interface between an application and a variety of HTTP client libraries via adapters. I had to dig through a lot of code to discover that RSolr is using Faraday's default HTTP adapter, which is the very-dated Net::HTTP library.

I suspect that RSolr and Faraday are using Net::HTTP because it's part of the Ruby Standard Library and doesn't introduce external dependencies. If you have Ruby, you probably also have Net::HTTP, so it's likely the safest option. Just not the best, especially in whatever year you're reading this.

Personally, I like Typhoeus because it wraps libcurl (so you can pass curl params directly to it), and it supports parallel requests. It's also much faster and more efficient at handling SSL/TLS requests. It's possible to run compression and HTTP Keep-Alive with Typhoeus. It's just generally better. And there’s a Typhoeus adapter for Faraday!

Adding Typhoeus🔗

Simply adding Typhoeus to your Gemfile will not be enough. Neither Sunspot nor RSolr will pick it up automatically, and Sunspot doesn’t let you specify Faraday adapters in the sunspot.yml configuration file.

The workaround I used was to create a custom connection class that invoked RSolr::Client with a Faraday object that had the Typhoeus adapter. It’s straightforward conceptually, but involves more code than I wanted to add to a Rails codebase. I like clean solutions. So, I packaged it into a gem.

It's dead-simple. The important bits are:

module Websolr
  class Railtie < ::Rails::Railtie
    initializer 'setup_solr' do
      require 'rsolr'
      Sunspot::Session.connection_class = Websolr::Connection.new({
        'X-Websolr-Routing': 'prefer-replica'
      })
    end
  end
end

This creates an initializer that overrides the default Sunspot::Session connection class with a custom class called Websolr::Connection. That class looks like this:

module Websolr
  class Connection
    attr_accessor :connection, :default_headers

    def initialize(default_headers = {})
      self.default_headers = default_headers
      self.connection      = create_connection
    end

    def connect(opts = {})
      RSolr::Client.new(connection, opts)
    end

    def create_connection
      conn_opts = { request: {} }
      conn_opts[:request][:params_encoder] = Faraday::FlatParamsEncoder
      conn_opts[:headers] = default_headers
      Faraday.new(conn_opts) do |conn|
        conn.response :raise_error
        conn.headers = {
          user_agent: 'Websolr Client (Faraday with Typhoeus)',
          'Keep-Alive': 'timeout=10, max=1000'
        }.merge(conn_opts[:headers])
        conn.adapter  :typhoeus
      end
    end
  end
end

Now when Sunspot runs, it will use the custom class, which is using Typhoeus, HTTP Keep-Alive, and a header that tells the Websolr proxy to load balance read/write operations between the replica and primary cores, respectively. I didn’t even implement compression because RSolr doesn’t support it at all.

Benchmarking🔗

I spun up a dummy Rails app with Sunspot using all the defaults, a basic User model, and generated 50K User records. When that was done, I ran a few quick tests using the Benchmark library:

# Get all the User records into memory:
users=User.all

# Benchmark a complete reindex:
puts Benchmark.measure { User.reindex }

# Benchmark 1,000 searches:
puts Benchmark.measure { 1_000.times { User.search { fulltext users.sample.first_name } } }

# Benchmark 1,000 random requests, roughly 1/3 searches and 2/3 updates:
puts Benchmark.measure {
  1_000.times do
    if ((rand() * 12).to_i % 3 == 0)
      User.search { fulltext users.sample.first_name }
    else
      User.where(id: users.sample.id).update_all(
        first_name: rand(36**10).to_s(36)
      )
    end
  end
}

These tests gave me some baseline numbers for how Sunspot performed by default. Then, because I’d shoved my code into a gem, I just added the gem to the project’s Gemfile:

gem 'websolr'

And ran bundle install. I cleared the test Websolr index and re-ran the benchmark tests. Here’s what I found, averaging the results over multiple trials:

That’s a crazy performance improvement over the Net::HTTP adapter! Especially considering that I didn't even implement compression. Typhoeus' libcurl-based solution, along with HTTP Keep-Alive was enough to give almost a 350% speed increase in searches.

Wrapping Up🔗

None of this is revelatory. Keep-alive, load balancing, and compression (if you can get RSolr to cooperate) are the same three things every performance guide already tells you to do. Now there's a number attached: 350%, on two of the three, with the easy one still on the table.

Next time someone asks why Solr feels slow when Solr isn't the problem, at least I've got a receipt.