Reverse Proxy with NGINX and EC2

For a little bit of background, I occasionally talk to customers who are looking for greater access control to their search engines. A common one is IP whitelisting: “Can I control which hosts are allowed to communicate with my cluster?” There are some technical limits to what we can offer, and while we’ve had internal discussions about it, the cost-benefit just hasn’t really made sense.

That said, when these questions come up, I will typically demur and offer that the user look into setting up a reverse proxy with NGINX. I realized recently that while this is solid advice, I didn’t actually know how to do it myself. I only had heard that it was possible and a realistic solution.

This is the kind of thing that is perfect for dogfooding: test out the solution you’re recommending to customers. It will help you to validate that the advice is good, and you can answer follow up questions. And, if it turns out to be really complicated and a poor strategy, then you can find something that works and make that the new answer.

What is a Reverse Proxy?

A reverse proxy is basically an intermediary between your application and a back end service. Imagine that the back end service is a fancy club: the reverse proxy is your bouncer at the door, keeping the sketchy people out. More technically, sending your traffic through a reverse proxy allows you to mask the location and authentication details from a malicious user, and denying all hosts except for your application ensures that the malicious user can’t access your cluster. Or maybe some other business logic for interlopers.

A common case is when a front end application or UI makes calls out to a back end service like a search engine, API or database. It’s not hard in that situation for someone to read the source code or monitor network calls to map out which services the application is using. Even if the back end service is protected with authentication requirements, which would only prevent something like an automated attack from a port scanner.

A reverse proxy would anonymize the back end services. The UI could call out to it, and the reverse proxy would handle authenticating and “cleaning” the request before passing it along to the appropriate service. The reverse proxy then returns the service’s response to the UI. That way, a malicious user has a substantially harder time mapping out infrastructure and accessing secure systems.

What is NGINX?

NGINX (“engine-X”) is a web server that has been around since the early 2000’s. It was originally written as a C10K frontend proxy for Apache, which to this day has some major performance limitations. NGINX also describes itself as a web server, reverse proxy and IMAP/POP3 proxy server. This kind of “ALL THE THINGS” description is always a huge red flag for me: the more things the software can do, the fewer things it’s actually good at.

I’ve worked with Apache on a number of projects, and while it’s definitely solid software, it also has a steep learning curve and can be a pain to configure. The XML configurations don’t help either. So with that as a baseline, I assumed NGINX would be painful.

Turns out it’s easy as pie. NGINX doesn’t use huge XML configuration files, which makes it much easier to read. Configurations are contained in modules, which have directives, contexts and comments. The documentation on this is very simple and straightforward.

Want to spin up a server on a special port? BAM:

server {
    listen 8080;
    root /data/dir_1;

    location / {
    }
}

That will listen on port 8080, and route all requests to the /data/dir_1 directory on the server.

What about a reverse proxy? Bruh:

server {
  listen 80
  location / {
    proxy_pass http://url-of-service.com;
  }
}

That listens on port 80 and passes all requests on that port to http://url-of-service.com. It’s the easiest freaking configuration I’ve ever seen for something this powerful. And it has a ton of options.

Running on EC2

The NGINX documentation has a guide on spinning up an EC2 instance with NGINX. The process is, basically:

1. Log into AWS (or set up an account if you don’t already have one)
2. Navigate to the EC2 dashboard and launch an instance
3. Use the Ubuntu Server 16.04 LTS (HVM), SSD Volume Type AMI (although I think basically any recent Linux AMI will be fine)
4. Use a t2.micro instance type. This is free for the first year, then a couple dollars a month after. It is minimal resource, but that’s exactly the kind of environment NGINX excels in. You can get a larger instance, of course, but you’ probably end up paying for a lot of unused resources.
5. Jump to “Configure security group” and allow the instance to accept HTTP and/or HTTPS traffic. You can do both or just one (or something else if the back end service isn’t using a REST API)
6. Launch that sucker
7. Once the new instance is online, you should be able to SSH in over the public IPv4 address
8. Run sudo apt-get install nginx
9. Profit!

This is a quick summary, feel free to peruse the docs linked above for more details and screen shots. I tested this a few times, and the whole process takes 5-10 minutes.

Sample Configuration

In our case, the reverse proxy would be connecting to a back end service protected with HTTP Basic authentication. The credentials are required to access the resource, and this is why the reverse proxy is needed for UI components that rely on it: the username and password are user-readable!

The secure option here is to create an encrypted connection to NGINX, which will create an encrypted connection to the back end. There is some overhead here, but we can configure the proxy to keep sessions open and reduce latency. NGINX will have a list of IPs that are allowed, and all others disallowed.

This strategy means the front end can make a simple call out to some random EC2 instance; NGINX will receive the request, approve the source, then inject the authentication headers into the request before it forwards it to the back end service. The back end service authenticates the request, processes it and responds. NGINX returns the response to the front end, and all of this happens voer an encrypted channel.

This is what I ended up using:

server {
  listen 443 ssl default_server;
  listen [::]:443 ssl default_server;

  # SSL Configuration
  server_name ec2-XX-XX-XX-XX.compute-1.amazonaws.com; # "Public DNS" of EC2 instance
  ssl_certificate /etc/nginx/ssl/nginx.crt;            # SSL CRT
  ssl_certificate_key /etc/nginx/ssl/nginx.key;        # SSL key

  # IP Whitelisting
  allow AA.BB.CC.DD; # The IP we want to whitelist
  deny all;

  location / {
    # The URL of the back end service
    proxy_pass https://some-crazy-service.us-east-1.example.com;

    # Keep alive settings for lower latency responses
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";

    # This is how your Bonsai cluster authentication is forwarded
    proxy_set_header Authorization "Basic dXNlcm5hbWU6cGFzc3dvcmQ=";
  }

}

A few notes on this:

1. EC2 Configuration

I configured my EC2 instance to allow SSH and HTTPS connections only. My thought process here was that I didn’t want NGINX listening to two ports, and I didn’t want to support unencrypted connections. I could have enabled HTTP connections and just not set up anything to listen to that port, but I’m a believer in not exposing any ports that aren’t being actively used.

2. Creating the encrypted connection between the UI and NGINX requires an SSL/TLS certificate

This certificate also needs to be available on both the application side and NGINX side. For the demonstrator, I simply created a self-signed certificate like so:

$ sudo mkdir /etc/nginx/ssl
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

In practice though, this would fail on a client side UI. For that, you would need a certificate signed by a recognized issuing authority. It turns out that free certs offered by companies like Let’s Encrypt are subject to some limitations on what domains are allowed. If you try and create one with ec2-XX-XX-XX-XX.compute-1.amazonaws.com as the domain, it will fail because the format is blacklisted.

The short version is that you would probably need to pay a company like Comodo or Verisign for a signed SSL certificate that wouldn’t raise security errors on the client side. Or you would need to put the EC2 instance behind a registered domain and use LetsEncrypt. Either way, there’s going to be a cost involved.

3. HTTP Basic Authentication

HTTP Basic Authentication is a string in the format “username:password”, which is converted into Base64 and included in Authorization header. Let’s say the back end service literally wants a user name of username and a password of password. Then you could generate the base64 header like so:

$ echo -n "username:password" | base64
dXNlcm5hbWU6cGFzc3dvcmQ=

This means that the HTTP header for the authentication credentials is Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

Summary

I’d set aside a day to work on this, and it took about an hour to read the docs, spin up the instance, install and configure NGINX. I tested it out and played around with it for another hour, trying to break it. It worked perfectly.

I’m honestly pretty stoked about it. I’m so used to tools and services being difficult to learn and manage, or working (or breaking!) in totally unexpected ways. It’s a treat to be able to find something this powerful be so intuitive and fast.