We use nginx to proxy to an Amazon S3 bucket to serve static content to customers. Yesterday, I noticed a high failure rate through the proxy but I couldn’t figure out why.

After some debugging, I finally discovered that the IP address nginx was hitting was different from the one that DNS was returning. It turns out that nginx resolves hostnames only once on load, meaning whatever IP address it got on load would stick around until reload.

So Amazon updated DNS for the S3 endpoint for our bucket but we were still using the old one (which was apparently failing).

Dynamic proxy_pass

Here’s how the directive might look:

server {
  listen 80;
  server_name files.example.com;

  location / {
    proxy_pass http://example.s3.amazonaws.com;
  }
}

The documentation for proxy_pass has this little gem:

In some cases, the part of a request URI to be replaced cannot be determined:

[…]

A server name, its port and the passed URI can also be specified using variables:

[…]

In this case, the server name is searched among the described server groups, and, if not found, is determined using a resolver.

Meaning if we change proxy_pass to use a variable instead, then nginx will be forced to resolve it using a resolver which will work how we want (i.e. DNS TTLs will work).

Here’s what the “fixed” block looks like:

server {
  listen 80;
  server_name files.example.com;

  // or use whatever resolver your machine is set to use
  resolver 8.8.8.8;

  set $proxy_pass_url http://example.s3.amazonaws.com;

  location / {
    proxy_pass $proxy_pass_url;
  }
}