Powering up HTTP clients with Train

Middlewares make it easy to write independent and reusable modules for HTTP servers. For instance, we've abstracted common reporting with our statsd and logger middlewares. This makes it easy for new services to be built with reliable reporting from day one.

h1 := statsd.New(stats)(app)
h2 := logger.New()(h1)
http.ListenAndServe(":12345", h2)

Although easy enough to write it ourselves, Alice is a tiny library that makes it easy to chain multiple middlewares.

chain := alice.New(logger.New(), statsd.New(stats))
http.ListenAndServe(":12345", chain.Then(app))

While working on sources, we realized we needed a similar solution for writing http clients. At their core, sources are pretty simple. Sources make http requests to a third party service, translate the responses into our warehouses format, and forward it to our warehouse objects API. A majority of the errors are from failing HTTP requests. To have complete visibility into the performance of sources, we needed to be able to record metrics and log activity of the HTTP client across multiple codebases.

Borrowing from OkHttp's interceptors, I wrote Train. At it's core, train takes a series of Interceptors and combines them to return a RoundTripper that runs the interceptors in order.

Powerful

Interceptors enable endless use cases - they can observe, modify and even retry calls.

Observing Requests and Responses

Interceptors can continue the chain as is and observe outgoing requests and incoming responses. This is useful for reporting purposes, such as logging and stats.

func Dump(chain train.Chain) (*http.Response, error) {
  req := chain.Request()
  fmt.Println(httputil.DumpRequestOut(req, true))

  resp, err := chain.Proceed(req)
  if err != nil {
    return nil, err
  }
  fmt.Println(httputil.DumpResponse(resp, true))

  return resp, err
})

Modifying Requests

Interceptors can modify outgoing requests. For example, you can compress the request body if your server supports it.

func Compress(chain train.Chain) (*http.Response, error) {
  req := chain.Request()

  contentEncoding := resp.Header.Get("Content-Encoding")
  if resp.Body != nil && contentEncoding != "" {
    z, err := zlib.NewReader(req.Body)
    if err != nil {
      return nil, err
    }
    req.Body = z
    req.Header["Content-Encoding"] = "zlib"
  }

  return chain.Proceed(req)
})

Modifying Responses

Interceptors can modify incoming responses. Similar to above, you can decompress the response body before your application processes the response.

func Decompress(chain train.Chain) (*http.Response, error) {
  req := chain.Request()
  resp, err := chain.Proceed(req)
  if err != nil {
    return nil, resp
  }

  contentEncoding := resp.Header.Get("Content-Encoding")
  if resp.Body != nil && contentEncoding == "zlib"  {
    z, err := zlib.NewReader(resp.Body)
    if err != nil {
      return nil, err
    }
    resp.Body = z
  }

  return resp, err
})

Short Circuiting

Interceptors can short circuit the chain — this makes it great for testing.

func Short(train.Chain) (*http.Response, error) {
  return nil, errors.New("somebody set up us the bomb")
})

Pluggable

Like HTTP server middlewares, interceptors make it easy to share common HTTP client logic. Interceptors can be plugged into HTTP client as a transport.

transport := train.Transport(logger.New(), statsd.New(stats))
client := &http.Client{
  Transport: transport,
}

This also makes it easy to plug into client libraries built by other developers. For instance, we plugged in our stats interceptor to the Intercom Go library and had logs and metrics for free without needing to modify the source.

t := train.Transport(logger.New(), statsd.New(stats))

return &interfaces.IntercomHTTPClient{
  Client: &http.Client{
		Transport: t,
  },
}

Chainable

Interceptors build upon the Chain interface.

Interceptors are consulted in the order they are provided. You'll need to decide what order you want your interceptors to be called in.

For example, this chain will record stats about the compressed request and response. The stats interceptor is invoked after the compression interceptor compresses the request and before the compression interceptor decompresses the response.

transport := train.Transport(compress, log, stats)

The second example will record stats about the decompressed request and response. The stats interceptor is invoked before the compression interceptor compresses the request and after the compression interceptor decompresses the response.

transport := train.Transport(log, stats, compress)

Extensible

Train is designed to be extensible. We've been using it for a while to power up the standard library http client in our Go sources — including adding logging, fixing server errors, and collecting stats for our invaluable Datadog dashboards.

Try it out and let me know what you think!