Character Encoding and Go on AWS

Ah character encoding, my old friend, I have not missed you. I was visited recently by a recurring demon from my past, character encoding. Sometimes it seems like every new environment is just waiting to spring some encoding problem on me. Since pretty much all modern environments use utf-8, these are at least getting rarer. This post describes how to ensure that responses from API Gateway endpoints from Go are properly interpreted by the receiver.

Lambda Proxy Response

The response from a Lambda proxy integration to API Gateway in Go is usually an APIGatewayResponse object. In my case I was creating some JSON for the Body of the response and sending it back. My response looked like this:

  var buf bytes.Buffer

  // ... do stuff to create the JSON content in buf

  resp := Response{
    StatusCode:      200,
    IsBase64Encoded: false,
    Body:            buf.String(),
    Headers: map[string]string{
    "Content-Type": "application/json",
    },
  }

Spotting the Mistake

The above version is subtly wrong because it leaves the interpretation of the characters in the body of the response up to the client. Depending on how that client determines the character encoding it might get that wrong. Providing a solid hint on how it should treat the bytes in the response is really helpful. In my case a character as simple as a single quote caused part of the response to be improperly decoded. The JSON worked but the content of one of the fields had a unicode conversion instead of a simple single quote.

Here’s the corrected version.

  var buf bytes.Buffer

  // ... do stuff to create the JSON content in buf

  resp := Response{
    StatusCode:      200,
    IsBase64Encoded: false,
    Body:            buf.String(),
    Headers: map[string]string{
    "Content-Type": "application/json; charset=utf-8",
    },
  }

The change is to add the charset=utf-8 in the Content-Type header. That makes the encoding unambiguous which makes the client happy.

Leave a Reply

Your email address will not be published. Required fields are marked *