Assumed reader level: Intermediate

Content level: Advanced beginner

JSON is JSON, it’s everywhere, and if you’re working with Go you’re most probably doing tons of JSON marshalling and unmarshalling. Having experience in languages that have nearly identical built-in syntax for JSON (Javascript and Python), I repeatedly ran into certain issues, having to do with Go’s idiosyncracies and my deep-seated habits. Keep in mind that these points apply to other encodings in the Go standard library, and generally to all packages that implement the same interfaces and patterns.

Only public fields are (un)marshalled

This is the gotcha that annoyed me the most until I got it carved into my mind after spending countless minutes debugging it. Traditionally, JSON object keys start with lowercase letters, whereas Go uses capitalization to determine public vs private. When code accesses the fields of a struct within the same module, you will not get into trouble with private fields, as they are treated as accessible within the same module. JSON is a different module, however, and it will not be able to write to these private fields. For example, the following will work:

var data struct {
    Key string
}
jsonData := []byte(`{"Key": "Value"}`)
json.Unmarshal(jsonData, &data)
fmt.Printf("%v\n", data)

This should print {Value}. But lest you forget that Key has to be capitalized, so that encoding/json can write to it; then it will not help you even to set the field tag, as follows:

var data struct {
    key string `json:"the_key"`
}
jsonData := []byte(`{"the_key": "Value"}`)
err := json.Unmarshal(jsonData, &data)
fmt.Printf("%v\n", data)
fmt.Printf("%s\n", err)

This will, of course, print {}, and a nil error. What does catch this error, however, is go vet, which prints the following friendly error message:

./jsong.go:10:3: struct field key has json tag but is not exported

You will not get this message, though, if you don’t have JSON tags. Long story short: Use tags even when keys match, and use go vet.

Unmarshaling is not for error checking

As encoding/json unmarshals a JSON-encoded byte array to a struct, you would expect some kind of error checking to happen. Let’s take the following example:

type Data struct {
       IntField  int  `json:"intfield"`
       BoolField bool `json:"boolfield"`
}
jsonData := []byte(`{"intfield": "yolo", "boolfield": "ctulhu ftaghn (whatever the hell that means)"}`)
var data Data
err := json.Unmarshal(jsonData, &data)
fmt.Printf("%v\n", data)
fmt.Printf("%s\n", err)

As you can see, we are packing all kinds of junk in the JSON object keys that correspond to the Data struct fields. Go deserializes this as far as it can, and when it can’t do so anymore, leaves the rest of the fields as they were beforehand. The error message reports the last field that could not be deserialized, resulting with the following output in the above case:

{0 false}
json: cannot unmarshal string into Go struct field Data.intfield of type int

So keep in mind: Deserialization is not validation. For purposes of validation, you should use a library such as https://github.com/go-playground/validator/, or even better, something that validates the input JSON directly (which I haven’t found a library for yet).

Struct tags are not error-checked in any manner

When logic is put into strings in a programming language, trouble is inevitable. Language capabilities go out the window, and you are left alone with your tired eyes and mind to catch errors. Go’s struct tags are no exception. Since their contents are not code, any errors you make go straight through the Go compiler without any warnings. Let’s have a look at this example:

type Data struct {
    IntField int `json:"int_field or something`
}
jsonData := []byte(`{"int_field": 43}`)
var data Data
err := json.Unmarshal(jsonData, &data)
fmt.Printf("%v\n", data)
fmt.Printf("%s\n", err)

This will print {0} and no error. One might think that struct tags are always simple, such as those for JSON deserialization, and an average programmer should be able to deal with them in a normal state. Unfortunately, tags are used by all kinds of libraries, which implement their own syntax embedded in the tag string. One example is the tag structure used by the validation library I linked to above, demonstrated in the following type definition:

type IntegrationInput struct {
    IntegrationTypeID int32 `json:"integration_type_id" validate:"gte:1"`
}

Can you see the error here? The validate tag has to be "gte=1" and not "gte:1". Things like this are difficult to get right and debug, especially when multiple tags are interacting, as in this example. As with unexported struct fields, go vet can help you with tags, generating the following error for the first example:

./tags.go:10:3: struct field tag `json:"int_field or something` not compatible with
  reflect.StructTag.Get: bad syntax for struct tag value

But go vet cannot help you with the validate tag, because those tags have their own logic. So use go vet to avoid type field tags, but also pay extra attention to the format of the more complex tags.

Bonus: struct tag matching is case-insensitive

Thanks to procach on /r/golang for this tip.

You would think that, if you use field tags to match JSON fields, you would be able to precisely match the case of fields in JSON data. This is not really the case, however. Even if you use tags, the match is case-insensitive, as the following example shows:

var data struct {
    Key string `json:"TheKey"`
}
jsonData := []byte(`{"thekey": "Value"}`)
err := json.Unmarshal(jsonData, &data)
fmt.Printf("%v\n", data)
fmt.Printf("%s\n", err)

This will output {Value}. Even though TheKey and thekey are differing strings, encoding/json will match the fields to each other. Another thing to keep in the back of your head, in case unmarshalling behaves in unexpected ways.

Conclusion

I consider it a useful restriction-cum-feature that Go requires you to convert JSON data to native structures to manipulate them conveniently. Languages like Python, which have built-in syntax for similar structure, can lead to JSON-driven development, which I had discussed in another blog post. If you want to have a good time converting between JSON and Go, make sure you don’t skip error checks, regularly use go vet, and pay attention to capitalization, and you should be all fine and dandy.