The Benchmark That Measured Itself

Two months ago I published a benchmark showing easyjson marshalling a 3.3 KB struct 4.1× faster than encoding/json.

The real figure is 1.54×.

The missing 2.65× wasn't a mistake in arithmetic or a bad machine. It was a code generator silently replacing the baseline with the thing the baseline was supposed to be compared against. My benchmark was measuring itself.

Worse, a second claim came out backwards. I reported sonic at 3.7× the standard library. On a clean baseline it is slower than encoding/json on Apple Silicon — and 2.27× faster on an Intel Xeon. Same code, same payload, opposite conclusion.

So I ran it on three CPUs. That turned up something better than a corrected number: the reason everyone gives for sonic being fast is wrong, mine included.

The bug

easyjson is a code generator. You point it at a struct and it writes a fast, reflection-free encoder. I ran it the obvious way:

easyjson -all internal/types/payload.go

That emitted, into the same package as my payload type:

func (v Payload) MarshalJSON() ([]byte, error) { ... }

A value receiver. Same package. From that line onward, Payload satisfies json.Marshaler — and encoding/json checks for exactly that interface before it does anything else. So json.Marshal(&payload) stopped reflecting and started calling easyjson's generated method, then ran a Compact pass with HTML escaping over the result.

My "standard library" row was easyjson plus overhead.

Nothing failed. Nothing warned. The numbers were plausible, internally consistent, and ordered the way I expected. The benchmark ran green for two months.

What made it invisible

This is the part worth generalising, because the bug itself is Go-specific and the failure mode is not.

The defect lived in the gap between two tools that were each behaving correctly. easyjson generated exactly what it promised. encoding/json honoured an interface exactly as documented. Neither had a bug. The bug was in my assumption that adding a file to a package doesn't change the behaviour of code that never references it.

And it was self-flattering. It inflated the baseline, which made every library look better, which matched what I expected to find. A defect that produces a surprising result gets investigated. A defect that confirms your hypothesis gets published.

Rebuilding it so this can't happen

I rewrote the harness from scratch. The governing constraint: make the defect structurally impossible rather than merely absent.

Every encoder now compiles against its own private copy of the payload type, generated from one canonical source. Not a type alias — an alias is the same type, so methods attach. Not a defined type — that has no field list for a generator to work from, and its nested types stay shared. A full source copy per variant is the only construct where a method generated for one implementation provably cannot reach another's type.

easyjson then runs with -no_std_marshalers, so even inside the one package entitled to generated code, the type never satisfies json.Marshaler.

Then four guards, and the important detail is that each one is proven to fire before it's trusted to assert an absence:

The recursion matters more than it sounds. I keep a deliberately poisoned fixture whose root type is clean and whose contamination sits four levels down, reached through a struct, a slice and a pointer. A guard that only inspected the root would call that type healthy. Mine reports it, and there's a test asserting that it does.

A guard nobody has watched fail is not a guard. It's a comment with a test function around it.

The corrected numbers

Apple M5 Pro, GOMAXPROCS=2, Go 1.26.2. Ten repetitions per cell, each a fresh process. Median with a 95% interval.

Before any of it: the noise floor. I register the same implementation twice under two names, so any measured difference between them is pure noise. On this laptop it peaked at 1.89%. A noop codec that encodes nothing gives the harness floor: 1.75 ns, about 0.05% of the smallest real measurement. Every figure below clears both.

1.54×. Arrived at independently, and it lands almost exactly where the old repository's own pre-codegen figure implied (1,574 ns ÷ 1,008 ns ≈ 1.56×). Two harnesses, built two months apart, agreeing that the published number was inflated by about 2.65×.

Two things I would not have predicted

segmentio/encoding wins marshal, and it wasn't in my original benchmark at all. Fastest at every payload size on every CPU I tried, with one allocation per operation against easyjson's twelve. I had simply never heard of it.

Decoding is where the standard library actually hurts. Marshal is a 1.6× story. Unmarshal is not — and this holds on all three CPUs:

If your service decodes more than it encodes — most do — this is the table that matters, and it gets less attention because benchmarks traditionally lead with marshal.

Three CPUs, one payload

Everything above is one laptop. A benchmark on one CPU tells you about that CPU, so I ran the identical commit on two AWS instances in ap-south-1 as well:

Matched on physical cores, not vCPU. Graviton has no SMT, so 2 vCPU is 2 cores; x86 vCPUs are hyperthreads, so c7i.large would have given me one core against Graviton's two. Getting two real x86 cores means a 4-vCPU instance with threads_per_core = 1 — 3.6× the hourly price for a comparison that means something.

Worth noting in passing: both $0.05–0.18/hour cloud instances are better instruments than my laptop, by a factor of five on noise floor. The M5 has asymmetric cores and a scheduler that will move a benchmark thread between them.

The encoded payload is byte-identical on all three at every size. That took two false starts to achieve, and the second one is a finding in itself — see the note below.

Marshal, ratio to encoding/json on the same machine

segmentio and easyjson are flat across architectures — 1.63–1.64× and 1.48–1.54×. Whatever they do, they do it the same everywhere.

sonic swings from 0.83× to 2.27×. On Apple Silicon it loses to the standard library. On Graviton it draws. On Xeon it wins by more than 2×. If you benchmark sonic on a MacBook and deploy to x86, or benchmark on x86 and deploy to Graviton, you will be wrong by a factor of two to three in either direction.

The AVX2 explanation is wrong

Everyone attributes sonic's speed to AVX2 SIMD. I attributed it to AVX2 SIMD, in the post this one corrects. It is the natural explanation: sonic is amd64-fast and arm64-slow, and AVX2 is the obvious amd64-only thing.

sonic honours SONIC_MODE=noavx, which forces its SSE kernel. So the claim is testable on a single machine, with no architecture change to confound it:

Turning AVX2 off makes sonic marginally faster. Every size, against a 0.34% noise floor. Decode moves 1–3% the other way. There is no AVX2 advantage here to speak of.

So sonic's 2.27× on this machine is not SIMD. It is the JIT — sonic compiles a specialised encoder at runtime on amd64, and that path doesn't exist on arm64. The instruction set is a red herring; the code generator is the mechanism.

Small payloads versus large

Four size classes, 571 B to 98 KB. Cost per encoded byte, so bigger-is-slower is visible rather than implied:

Small documents are cheaper per byte, not more expensive. That is the opposite of the usual intuition about per-call overhead, and it holds on all three CPUs. segmentio runs at 1,645 MB/s on a 571-byte document and 884 MB/s on a 98 KB one.

Everything plateaus after 25 KB. 1.125 against 1.132 ns/byte; 1.999 against 1.978. Past about 25 KB, payload size stops telling you anything new — which is why the ladder stops there and why a 25 KB class was worth adding to a benchmark that previously jumped from 3 KB to 98 KB.

Advantages compress as documents grow. segmentio goes 2.22× → 1.41× on the M5, 2.29× → 1.38× on Xeon. The gap between libraries is largest exactly where per-call overhead dominates, so a benchmark run only on small payloads overstates what you will see on big ones.

Decode is the exception: the stdlib penalty is essentially flat at 3.0–5.0× across every size and every CPU.

encoding/json is two implementations

While rebuilding this I found something that changes how you should read any Go JSON benchmark, including this one.

Go 1.26 ships encoding/json/v2 behind GOEXPERIMENT=jsonv2. What isn't obvious is what the flag does to v1:

$(go env GOROOT)/src/encoding/json/encode.go     //go:build !goexperiment.jsonv2
$(go env GOROOT)/src/encoding/json/v2_encode.go  //go:build goexperiment.jsonv2

encoding/json is a different implementation depending on that flag. Measure "the standard library" without recording GOEXPERIMENT and you don't know which one you measured. So I measured all three:

v2 costs 1.24× on marshalling and is 2.02× faster on unmarshalling. Even the compatibility shim — same v1 API, new engine underneath — already buys 1.67× on decode without a line of your code changing, and pays 1.20× for it on encode.

I can't find this comparison published anywhere. If you decode more than you encode, that trade is worth knowing about before v2 arrives as the default.

The ranking doesn't survive concurrency

Single-operation cost is not what a service experiences. So I built a second mode: N workers, each doing decode-a-request-then-encode-a-response as one transaction, which is what an HTTP handler actually does.

Transactions per second, 3.3 KB payload, two cores, on the M5 Pro:

segmentio wins the single-operation benchmark. goccy wins under load. And sonic — slower than stdlib on one core — has the best tail latency of anything here at 16.2 µs p99.

The mechanism is scaling from one core to two:

encoding/json barely uses the second core. It then keeps climbing 1.56× from concurrency 2 to 500 — not because it got better, but because it had been leaving the machine idle the whole time.

The finding I'd have missed entirely

At 500 workers on two cores, a 2-second measurement window reached only 175 of 500 goroutines. Eight seconds reached 492.

A latency number from that short run would be labelled "concurrency 500" while describing roughly 175-way concurrency. The harness now flags any cell where a worker completed zero operations, and refuses to present it as the configured level.

A short window at high concurrency does not measure the concurrency you configured.

One more thing the fixture caught

jsoniter.ConfigFastest is 0.77× the baseline here — respectable. It also sets MarshalFloatWith6Digits, whose comment in jsoniter's own source reads "will lose precession".

It truncates your floats.

I found this because the fixture carries floats with nine significant digits specifically so a truncating encoder gets caught rather than believed. The correctness gate localises it:

$.sensor_report.channels[0].calibration.gain: 0.902394 != 0.9023941639999999

That's not a faster configuration. It's a different product, and its output is 213 bytes smaller because it's carrying less of your data.

The bug that only appeared on the second architecture

Getting one payload onto three CPUs took two attempts, and the failure is worth the paragraph.

The first x86 run was rejected by its own correctness gate:

$.sensor_report.channels[0].calibration.offset:
    1.7927299100000003 != 1.7927299100000005

One unit in the last place. The fixture was generated on arm64; x86 regenerated it from the same seed and got a different float.

My corpus builder computed lo + frac*(hi-lo). The Go spec permits an implementation to fuse floating-point operations into one, "possibly across statements" — and arm64 has a fused multiply-add that the compiler emits here, while amd64 does not fuse by default. One expression, two results, both correct per the spec.

This would not have surfaced as a wrong number. It would have surfaced as two architectures measuring different documents while both reported the same size class — and the cross-platform comparison above would have been quietly meaningless. math.FMA fixes it by making the fusion explicit: IEEE 754 specifies it as a single rounding and Go guarantees those semantics everywhere, with a software fallback where the hardware lacks the instruction.

Two things I take from that. Determinism you have only tested on one architecture is not determinism. And the gate caught it on its first ever cross-architecture run, before any number was published, which is the entire argument for asserting properties instead of assuming them.

What I'd tell you to do differently

The uncomfortable part of this isn't that I shipped a wrong number. It's that the wrong number was more believable than the right one — bigger, rounder, and pointing the way I already expected. That's the shape of most measurement error I've made: not noise, but a plausible story with a mechanism I didn't check.

make verify && make bench REPS=10 BENCHTIME=500ms && make report