Casey Barajas/05 Log
/ Log/chunk-loader-postmortem
/Tech/~1240 Words

Chunk Loader: A Postmortem on a Bug That Didn't Exist

The Symptom

A small Qubica world loaded fine. Anything past about a 4km perimeter ate memory until the OS killed it. The leak grew with world size, not with playtime, which is a strange shape for a leak.

We profiled. The arenas looked clean. The chunk LRU evicted on schedule. Memory still climbed.

The Wrong Story

For two weeks, I was sure it was the LRU. I rewrote the eviction policy twice. I added telemetry. I built a small repro that loaded a 12km world and watched the heap. The heap kept climbing and the LRU kept reporting that it was, by every metric I cared about, perfectly empty.

The lesson here is one I keep relearning: if every metric you care about says the bug is not there, you are looking at the wrong metric.

The Actual Bug

Chunk coordinates are (i64, i64). Internally, my chunk hash function took two i64s, mixed them, and returned a u64. Fine.

The cache key in the LRU was a ChunkId newtype, which I had defined two years ago, in a hurry, as (i32, i32).

// What I Thought I Had
struct ChunkId(i64, i64);

// What I Actually Had
struct ChunkId(i32, i32);

For small worlds, anything inside i32::MAX / chunk_size works. The conversion from i64 to i32 is silent and lossless. The hash function is happy. The LRU is happy. I was happy.

For large worlds, two distinct chunks at, say, (2^32 + 4, 17) and (4, 17) would produce the same ChunkId. The LRU thought it had evicted the chunk. It had not; it had just lost track of one copy. That copy was retained by a long-lived reference somewhere upstream of the cache. The leak grew with world size because the number of collisions grew with world size.

What I Changed

One line:

struct ChunkId(i64, i64);

And one test:

#[test]
fn chunk_ids_dont_collide_at_world_extents() {
    let a = ChunkId::from_world(i64::MAX - 4, 17);
    let b = ChunkId::from_world(4, 17);
    assert_ne!(a, b);
}

The fix took 30 seconds. Finding it took two weeks.

What I Am Taking From This

  • A type that "happens to work" is a bug with a longer fuse. The original (i32, i32) was fine in 2024. It became wrong the moment the world stopped being small. No compiler warning fires for that.
  • Lossy conversions should be loud. Rust's as is too quiet for cross-precision integer casts in code that handles user-bounded values. I am switching the chunk path to try_into() and a Result.
  • If your metrics all say "no bug here," your bug is not in the thing your metrics measure. I knew this. I forgot it. I will forget it again. Putting it here so future me has somewhere to find it.

A two-week bug is annoying. A two-week bug whose fix is one line is annoying and humbling. That is probably the right ratio.

skip to content