Packing Levels Into 16 Bits with unorm16 and snorm16

Auto-Editor's whole job starts with a sequence of levels: for every unit of the timebase, how loud is the audio, how much motion is on screen, and so on. A threshold turns that sequence into keep/cut decisions. We also cache those levels on disk so re-running the same edit is instant.

For a long video, "one value per frame" is a lot of values. A two-hour clip at a 30fps timebase is over 200,000 audio levels alone. Every one of them was a float32. Four bytes both in memory and in the cache file on disk.

The thing is, almost none of these numbers need 32 bits of range. Loudness and motion are normalized to [0.0, 1.0]. Opacity is [0.0, 1.0]. Brightness is [-1.0, 1.0]. These are bounded quantities, and a 32-bit float spends most of its bits representing values these numbers will never take.

Why not just use float16?

The obvious move is to reach for a half-precision float. float16 is two bytes, so it halves the storage just the same. But it's the wrong tool here, for four reasons.

First, most CPUs don't do float16 arithmetic natively; any math means widening to float32 first. Nim doesn't even have it in its type system.

Second, it spends its bits on range we don't have. A float divides its bits between an exponent (range) and a mantissa (precision). float16 can represent values out past 65000 and denormals down near 10⁻⁸ — an enormous dynamic range that a number clamped to [0, 1] will never touch. Every bit in that exponent is a bit not spent on precision.

Third, float precision is nonuniform, and that matters when you only have a handful of bits to begin with. Floats are dense near zero and sparse near one. Up around 1.0, consecutive float16 values are about 0.00049 apart, while down near 0.04 (where the default audio threshold sits) they're far finer. So a half-float would give you wildly different resolution depending on how loud the moment is.

Fourth, floating point is noncanonical. Zero is represented by both +0.0 and -0.0, and the all-ones exponent gives you ~2046 distinct NaN bit patterns plus two infinities. None of that is a large fraction of the 65536 codes, but it means two values that compare equal can have different bytes. That bites later, when the cache blits these straight to disk and trusts the bytes — a canonical, every-pattern-is-a-real-value type is simply safer to store.

Borrowing from graphics

GPUs solved this problem a long time ago. Texture formats have a pair of types called UNORM and SNORM, "unsigned normalized" and "signed normalized" respectively. A unorm16 is just a uint16 that you agree to interpret as a fraction: 0 means 0.0, 65535 means 1.0, and everything in between is linear. A snorm16 does the same with a int16 over [-1.0, 1.0].

So I added src/util/dnorm16.nim ("d" for the two of them together) with a pair of distinct types:

type Unorm16* = distinct uint16
type Snorm16* = distinct int16

func toUnorm16*(f: float32): Unorm16 =
  let c = max(0.0'f32, min(1.0'f32, f))
  Unorm16(uint16(c * 65535.0'f32 + 0.5'f32))

func toFloat32*(u: Unorm16): float32 =
  uint16(u).float32 * (1.0'f32 / 65535.0'f32)

Using distinct improves safety by making sure the compiler won't let you accidentally add a Unorm16 to a raw uint16, or pass a snorm where an unorm is expected. The operators that I want unchanged from the base type are pulled in Nim's borrow pragma:

func `==`*(a, b: Unorm16): bool {.borrow.}
func `<=`*(a, b: Unorm16): bool {.borrow.}
func `+`*(a, b: Unorm16): Unorm16 {.borrow.}
func `-`*(a, b: Unorm16): Unorm16 {.borrow.}

That gives me a type that's two bytes wide, compares as cheaply as an integer, and can't be confused with anything else.

First stop: the action bytecode

The first place this landed was opacity. Auto-Editor serializes actions into a compact byte stream, and opacity was being stored as a tag byte plus a 4-byte float:

of actOpacity:
  var u = toUnorm16(a.val)
  copyMem(addr base[i + 1], addr u, sizeof(Unorm16))
  i += 3

Five bytes became three. Opacity is a fraction between 0 and 1, perfect for unorm16. Brightness followed as a snorm16, since it ranges over [-1.0, 1.0].

The cache gets a codec byte

Next was the on-disk level cache. The trick here is that a cache file needs to be self-describing — if the format changes, or you read a unorm file expecting floats, you want it to fail cleanly and recompute rather than silently hand back garbage. So every cache file now starts with a one-byte codec tag:

type CacheCodec = enum
  ccFloat32 = 0'u8  # raw float32
  ccUnorm16 = 1'u8  # [0.0, 1.0]
  ccSnorm16 = 2'u8  # [-1.0, 1.0]

At first I picked the codec from a string ("audio" and "motion" → unorm). That worked, but it was a lookup that could drift out of sync with reality. The better version lets the type decide:

func codecOf(T: typedesc): CacheCodec =
  when T is Unorm16: ccUnorm16
  elif T is Snorm16: ccSnorm16
  elif T is float32: ccFloat32
  else: {.error: "cache: unsupported element type".}

Once the cache was generic over T, save and load stopped iterating value-by-value and just blasted the whole buffer to disk in one shot:

proc saveCache[T](filename: string, data: seq[T]) =
  ...
  fs.write(uint8(codecOf(T)))
  fs.write(data.len.int32)
  if data.len > 0:
    fs.writeData(unsafeAddr data[0], data.len * sizeof(T))

Loading checks the codec byte matches the type you asked for, then reads the bytes straight back into the seq. A mismatch or a truncated file raises, the Option comes back empty, and the levels are simply recomputed.

Keeping it 16-bit in memory too

The early commits converted to unorm16 only at the disk boundary — levels were still seq[float32] while the program held them. That meant doubling the memory the moment you loaded a cache. So the next step was to keep them narrow all the way through: thresholding now operates on Unorm16 directly.

const
  defaultAudioThres = toUnorm16(0.04)
  defaultMotionThres = toUnorm16(0.02)

proc orWithThreshold(result: var seq[bool], levels: seq[Unorm16], t: Unorm16) =
  ...
  result[i] = levels[i] >= t

Because the threshold is itself quantized into a Unorm16, the comparison is an integer compare against integer data — no per-sample float conversion just to decide keep-or-cut. The in-memory footprint of the levels is halved, and for the cache, the file on disk is halved too.

The annoying part: printing them back

1/65535 is not a round number. When you quantize 0.5 into a unorm16 and read it back, you get 0.5000076. If you ever need to turn one of these back into text — and Auto-Editor does, because action parameters get serialized into a human-readable form — naive formatting splatters that quantization noise everywhere.

The fix is to find the shortest decimal string that re-quantizes to the same bucket. Walk up the precision until a rounded decimal snaps back to exactly the value you started with:

func `$`*(u: Unorm16): string =
  let n = uint16(u).int
  const d = 65535
  for prec in 1 .. 5:
    let scale = 10 ^ prec
    let rounded = (n * scale + d div 2) div d  # round(n/d * 10^prec)
    let f = rounded.float32 / scale.float32
    if toUnorm16(f) == u:
      # ...format `rounded` with a decimal point and trim trailing zeros...
      return result

The first version of this leaned on formatFloat + parseFloat to do the round-tripping. It worked, but it was doing string formatting and parsing in a hot path. The faster rewrite does the whole thing in integer arithmetic and only builds a string once, at the end, for the precision that actually round-trips. So 0.5 prints as "0.5", not "0.5000076", and it gets there without touching the float formatter.

Letting actions hold dnorms directly

The last bit of cleanup was removing a round-trip I'd left in by accident. Actions were storing a float32 and converting unorm16 → float32 → string every time they were serialized. Once the storage type is the dnorm, the action can hold it directly and skip the hop — fewer conversions, and the value you print is exactly the value you stored. A small set of dnorm literals and tighter range guards (opacity must be in [0, 1], brightness in [-1, 1]) finished it off.

Was it worth it?

For the data that dominates Auto-Editor's work, yes. Memory and cache size for levels sequences dropped by half, and the threshold loop got cheaper in the process because it compares integers instead of floats.

The quantization is lossless for the purpose: 16 bits over [0, 1] gives a resolution of about 0.0000153, far finer than any threshold a human would set or any difference that changes a keep/cut decision. You give up precision you were never using, and you get back half your bytes and a faster inner loop.

Pick the data structure that fits the data. A loudness value between 0 and 1 never needed to be a 32-bit float. It just took borrowing a 30-year-old idea from graphics hardware to notice.

---