Zstd Window Size

April 25th, 2025
tech
At work we've recently been using zstd as a better-compressing alternative to gzip, and overall I've been pretty happy with it. A minor documentation gripe, though, is that the behavior around multithreaded compression is a bit unclear. I understand it's chunking the work and sending chunks to different threads to parallelize the compression process, and this means that I should expect to see better use of threads on larger files because there are more chunks to spread around, but what is the relationship?

When I look in man zstd I see that you can set -B<num> to specify the size of the chunks, and it's documented as "generally 4 * windowSize". Except the documentation doesn't say how windowSize is set.

From a bit of poking at the source, it looks to me like the way this works is that windowSize is 2**windowLog, and windowLog depends on your compression level. If I know I'm doing zstd -15, though, how does compressionLevel=15 translate into a value for windowLog? There's a table in lib/compress/clevels.h which covers inputs >256KB:

Level windowLog chainLog hashLog searchLog minMatch targetLength strategy
<1 19 12 13 1 6 1 fast
1 19 13 14 1 7 0 fast
2 20 15 16 1 6 0 fast
3 21 16 17 1 5 0 dfast
4 21 18 18 1 5 0 dfast
5 21 18 19 3 5 2 greedy
6 21 18 19 3 5 4 lazy
7 21 19 20 4 5 8 lazy
8 21 19 20 4 5 16 lazy2
9 22 20 21 4 5 16 lazy2
10 22 21 22 5 5 16 lazy2
11 22 21 22 6 5 16 lazy2
12 22 22 23 6 5 32 lazy2
13 22 22 22 4 5 32 btlazy2
14 22 22 23 5 5 32 btlazy2
15 22 23 23 6 5 32 btlazy2
16 22 22 22 5 5 48 btopt
17 23 23 22 5 4 64 btopt
18 23 23 22 6 3 64 btultra
19 23 24 22 7 3 256 btultra2
20 25 25 23 7 3 256 btultra2
21 26 26 24 7 3 512 btultra2
22 27 27 25 9 3 999 btultra2

See the source if you're interested in other sizes.

So it looks like windowSize is:

  • ≤1: 524k
  • 2: 1M
  • 3-8 (default): 2M
  • 9-16: 4M
  • 17-19: 8M
  • 20: 32M
  • 21: 64M
  • 22: 128M

Probably best not to rely on any of this, but it's good to know what zstd -<level> is doing by default!

Comment via: facebook, lesswrong, mastodon, bluesky, substack

Recent posts on blogs I like:

Elixir's Last Dance

On May 18th, the contra dance band Elixir had their last gig ever. The dance was packed: there were three hundred people. It was the only dance BIDA has ever done where they sold tickets. People flew from across the country just to hear Elixir play one la…

via Lily Wise's Blog Posts June 5, 2025

Body Language For Trans People

When I first came out as trans, resources for trans people were full of advice about body language.

via Thing of Things June 2, 2025

Workshop House case study

Lauren Hoffman interviewed me about Workshop House and wrote this post about a community I’m working on building in DC.

via Home April 30, 2025

more     (via openring)