Hit frequencies |
December 2nd, 2016 |
| tech [html] |
- Stress all endpoints evenly
- Stress one endpoint in isolation
Why? How well caching works depends on your hit rate, which depends on what the distribution of your requests looks like. Situation (1) is a caching worst-case, (2) is a caching best case, and your real situation is somewhere in the middle. For example, from my access logs over the past couple years, here's the distribution of requests I've seen:
| 1275358 |
/news.rss
|
| 562369 |
/
|
| 437312 |
/favicon.ico
|
| 406583 |
/index
|
| 168296 |
/robots.txt
|
| 162728 |
/p/
|
| 116353 |
/simple_piano_recordings/piano_chords.svg
|
| 94048 |
/icdiff
|
| 92680 |
/p/mercury-spill
|
| 63728 |
/wsgi/json-comments-cached/gp/i6WrmjSHXLg
|
| [snip ~1k entries] | |
| 1179 |
/news/2011-11-02
|
| 1178 |
/wsgi/json-comments/gp/YTnbXQoRPRN
|
| 1178 |
/fiddle-clip-on/pictures/11-top.jpg
|
| [snip ~10k entries] | |
| 113 |
/news/back_from_564.rss
|
| 113 |
/news/2012-07-24
|
| 113 |
/news/2012-07-09.html
|
| [snip ~100k entries] | |
| 3 |
/nextbus/omnitrans/82/7387/
|
| 3 |
/nextbus/art/1135/759763/
|
| 3 |
/nextbus/mbta/39/6460/
|
| [snip ~1M entries] | |
| 1 |
/ngx_pagespeed_beacon?ets=load:906&rload=1605...
|
| 1 |
/nextbus/jtafla/17/1433/next/
|
| 1 |
/news/all/trillion-dollar-platinum-coin
|
This is kind of a mess, but the main idea is that we have a few "hot" endpoints, and then a long tail of less popular entries. I can use this distribution in load testing, to get something between uniform sampling (option 1) and singleton sampling (option 2) that better represents what real load on the site would look like.
In case this is useful to other people, here's the frequency list (hits-frequency.txt.gz) and a short script to sample from it (generate_urls.py).
Comment via: google plus, facebook