It is probably better to dynamically size the default number of benchmark runs based on the time to run each benchmark; rather than hardcode the default to a value like five. That's because short run times are more likely to be significantly affected by transient factors outside the control of the benchmark. Long benchmark run times are more likely to have transient external factors amortized and thus result in less perturbation from the optimal run time. TBD is how formal this scaling of runs should be versus a "seat of the pants" approach that yields good enough results. One solution is to calculate the variance of run times and perform more runs if the variance is above some threshold while ensuring a minimum number of runs (e.g., totaling ten seconds) when each individual run is an extremely short duration.

This issue is interesting. It's semi-relevant since it applies to a C++ benchmark framework but the issues it explores are relevant to benchmarks of any other language. I've read many similar threads regarding optimizing the optimal number of benchmark runs. As far as I can tell there is no consensus on the optimal number of times to execute a benchmark. So I'm inclined to implement a &benchmark option that works for my situation and leave optimizations for future changes.