Document benchmarking CLI (#11246)

* Decrese default repeats Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add benchmarking READMEs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update README Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: parity-processbot <> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
2026-04-26 22:47:56 +00:00 · 2022-05-25 05:47:21 +02:00
parent 35af8fd726
commit 29474f9893
9 changed files with 505 additions and 8 deletions
@@ -0,0 +1,118 @@
+# The `benchmark block` command
+
+The whole benchmarking process in Substrate aims to predict the resource usage of an unexecuted block.  
+This command measures how accurate this prediction was by executing a block and comparing the predicted weight to its actual resource usage.  
+It can be used to measure the accuracy of the pallet benchmarking.
+
+In the following it will be explained once for Polkadot and once for Substrate.  
+
+## Polkadot # 1
+<sup>(Also works for Kusama, Westend and Rococo)</sup>
+
+
+Suppose you either have a synced Polkadot node or downloaded a snapshot from [Polkachu].  
+This example uses a pruned ParityDB snapshot from the 2022-4-19 with the last block being 9939462.  
+For pruned snapshots you need to know the number of the last block (to be improved [here]).    
+Pruned snapshots normally store the last 256 blocks, archive nodes can use any block range.  
+
+In this example we will benchmark just the last 10 blocks:  
+```sh
+cargo run --profile=production -- benchmark block --from 9939453 --to 9939462 --db paritydb
+```
+
+Output:
+```pre
+Block 9939453 with     2 tx used   4.57% of its weight (    26,458,801 of    579,047,053 ns)    
+Block 9939454 with     3 tx used   4.80% of its weight (    28,335,826 of    590,414,831 ns)    
+Block 9939455 with     2 tx used   4.76% of its weight (    27,889,567 of    586,484,595 ns)    
+Block 9939456 with     2 tx used   4.65% of its weight (    27,101,306 of    582,789,723 ns)    
+Block 9939457 with     2 tx used   4.62% of its weight (    26,908,882 of    582,789,723 ns)    
+Block 9939458 with     2 tx used   4.78% of its weight (    28,211,440 of    590,179,467 ns)    
+Block 9939459 with     4 tx used   4.78% of its weight (    27,866,077 of    583,260,451 ns)    
+Block 9939460 with     3 tx used   4.72% of its weight (    27,845,836 of    590,462,629 ns)    
+Block 9939461 with     2 tx used   4.58% of its weight (    26,685,119 of    582,789,723 ns)    
+Block 9939462 with     2 tx used   4.60% of its weight (    26,840,938 of    583,697,101 ns)    
+```
+
+### Output Interpretation
+
+<sup>(Only results from reference hardware are relevant)</sup>
+
+Each block is executed multiple times and the results are averaged.  
+The percent number is the interesting part and indicates how much weight was used as compared to how much was predicted.  
+The closer to 100% this is without exceeding 100%, the better.  
+If it exceeds 100%, the block is marked with "**OVER WEIGHT!**" to easier spot them. This is not good since then the benchmarking under-estimated the weight.  
+This would mean that an honest validator would possibly not be able to keep up with importing blocks since users did not pay for enough weight.  
+If that happens the validator could lag behind the chain and get slashed for missing deadlines.  
+It is therefore important to investigate any overweight blocks.  
+
+In this example you can see an unexpected result; only < 5% of the weight was used!  
+The measured blocks can be executed much faster than predicted.  
+This means that the benchmarking process massively over-estimated the execution time.  
+Since they are off by so much, it is an issue [polkadot#5192].  
+
+The ideal range for these results would be 85-100%.
+
+## Polkadot # 2
+
+Let's take a more interesting example where the blocks use more of their predicted weight.  
+Every day when validators pay out rewards, the blocks are nearly full.  
+Using an archive node here is the easiest.  
+
+The Polkadot blocks TODO-TODO for example contain large batch transactions for staking payout.  
+
+```sh
+cargo run --profile=production -- benchmark block --from TODO --to TODO --db paritydb
+```
+
+```pre
+TODO
+```
+
+## Substrate
+
+It is also possible to try the procedure in Substrate, although it's a bit boring.  
+
+First you need to create some blocks with either a local or dev chain.  
+This example will use the standard development spec.  
+Pick a non existing directory where the chain data will be stored, eg `/tmp/dev`.
+```sh
+cargo run --profile=production -- --dev -d /tmp/dev
+```
+You should see after some seconds that it started to produce blocks:  
+```pre
+…
+✨ Imported #1 (0x801d…9189)
+…
+```
+You can now kill the node with `Ctrl+C`. Then measure how long it takes to execute these blocks:  
+```sh
+cargo run --profile=production -- benchmark block --from 1 --to 1 --dev -d /tmp/dev --pruning archive
+```
+This will benchmark the first block. If you killed the node at a later point, you can measure multiple blocks.
+```pre
+Block 1 with     1 tx used  72.04% of its weight (     4,945,664 of      6,864,702 ns)
+```
+
+In this example the block used ~72% of its weight.  
+The benchmarking therefore over-estimated the effort to execute the block.  
+Since this block is empty, its not very interesting.
+
+## Arguments
+
+- `--from` Number of the first block to measure (inclusive).
+- `--to` Number of the last block to measure (inclusive).
+- `--repeat` How often each block should be measured.
+- [`--db`]
+- [`--pruning`]
+
+License: Apache-2.0
+
+<!-- LINKS -->
+
+[Polkachu]: https://polkachu.com/snapshots
+[here]: https://github.com/paritytech/substrate/issues/11141
+[polkadot#5192]: https://github.com/paritytech/polkadot/issues/5192
+
+[`--db`]: ../shared/README.md#arguments
+[`--pruning`]: ../shared/README.md#arguments