Compression on the client side
Please refer to the Java client - Data Serialization and Java client - Data Compression.
Compression on the server side
It’s recommended:
- For the server with low CPU load, use the
zstdalgorithm with the highest compression ratio. - For the server with high CPU load, use the
lz4algorithm with good compression rate and speed.
The compression algorithms supported by the Pegasus server:
- snappy
- lz4 (Since Pegasus v1.11.2)
- zstd (Since Pegasus v1.11.2)
Configure the compression algorithm through Configurations, for example:
[pegasus.server]
rocksdb_compression_type = lz4
Comparison of different compression algorithms (from zstd official benchmark):
| Compressor name | Ratio | Compression (MB/s) | Decompress (MB/s) |
|---|---|---|---|
| zstd 1.3.4 -1 | 2.877 | 470 | 1380 |
| zlib 1.2.11 -1 | 2.743 | 110 | 400 |
| brotli 1.0.2 -0 | 2.701 | 410 | 430 |
| quicklz 1.5.0 -1 | 2.238 | 550 | 710 |
| lzo1x 2.09 -1 | 2.108 | 650 | 830 |
| lz4 1.8.1 | 2.101 | 750 | 3700 |
| snappy 1.1.4 | 2.091 | 530 | 1800 |
| lzf 3.6 -1 | 2.077 | 400 | 860 |
This result is consistent with lz4 official benchmark.
Compression algorithm suggestions from RocksDB official:
Use options.compression to specify the compression to use. By default it is Snappy. We believe LZ4 is almost always better than Snappy. We leave Snappy as default to avoid unexpected compatibility problems to previous users. LZ4/Snappy is lightweight compression so it usually strikes a good balance between space and CPU usage.
If you want to further reduce the in-memory and have some free CPU to use, you can try to set a heavy-weight compression in the latter by setting options.bottommost_compression. The bottommost level will be compressed using this compression style. Usually the bottommost level contains majority of the data, so users get almost optimal space setting, without paying CPU for compress all the data ever flowing to any level. We recommend ZSTD. If it is not available, Zlib is the second choice.
If you want have a lot of free CPU and want to reduce not just space but write amplification too, try to set options.compression to heavy weight compression type. We recommend ZSTD. Use Zlib if it is not available.