2026-10-11
Capillaries: scalability update
Basic test settings remain the same:
- number_of_daemons = number_of_cassandra_nodes
- Cassandra nodes are 4x as powerful as Capillaries instances
- the daemon's GOMEMLIMIT is still set to 75%
What has changed
There have been a few changes since the last scalability update in May 2026 (see also the Jul 2025 update and the original scalability testing back in Aug 2024).
Lookups with async SELECTs
When handling lookups, Capillaries used an "IN" condition in SELECT statements, which is widely considered an anti-pattern. Since version 1.2.2, Capillaries has been running multiple async SELECT commands to retrieve index records by key and data records by rowid. As a result, we get much better resource utilization - Cassandra does not have to perform internal read requests. The CPU use chart 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) for the lookup nodes in the portfolio test looks much better: those Cassandra nodes that used to be loaded at 5-10% mark now do some work, and daemons that were in the 10-45% range are now in the 35-70% range:
The Cassandra read metric sum(irate(cassandra_clientrequest_localrequests_count{clientrequest="Read"}[1m])) also reflects the change: lookups now perform many more reads against the Cassandra cluster (although now they return only a single data point instead of a group of data points that match those IN conditions):
Test deployment changes
GOGC=25
For daemons, the GOGC=100 setting was replaced with GOGC=25, which gives us a somewhat more stable picture in the go_memstats_heap_inuse_bytes{job="go_exporter"} Prometheus metric. Memory consumption does not jump back and forth as it used to. See the following comparison for runs with 32 daemons and 32 Cassandra nodes, before and after:
Daemon threads per CPU
More importantly: daemon_thread_pool_factor setting was switched from 3 to 2. This means 2 worker threads per daemon instance CPU instead of 3. Three threads turned out to be too aggressive: during py_calc node handling, each thread has to spawn its own copy of Python, and too many Python processes tend to cause OOM errors (especially with Python 3.13 and up because of new featires like JIT compiler and no-GIL mode). To compensate, the number of Cassandra writers was slightly increased (see the daemon_writer_workers variable).
c8g instances
One more change to mention: c7g AWS EC2 instances were replaced with the newer c8g instances.
Updated test results
As before, the portfolio test (996 ARK-inspired stock portfolios with 14,683,696 transactions and 713,800 end-of-month holding records) is used for performance benchmarking. The updated results are presented below.
| Deployment flavor | AWS instance | 4 Cassandra nodes | 8 Cassandra nodes | 16 Cassandra nodes | 32 Cassandra nodes | Cassandra | Daemon | Bastion/RabbitMQ/Prometheus | Total cores | Hourly cost | Total cores | Hourly cost | Total cores | Hourly cost | Total cores | Hourly cost |
|---|---|---|---|---|---|---|---|---|---|---|---|
| aws.arm64.c8g.8 | c8gd.2xlarge | c8g.large | c8g.large | 42 | $1.9665 | 82 | $3.8532 | 162 | $7.6266 | 322 | $15.1735 |
| aws.arm64.c8g.16 | c8gd.4xlarge | c8g.xlarge | c8g.large | 82 | $3.8532 | 162 | $7.6266 | 322 | $15.1735 | ||
| aws.arm64.c8g.32 | c8gd.8xlarge | c8g.2xlarge | c8g.large | 162 | $7.6266 | 322 | $15.1735 | ||||
| aws.arm64.c8g.64 | c8gd.16xlarge | c8g.4xlarge | c8g.large | 322 | $15.1735 |
_10_output_daemon_instances = 4
_12_output_daemon_cpus = "2 cpus ($0.079760/hr)"
_13_output_daemon_total_cpus = 8
_14_output_daemon_memory_gb = "4 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "3072 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 4
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 4
_21_output_cassandra_cpus = "8 cpus ($0.391920/hr)"
_23_output_cassandra_total_cpus = 32
_30_output_total_ec2_hourly_cost = "$1.966480/hr"
_10_output_daemon_instances = 8
_12_output_daemon_cpus = "2 cpus ($0.079760/hr)"
_13_output_daemon_total_cpus = 16
_14_output_daemon_memory_gb = "4 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "3072 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 4
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 8
_21_output_cassandra_cpus = "8 cpus ($0.391920/hr)"
_23_output_cassandra_total_cpus = 64
_30_output_total_ec2_hourly_cost = "$3.853200/hr"
_10_output_daemon_instances = 16
_12_output_daemon_cpus = "2 cpus ($0.079760/hr)"
_13_output_daemon_total_cpus = 32
_14_output_daemon_memory_gb = "4 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "3072 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 4
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 16
_21_output_cassandra_cpus = "8 cpus ($0.391920/hr)"
_23_output_cassandra_total_cpus = 128
_30_output_total_ec2_hourly_cost = "$7.626640/hr"
_10_output_daemon_instances = 32
_12_output_daemon_cpus = "2 cpus ($0.079760/hr)"
_13_output_daemon_total_cpus = 64
_14_output_daemon_memory_gb = "4 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "3072 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 4
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 32
_21_output_cassandra_cpus = "8 cpus ($0.391920/hr)"
_23_output_cassandra_total_cpus = 256
_30_output_total_ec2_hourly_cost = "$15.173520/hr"
_10_output_daemon_instances = 4
_12_output_daemon_cpus = "4 cpus ($0.159520/hr)"
_13_output_daemon_total_cpus = 16
_14_output_daemon_memory_gb = "8 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "6144 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 8
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 4
_21_output_cassandra_cpus = "16 cpus ($0.783840/hr)"
_23_output_cassandra_total_cpus = 64
_30_output_total_ec2_hourly_cost = "$3.853200/hr"
_10_output_daemon_instances = 8
_12_output_daemon_cpus = "4 cpus ($0.159520/hr)"
_13_output_daemon_total_cpus = 32
_14_output_daemon_memory_gb = "8 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "6144 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 8
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 8
_21_output_cassandra_cpus = "16 cpus ($0.783840/hr)"
_23_output_cassandra_total_cpus = 128
_30_output_total_ec2_hourly_cost = "$7.626640/hr"
_10_output_daemon_instances = 16
_12_output_daemon_cpus = "4 cpus ($0.159520/hr)"
_13_output_daemon_total_cpus = 64
_14_output_daemon_memory_gb = "8 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "6144 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 8
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 16
_21_output_cassandra_cpus = "16 cpus ($0.783840/hr)"
_23_output_cassandra_total_cpus = 256
_30_output_total_ec2_hourly_cost = "$15.173520/hr
_10_output_daemon_instances = 4
_12_output_daemon_cpus = "8 cpus ($0.319040/hr)"
_13_output_daemon_total_cpus = 32
_14_output_daemon_memory_gb = "16 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "12288 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 16
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 4
_21_output_cassandra_cpus = "32 cpus ($1.567680/hr)"
_23_output_cassandra_total_cpus = 128
_30_output_total_ec2_hourly_cost = "$7.626640/hr"
_10_output_daemon_instances = 8
_12_output_daemon_cpus = "8 cpus ($0.319040/hr)"
_13_output_daemon_total_cpus = 64
_14_output_daemon_memory_gb = "16 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "12288 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 16
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 8
_21_output_cassandra_cpus = "32 cpus ($1.567680/hr)"
_23_output_cassandra_total_cpus = 256
_30_output_total_ec2_hourly_cost = "$15.173520/hr"
_10_output_daemon_instances = 4
_12_output_daemon_cpus = "16 cpus ($0.638080/hr)"
_13_output_daemon_total_cpus = 64
_14_output_daemon_memory_gb = "32 GiB"
_15_output_daemon_gomemlimit_ratio = 0.75
_16_output_daemon_gomemlimit_mb = "24576 MiB"
_17_output_daemon_gogc = 25
_18_output_daemon_thread_pool_size = 32
_19_output_daemon_writer_workers = 16
_1A_output_daemon_lookup_key_workers = 6
_1B_output_daemon_lookup_rowid_workers = 6
_1C_output_daemon_writers_per_cassandra_cpu = 8
_20_output_cassandra_hosts = 4
_21_output_cassandra_cpus = "64 cpus ($3.135360/hr)"
_23_output_cassandra_total_cpus = 256
_30_output_total_ec2_hourly_cost = "$15.173520/hr"
| Deployment flavor | Cassandra node cores | Cassandra nodes | Run time, s | Cost of run |
|---|---|---|---|---|
| aws.arm64.c8g.8 | 8 | 4 | 711 | $0.388 |
| 8 | 373 | $0.399 | ||
| 16 | 190 | $0.403 | ||
| 32 | 113 | $0.476 | ||
| aws.arm64.c8g.16 | 16 | 4 | 371 | $0.397 |
| 8 | 176 | $0.373 | ||
| 16 | 113 | $0.476 | ||
| aws.arm64.c8g.32 | 32 | 4 | 195 | $0.413 |
| 8 | 115 | $0.485 | ||
| aws.arm64.c8g.64 | 64 | 4 | 131 | $0.552 |
As usual, the colors (green, orange, red, and black) represent scale-up options, while the number of Cassandra nodes (4, 8, 16, and 32) represents scale-out options. It is worth mentioning that the cost per run was reduced by almost half since Aug 2024, thanks to:
- Capillaries improvements
- test environment optimization
- AWS EC2 pricing changes
Those async SELECTs are definitely the way to go. Extra thread pool workers do not seem to help. The GOGC value is really up to the DevOps team. Also, please keep in mind that this time, all queue messages are encrypted by Webapi and decrypted by the daemons (new in Capillaries 1.2.2). This is not expected to add much to the daemon CPU usage, but it brings us closer to the production setup.