Recurring Disk Latency Spikes While Pyramid In-Memory Service Is Running

Hello Pyramid team,

I am investigating recurring disk read/write latency spikes on multiple Pyramid in-memory servers.

The storage was moved to a higher-performance network SSD type, but the latency spikes remained.

Monitoring shows recurring write latency spikes, with p99 frequently reaching approximately 150-200 ms, while the number of IOPS and the actual write throughput remain relatively low.

What I observed:

1. The latency spikes occur while the Pyramid in-memory service is running.
2. When the service is stopped, the spikes almost completely disappear.
3. RAM usage remains within the normal range, approximately 40-60%, with no signs of memory exhaustion.
4. Separate fio tests were performed on the same volume while the Pyramid services were stopped.
5. The fio tests did not reproduce the high latency shown in monitoring.

fio results:

Random read:

• Average latency: approximately 0.62 ms
• p99 latency: approximately 2.15 ms

Random write:

• Average latency: approximately 0.59 ms
• p99 latency: approximately 2.06 ms
• Maximum latency: approximately 12.8 ms

No fio errors were reported.

Based on these results, the storage does not show the same high latency under a simple synthetic read/write workload. However, I cannot determine whether the issue is caused by a specific Pyramid I/O pattern or by the way the storage handles this workload.

Could you please advise:

1. Which Pyramid components or background processes can generate this disk I/O pattern?
2. Are there any known issues related to high write latency on Pyramid in-memory servers?
3. Which Pyramid logs or diagnostics should I collect during a latency spike?
4. Is there a recommended way to correlate Pyramid operations with disk latency?
5. Are there any recommended settings for the in-memory service or storage configuration that should be checked?

I have attached the monitoring screenshots and the complete fio read/write logs.

Thanks!