Tuesday 23rd July 2024

Metrics Metrics latencies and errors

We identified a bottleneck on our FoundationDB cluster for warp10-c2. Writing is impacted and might occur a lag in read metrics. We enabled sampling on data.

6:32 UTC: We identified unusual usage that was harming the system

6:35 UTC: Unusual usage stopped, the storage layer is starting to recover

6:58 UTC: Storage layer fully recovered, we still investigate & watch over the system

7:45 UTC: System is back to normal