Past Incidents

A cleanup process has triggered some durability lag on our storage layer. You may experience query instability.

Mon Mar 25 20:32:34 2024 UTC: all components are back to normal

(times in UTC)

Around 21:00, a part of the logs drains stack broke in a way that our monitoring did not see right away. It started to fill up the disk of the underlying RabbitMQ. At 21:37, We were alerted by the lack of space on RabbitMQ. We started investigating it around 22:10. At 22:57: the log drain stack is back up! However, to fix the RabbitMQ, we had to drop the pending queues. Our logs are still collected in our new logs infrastructure, but all drains lost the logs between 21:00 and 22:57.

Scope:

• Database Load Balancer (configuration update)

Expected Impact:

• Brief disconnections or connection drops during the update process.
• Potential minor performance fluctuations.

Additional Information:

• We will deploy a patch on the load balancer that reduce memory consumption and enable more telemetry.
• Please report any issues with a method for reproducing the problem
• This maintenance is a direct follow up of the incident to propagate the patch https://www.clevercloudstatus.com/incident/826

EDIT 16:25 UTC : We have patched RBX, RBXHDS and MTL regions

EDIT 16:25 UTC : We are rolling out the patch on PAR region.

EDIT 16:45 UTC: We have patched PAR region, we start the WSW region

EDIT 16:55 UTC: We have patched WSW region, we start the SYD region

EDIT 17:05 UTC: We have patched SYD region, we start the GRAHDS region

EDIT 17:05 UTC: We have patched GRAHDS region, we start the SCW region

EDIT 17:15 UTC: We have patched SCW region, we start the SGP region.

EDIT 17:35 UTC: We have patched the SGP region as well. The maintenance is over.