Some systems are experiencing issues

Past Incidents

Wednesday 20th March 2024

Reverse Proxies [Global] Database load balancers maintenance

Scope:

  • Database Load Balancer (configuration update)

Expected Impact:

  • Brief disconnections or connection drops during the update process.
  • Potential minor performance fluctuations.

Additional Information:

  • We will deploy a patch on the load balancer that reduce memory consumption and enable more telemetry.
  • Please report any issues with a method for reproducing the problem
  • This maintenance is a direct follow up of the incident to propagate the patch https://www.clevercloudstatus.com/incident/826

EDIT 16:25 UTC : We have patched RBX, RBXHDS and MTL regions

EDIT 16:25 UTC : We are rolling out the patch on PAR region.

EDIT 16:45 UTC: We have patched PAR region, we start the WSW region

EDIT 16:55 UTC: We have patched WSW region, we start the SYD region

EDIT 17:05 UTC: We have patched SYD region, we start the GRAHDS region

EDIT 17:05 UTC: We have patched GRAHDS region, we start the SCW region

EDIT 17:15 UTC: We have patched SCW region, we start the SGP region.

EDIT 17:35 UTC: We have patched the SGP region as well. The maintenance is over.

Tuesday 19th March 2024

No incidents reported

Monday 18th March 2024

No incidents reported

Sunday 17th March 2024

No incidents reported

Saturday 16th March 2024

No incidents reported

Friday 15th March 2024

No incidents reported

Thursday 14th March 2024

Cellar Cellar North: Requests slowness

We are currently investigating requests slowness on the Cellar north service.

EDIT 19:46 UTC+1: The underlying storage system is currently having issues and is rebalancing the data. No data loss is to be expected but timeouts may occur. We are looking to stabilize the system.

EDIT 20:09 UTC+1: The underlying storage system has stabilized the last 5 minutes. We keep an eye to make sure everything is okay

EDIT 21:47 UTC+1: The service is now stable. We will need to perform additional maintenance to fully fix the underlying issue. We will create the maintenances in the following days accordingly.

Reverse Proxies [RBX][RBXHDS] database load balancer instabilities

One instance of the load balancer has been unreachable, we have performed a patch and rebooted it. We are watching the load balancer.

EDIT 14:00 UTC : We have updated the load balancer configuration, you may have seen some connections cut during the reload.

EDIT 14:30 UTC : We have seen the same instabilities on RBX HDS database load balancer configuration, we have applied the patch that thr RBX database load balancer.

EDIT 2024-03-19T10:00:00Z : After a few days of observation, the currently deployed patched does not handle the issue correctly, we are are working on improving it.

EDIT 2024-03-19T11:30:00Z : We have deployed a new version of the patch that should handle the issue, we are watching the metrics to validate it

EDIT 2024-03-19T18:30:00Z : We have rolled out the patch as it handles the issue correctly. We are still watching metrics. However, we will need a few days to validate the behavior.

EDIT 2024-03-19T16:00:00Z : The patch is validated as we do not see the issue occurring.