Past Incidents

Thursday 14th March 2024

Cellar Cellar North: Requests slowness

We are currently investigating requests slowness on the Cellar north service.

EDIT 19:46 UTC+1: The underlying storage system is currently having issues and is rebalancing the data. No data loss is to be expected but timeouts may occur. We are looking to stabilize the system.

EDIT 20:09 UTC+1: The underlying storage system has stabilized the last 5 minutes. We keep an eye to make sure everything is okay

EDIT 21:47 UTC+1: The service is now stable. We will need to perform additional maintenance to fully fix the underlying issue. We will create the maintenances in the following days accordingly.

Reverse Proxies [RBX][RBXHDS] database load balancer instabilities

One instance of the load balancer has been unreachable, we have performed a patch and rebooted it. We are watching the load balancer.

EDIT 14:00 UTC : We have updated the load balancer configuration, you may have seen some connections cut during the reload.

EDIT 14:30 UTC : We have seen the same instabilities on RBX HDS database load balancer configuration, we have applied the patch that thr RBX database load balancer.

EDIT 2024-03-19T10:00:00Z : After a few days of observation, the currently deployed patched does not handle the issue correctly, we are are working on improving it.

EDIT 2024-03-19T11:30:00Z : We have deployed a new version of the patch that should handle the issue, we are watching the metrics to validate it

EDIT 2024-03-19T18:30:00Z : We have rolled out the patch as it handles the issue correctly. We are still watching metrics. However, we will need a few days to validate the behavior.

EDIT 2024-03-19T16:00:00Z : The patch is validated as we do not see the issue occurring.

Wednesday 13th March 2024

No incidents reported

Tuesday 12th March 2024

No incidents reported

Monday 11th March 2024

No incidents reported

Sunday 10th March 2024

No incidents reported

Saturday 9th March 2024

No incidents reported

Friday 8th March 2024

Infrastructure [RBX] Network instabilities

We have seen network instabilities which implies slowness and errors. We are investigating the issue.

We suspect that we may have impacted by one of this maintenance of our infrastructure provider. See:

https://network.status-ovhcloud.com/incidents/ym5hrzs2cn3b
https://network.status-ovhcloud.com/incidents/np3j0flx9w24

EDIT 14:00 UTC : we are confident that was not link to the infrastructure provider and was an isolated incident. We are still monitoring the issue, but it seems to be solved.