Past Incidents

We may be impacted by https://network.status-ovhcloud.com/incidents/nnhpfdw50vsn which we are investigating. Only OVH regions based services are concerned.

Update 13:33 UTC: we are indeed impacted by OVHcloud's Backbone incident. Some network routes cannot reach OVHcloud's datacenters. We are working on it. More info can be found on https://x.com/olesovhcom/status/1819742478586528146

Update 14:42 UTC: network seems more reliable now. We are still watching the network links

Update 15:16 UTC: The services are getting operational according to OVHcloud and we are not seeing network issues anymore.

We are experiencing a global outage. We observed a network split in addition to an event bus outage. The effect has been inpactful for some core services.

EDITS :

• 2:00 PM CEST - Core services are being recovered and Deployments are being reloaded. This will synchronize back load balancers for customer's application trying to reach their new deployments.
• 2:08 PM CEST - Some services are being shut to accelerate the recovery process. Expect disturbed experience for observability and deployments for a few minutes
• 2:29 PM CEST - Criticial Core services are OK. Deployments are being rolled out.
• 3:07 PM CEST - Some workload queues have still difficulties to be processed. Some components may still be in an unstable state. Current effort is to identify them, then reload them.
• 3:40 PM CEST - Some hypervisors have experienced some crashes. Recovery process is occuring and will take a couple of minutes
• 3:56 PM CEST - Some hypervisors seems still experiencing network issues.
• 4:16 PM CEST - Apps are being deployed for premium customers. All apps are going to be deployed. Anyone can accelerate the process for its own application by manually deploying them.
• 4:24 PM CEST - In the meantime, we continue to identify noisy VMs that have been impacted by the outage
• 5:15 PM CEST - Metrics API is being restarted.
• 6:20 PM CEST - Last deployments are being rolled out. Reminder : accelerate by triggering a redeploy action
• 6:30 PM CEST - Still a few hundreds of VMs are consuming very high CPU rates and being cleaned.
• 6:35 PM CEST - We estimate approximately 40min to have full recovered all deployment of applications (MANUALY REDEPLOY FOR FASTER RECOVERY)
• 7:05 PM CEST - All IPSec links should be back online