Past Incidents

For security reasons, we will update the kernel of 4 Hypervisors in the Paris (PAR) region, more precisely in the PAR6 datacenter. Services (in particular databases) hosted on those hypervisors will be impacted : they will be unavailable between 5 and 10 minutes. Impacted hypervisors are:

On Wednesday 20 November

• hv-par6-012
• hv-par6-020

On Thursday 21 November:

• hv-par6-008
• hv-par6-011

Affected clients are directly and individually contacted by email with the list of impacted services, and options to avoid any impact. The maintenance will be planned in 2 operations of 2 hypervisors each, during the week of 18 to 22 Novembre 2024 between 22:00 and 24:00 UTC+1.

EDIT 2024-11-20 22:30 UTC: Both hypervisors were rebooted. All services are available again.

EDIT 2024-11-21 22:00 UTC: Both hypervisors were rebooted. All services are available again. This maintenance is now over.

Following yesterday's incident (https://www.clevercloudstatus.com/incident/911), we took actions to solve the root cause issue.

When we are performing some actions the ZooKeeper cluster becomes unstable and fails. Access logs, logs and deployments stack are affected as well as all services interacting with the Pulsar cluster.

16:20 UTC : The cluster pulsar is up and running, the deployment stack, logs is running as well, we are restarting the access logs stack.

16:30 UTC : The access logs stack is up and running.

The monitoring report that pulsar is in a unhealthy state, we are investigating.

16:38 UTC: there seems to be an inconsistency in the underlying bookkeeper cluster. We are looking into it.

16:40 UTC: we are now looking into the zookeeper service that seems to fail.

17:30 UTC: we have fix the zookeeper issue, and we begin the recovery process of the cluster bookeeper and then pulsar.

18:10 UTC : we are rolling open the access to the pulsar cluster.

18:45 UTC : we have rolled open the access to the pulsar cluster to half of our hypervisors.

19:15 UTC : the pulsar cluster is running and available for everyone. We are running the recovery process of the platform to ensure that every applications is up and running as well.

21:30 UTC : we have finished to redeploy applications. We are investigating the access logs stack that got offloaders errors on pulsar-side.

22:10 UTC : we have finished to restart the access logs stacks.