Past Incidents

Tuesday 12th September 2023

Access Logs Metrics and access logs storage layer issue

The storage layer has lost some nodes. We are investigating the issue.

EDIT 13:45 UTC : We have found that we have a network issue which cause storage nodes to timeout and then crash. Those nodes are now up and running, we are beginning the recovery process

EDIT 15:10 UTC : We have finished the recovery process and we are consuming the lag.

EDIT 18:52 UTC : We have almost consume all the data lag (estimate duration is 30 mins left), but there is still 2h of metadata lag.

EDIT 21:00 UTC: We have catched up the data and metadata lag, the query is now open

API Main API unreachability

Our main API is currently unreachable. We are aware of the issue and working towards bringing it back.

EDIT 12:56 UTC: The main issue is now resolved and the API is back online. We continue to see some errors and are working towards identifying their source.

EDIT 14:25 UTC: The API has stabilized but we are still looking for the origin of the troubles.

EDIT 13/09 09:03 UTC: The API is unreachable again, we are working on it

EDIT 13/09 09:15 UTC: The API is now operational, the root cause has been identified.

API Main API unavailability

We are performing security updates on some core components.

Our main API may be unavailable for 1 hour.

EDIT 00:30 UTC: The maintenance is now over since 25 minutes ago. We are monitoring the results.

Monday 11th September 2023

[Maintenance] Main API planned unavailability, scheduled 1 year ago

On Monday 2023-09-11 around 20:00 UTC, our main API (api.clever-cloud.com) will be unavailable. The CLI and Console will be impacted and may display errors during some requests. Deployments will also be impacted and won't be available either through the Console/CLI or using git.

The maintenance is planned for one hour but is expected to last a few minutes at most.

EDIT 20:00 UTC: The maintenance is starting.

EDIT 20:02 UTC: The API is now unavailable as well as the Console.

EDIT 20:16 UTC: One of the steps took a bit more time than expected, we are back on track.

EDIT 20:44 UTC: Unexpected problems occurred and we are currently doing a rollback of the changes.

EDIT 20:54 UTC: The maintenance is over, changes were rollback and everything should now be operational again.

Sunday 10th September 2023

No incidents reported

Saturday 9th September 2023

No incidents reported

Friday 8th September 2023

No incidents reported

Thursday 7th September 2023

No incidents reported

Wednesday 6th September 2023

Infrastructure [JED] Hypervisors unreachability

We are currently experiencing unreachable hypervisors on the JED region. We are investigating the issue.

EDIT 18:50 UTC: The hypervisors are back online since 25 minutes now, all services were restarted by our monitoring.

Infrastructure [SCW] Hypervisor has crashed

An hypervisor has crashed, we are currently investigating the root cause

EDIT 18:45 - The hypervisor had a kernel panic. During the reboot operation the kernel has been upgraded and this issue should not occur again.