Past Incidents

Tuesday 10th December 2024

No incidents reported

Monday 9th December 2024

No incidents reported

Sunday 8th December 2024

No incidents reported

Saturday 7th December 2024

No incidents reported

Friday 6th December 2024

No incidents reported

Thursday 5th December 2024

Pulsar Pulsar storage layer issue

A few nodes of the pulsar storage layer known as bookkeeper crashed and propagate the pulsar cluster with them. We are restoring the bookkeeper cluster and then we will help the cluster pulsar to recover.

EDIT 19:15 UTC : We have deployed a patch to fix the bookkeeper cluster, we have deployed the new configuration and we are rolled out the cluster. The pulsar cluster should be available.

EDIT 20:20 UTC : Some nodes of the bookkeeper cluster are under memory pressure, we are investigating the issue.

EDIT 21:20 UTC: We found the issue and are deploying the patch.

EDIT 21:50 UTC: Situation is back to normal.

Metrics Trouble to access Metrics in Grafana

We observed an issue while accessing Grafana Metrics dashboards with the message Access denied to this dashboard

A patch is currently beiing deployed

[ 12:30 CET]: All organisations have been patched

Wednesday 4th December 2024

Infrastructure [PAR] An hypervisor is experiencing degraded I/O operations

An hypervisor on the Paris region is experiencing degraded I/O operations. We are looking into it.

EDIT 20:25 UTC+1: The hypervisor is back to normal levels since 20:08 UTC+1. We keep investigating the reason of the slow I/O. Applications on this hypervisor were redeployed elsewhere to avoid any issues.

Infrastructure [PAR] Hypervisor crashed

An hypervisor has crashed on the PAR region. Applications are currently redeploying. We are investigating the reason of the crash, probably an issue with the RAID array.

EDIT 11:50 CET : hypervisor is up and running, we are still investigating the root cause.