Past Incidents

The monitoring system has detected that an hypervisor is unreachable. We are investigating.

EDIT 08:32 UTC : We have found the issue and the hypervisor is rebooting

EDIT 08:50 UTC: The hypervisor has finished to reboot and services is working

An hypervisor rebooted on the Paris zone. Impacted applications are redeployed on other servers. We are monitoring the situation.

EDIT 11:40 UTC: All impacted applications have been redeployed automatically. We will investigate further why this server rebooted. The incident is now over.

Our metrics system's hbase cluster is in an inconsistent state. We found out which nodes are responsible for it and are fixing them.

12:26 UTC: we restarted the node responsible for the issue. While it re-converges, we stop the egress servers. We will put them back on in a few minutes.

13:31 UTC: Query is back online. We are still catching up the lag, so new datapoints may not be available

14:35 UTC: lag has ben catched up

Our monitoring has detected failure on the storage layer of metrics and access logs. We have found that a storage node has lost several disk. We have remove faulty disks and restarted the storage node.

EDIT 16:00 UTC : The storage layer is restarted and we are consuming the ingestion lag