Past Incidents

Saturday 13th April 2024

Cellar Cellar on Paris is experiencing trouble

Ceph (the software we are running Cellar on) is rebalancing some shards due to a change in its storage capacity. Some requests might fail while doing so.

edit: after a few alerts, everything has been running smoothly.

Friday 12th April 2024

No incidents reported

Thursday 11th April 2024

Metrics [GLOBAL] Metrics query unavailable

The metrics query is currently unavailable as some indexing shared are offline. We are working to get them up as quickly as possible. There is no impact on ingestion pipeline and storage layer.

UTC 11:00: Queries are available

Wednesday 10th April 2024

No incidents reported

Tuesday 9th April 2024

No incidents reported

Monday 8th April 2024

Metrics [GLOBAL] Metrics query unavailable

The metrics query is currently unavailable as some indexing shared are offline. We are working to get them up as quickly as possible. There is no impact on ingestion pipeline and storage layer.

UTC 14:35 Services are back to normal

Access Logs [Global] Access logs ingestion lags

We have some lags on the ingestion of access logs. We are working on it.

EDIT 12:00 UTC : we have consume the lag for 75% of the access logs, we are working on the remaining ones.

EDIT 14:45 UTC : we are in-sync for 80% of access logs, we are consuming the remaining ones (eta 22h to consume the lag)

EDIT 16:41 UTC : we have consumed all remaining access logs

Sunday 7th April 2024

cleverapps.io domains Cleverapps.io certificate has expired

The *.cleverapps.io wildcard certificate failed to renew. We are currently renewing it.

Edit 15:05 UTC: we renew the certificate. It should appear on the load balancers in a few minutes. Edit 15:11 UTC: the certificate has been deployed on all cleverapps.io load balancers. Incident is over.