Some systems are experiencing issues

Past Incidents

Tuesday 14th November 2023

Access Logs Metrics kafka cluster maintenance

We are currently scaling up our kafka cluster for metrics usage. You may not see latest datapoints

EDIT 15th of november 09:09 AM UTC: cluster has been scaled up and partitions distributed among new brokers

API API is slow

Our main API responds slowly. We are investigating to find out why.

(times in CET)

  • EDIT 11:54 - API has been redeployed and it solved the issue. We are still investigation the root cause.
  • EDIT 12:09 - API is down, we are working on it.
  • EDIT 12:23 - The API is recovering, our load balancers on Paris and Scaleway regions are down
  • EDIT 12:30 - Load balancers on Scaleway region is now operational
  • EDIT 12:33 - Load balancers on PAR region is now operational. API is now fully operational
  • EDIT 14:17 - We found and fixed a cause of the API problems. The API seems to be running fine, we may have found the root cause of the issue. We are keeping an eye on it.
API maintenance, scheduled 9 months ago

A maintenance is planned for our API, some API calls will probably not be available for a couple of minutes. This will not impact any deployed service.

The maintenance will start today November 14, 2023 at 12:00 UTC+1.

EDIT 2023-11-14 12:09 UTC+1: API is down, we report this maintenance

EDIT 2023-11-14 15:20 UTC+1: Maintenance is starting

EDIT 2023-11-14 17:10 UTC+1: Maintenance has been correctly completed

Monday 13th November 2023

Access Logs Metrics instability

We are detecting a performance issues on our new metrics storage layer. You may not see fresh datapoints. We are working on it.

EDIT Mon Nov 13 18:51:01 2023 UTC: config tuning has been made, cluster is now fully recovered. Lag will be resolved within minutes

Reverse Proxies Cleverapps load balancer instability

We are detecting a high number of errors on our reverse proxies towards cleverapps domains, we are investigating

EDIT: fixed

Sunday 12th November 2023

No incidents reported

Saturday 11th November 2023

No incidents reported

Friday 10th November 2023

Reverse Proxies SSL Handshake error on one of our Load Balancer

One of our load balancers randomly close connection during TLS handshake

EDIT 16h35: This issue has been fixed.

Thursday 9th November 2023

Addon API maintenance, scheduled 10 months ago

This maintenance is planned on our API, this maintenance will only impact the view of the add-ons and the creation of new one. Each already deployed add-ons will still be available.

This concern only Jenkins, ElasticSearch, MySQL, PostgreSQL, MongoDB and Redis API.

For each kind of add-ons expect a downtime of 20 to 30 minutes.

The maintenance will start tonight November 9, 2023 at 21:00 UTC.

EDIT 2023-11-09 22:00 UTC+1: Maintenance is starting.

EDIT 2023-11-10 01:00 UTC+1: Maintenance is now completed.

Wednesday 8th November 2023

API APIs are experiencing issues

We are seeing issues with our APIs. We are investigating this.

Edit 2023-11-09 : we are keeping this incident open as the performance issues seem to have lowered but not vanished. There seems to be a seasonality with these issues, we are still searching why we have these surges in load.

Cellar [Cellar] random lag on Cellar connections

Customers have complained about slow request handling on Cellar for a few days. On random requests, Cellar might take up to a few seconds to send the first byte.

We are investigating this issue.