Some systems are experiencing issues
Scheduled Maintenance
Cellar Paris: Service upgrade might increase requests latencies

We are currently in the process of upgrading our Cellar Paris service. During this upgrade, customers might experience increased requests latencies when we restart certain components. No service interruption is to be expected. The upgrade should take place throughout the week starting on Monday 13/01/2025 around 09:00 UTC . We will update this maintenance once it is over or if anything comes up.

Past Incidents

Wednesday 10th January 2024

PostgreSQL Issue when deploying a PostgreSQL 10 add-on

Trouble to deploy new PostgreSQL add-on in version 10. This version is temporarily disabled for migration and new add-on order.

Tuesday 9th January 2024

[Paris] Datacenter updates, scheduled 1 year ago

We are planning to do various updates on one of our datacenter in the Paris region starting at 10:35 UTC. It will last for a few hours. No issue is to be expected during this maintenance.

We will update this status accordingly.

EDIT 2024-01-10 20:00 UTC: Maintenance is over, no impact during the operations.

Monday 8th January 2024

No incidents reported

Sunday 7th January 2024

No incidents reported

Saturday 6th January 2024

Deployments Elevated rate of failed deployments

We are seeing an elevated rate of failed deployments. We are investigating the issue.

EDIT 15:58 UTC: The issue has been identified and deployments should be back to normal since 15:40 UTC.

Friday 5th January 2024

No incidents reported

Thursday 4th January 2024

Access Logs [Metrics] Elevated queries error rate

We are seeing elevated error rate for metrics read queries due to the underlying storage system. The problem has been identified and we are working toward its resolution. This can impact some of the grafana dashboards or API queries. Write performance is not impacted.

Update Thu Jan 04 14:48:00 2024 UTC: We have triggered some data balancing. Some queries may take longer than expected. This can impact some of the grafana dashboards or API queries. Write performance may be impacted.

Update Thu Jan 04 20:44:01 2024 UTC: data balancing is more aggressive than expected, overloading some components. Query may be unavailable during that time

Update Fri Jan 05 02:26:05 2024 UTC: some components are still overloaded. We are currently catching up the lag, but query is disabled for now.

Update Fri Jan 05 08:01:45 2024 UTC: our write-path is still overloaded. We are searching for the bottleneck

Update Fri Jan 05 16:03:48 2024 UTC: a cleanup subroutine has been triggered to balance and remove slack space from our internal Btree storage. Query is still disabled to speed-up the process.

Update: Sat Jan 06 11:25:28 2024 UTC: lag has been absorbed. Query is now up, the cleanup subroutine is still in-progress. You may notice latency spikes during query.

Update: Mon Jan 08 14:36:57 2024 UTC: cleanup subroutine is still in-progress, and some workloads triggered an overloading of some components. Query is disabled to speed-up recovery

Update: Mon Jan 08 16:36:18 2024 UTC: query is now open.

Update Tue Jan 09 14:38:34 2024 UTC: Some StorageServers are late, meaning that a really small portion of the data is not available for the query. We are currently catching up with the lag

Update Tue Jan 16 14:56:55 2024 UTC: closing the ticket.