Some systems are experiencing issues
Scheduled Maintenance
[PAR] Security maintenance on 4 hypervisors

For security reasons, we will update the kernel of 4 Hypervisors in the Paris (PAR) region, more precisely in the PAR6 datacenter. Services (in particular databases) hosted on those hypervisors will be impacted : they will be unavailable between 5 and 10 minutes. Impacted hypervisors are:

On Wednesday 20 November

  • hv-par6-012
  • hv-par6-020

On Thursday 21 November:

  • hv-par6-008
  • hv-par6-011

Affected clients are directly and individually contacted by email with the list of impacted services, and options to avoid any impact. The maintenance will be planned in 2 operations of 2 hypervisors each, during the week of 18 to 22 Novembre 2024 between 22:00 and 24:00 UTC+1.

Past Incidents

Wednesday 10th January 2024

PostgreSQL Issue when deploying a PostgreSQL 10 add-on

Trouble to deploy new PostgreSQL add-on in version 10. This version is temporarily disabled for migration and new add-on order.

Tuesday 9th January 2024

[Paris] Datacenter updates, scheduled 10 months ago

We are planning to do various updates on one of our datacenter in the Paris region starting at 10:35 UTC. It will last for a few hours. No issue is to be expected during this maintenance.

We will update this status accordingly.

EDIT 2024-01-10 20:00 UTC: Maintenance is over, no impact during the operations.

Monday 8th January 2024

No incidents reported

Sunday 7th January 2024

No incidents reported

Saturday 6th January 2024

Deployments Elevated rate of failed deployments

We are seeing an elevated rate of failed deployments. We are investigating the issue.

EDIT 15:58 UTC: The issue has been identified and deployments should be back to normal since 15:40 UTC.

Friday 5th January 2024

No incidents reported

Thursday 4th January 2024

Access Logs [Metrics] Elevated queries error rate

We are seeing elevated error rate for metrics read queries due to the underlying storage system. The problem has been identified and we are working toward its resolution. This can impact some of the grafana dashboards or API queries. Write performance is not impacted.

Update Thu Jan 04 14:48:00 2024 UTC: We have triggered some data balancing. Some queries may take longer than expected. This can impact some of the grafana dashboards or API queries. Write performance may be impacted.

Update Thu Jan 04 20:44:01 2024 UTC: data balancing is more aggressive than expected, overloading some components. Query may be unavailable during that time

Update Fri Jan 05 02:26:05 2024 UTC: some components are still overloaded. We are currently catching up the lag, but query is disabled for now.

Update Fri Jan 05 08:01:45 2024 UTC: our write-path is still overloaded. We are searching for the bottleneck

Update Fri Jan 05 16:03:48 2024 UTC: a cleanup subroutine has been triggered to balance and remove slack space from our internal Btree storage. Query is still disabled to speed-up the process.

Update: Sat Jan 06 11:25:28 2024 UTC: lag has been absorbed. Query is now up, the cleanup subroutine is still in-progress. You may notice latency spikes during query.

Update: Mon Jan 08 14:36:57 2024 UTC: cleanup subroutine is still in-progress, and some workloads triggered an overloading of some components. Query is disabled to speed-up recovery

Update: Mon Jan 08 16:36:18 2024 UTC: query is now open.

Update Tue Jan 09 14:38:34 2024 UTC: Some StorageServers are late, meaning that a really small portion of the data is not available for the query. We are currently catching up with the lag

Update Tue Jan 16 14:56:55 2024 UTC: closing the ticket.