Some systems are experiencing issues

Past Incidents

Monday 23rd October 2023

[PAR] MySQL in plan `DEV` maintenance, scheduled 11 months ago

For security reasons, we will migrate our DEV MySQL services on the Paris (PAR) region. Applications using those services will be impacted.

Only the par dev cluster will be updated during this maintenance.

The maintenance will take place on Monday 23rd of October 2023, between 11:45 UTC+2 and 15:00 UTC+2.

For the entire duration of the update, the services will not be available.

The time required to perform the update is estimated between 1 and 2 hours. However, total downtime might be longer as every application using the cluster will need to be restarted.

In case you have connection issues after those updates, you can manually trigger a redeployment of your linked applications.

If you do not want to be impacted by your DEV addon being offline, you can still order or migrate to a dedicated one before this maintenance starts.

Our support team is available for any questions via the ticket center in the console.

EDIT 2023-10-23 11:50 UTC+2: Maintenance is starting.

EDIT 2023-10-23 12:20 UTC+2: Dev addons are now available again. We will restart linked applications

EDIT 2023-10-23 12:22 UTC+2: We investigate an error while creating new DEV addons

EDIT 2023-10-23 12:40 UTC+2: New DEV addons can now be created. All applications linked to DEV addons are currently restarting

EDIT 2023-10-23 13:00 UTC+2: All applications linked to DEV addons have restarted

Sunday 22nd October 2023

[PAR} Load balancer security maintenance, scheduled 11 months ago

For security reasons, we will migrate our public load balancers on the Paris (PAR) region including cleverapps.io domains.

The maintenance will take place on Sunday 22 October 2023, between 14:00 UTC+2 and 20:00 UTC+2.

During the maintenance, applications and add-ons on this region will experience unexpected connection closed or reset, specifically on long running connections, beginning at 16:00 UTC+2. To prevent issues, you could restart your application if you see connection issues.

To check which of your services are impacted, you can consult the information section of your applications and see the region where your application is deployed.

14:45 UTC+2 : we are beginning the preparation steps to update load balancer that received cleverapps.io traffic 16:00 UTC+2 : we have identified a bug, so we will skip the update for now of cleverapps.io load balancers 16:30 UTC+2 : we are beginning the update of the last load balancer. 18:00 UTC+2 : we will soon update dns records to send traffics to new load balancer. 18:15 UTC+2: dns records has been updated 18:20 UTC+2 : monitoring is green, the maintenance is done

Saturday 21st October 2023

Deployments off for ~2 hours, scheduled 11 months ago

Due to security updates, we will need to shutdown and upgrade a component of our deployment infrastructure.

As a result, we will need to shutdown the deployment component for approx. 1 hour.

The maintenance is over, deployments are now usable again.

[PAR} Load balancer security maintenance, scheduled 11 months ago

For security reasons, we will migrate our public load balancers on the Paris (PAR) region including cleverapps.io domains.

The maintenance will take place on Saturday 21 October 2023, between 14:00 UTC+2 and 20:00 UTC+2.

During the maintenance, applications and add-ons on this region will experience unexpected connection closed or reset, specifically on long running connections, beginning at 16:00 UTC+2. To prevent issues, you could restart your application if you see connection issues.

To check which of your services are impacted, you can consult the information section of your applications and see the region where your application is deployed.

14:15 UTC+2 : The maintenance will start soon, we are ending preparation steps 15:15 UTC+2: Preparation steps took more time than estimated, we are rolling some configuration update on dedicated load balancers 16:15 UTC+2: Update in rolling of dedicated load balancers is terminated, we are beginning the public shared load balancer. 17:15 UTC+2: We are udpating the domain name resolutions for public shared load balancer of addons 18:30 UTC+2: We have updated two of eights servers of public shared load balancer of addons. 19:00 UTC+2: We have updated four of eights servers of public shared load balancer of addons. 19:15 UTC+2: We have updated six of eights servers of public shared load balancer of addons. 19:15 UTC+2: We have updated seven of eights servers of public shared load balancer of addons. 19:50 UTC+2: We have updated all servers of public shared load balancer of addons. As it is late and we are reaching the end of the window, we will update last load balancers tomorrow afternoon

Friday 20th October 2023

No incidents reported

Thursday 19th October 2023

No incidents reported

Wednesday 18th October 2023

Infrastructure [PAR] Network connectivity issue

This night, we had network connectivity issues when adding new bgp peer. Those connectivity issues has resulted in desynchronisation of our load balancers. You may have experienced connectivity issue to your database and unavailability of your applications.

Notifications, scheduled 11 months ago

A maintenance on our notification stack is scheduled for this afternoon. This means services such as WebHooks and Email Notifications will not work.

Downtime is expected to last between 30 minutes to 1 hour.

[14:15 UTC] All notifications services are now up and running

Tuesday 17th October 2023

FS Buckets [RETROACTIVE][PAR] PHP+FTP applications timeouts / slowness

PHP+FTP applications may have seen degraded performances and / or timeouts those past few hours on the Paris region due to a configuration issue of the underlying disk.

The configuration has been fixed at 21:00 UTC and disk access time are now in the normal range. We will keep monitoring the situation in the upcoming days to make sure performance stays in normal ranges.

[MongoDB] DEV Cluster security update on PAR, scheduled 11 months ago

Our security updates will need us to update nodes from our DEV mongodb cluster on PAR.

If you use the mongodb uri correctly, it should only disrupt your application for a few seconds. Otherwise, expect up to two hours of maintenance.

Monday 16th October 2023

Services Logs Logs are experiencing issues

We are encountering issues with our log systems.

The fetch of logs can take a while.

EDIT 21:37 UTC - fixed.

Sunday 15th October 2023

No incidents reported