Some systems are experiencing issues

Past Incidents

Friday 20th October 2023

[PAR] FS Bucket maintenance on n19 and n20 servers, scheduled 11 months ago

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n19 and n20.

The maintenance will take place on Friday 20 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-20 12:00 UTC+2: Maintenance is starting.

EDIT 2023-10-20 13:24 UTC+2: Applications are currently redeploying.

EDIT 2023-10-20 14:20 UTC+2: Applications have redeployed. We are cleaning things up.

EDIT 2023-10-20 16:12 UTC+2: The maintenance is over.

Thursday 19th October 2023

[PAR] FS Bucket maintenance on n10 and n17 servers, scheduled 11 months ago

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n10 and n17.

The maintenance will take place on Thursday 19 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

Wednesday 18th October 2023

[PAR] FS Bucket maintenance on n15 and n16 servers, scheduled 11 months ago

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n15 and n16.

The maintenance will take place on Wednesday 18 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

Tuesday 17th October 2023

[PAR] FS Bucket maintenance on n12 and n13 servers, scheduled 11 months ago

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n12 and n13.

The maintenance will take place on Tuesday 17 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-17 12:10 UTC+2: The maintenance is starting. FSBucket servers are set in read-only mode.

EDIT 2023-10-17 12:47 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-17 16:15 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 14:00 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

Monday 16th October 2023

[PAR] PHP FTP maintenance on n11 and n18 servers for PHP+FTP applications, scheduled 11 months ago

For security reasons, we will update the PHP FTP services on the Paris (PAR) region used by PHP+FTP applications. PHP applications using the Git deployment method are not impacted by this maintenance.

PHP FTP hosts that will be updated during this maintenance are: n11 and n18.

The maintenance will take place on Monday 16 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your PHP+FTP applications in order to avoid waiting for the automatic redeployment.

Our support is available for any questions via the ticket center in the console. This maintenance will be updated during the maintenance window.

EDIT 2023-10-16 12:03 UTC+2: The maintenance will begin shortly. FSBucket add-on hosted on those servers will soon become read-only.

EDIT 2023-10-16 12:09 UTC+2: FSBuckets are now read-only

EDIT 2023-10-16 12:54 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-16 14:04 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 13:30 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

Sunday 15th October 2023

No incidents reported

Saturday 14th October 2023

No incidents reported

Friday 13th October 2023

No incidents reported

Thursday 12th October 2023

No incidents reported

Wednesday 11th October 2023

[ALL] Logs collection maintenance, scheduled 11 months ago

For security reasons, we will update our logs collection systems.

The logs collection (logs drains too) will be unavailable during the maintenance.

EDIT 00:00 UTC: The maintenance is now over.

Tuesday 10th October 2023

Infrastructure [PAR] Hypervisor network unreachability

An hypervisor had its internal network unreachable between 15:16 and 15:20 UTC. During that time, services on the hypervisors may have suffered a total loss of network connectivity. Applications have been redeployed because of the unreachability. The issue has been fixed and we are monitoring the situation.

EDIT 15:49 UTC: All services are now reachable again, the incident is now over.

Monday 9th October 2023

Access Logs [Metrics/AccessLogs] Ingestion lag

We are detecting a data lag on our Metrics / AccessLog stack. You may not experience the latest datapoints. We are working on it.

EDIT 10:00 PM: lag has been fully absorbed