Some systems are experiencing issues

Past Incidents

Monday 16th October 2023

[MongoDB] DEV Cluster security update on PAR

Our security updates will need us to update nodes from our DEV mongodb cluster on PAR.

If you use the mongodb uri correctly, it should only disrupt your application for a few seconds. Otherwise, expect up to two hours of maintenance.

Services Logs Logs are experiencing issues

We are encountering issues with our log systems.

The fetch of logs can take a while.

EDIT 21:37 UTC - fixed.

Sunday 15th October 2023

No incidents reported

Saturday 14th October 2023

No incidents reported

Friday 13th October 2023

No incidents reported

Thursday 12th October 2023

No incidents reported

Wednesday 11th October 2023

[ALL] Logs collection maintenance

For security reasons, we will update our logs collection systems.

The logs collection (logs drains too) will be unavailable during the maintenance.

EDIT 00:00 UTC: The maintenance is now over.

[PAR] FS Bucket maintenance on n19 and n20 servers

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n19 and n20.

The maintenance will take place on Friday 20 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-20 12:00 UTC+2: Maintenance is starting.

EDIT 2023-10-20 13:24 UTC+2: Applications are currently redeploying.

EDIT 2023-10-20 14:20 UTC+2: Applications have redeployed. We are cleaning things up.

EDIT 2023-10-20 16:12 UTC+2: The maintenance is over.

[PAR] FS Bucket maintenance on n10 and n17 servers

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n10 and n17.

The maintenance will take place on Thursday 19 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

[PAR] FS Bucket maintenance on n15 and n16 servers

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n15 and n16.

The maintenance will take place on Wednesday 18 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

[PAR] FS Bucket maintenance on n12 and n13 servers

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n12 and n13.

The maintenance will take place on Tuesday 17 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-17 12:10 UTC+2: The maintenance is starting. FSBucket servers are set in read-only mode.

EDIT 2023-10-17 12:47 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-17 16:15 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 14:00 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

[PAR] PHP FTP maintenance on n11 and n18 servers for PHP+FTP applications

For security reasons, we will update the PHP FTP services on the Paris (PAR) region used by PHP+FTP applications. PHP applications using the Git deployment method are not impacted by this maintenance.

PHP FTP hosts that will be updated during this maintenance are: n11 and n18.

The maintenance will take place on Monday 16 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your PHP+FTP applications in order to avoid waiting for the automatic redeployment.

Our support is available for any questions via the ticket center in the console. This maintenance will be updated during the maintenance window.

EDIT 2023-10-16 12:03 UTC+2: The maintenance will begin shortly. FSBucket add-on hosted on those servers will soon become read-only.

EDIT 2023-10-16 12:09 UTC+2: FSBuckets are now read-only

EDIT 2023-10-16 12:54 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-16 14:04 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 13:30 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

Tuesday 10th October 2023

Infrastructure [PAR] Hypervisor network unreachability

An hypervisor had its internal network unreachable between 15:16 and 15:20 UTC. During that time, services on the hypervisors may have suffered a total loss of network connectivity. Applications have been redeployed because of the unreachability. The issue has been fixed and we are monitoring the situation.

EDIT 15:49 UTC: All services are now reachable again, the incident is now over.

Monday 9th October 2023

Access Logs [Metrics/AccessLogs] Ingestion lag

We are detecting a data lag on our Metrics / AccessLog stack. You may not experience the latest datapoints. We are working on it.

EDIT 10:00 PM: lag has been fully absorbed