Some systems are experiencing issues

Past Incidents

Thursday 7th March 2024

[PAR Scaleway] Load balancer maintenance, scheduled 9 months ago

Maintenance Window: 2024-03-07T13:00:00Z - 2024-03-07T17:00:00Z (UTC)

Scope:

  • Database Load Balancer (software and hardware upgrade)
  • Application Load Balancer (software and hardware upgrade)

Expected Impact:

  • Brief disconnections or connection drops during the upgrade process.
  • Potential minor performance fluctuations.

Additional Information:

  • Software upgrade already record on cleverapps.io (https://www.clevercloudstatus.com/incident/803) and on Paris load balancers (https://www.clevercloudstatus.com/incident/807 and https://www.clevercloudstatus.com/incident/805)
  • Please report any issues with a method for reproducing the problem (e.g., curl command for application load balancer issues and/or psql / redis / mysql queries for database load balancer).

EDIT : We will do the maintenance next week on 2024-03-07.

EDIT 17:30 UTC : We have finished to deploy the load balancers alongside the current ones. We will switch the traffic from the old one to the new one.

EDIT 17:32 UTC : We have switch the traffic from old to new database load balancer with a unexpected behavior which is fixed now. We may have seen unexpected connections refused.

EDIT 17:35 UTC : We will begin to switch the application load balancer soon.

EDIT 17:50 UTC : We have switched the first instance of application load balancer.

EDIT 18:00 UTC : We have finished to roll the application load balancer. We are watching.

Friday 23rd February 2024

[MTL] Load balancer maintenance, scheduled 10 months ago

Maintenance Window: 2024-02-23T13:00:00Z - 2024-02-23T17:00:00Z (UTC)

Scope:

  • Database Load Balancer (software and hardware upgrade)
  • Application Load Balancer (software and hardware upgrade)

Expected Impact:

  • Brief disconnections or connection drops during the upgrade process.
  • Potential minor performance fluctuations.

Additional Information:

  • Software upgrade already record on cleverapps.io (https://www.clevercloudstatus.com/incident/803) and on Paris load balancers (https://www.clevercloudstatus.com/incident/807 and https://www.clevercloudstatus.com/incident/805)
  • Please report any issues with a method for reproducing the problem (e.g., curl command for application load balancer issues and/or psql / redis / mysql queries for database load balancer).

EDIT : We move the maintenance to 2024-02-23 instead of 2024-02-22.

EDIT 15:00 UTC: We are beginning the maintenance alongside the current installation. We will do the failover next week.

EDIT 2024-02-26 16:30 UTC : We have added two new IP address to domain.mtl.clever-cloud.com. DNS records.

EDIT 2024-02-26 17:00 UTC : We have removed the two old IP address from domain.mtl.clever-cloud.com. DNS records.

EDIT 2024-02-26 17:15 UTC : We will update the DNS records for database load balancers.

EDIT 2024-02-26 17:30 UTC : We have updated the DNS records for database load balancers, we are watching

Tuesday 20th February 2024

[WSW] Load balancer maintenance, scheduled 10 months ago

Maintenance Window: 2024-02-20T13:00:00Z - 2024-02-20T17:00:00Z (UTC)

Scope:

  • Database Load Balancer (software and hardware upgrade)
  • Application Load Balancer (software and hardware upgrade)

Expected Impact:

  • Brief disconnections or connection drops during the upgrade process.
  • Potential minor performance fluctuations.

Additional Information:

  • Software upgrade already record on cleverapps.io (https://www.clevercloudstatus.com/incident/803) and on Paris load balancers (https://www.clevercloudstatus.com/incident/807 and https://www.clevercloudstatus.com/incident/805)
  • Please report any issues with a method for reproducing the problem (e.g., curl command for application load balancer issues and/or psql / redis / mysql queries for database load balancer).

EDIT 13:30 UTC : We are preparing the hardware and software upgrade along-side the current stack.

Thursday 15th February 2024

[SGP] Load balancer maintenance, scheduled 10 months ago

Maintenance Window: 2024-02-15T13:00:00Z - 2024-02-15T17:00:00Z (UTC)

Scope:

  • Database Load Balancer (software and hardware upgrade)
  • Application Load Balancer (software and hardware upgrade)

Expected Impact:

  • Brief disconnections or connection drops during the upgrade process.
  • Potential minor performance fluctuations.

Additional Information:

  • Software upgrade already record on cleverapps.io (https://www.clevercloudstatus.com/incident/803) and on Paris load balancers (https://www.clevercloudstatus.com/incident/807 and https://www.clevercloudstatus.com/incident/805)
  • Please report any issues with a method for reproducing the problem (e.g., curl command for application load balancer issues and/or psql / redis / mysql queries for database load balancer).

EDIT 13:30 UTC : We are starting the preparation to do the upgrades

EDIT 13:50 UTC : We have finished the preparation, we are beginning the rolling of the application load balancer

EDIT 14:20 UTC : We have finished to do the rolling of the application load balancer

EDIT 14:30 UTC : We are starting to roll the database load balancer

EDIT 14:35 UTC : We have finished to roll the database load balancer.

Tuesday 13th February 2024

[SYD] Load balancer maintenance, scheduled 10 months ago

Maintenance Window: 2024-02-13T13:00:00Z - 2024-02-13T17:00:00Z (UTC)

Scope:

  • Database Load Balancer (software and hardware upgrade)
  • Application Load Balancer (software and hardware upgrade)

Expected Impact:

  • Brief disconnections or connection drops during the upgrade process.
  • Potential minor performance fluctuations.

Additional Information:

  • Software upgrade already record on cleverapps.io (https://www.clevercloudstatus.com/incident/803) and on Paris load balancers (https://www.clevercloudstatus.com/incident/807 and https://www.clevercloudstatus.com/incident/805)
  • Please report any issues with a method for reproducing the problem (e.g., curl command for application load balancer issues and/or psql / redis / mysql queries for database load balancer).

EDIT 13:15 UTC : We are beginning the hardware upgrade along side the current hardware.

EDIT 14:20 UTC : We have finished the hardware upgrade, we will start the rolling by application load balancer.

EDIT 14:35 UTC : We have rolled the first application load balancer, we are beginning the second one.

EDIT 15:00 UTC : We have finished to roll the application load balancer, we are beginning the database load balancer.

EDIT 15:15 UTC : We have rolled the first database load balancer, we are watching.

EDIT 15:25 UTC : We have rolled the second database load balancer.

EDIT 15: 25 UTC : We have rolled all load balancers, we are keeping an eye on them, but the maintenance is over.

Monday 12th February 2024

No incidents reported

Sunday 11th February 2024

No incidents reported

Saturday 10th February 2024

No incidents reported

Friday 9th February 2024

No incidents reported

Thursday 8th February 2024

No incidents reported

Wednesday 7th February 2024

[Heptapod Cloud] Planned upgrade to 1.0, scheduled 10 months ago

We are planning an upgrade of our Heptapod Cloud offer heptapod.host on Wednesday 2024-02-07 at 14:00 UTC. Heptapod will be updated to the 1.0 version.

Expected downtime of the service is 30 minutes. During that time, git and mercurial operations might fail as well as loading the UI.

EDIT 16:45 UTC: The update is over.

Tuesday 6th February 2024

Reverse Proxies [PAR] Load balancer maintenance

We will proceed to software upgrade of the database load balancer which should be transparent. You may observed a few connection cut during the operation. If you have an issue during this maintenance, please contact the support with a way to reproduce the issue (a curl command or a psql / mysql / redis example will be great).

EDIT 10:15 UTC : We have started the maintenance

EDIT 11:15 UTC : We have finished the maintenance