Some systems are experiencing issues

Past Incidents

Monday 9th October 2023

Access Logs [Metrics/AccessLogs] Ingestion lag

We are detecting a data lag on our Metrics / AccessLog stack. You may not experience the latest datapoints. We are working on it.

EDIT 10:00 PM: lag has been fully absorbed

Sunday 8th October 2023

No incidents reported

Saturday 7th October 2023

Infrastructure A Hypervisor is not responding

One hypervisor is unreachable. We are investigating it.

It may impact some databases that are hosted on top of this hypervisor.

06:20 The hypervisor seems to have encountered a kernel panic. It has been rebooted and we fixed the kernel version to avoid future Kernel Panic.

06:45 Now that the hypervisor is back up, we are cleaning the situation: checking all add-ons instances have rebooted successfully, that all applications have redeployed successfully.

07:21 Everything is now back to normal

Migrating parts of our deployment stack, scheduled 11 months ago

Starting 2023-10-06 23:00 UTC, we will conduct a maintenance operation on one of the key parts of the deployment system. With the security margins, we expect it to take 2 hours.

As a result, deployments on all our zones will be disabled between 11:00 and 01:00 (2023-10-07).

EDIT 2023-10-07 01:32 UTC: The maintenance is over. Deployments are now working again.

Friday 6th October 2023

No incidents reported

Thursday 5th October 2023

No incidents reported

Wednesday 4th October 2023

No incidents reported

Tuesday 3rd October 2023

Infrastructure [PARIS] Network unreachability

Part of our network infrastructure in Paris is unreachable. We are looking into it.

09:25 PM UTC: Network is back online. We are bringing back services which are not healthy.

09:30 PM UTC: Network is still flappy, we are on it.

10:06 PM UTC: Network seems stable. We are bringing back services which are not healthy.

10:37 PM UTC: We are bringing back services which are not healthy.

10:52 PM UTC: all services are back online. Good night.

Deployments New deployments may not be triggered on git push

Some pushes are not starting a new deployments. We are investigating.

EDIT 09:06 UTC: We implemented a fix and are monitoring the results. If you pushed new commits that didn't get deployed, you can either contact us through the support with your application id and the associated commit, or use our CLI with clever restart --commit <commit>.

EDIT 12:25 UTC: The incident is now resolved.

SSH Gateway planned unavailability, scheduled 11 months ago

The SSH Gateway will be unavailable on October 3rd, 2023 starting at 20:00 UTC. During that time, SSH access to services using either ssh -t [email protected] or using our CLI command clever ssh will be unavailable. Existing SSH connections through the gateway to services will be interrupted.

Once the maintenance is over, it is possible that some applications will need to be restarted to be able to be accessed through the SSH Gateway again.

The maintenance is planned to last less than 30 minutes.

EDIT 20:05 UTC: The maintenance is starting.

EDIT 20:27 UTC: The maintenance is now over. We are monitoring the results. You should now be able to access your services using the SSH gateway.

EDIT 20:27 UTC: Everything is working as intended. If you have any issues using the SSH gateway, you can try to redeploy your service and contact our support team.