Past Incidents

We are currently experiencing unreachable hypervisors on the JED region. We are investigating the issue.

EDIT 18:50 UTC: The hypervisors are back online since 25 minutes now, all services were restarted by our monitoring.

We've seen network instabilities on the PAR region. It is currently resolved but we are still investigating the root cause.

EDIT 06:34 UTC: The problem is back with elevated packet loss. Our network provider is currently having an incident and is looking into the issue.

EDIT 06:46 UTC: Some DNS domains for services hosted on other regions may also have issues to resolve because their authoritative server is currently hosted on the PAR region.

EDIT 06:55 UTC: The incident is still ongoing and our network provider is still looking into the issue.

EDIT 07:20 UTC: Our upstream network provider is currently experiencing a DDoS attack. We are currently looking to use an alternative network transit to avoid going through the upstream network provider.

EDIT 07:47 UTC: We are seeing improvements for the last 20 minutes. We still are waiting for a confirmation of the issue resolution.

EDIT 07:58 UTC: We are seeing some loss again.

EDIT 08:15 UTC: The DDoS is still happening. It's partially mitigated. We still see some loss, but there is less impact globally.

EDIT 10:54 UTC: We still see loss from time to time, but much less that before. We keep an eye on the situation.

EDIT 15:45 UTC: Most of the ddos is mitigated, we didn't have any loss those past few hours, we still monitor the situation.

EDIT 2023-09-06 15:24 UTC: No more instabilities were detected since yesterday. The incident is now over.

When ordering a new database it can take time to reach them (databases are correctly created)

[EDIT] 21:38 UTC the root cause was identified and a path deployed

Some websites went unreachable by external monitoring. This indicates reverse proxies are not taking connections as they should.

Metrics on the proxies seem ok. We are investigating why they are acting like that.

Seems some applications where causing connections to enqueue and blocking new connections. We are looking into ways to avoid this to happen.

The issue is resolved