Some systems are experiencing issues

Core

Infrastructure

Operational

API

Operational

Git repositories

Operational

Deployments

Operational

Reverse Proxies

Operational

Logs

Drains

Partial Outage

Services Logs

Operational

Access Logs

Operational

Products

Cellar

Operational

SSH Gateway

Operational

VPN

Operational

MySQL shared cluster

Operational

Let's Encrypt certificate generation

Operational

MongoDB shared cluster

Performance Issues

Unique IP for outgoing traffic

Operational

RabbitMQ shared cluster

Operational

FS Buckets

Performance Issues

Console

Operational

PostgreSQL shared cluster

Operational

Elasticsearch

Operational

Customer support

Operational

Redis

Operational

Pulsar

Operational

Matomo add-on

Operational

PostgreSQL

Performance Issues

MySQL

Operational

Metrics

Operational

Heptapod Cloud

Operational

Other Components
cleverapps.io domains
Operational
Mails
Operational

Scheduled Maintenance

[PAR] Security maintenance on 4 hypervisors

For security reasons, we will update the kernel of 4 Hypervisors in the Paris (PAR) region, more precisely in the PAR6 datacenter. Services (in particular databases) hosted on those hypervisors will be impacted : they will be unavailable between 5 and 10 minutes. Impacted hypervisors are:

hv-par6-008 hv-par6-011 hv-par6-012 hv-par6-020

Affected clients are directly and individually contacted by email with the list of impacted services, and options to avoid any impact. The maintenance will be planned in 2 operations of 2 hypervisors each, during the week of 18 to 22 Novembre 2024 between 22:00 and 24:00 UTC+1.

Past Incidents

Thursday 26th September 2024

No incidents reported

Wednesday 25th September 2024

Infrastructure Platform instability

We have detected some latency and a few instabilities to connect to our platform, we are investigating.

EDIT 17h40 - Root cause has been identified, and network is now stabilized. We are closely monitoring the platform to be sure this incident is closed

Tuesday 24th September 2024

No incidents reported

Monday 23rd September 2024

Services Logs Reading logs is experiencing issues

We detected an issue on log reads.

EDIT 13:00 UTC: identified and patched. We are currently deploying the fix.

EDIT 13:15 UTC: fixed.

Sunday 22nd September 2024

No incidents reported

Saturday 21st September 2024

No incidents reported

Friday 20th September 2024

Pulsar Pulsar instability

We are experiencing a pulsar outage, which impacts logs and access logs and other components of the platform. Preliminary root cause seems like a zookeeper problem. We are working on it.

EDIT Fri Sep 20 18:16:00 2024 UTC Deployments have been disabled. We are still investigating the Zookeeper outage, causing Pulsar outage.

EDIT Fri Sep 20 19:57:09 2024 UTC: The zookeeper quorum is back online, and therefore Pulsar. Deployments have been enabled, we are watching the situation.

EDIT Fri Sep 20 22:09:40 2024 UTC: Pulsar cluster is still unstable, deployment have been disabled.

EDIT Fri Sep 20 23:36:10 2024 UTC: Deployments queue is back, we are ramping up logs's data usage to avoid bursting Pulsar too much.

EDIT Sat Sep 21 00:49:10 2024 UTC: Pulsar cluster is now stable. Applications should now have their logs available in the console / CLI as well as the drains. Access logs lag is currently catching up. We continue to monitor the situation.