Some systems are experiencing issues
Scheduled Maintenance
[PAR] Security maintenance on 4 hypervisors

For security reasons, we will update the kernel of 4 Hypervisors in the Paris (PAR) region, more precisely in the PAR6 datacenter. Services (in particular databases) hosted on those hypervisors will be impacted : they will be unavailable between 5 and 10 minutes. Impacted hypervisors are:

hv-par6-008 hv-par6-011 hv-par6-012 hv-par6-020

Affected clients are directly and individually contacted by email with the list of impacted services, and options to avoid any impact. The maintenance will be planned in 2 operations of 2 hypervisors each, during the week of 18 to 22 Novembre 2024 between 22:00 and 24:00 UTC+1.

Past Incidents

Wednesday 24th April 2024

Heptapod Cloud Heptapod: Email notifications failures

Some emails issued by the heptapod service weren't correctly delivered to their recipients the last few days. The underlying issue has been fixed and the mail backlog is currently being processed. Additional monitoring will be put in place to monitor the email queue.

We will update this incident once the backlog is fully processed.

EDIT 2024-04-25 16:00 UTC: The backlog has been fully ingested. The incident is now over.

Tuesday 23rd April 2024

No incidents reported

Monday 22nd April 2024

Metrics [Global] Metrics infrastructure improvement

An operation on the metric cluster is pending which will make it more resilient to spikes and load. It shouldn't impact read queries of metrics, it can generate lag in the writing path.

EDIT UTC 18:29 : Operation is done, services weren't disturbed.

Sunday 21st April 2024

Access Logs [Global] Access logs ingestion issue

Beginning at 5h00 UTC, we seen a drop in the rate of access logs consumption which seems to be caused to difficulty to produce them. We are investigating the issue. You may see delays to retrieve your access logs.

EDIT 10:30 UTC : We are performing a rolling restart of the underlying pulsar brokers, you may seen disconnection.

EDIT 16:00 UTC : The rolling restart is performed. We still have ingestion issues we will keep investigating

EDIT D+1 08:50 UTC : We have still ingestion issues on few partitions which may be related to an underlying trouble, we are digging into it.

EDIT D+2 14:00 UTC : We have found the underlying issue and solve it, we are consuming the remaining lags.

EDIT D+3 13:00 UTC : We are still consuming the remaining lags, the current eta of full recovery is targeting tomorrow during the night

EDIT D+4 06:00 UTC : We have done consuming the remaining lag.

Saturday 20th April 2024

No incidents reported

Friday 19th April 2024

No incidents reported

Thursday 18th April 2024

Mails Platform email services delay

We are currently experiencing a disruption in our email services due to an unforeseen issue, emails will be delayed until this issue is resolved. Our team is actively working to restore access as quickly as possible. We will keep you updated on our progress and notify you as soon as services are fully operational again.

EDIT 20:04 UTC+2: We are still working on the issue.

EDIT 2024-04-19 12:17 UTC+2: The issue has been fixed, we continue to monitor the situation.