Some systems are experiencing issues
Scheduled Maintenance
[PAR] Security maintenance on 4 hypervisors

For security reasons, we will update the kernel of 4 Hypervisors in the Paris (PAR) region, more precisely in the PAR6 datacenter. Services (in particular databases) hosted on those hypervisors will be impacted : they will be unavailable between 5 and 10 minutes. Impacted hypervisors are:

On Wednesday 20 November

  • hv-par6-012
  • hv-par6-020

On Thursday 21 November:

  • hv-par6-008
  • hv-par6-011

Affected clients are directly and individually contacted by email with the list of impacted services, and options to avoid any impact. The maintenance will be planned in 2 operations of 2 hypervisors each, during the week of 18 to 22 Novembre 2024 between 22:00 and 24:00 UTC+1.

Past Incidents

Monday 5th June 2023

No incidents reported

Sunday 4th June 2023

Infrastructure [RBX] lost connectivity with an hypervisor

We lost connectivity with an hypervisor on RBX. Applications have been redeployed but some databases may not be reachable. We are investigating.

EDIT 03:58 UTC: server is back online. All databases should now be reachable.

Saturday 3rd June 2023

No incidents reported

Friday 2nd June 2023

Access Logs Metrics/access logs storage layer issue

We are detecting some errors on our storage layer responsible for storing metrics and access logs data. We are investigating.

EDIT Lag has been catched up

Thursday 1st June 2023

Cellar [RBXHDS] Cellar Load Balancers partially unavailable

2023-06-01 16:20 UTC : During the RBXHDS incident, one of the Cellar LB lost its configuration. The configuration of each LB was not correctly monitored. Only the whole service availability was.

2023-06-02 09:15 UTC : after customer complaints we found out about the LB misconfiguration and fixed it.

2023-06-02 09:28 UTC : monitoring checks have been added to catch this kind of issues right away.

Infrastructure [RBXHDS] Load balancers metrics show abnormal response status code

Monitoring of load balancers is detecting an abnormal amount of http 404 status. We are investigating.

EDIT 17:51 UTC : We have found the issue and the fix is passed. Everything is operating normally

Customer support Ticket center availability issue

We are currently aware of an issue impacting our Ticket center service. This may impact our customers to open, view and reply to the tickets opened with our support team.

EDIT 13:30 UTC: Our ticket center provider told us that the issue has been mitigated on their end and that it is now resolved. We keep monitoring the situation for now but we can indeed see that service are operating normally those last few minutes.

EDIT 14:47 UTC: We did not see any other issues. We consider this incident to be over.

Wednesday 31st May 2023

Access Logs The metrics storage layer is unavailable

The monitoring detect errors on the metrics / access logs storage layer. We are investigating.

EDIT 11:46 UTC : We have found the issue and fixed it. We are recovering the lag.

EDIT 13:19 UTC: The lag has been consumed, everyhting is operating normaly

Tuesday 30th May 2023

No incidents reported