At 09:30 UTC, we received an alert from the monitoring about deployments not going through. We checked and saw that the orchestrator failed silently in a way that wasn’t detected by the service controller. We restarted the orchestrator and the deployments started to be handled again.
09:33 UTC: end of alert.
We will find a way to detect this kind of failure and handle it before the deployments get blocked.