Resolved -
Following the recent Lustre-related issues, we decided to further improve the quality of our monitoring infrastructure by installing additional hardware in our cluster.
Unfortunately, we underestimated the additional power requirements of the new nodes. This resulted in an overload of one of our uninterruptible power supplies (UPS), which also provides power to parts of our service, login, and storage nodes. As a result, these systems became unavailable, causing an interruption to our services.
Because the outage affected both the service and storage infrastructure, we recommend reviewing any jobs that were running or pending at the time of the incident. Depending on their state, some jobs may need to be resubmitted.
To prevent a similar incident in the future, we have already rewired part of our infrastructure to better distribute the power load and eliminate this single point of failure.
We sincerely apologize for the disruption and any inconvenience this incident may have caused. Thank you for your patience and understanding.
Jul 16, 18:14 CEST
Investigating -
we are currently experiencing issues with our service, login and storage nodes, which are affecting job processing, login attempts and running/pending jobs.
At this time, the root cause has not yet been identified. We are actively investigating the issue and working to restore normal service as quickly as possible.
We apologize for the disruption and appreciate your patience while we work to resolve this. We will provide another update as soon as we have more information or when the service has been fully restored.
Thank you for your understanding.
Jul 16, 16:44 CEST