{"id":155993,"date":"2024-04-24T14:58:26","date_gmt":"2024-04-24T14:58:26","guid":{"rendered":"https:\/\/monday.com\/blog\/?p=155993"},"modified":"2024-04-24T14:58:26","modified_gmt":"2024-04-24T14:58:26","slug":"core-service-update","status":"publish","type":"post","link":"https:\/\/monday.com\/blog\/product\/core-service-update\/","title":{"rendered":"Core service update"},"content":{"rendered":"

To our monday.com community,<\/h2>\n

On April 11th and 16th, we experienced some service interruptions on our US server that may have impacted your account. We want to share a more detailed update on what happened and how we plan to improve your experience moving forward.<\/p>\n

The incidents resulted in a total of 59 minutes of system downtime and 62 minutes of degraded performance across the two dates. There was no data loss or security risk and EU and APAC servers were not affected.<\/p>\n

We know that you rely on our platform and that any disruption to the platform impacts your workflow and for that we deeply apologize. We take these interruptions very seriously and we\u2019ll continue to invest in preventing them from happening again.<\/p>\n

A more detailed update follows below on exactly what happened and our plans for improving your experience moving forward.<\/p>\n

What happened<\/h2>\n

Thursday, April 11th – Status page retro<\/span><\/a><\/h3>\n

The root cause of the incident on April 11th was a rare service issue in one of our service provider\u2019s infrastructure, which resulted in 20 minutes of downtime.<\/p>\n

Our monitoring system alerted our R&D teams to the issue, and we subsequently began moving the system to a backup. Some technical difficulties in doing so caused system instability and read-only mode for another 62 minutes.<\/p>\n

At this stage, we\u2019ve already updated our system readiness processes and fine-tuned our backup servers, enabling us to recover faster and reduce interruptions.<\/p>\n

Total downtime on the US server: 20 minutes<\/p>\n

Total time of degraded performance on the US server: 62 minutes<\/p>\n

No service interruption to EU and APAC servers<\/p>\n

Tuesday, April 16th – Status page retro<\/span><\/a><\/h3>\n

In the early hours of Tuesday, we experienced a 20-minute platform downtime. Our monitors detected the issue, and our teams worked to reboot the platform and begin investigating the cause.<\/p>\n

During the investigation, a second issue occurred, causing fluctuations in the platform\u2019s availability across one hour, totaling an additional 19 minutes of downtime.<\/p>\n

Our investigations confirmed that the issue was caused by a recently deployed monitoring service designed to flag large or complex queries. While the new monitoring service had initially been running smoothly within the platform, it was identified as the cause of the infrastructure lock, and we promptly reverted to an older version of the service that had been operating for many months without any issues.<\/p>\n

Total downtime on the US server: 39 minutes<\/p>\n

No service interruption to EU and APAC servers<\/p>\n

What\u2019s next<\/h2>\n

Prevention and faster recovery<\/h3>\n

We continue to take a proactive approach to ensuring platform stability, with both immediate action items and long-term plans in place to reduce risk, improve our ability to recover quickly, and, most importantly, prevent platform instability through enhanced resilience.<\/p>\n

Prevention & reduced risk:<\/p>\n