Notifications are stuck in WAITING state.
Incident Report for Mambu

Problem with processing notifications for the Dublin, Ireland Region


On 2020-01-14 webhook notifications were not sent in production environment in Dublin, Ireland between 03:11 and 09:29 GMT. The notifications were being queued up, causing delay to tenants' business processes that rely on these notifications. Mambu restarted the notifications process at 09:18 GMT which resolved the issue and confirmed that the system returned to normal by 10:02 GMT.

What Happened?

The process responsible for sending notifications became blocked and did not respond to new notifications queued. Restarting the service unblocked the process and allowed notifications to be sent.

What Are We Doing About This?

We will implement additional monitoring for notifications service in order to improve response time. We will also add a feature to automatically skip batches of notifications that become blocked and requeue them for sending. This feature will make the notification service more resilient to errors and minimize the impact if this situation should arise again.

We assure you that at Mambu we take our commitment to deliver a high quality service very seriously. We apologize for the inconvenience and will take the appropriate actions to prevent future incidents of this nature. As always, if you have any questions or concerns, feel free to contact us via the usual support channels.

Posted Jan 27, 2020 - 15:32 UTC

Mambu has determined that the fix was effective and full functionality for notifications has been restored.
Posted Jan 14, 2020 - 10:00 UTC
Mambu has identified the root cause of the incident and is resolving the issue.
Posted Jan 14, 2020 - 09:49 UTC
Mambu has become aware of a situation affecting Notifications. Users may experience issues with notifications being stuck in WAITING state. We are currently investigating the root cause and will update you when have identified it.
Posted Jan 14, 2020 - 09:14 UTC
This incident affected: Mambu Production 2 (Dublin, Ireland).