Problem with processing notifications for the North Virginia, US Region
Incident Report for Mambu

Post Mortem - Problem with processing notifications for the North Virginia, US Region


On November 13, 2019 between 050:00 EST and 09:38 EST, notifications were queuing up ready to be sent in production environment in N. Virginia, USA. During the incident, 180 notifications were delayed up to 4 hr 38 min before being sent. Mambu restarted the notification service at 09:38 EST which caused the queued notifications to be sent out and which resolved the incident.

What Happened?

The process responsible for sending notifications became blocked and did not respond to new notifications queued. Restarting the service unblocked the process and allowed notifications to be sent.

What Are We Doing About This?

We will implement additional monitoring for notifications service in order to improve response time.

We assure you that at Mambu we take our commitment to deliver a high quality service very seriously and sincerely apologize if this has caused any inconvenience.

Posted Dec 06, 2019 - 11:34 UTC

Mambu has determined that the fix was effective and full functionality has been restored.

We apologise for the inconvenience and will take the appropriate actions to prevent future incidents of this nature. As always, if you have any questions or concerns, feel free to contact us via the usual support channels
Posted Nov 13, 2019 - 15:26 UTC
Mambu has implemented the fix and is monitoring the effectiveness of the solution. Full functionality should now be restored; please contact us at if you experience any further issues. We will update you when we have determined that the situation is fully resolved.
Posted Nov 13, 2019 - 14:25 UTC
Mambu has identified the root cause of the incident and we are currently working on a fix in order to restart processing notifications.
Posted Nov 13, 2019 - 10:01 UTC
Mambu has become aware of a situation affecting notifications. We are currently investigating the root cause and will update you when have identified it.
Posted Nov 13, 2019 - 09:02 UTC
This incident affected: Mambu Production (N. Virginia, USA).