Problem with processing notifications for the North Virginia, US Region

Incident Report for Mambu

Postmortem

Post Mortem - Problem with processing notifications for the North Virginia, US Region

Summary

On November 13, 2019 between 050:00 EST and 09:38 EST, notifications were queuing up ready to be sent in production environment in N. Virginia, USA. During the incident, 180 notifications were delayed up to 4 hr 38 min before being sent. Mambu restarted the notification service at 09:38 EST which caused the queued notifications to be sent out and which resolved the incident.

What Happened?

The process responsible for sending notifications became blocked and did not respond to new notifications queued. Restarting the service unblocked the process and allowed notifications to be sent.

What Are We Doing About This?

We will implement additional monitoring for notifications service in order to improve response time.

We assure you that at Mambu we take our commitment to deliver a high quality service very seriously and sincerely apologize if this has caused any inconvenience.

Posted Dec 06, 2019 - 11:34 UTC

Resolved

Mambu has determined that the fix was effective and full functionality has been restored.

We apologise for the inconvenience and will take the appropriate actions to prevent future incidents of this nature. As always, if you have any questions or concerns, feel free to contact us via the usual support channels

Posted Nov 13, 2019 - 15:26 UTC

Monitoring

Mambu has implemented the fix and is monitoring the effectiveness of the solution. Full functionality should now be restored; please contact us at support@mambu.com if you experience any further issues. We will update you when we have determined that the situation is fully resolved.

Posted Nov 13, 2019 - 14:25 UTC

Identified

Mambu has identified the root cause of the incident and we are currently working on a fix in order to restart processing notifications.

Posted Nov 13, 2019 - 10:01 UTC

Investigating

Mambu has become aware of a situation affecting notifications. We are currently investigating the root cause and will update you when have identified it.

Posted Nov 13, 2019 - 09:02 UTC

This incident affected: Mambu Production (N. Virginia, USA).