On 2020-01-14 webhook notifications were not sent in production environment in Dublin, Ireland between 03:11 and 09:29 GMT. The notifications were being queued up, causing delay to tenants' business processes that rely on these notifications. Mambu restarted the notifications process at 09:18 GMT which resolved the issue and confirmed that the system returned to normal by 10:02 GMT.
The process responsible for sending notifications became blocked and did not respond to new notifications queued. Restarting the service unblocked the process and allowed notifications to be sent.
We will implement additional monitoring for notifications service in order to improve response time. We will also add a feature to automatically skip batches of notifications that become blocked and requeue them for sending. This feature will make the notification service more resilient to errors and minimize the impact if this situation should arise again.
We assure you that at Mambu we take our commitment to deliver a high quality service very seriously. We apologize for the inconvenience and will take the appropriate actions to prevent future incidents of this nature. As always, if you have any questions or concerns, feel free to contact us via the usual support channels.