Mambu Status - API v1.0 - Posting deposit transactions with required custom fields returns inconsistent error

API v1.0 - Posting deposit transactions with required custom fields returns inconsistent error

Incident Report for Mambu

Postmortem

Summary

Mambu experienced an incident, where custom field data was lost when performing deposit transactions via API. The incident affected the API calls received between, Thursday Nov 29th 9:30am and Tuesday Dec 4th 6:45 pm UTC.

What Happened?

On Thursday, Nov 29th, 2018 at 9:30am UTC we released a patch with multiple improvements. Unfortunately, this build contained defects in the handling of custom fields for deposit transactions.

On Friday, Nov 30th it was noticed that deposit transactions API calls, with the custom fields that are marked required, were rejected. In the initial analysis , we suggested marking all custom fields as non-required, as a workaround to be able to execute Deposit Transactions, while the permanent fix was underway.

On Monday, Dec 3rd we have identified that all custom fields for deposit transactions posted via APIs were not saved. We started to work on the fix immediately well and by Dec 4th the fix was released to live environments..

On Monday, Dec 10th we have conducted the Root Causes analysis, that confirmed shortcomings in the release flow, due to which the defect was not captured. As the result, relevant automated tests were skipped, which would have normally caught this.

What Are We Doing About This?

In the effort to minimise the impact we have identified the transactions that might have been affected by the defect. A file containing a list of potentially impacted deposit transactions was provided to the affected tenants. We are working closely with each tenant to find best possible remediation to recover the possible lost Custom Field data.

The shortcoming in the release flow has been identified and fixed immediately, which will ensure a similar situation will not happen again and all the automated tests will be executed for each minor release. In addition to that, we are significantly improving Quality Assurance practices and controls, increasing automated tests coverage, as well as introducing Consumer-Driven Tests to capture customer-specific API usage scenarios and test all these scenarios before releasing the changes.

Furthermore, Mambu is also looking at improving the overall rollback process, which would allow us to immediately rollback a version that is causing problems. This would give us time to address the issue and re-deploy the release incident-free, without impacting the overall production flows and data while we are busy fixing the issue.

We assure you that, at Mambu, we take our commitment to deliver a high quality service very seriously and sincerely apologise if this has caused any inconvenience.

Posted Dec 11, 2018 - 17:07 UTC

Resolved

This incident has been resolved.

Posted Dec 07, 2018 - 15:31 UTC

Monitoring

We fixed the issue and deployed version v8.12 via an urgent minor release on the 4th of December 2018.

All deposit transactions with required custom fields for specific transaction channels can again be posted via API v1.0 and all non-required custom fields are stored in the database.

We are currently in the process of compiling a list of deposit transactions for affected tenants to remediate any data loss.

If you are an affected tenant, our Customer Support Team will reach out to you shortly.

For any questions and issues, please do not hesitate to reach out to support@mambu.com

Posted Dec 05, 2018 - 10:20 UTC

Update

We identified an issue that is impacting deposit transactions custom fields posted with API v1.0 if the custom fields are marked as not required - the transaction is posted but without any custom fields being stored in the database.

Our development team has implemented a fix and is currently doing all the preparatory steps for the release, such as merging and testing the solution. We expect by end of day to release into production, if not latest tomorrow morning.

In the meantime, as a temporary workaround, we suggest performing a second PATCH request after posting Deposit transaction to add the custom fields values.

We are currently investigating the extent of any data loss. We have compiled a list of affected tenants, which will be contacted privately in order to perform remediation steps.

We apologise for any inconvenience this might have caused. If you have question or issues, feel free to contact us via our usual support channels.

Posted Dec 04, 2018 - 15:28 UTC

Identified

Mambu has identified an issue impacting deposit transaction posting with required custom fields via API v1.0. We are already working towards finding a solution as quickly as possible, but in the meantime recommend modifying the configuration of your required custom fields to not required. This will ensure your deposit transactions are posted successfully.

Please note, this issue only concerns required custom fields for transaction channels used to POST deposit transactions. Other fields or channels should not be impacted.

We are treating this with the highest priority and apologise for any inconvenience caused. If you have any questions, please don’t hesitate to contact support@mambu.com.

Posted Nov 30, 2018 - 15:38 UTC

This incident affected: Mambu Production (N. Virginia, USA), Mambu Production (Dublin, Ireland), Mambu Production (Singapore, Asia), Mambu Production (Sydney, Australia), Mambu Sandbox (N. Virginia, USA), Mambu Sandbox (Dublin, Ireland), Mambu Sandbox (Singapore, Asia), and Mambu Sandbox (Sydney, Australia).