Root Cause

During planned maintenance to remove an unused service instance, the AWS Auto Scaling Group (ASG) determined that our production messaging service instances did not match its specified configuration and terminated them. The loss of those instances caused upstream services to behave badly, resulting in cascading failure.

Impact

Sense REST APIs were unavailable from approximately 4:24pm to 5:20pm, disabling most client application functionality.

Because monitors buffer and retry data uploads, no data should have been lost during this period.

Improvements

1. Ensure that termination protection is enabled on critical instances.
2. Improve process around ASG changes.
3. Improve resiliency of upstream services to unavailability of the messaging service.