Pulsara experienced push notification issues for some Android users due to an error with Google's Firebase Messaging System.
Here's the full details in chronological order. All times are US Mountain.
---
Updated 4:50 PM 5/16/17
Pulsara engineers have updated the Google messaging key from the legacy server key, to a the FCM key per Google’s recommendation, verified the change in test environments, and finally rolled this change out to Production.
This issue is fully resolved to the best of our knowledge. With the additional changes to our update our messaging token, we expect that operations are fully restored, and push notifications to all Android users will be sent and delivered as normal going forward. We will keep monitoring server logs this evening to keep an eye on things.
---
Updated 3:22 PM 5/16/17
Google has posted an update that the outage is affecting customers using legacy GCM keys. Pulsara Engineering is initiating steps to upgrade our GCM keys to FCM keys, per Google’s recommendation.
---
Updated 3:02 PM 5/16/17
Google has officially acknowledged the service disruption, and has posted updated (and backdated) status at the Firebase status console at https://status.firebase.google.com
Apologies for the service disruption and for not updating the Firebase status earlier. The rate of errors has been small enough that we didn't believe was having an impact on FCM users. Clearly some developers are being affected disproportionately. We updated the Firebase status at this page: https://status.firebase.google.com/incident/Cloud%20Messaging/17005
---
Updated 2:51 PM 5/16/17
Pulsara Engineering is continuing to monitor the issue.
---
Updated 1:49 PM 5/16/17
Pulsara has released hotfix Server 44.2 to production, which will resolve the errors being seen in our mobile applications, by continuing past any sporadic errors from Google FCM.
NOTE: some Android users will continue to have intermittent push notification delivery until Google resolves their issues.
---
Updated 12:36 AM 5/16/17
ETA for build complete is 1:20 PM. ETA for hotfix deployed to production is about 2 PM.
---
Updated 12:35 AM 5/16/17
Google has a Google FCM service outage on public message boards. However, they have not yet updated the Firebase status console at https://status.firebase.google.com
---
Updated 12:10 AM 5/16/17
We have confirmed on developer message boards that we are not the only company to be experiencing this issue with Google messaging services.
---
Updated 11:43 AM 5/16/17
Next Steps: Pulsara Engineering is working on a code change that will stop fatal errors from happening when our connection to Google FCM fails. After confirming the fix, we will roll out the changes to test stacks (environments), and then to Production.
---
Updated 10:54 AM 5/16/17
Errors from Google FCM have started occurring again. Our engineering team is continuing to monitor the issue.
---
Updated 10:27 AM 5/16/17
Issues at Google FCM appear to be resolved. We have not observed an error in 20+ minutes.
---
Updated 9:51 AM 5/16/17
We are currently experiencing intermittent errors connecting to Google Firebase Cloud Messaging (FCM). Pulsara Engineering is investigating the root cause. On an intermittent basis, affected users will not receive push notifications and other mobile application users will see errors when performing operations that send push notifications as a side effect. More info will be posted here when available.