Prisoners of Google Android Development

Update #1

I submitted this article to Hacker News on 27th of August. It sparked some pretty interesting discussions and release strategies for Android among victim blaming and ended up on the first page of HN so we also managed to experience Hacker News effect - this brought around 130k unique visitors on this article during a few day period and caused no problems on our ~5EUR VPS because it is a static site. However, it did not get us any attention from anyone from Google and app stayed "in review".

Update #2

On 1st of September I accidentally found out that there is a possibility to start a chat session with Google Support hidden behind the question mark icon at the Google Play Console. However, that chat session, did not end up too promising even though support person promised that this review will be expedited. I'm surprised that not a single person in the Hacker News thread didn't bring out a possibility to even contact support - Google has been famous for it's non-existent and/or bad support, maybe that's the reason.

Update #3

On 4th of September, after being "in review" for 11 days, update was finally published to production. I created another version update with smaller fixes using staged rollout strategy instead of 100% rollout. It was "in review" for less than an hour before being published.

Update #4

On 7th of September another small update was sent to "in review" to replace previous version which was rolled out to 99.9999% of users. Another surprising blowback occurred - as soon as new version was sent to "in review", the previous version with 99.9999% rollout percentage was halted automatically without any warning from Google. This meant that there was a chance that update would be "in review" for a longer period, but Google pulled back previous release immediately. Taking update back from review did not allow to restore previous version. Fortunately this update was "in review" for less than an hour.

We have been in charge of maintaining one legacy Android app for our customer. It is an app, which is used by end-customers in production, but it does not have any active development going on because it's been ready for years now. If it would be up to us, then we would not touch that app and would let it live its life happily ever after.

The Beginning

It all started with the following innocent-looking e-mail by Google received at 18th of August 2023:

Trying to find more information about this requirement I found the following on Google's website:

The following sentence caught our eye: Existing apps must target API level 31 or above to remain available to users on devices running Android OS higher than your app's target API level. Reading this it's hard to understand what will happen exactly to the apps installed on newer Android devices targeting lower API levels. I took the approach of better safe than sorry and prioritized this task even though it would add no business value — it was only required to be completed because Google said so. And the deadline set by Google was less than 3 weeks from the time when e-mail was received. No other e-mails have been sent by Google before regarding this issue.

Implementing Change

I started with this task on 23th of August. I updated targetSdkVersion from API level 30 to 33 and tried to compile/run the application in Android emulator. First run failed due to incompatible dependency. Fortunately I could just remove that dependency since it was related to analytics and not tightly coupled to the business logic itself so it was not that critical one.

Ran the application successfully and tried core functionality which seemed to work the same way as before without any problems. It was time to put it up into Google Play Store.

Play Store

Putting application up into Play Store went mostly without any hiccups. Of course, since this is a legacy app not having releases happening often, then I needed to fill some questionnaires created by Google, but this is pretty normal thing to happen when apps are updated at most once or twice per year. After completing these, app was sent to review. It took less than an hour until app was reviewed, accepted and released to production. Little did I know what would happen next. It all happened later in the evening after working hours.

Problems Ahead

Around 21:30 my phone received a message from our customer who had problems logging into their account using newest application version. At first I was not that alarmed because it didn't seem to be related anyhow to the app update. However, I double-checked first time with a physical Android device (did not have access to that before) to see if I can successfully log in. Right after logging in, application crashed and closed. At that moment chills started to move down my spine and I started to investigate the experienced problem in more detail.

After some troubleshooting it was pretty clear that there exists a problem with newest Android (13 at that moment), which will crash the application right after logging in. Using older Android versions was okay. One of our problems were that I did not use newest Android version in emulator while testing, thus these problems slipped through. It's not unheard of that something will break when updating, but this time there was a time-pressure set by Google and changes were minimal making it less expected. I could have done better by using different Android emulator versions, but it was already too late.

Solving the Problem

First idea was to roll back to the older working version in the Google Play Store so that only users who were running latest Android and had the latest version of the app would be affected and then deal with that problem in a proper way at the next day. For my surprise I found out that this is not possible — there is no way using Android eco-system to pull back or cancel latest release.

Second idea was to revert targetSdkVersion to API level 30, bump application version and create a new release in Play Store. This was also not possible due to the error message shown by Google about API level 33 being mandatory (remember the statement from Google that application updates below level 33 is possible until 1st of September?). At this moment I also realized that I could ask extension from Google to be able to use API level 30 until 1st of November — I did that, but unfortunately error message stayed the same. This meant that there's no way of going back to the old version and only way forward is to fix crashes for the newest Android version and create a new release.

Instead of rolling back to the previous working version and fixing crashes at the next day, I needed to start fixing it immediately because users were slowly automatically getting newest version of the app into their phone.

Fortunately to me these crashes also happened on newest Android emulator and there were not many code changes needed to fix them. But still, mistakes could happen when working under time-pressure situation during evening hours and there's not that much time for thorough testing before putting fixed version into Play Store. Since apps were crashing right after logging in then it seemed that whatever update I can put into Play Store as soon as possible then it would be better compared to current state. In short — the plan was to fix all of the known crash issues, release a new version and then after more thorough testing release a new version with possible fixes. That was the idea at least. After creating a new release on Play Store I anxiously waited until Google finishes their review.

It had been in review for two hours and I decided to go to sleep around one in the night in the hopes of app being released by the morning when I wake up.

Next Day

After waking up, application was still in "In review" status.

Most of the day was spent refreshing Google Play Store page to see if application can be released and testing on Android 13. Found few smaller issues, which were fixed, but nothing critical.

Day ended with application being "In review" status.

Aftermath

I've read multiple articles about something similar happening with mobile apps development where Google (or Apple) prevent fixing problems in production or even worse where apps are taken down from the appropriate app store for no apparent reason.

There's nothing we as developers can do to speed up the reviewal process nor contact Google support in any way. There are no possible workarounds and we just have to wait. Wait until we're excused to put our fixes to production.

I personally have been against developing mobile apps for years now for the exact same reasons described in here and other similar articles — as soon as you decide to develop mobile apps then you give control of your product/service away to a third party, which you can't replace when problems happen. It's been controlled by this anonymous big company Google/Apple and as long as there are no problems then everyone is happy. But when things go south then you're on your own and you can't solve the problems no matter how good you are from a technical point of view. Usually there's not even any temporary workarounds you could do. It's just you against some multi-billion dollar company.

Nowadays I'm not even sure why are we, as developers, allowing this to happen — there's usually not any good reason to develop mobile applications at all anymore. It's time to move back to open (web) standards and take control back into our own hands! Technology is more than ready. Until that happens we just need to keep refreshing "Google Play Console" web page in the hopes of the update moving away from "In review" status into production.

It has been about 72 hours and update is still "In review". I just have to wait and hope that someone at Google presses correct buttons to allow the application update into the store. It's first time when I see that long review process from Google (usually that's the case with Apple). Murphy's law — of course it happens at the most critical moment.

Hope is my only option now. I don't know about you, but that doesn't sound professional way to solve any problems.

---

Solutional is an agile software development company which has a team of professional engineers who are able to solve all software problems from beginning to the end without any middlemen.

Contact us at info@solutional.ee in case you have any new or existing projects needing help with successful execution.

---