Google Summer of Code 2021 @Oppia

End-to-End Testing Support

Introduction:

End-to-End testing is a software testing method that tests the entire product from beginning to end to verify the application flow. This helps the project to prepare for the global launch. In Oppia-android, the current testing corpus includes Unit tests using Robolectric & Integration tests using Espresso. These tests are limited to one or two activities, which doesn’t facilitate cross-activity navigation flows and doesn’t simulate the real user scenario.

This project introduces End-to-End testing support to Oppia-android and an infrastructure that builds a separate test app that uses the local development server for the network requests. These tests and the test app are implemented under a new module called instrumentation. The tests are written using UI Automator library, the app and the test app are build using Bazel and the tests are run using adb.

Important Links:
- Project description
- Proposal

Milestone 1:

Introduce developer-only functionality in Oppia backend to populate test topic and exploration to be tested in Oppia-android. This involves creating an Android-specific exploration that supports all the android supported interactions.

1. Create exploration for Android-specific interactions: PR#13029.
2. Controller to initialize Android-specific structures: PR#13174.

Milestone 2:

Introduce End-to-End testing support to Oppia-android using UI Automator. Setup an Infrastructure to test using the local development server. Writing a test to play through one exploration using e2e tests.

1. Setup UI Automator with Bazel: PR#3505.
2. Exploration Player End to End tests: PR#3608.
3. Setup Infrastructure to test using local dev server: PR#3529.

Issues Closed

• Issue#13170: Add dummy exploration for android specific interactions.
• Issue#13172: Initialize android structures using a controller.
• Issue#3315: Introduce UIAutomator with bazel.
• Issue#3623: Move DEVELOPER_URL to DI Graph.
• Issue#3607: Write e2e tests for explorationplayer.
• Issue#3528: Setup infrastructure to download explorations from the local dev server.

Final Outcome

1. Created an exploration that contains all the interactions supported in Android i.e, android_interactions.
2. Create a developer-only controller to initialize the topic with android_interactions exploration to the local datastore.
3. Added tests to play through an exploration i.e, ExplorationPlayerTest.
4. Created a test-only android binary oppia_test for e2e tests which uses the local development server for the API requests.

Challenges faced

1. The e2e tests were planned to run using android_instrumentation_test target but it doesn’t give the proper test report after running the tests. The project was about to shift to Gradle since it was some issue with android/android-test#682 and the bazelbuild/bazel/#5876 then we have to come up with an approach that uses Bazel to build the test apk and use the adb to run the instrumentation test, This wouldn’t be possible if I didn’t know how actually the instrumentation tests are run.
2. The e2e test doesn’t reset the app after each test case, Tried multiple ways to clear the app data to reset the app between each test case but the test either crashes or the data remains the same. Finally, we discovered that the Gradle had the same issue and it needed an argument to reset the app between the test cases then we found that to do the same with adb using the Android Test Orchestrator in command line itself.

Summary

It was an amazing learning experience working with Oppia this summer. Thanks to my mentors Akshay Nandwana and Ben Henning for helping me out with the challenges I’ve faced during my project. It was really fun working with a professional codebase. I’ve learned many new things during my GSoC period. I even contributed to Oppia’s web repository without even having a basic understanding of the codebase, I had help from other contributors understanding what exactly I need to know to complete my work within a short span. I now have a core understanding of how AndroidManifest, modules, instrumentation tests, and unit tests actually work.

Thanks for Reading!

Feel free to reach out to me.
GitHub: @FareesHussain
Twitter: @farees_hussain
LinkedIn: @farees-hussain