Automated Testing on Live Environments

TLDR

Set up automated testing on live environments for 3 of our systems to catch issues faster and increase trust in code changes.

My Role

Implementor of the change

Context

Employee Referrals' (ER) back-end is powered by a fair number of microservices. Often, several services are needed to fulfil a user request. As such, code tests (unit or integration) don't provide full guarantees that the functionalities offered will work live environments. I saw tests running on these live environments (something I termed "live-tests") as a good way to increase confidence in the code we were running.

Details

The tests basically invoke the APIs our clients use to fulfil a certain functionality under several scenarios. By invoking the real environments, we also test system-to-system network calls, pushes/pulls from messaging systems, and any other external communication.

I set up dashboards to monitor these tests using Allure and a tool I wrote for this particular need: act3.