If you’re like me, one of the first things you do in the morning is check the results of the previous nights automated test runs. You site at your desk, cross your fingers, open up Jenkins, and hope to see a bunch of blue dots (PASS).

When a build fails it’s time to go into debugging mode. Automated test can fail for a lot of reasons. Sometimes just knowing what to look for can help you avoid the problem in the first place.

7 Common Automated Nightly Regression Failures

1. Test environment can’t handle the load generated by nightly regression tests. This is mainly found when the system stops responding. If you’re running a lot of things concurrently make sure your environment can handle the load.
2. Browser timeouts. Some calls/actions in a test will timeout. Under heavy load, test environment has slower response (see above). This could also be a possible performance/scalability bug with the system. If you think you found a bug checkout my post about 6 Things You Should Do After Finding A Bug.
3. Test fails because of odd conditions detected in the test script. These are just straight up bugs! Something changed with the application and now your test is failing. Congratulations! What’s drove me crazy before is when these types of issues  are intermittent.
4. Poor test script implementation causes framework timeouts. In a selenium script, this happens when the clicking action on a button + waiting for response for more than 3min. Most test framework will terminate selenium session if no session activity (calls from the test script) for 3 mins. A solution for this is to, remove pause/waits from test script, and use loop to periodically query browser session. By doing this you can check if an action results in expected result. 
5. UI element changes. What I’ve mainly seen is that element id’s get changes or the paths to buttons is different because of some minor update. Normally not a bug with product but still common cause of test failures.
6. Bad test preconditions: bad test data, poor SQL select performance. To fix this issue, refine queries to return “good” data, and optimize to run faster to avoid timeouts. If you have test the hit a database you want to make sure that you’re getting the correct response before continuing.
7. Too many tests bundled into single job. Split your tests into few smaller components, so they can be executed in parallel and run faster. Aim for nightly jobs to run under one hour, and start killing jobs that run longer than three hours.