So your got a few automated test under your belt and are ready to take it to the next level.

Its time to start scaling things out.

There are a few things you’ll want to keep an eye out for.
Don’t get caught of guard by these common pitfalls.

Pitfall #1 Parallelization Issues

The first common issue are those dealing with parallelization.
One of the big benefits of automated test saving time. Automation is typically a lot faster then manual test.
Most teams want to push this even more. So a common question is how can we run these test faster?

Parallelization is a natural response, but there are some easily overlooked issues.
If you are using shared state, you can run into big trouble when your tests are executing in parallel.

Problem:

There are many manifestations of this. You could have issues at the database level or at the local machine memory level. Locking issues with files or even particular user accounts. Also ‘data corruption’ issues come up when values are changed by other running test.

Solution:

The best way to avoid this kind of problem is isolate each test as much as possible. Don’t reuse data setups between test cases. You could even go as fare as to use separate virtual machines for each test execution.

Pitfall #2 Ineffective Error Reporting

Automated test tend to fail a lot more compared to other pieces of code. That’s because they normally have multiple dependencies that are not under the control of testers.

You’ll always be asking did this test fail because of an issue the AUT (Application Under Test) or because of problem with the script.
Debugging and troubleshooting become an important part of the job.

Problem:

Reading through default error messages / logs don’t give all the needed information sometimes. This because even more of a problem when running a huge volume of tests.

Not having a good way to sort through the results also comes up here.
Add in the fact that it’s much harder to figure out why things failed when they are run across multiple machines, and you just upped the difficulty level.

Solution:

Thankfully Unit test frameworks & Continues Integration Tools have a lot of built in functionality that will help with this. So for example by writing test in JUnit & having them triggered to run with Jenkins you’ll already be in a good spot.

You’ll also ant to log application specific information that you feel will help the team debug failed test (usernames, action attempted, error thrown, etc).

Setting up your framework to taking screenshot during failure also helps tremendously.

Pitfall #3 Environment setup

This issue is so common that it needs very little introduction.

If I had a dollar for every time someone told me that a test passes locally and they can’t reproduce, I would be a very rich man.

Problem:

Production environments, are different from testing environments, which are different from developer environments, which are different from everyones local environment.

Solution:

Try to make each environment as similar as possible. If there are major differences note them down so everyone’s on the same page and accounts for them during testing.

Try to make it so all test execution environments are exactly the same unless you are specifically testing different environments for execution. You don’t want tests failing on one machine but passing on another.

Time to start scaling like a pro

Now that you’re aware of these 3 pitfalls you should be ready to scale out your automated test even faster.