Pros and Cons of Automated and Manual Testing

Software testing is an important part of building an IT product.

We test software to detect bugs and find issues that may negatively affect the user experience.

Generally speaking, we can distinguish automated and manual testing. In one of our previous articles we explained why automated testing is good for your startup.

In this article we are going to talk about advantages of automated testing over manual testing. But first, let’s clarify these terms.

What is automated and manual testing?

Manual testing is performed by hand. Quality Assurance (QA) specialists ensure that applications work properly by following conditions written in test cases. Despite its primitive nature, manual testing is still important, as certain functionality simply cannot be automatically tested. For example, wearables and mobile devices can require field testing in a physical environment. Mobile applications often undergo ‘monkey tests’ that detect bottlenecks during unpredicted stressful conditions. For instance, what happens if a user forgets a mobile device in their pocket with an application running and they tap the screen unintentionally? Is it possible that they’ll crash the app? Only manual testing can account for these scenarios.

Automated testing relies on pre-scripted tests that are run automatically. Their function is to compare actual results with expected results. By doing so, they can determine whether or not an application performs as expected. Automated testing can execute repetitive tasks and regression tests to check if an application performs correctly after newly implemented changes.

Now let’s discuss automated and manual testing in detail.

Automated Testing

Automated testing is a crucial element of Test-Driven Development (TDD). Test-driven development is characterized by the writing of test cases for each unit (Unit Tests) <>before the actual code is written. After the code is written, these unit tests are run, and based on the results of these tests, code might be refactored (if necessary) or new tests added and executed. Here at RubyGarage, we use a TDD approach to build most of our products; this approach helps us to integrate tests into each unit of an application and rapidly build quality software products.

As a rule, our QA team automates the following types of testing:

• Unit testing – Testing of discrete parts (units) of a software product.
• Functional testing – Software QA testing is performed in order to detect actual performance of an application’s functional requirements. Functional testing usually considers accuracy, interoperability, compliance, security and suitability.
• Regression testing (a type of functional testing) – Checks if an application performs correctly after being modified or integrated with other software.
• Graphical user interface (GUI) testing – Testing of software interfaces to check if the GUI meets all requirements.

These are the most common types of testing that we automate. Manual testing requires people and time, while automated testing reduces the demand on both of these resources. Also, automatic tests eliminate the risk of human error, making code more reliable. Now let’s see what tasks we can solve by automated testing.

Why you need automated testing?

1. To handle time-consuming tasks. After changes and modifications are made to an application, you need to check if it works properly. In order not to check all of the application’s functionality again and again, we use regression tests. By running automated regression tests we can identify new bugs with less effort.
2. To handle repetitive tasks. If you're constantly adding the same features or executing the same operations in a project, then the flexible nature of automated tests can be a great time saver.
3. To eliminate human error. Since automated tests are powered by tools and scripts, the chance of missing a bug or a defect decreases. This makes the testing process more reliable and time-efficient.
4. For advanced GUI testing. Automated GUI tests detect and record differences in behaviour between platforms — for example, across different web browsers or operating systems. Also, automated GUI testing effectively finds regression errors.
5. To test load and performance. There’s essentially no manual alternative for testing load and performance. Automated tests simulate hundreds and thousands of simultaneous users.

Manual Testing

Despite the seemingly broad coverage of automated testing, manual testing shouldn’t be underestimated. In fact, you have to test software manually before you run automated testing. Even though manual testing requires much effort, without it you cannot be sure that automation is possible. One of the key testing principles is that 100% testing automation is impossible. Manual testing is still necessary.

Let’s see which tests cannot be automated (or would take too much effort to automate):

• Ad hoc testing – Testing which is executed without preparation and written test cases. During ad hoc testing, a QA specialist randomly tests functionality of the system; his aim is to be creative, “break” the system and discover flaws.
• Exploratory testing – This testing technique is sometimes confused with ad hoc testing, but they’re actually different. During exploratory testing, a QA specialist tests a product like an explorer, relying on his personal experience. After getting the full picture of a product’s functionality, a tester designs test cases which he uses to further test the product. That’s how exploratory testing differs from ad hoc testing: in exploratory testing you rely on your personal experience and design test cases in the testing process. An ad hoc testing can be done by any person without preparation.
• User interface testing – By executing user interface tests, a QA specialist validates the properties and states of interface elements. Also, the QA team ensures that design elements in the interface match elements in the final layout.

As we can see, some tests should be performed manually. This especially relates to tests that focus on user interfaces and usability. Although we could automate basically everything, manual testing still provides an effective and quality check of bugs and improprieties.

The cases where manual testing is best

1. UI Acceptance – Manual testing comes in handy when you test user interfaces, especially their visual aspects. Automated tests simply cannot detect colors of images or links, font sizes, gestures, and highlighting. Imagine that your application has a bug that causes a link to be so tiny that you can barely see it. Manual testers will immediately detect this bug, but automated tests are not able to notice the defect.
2. Initial development stage — Manual testing helps testers to detect bottlenecks during the initial development stage which results in less time and resources being spent on fixing bugs. Sometimes it’s not rational to invest money and human resources in automated testing during an unstable phase of a product’s development. However, in some approaches to software development, automated testing is a part of the process at any stage.
3. Short-term projects – Even though automated tests are aimed at saving time and resources, it takes time and resources to design and maintain them. In some cases, such as building a small promotional website, it can be much more efficient to rely on manual testing.
4. Non-automatable cases – Some tests cannot be automated; this especially concerns hardware-related tests. Writing scripts for automated checks of sensors, touchpads, and screens is almost impossible, and rarely viable. Also, some data can be tested only manually. For example, you can automatically test if a printer prints, but checking the quality and content of printed images is best done manually.

Conclusion

There’s no silver bullet for testing during the development process. Despite the wide variety of testing techniques and tools, we cannot rely on a single approach. Automated and manual testing each have their strengths and weaknesses. What we want to stress is that no matter how great automated tests are, you cannot automate everything. Manual tests play important role in software development and come in handy whenever you cannot automate the process.

At RubyGarage we apply automatic tests as part of our Agile development approach. Automatic testing enables us to deliver high-quality software, save time and human resources, and reduce development costs. If you want to find out more about how we use automated testing, feel free to contact us.