Software testing plays a vital role in the software development lifecycle, and automation has emerged as a critical component within this process. The rapid pace of software development and the demand for high-quality software products have made automation testing an indispensable tool for testers.
Why is Automation Testing Important? Automation testing offers numerous benefits, including:
- Time-saving: By automating repetitive and time-consuming tasks, automation testing reduces the time required for manual testing. This enables engineers to focus on more complex and critical aspects of the software.
- Improved Accuracy: Automated testing minimizes the risk of human error, resulting in more accurate and reliable test results. This is particularly crucial for regression testing, as even minor errors can significantly impact the software's performance.
- Increased Test Coverage: Automation testing facilitates a higher level of test coverage. Testers can execute a large number of tests within a short span of time, ensuring comprehensive testing of all functionalities.
- Enhanced Efficiency: Automated testing improves efficiency by enabling testers to run tests in parallel and simultaneously on multiple machines. This reduces overall testing time and expedites the software development process.
- Cost-effectiveness: In the long run, automated testing proves to be more cost-effective compared to manual testing. It eliminates the need for manual testers and reduces the testing time required.
Addressing Automation Testing Bottlenecks However, despite the aforementioned advantages, automation testing can face challenges that render its benefits ineffective. At Contra, we encountered a significant issue related to flakiness caused by third-party rate limit errors, particularly due to the high number of pull requests (PRs) we handle.
To mitigate this issue, we implemented the following measures:
- Conducting an audit of our website to ensure unnecessary calls to third-party libraries are minimized.
- Reducing the number of users created in Playwright that make API calls by employing the same user across multiple test specs while ensuring test cases remain atomic.
- Creating a separate project in Playwright specifically for test cases that trigger rate limit errors and running them in fewer shards.
- Utilizing the retry code block from Playwright to retry the API call if a rate error is encountered. An example of this implementation is as follows:
await expect(async () => { const response = await page.goto('test'); expect(response.status()).toBe(200);}).toPass({ intervals: [1_000, 2_000, 10_000], timeout: 30_000,});
5. Introducing delays in the pages that make API calls to the third party using the route functionality:
await page.route('**', async (route) => { setTimeout(() => { route.continue(); }, Math.floor(Math.random() * 1_000 + 5_000));});await thirdPartyFunctionExample();await page.unroute('**');
By implementing these measures, we have successfully enhanced the stability of our pipeline, resulting in smoother PR merges.
In Summary, Automation testing, when executed strategically and in a well-optimized manner, proves to be a valuable asset for delivering high-quality software products within shorter timeframes. As software development continues to evolve, embracing automation testing and continuously refining testing processes will remain essential for meeting the demands of the ever-changing software landscape.
