Skip to main content

Flaky Test

 

Summary

  • Definition: A flaky test is a test that passes and fails intermittently without changes to the environment or codebase.
  • Causes: Common causes include timing issues, race conditions, external dependencies, shared state, resource constraints, and environment differences.
  • Handling Flaky Tests: Use isolation, retries, stabilization, mocking, environment parity, and resource management to reduce flakiness.
  • Example: Adding retries and proper waits in Playwright to handle flaky tests.

By understanding and addressing the causes of flaky tests, you can improve the reliability of your test suite and ensure more consistent test results.


How to Handle Flaky Tests

  1. Isolation: Ensure each test is independent and doesn't rely on shared state or other tests.

  2. Retries: Implement retry logic in your test runner to re-run flaky tests automatically.

    • Example: In Playwright, you can configure retries in the playwright.config.ts file:

      typescript

    • Stabilization: Add proper waits and timeouts to ensure the application state is ready before performing assertions.

      • Example: Using await page.waitForSelector(selector) in Playwright to wait for an element to be ready.

    • Mocking: Mock external services to avoid dependency on unreliable or rate-limited services.

      • Example: Use a mock server to simulate API responses in your tests.

    • Environment Parity: Ensure that local, CI, and production environments are as similar as possible.

Comments

Post a Comment

Popular posts from this blog

Get OTP from email

/** * Retrieves an OTP from Gmail in a headless tab. * @param {Object} options - The options for the function. * @param {string} options.email - The Gmail email address. * @param {string} options.password - The Gmail password. * @returns {Promise<string | null>} The OTP retrieved from the email. */ export async function getOtpFromGmail ({ email , password , } : { email : string ; password : string ; }) : Promise < string | null > { const { chromium } = require ( "playwright" ); // Launch a headless browser context for Gmail login const browser = await chromium . launch ({ headless : true }); const context = await browser . newContext (); const page = await context . newPage (); await page . goto ( "https://mail.google.com" ); const emailInput = await page . waitForSelector ( "#identifierId" ); await emailInput . fill ( email ); const emailNextButton = await page . wa...
Post a Comment
Read more

Playwright Solution

 There is a web site need to run with browser. and which has zero downtime and alert and notifications pop up too. Third party applications are involved in and need to fix bugs too.What is the solution and how to create a framework for this scenario : need to use playwright as well 1. Understand the Requirements and Challenges: Zero Downtime: Ensure tests do not interrupt the service. Alerts and Notifications: Handle unexpected pop-ups gracefully. Third-Party Integrations: Test interactions with third-party services. Bug Fixing: Implement a process to identify and log bugs efficiently. 2. Set Up the Playwright Framework: Installation: Ensure Playwright is installed and configured. bash Copy code npm install playwright 3. Structure the Test Framework: Test Suites: Organize tests into suites (e.g., smoke tests, regression tests, integration tests). Page Object Model (POM): Use POM to manage page elements and actions. Configuration: Set up configurations for different envir...
Post a Comment
Read more

Challenges

 One of the challenging situations I faced was identifying the correct locators for elements in our Angular application. Despite using developer tools like Chrome DevTools, it was often difficult to pinpoint the exact locators due to the complexity of the application and the dynamic nature of Angular's rendering. To overcome this, I frequently had to review the application's code in Bitbucket to understand the structure and find reliable locators. This approach allowed me to write more robust and reliable tests. Additionally, I communicated with developers to gain insights and confirm my findings, which helped streamline the process." As our application grew, the number of test scripts also increased, making the scripts larger and more chaotic. This was especially true for our end-to-end tests using Playwright and TypeScript. The test scripts became hard to manage and maintain because they contained both the test logic and the test data, leading to a lot of redundancy and ...
Post a Comment
Read more