Skip to main content

Flaky Test

 

Summary

  • Definition: A flaky test is a test that passes and fails intermittently without changes to the environment or codebase.
  • Causes: Common causes include timing issues, race conditions, external dependencies, shared state, resource constraints, and environment differences.
  • Handling Flaky Tests: Use isolation, retries, stabilization, mocking, environment parity, and resource management to reduce flakiness.
  • Example: Adding retries and proper waits in Playwright to handle flaky tests.

By understanding and addressing the causes of flaky tests, you can improve the reliability of your test suite and ensure more consistent test results.


How to Handle Flaky Tests

  1. Isolation: Ensure each test is independent and doesn't rely on shared state or other tests.

  2. Retries: Implement retry logic in your test runner to re-run flaky tests automatically.

    • Example: In Playwright, you can configure retries in the playwright.config.ts file:

      typescript

    • Stabilization: Add proper waits and timeouts to ensure the application state is ready before performing assertions.

      • Example: Using await page.waitForSelector(selector) in Playwright to wait for an element to be ready.

    • Mocking: Mock external services to avoid dependency on unreliable or rate-limited services.

      • Example: Use a mock server to simulate API responses in your tests.

    • Environment Parity: Ensure that local, CI, and production environments are as similar as possible.

Comments

Post a Comment

Popular posts from this blog

Get OTP from email

/** * Retrieves an OTP from Gmail in a headless tab. * @param {Object} options - The options for the function. * @param {string} options.email - The Gmail email address. * @param {string} options.password - The Gmail password. * @returns {Promise<string | null>} The OTP retrieved from the email. */ export async function getOtpFromGmail ({ email , password , } : { email : string ; password : string ; }) : Promise < string | null > { const { chromium } = require ( "playwright" ); // Launch a headless browser context for Gmail login const browser = await chromium . launch ({ headless : true }); const context = await browser . newContext (); const page = await context . newPage (); await page . goto ( "https://mail.google.com" ); const emailInput = await page . waitForSelector ( "#identifierId" ); await emailInput . fill ( email ); const emailNextButton = await page . wa...
Post a Comment
Read more

Mock server

 Using a mock server in end-to-end (E2E) testing is indeed common, and it can be used for more than just reading values. Mock servers simulate the behavior of real servers and can handle various HTTP methods, including POST , PUT , DELETE , and PATCH , in addition to GET . This means you can use a mock server to simulate interactions that involve writing data, such as filling out and submitting forms. Automating Form Interactions with Mock Servers Simulating Form Submissions : Mock servers can be set up to handle form submissions. When you submit a form in your E2E test, the mock server can capture the request and respond with a predefined response. Validating Data : You can use the mock server to validate the data sent by your application. The mock server can check if the request body matches the expected format and values. Response Handling : After submitting a form, your application typically processes the server's response. The mock server can provide specific responses to test...
Post a Comment
Read more

Browser Context

 In the context of Playwright, understanding how pages and browser contexts work is crucial for managing test isolation and ensuring reliable test execution. Here’s an explanation: Browser Context in Playwright Definition : A browser context in Playwright is akin to a new browser profile. Each browser context provides a separate environment where cookies, local storage, and other session data are isolated from other contexts. Isolation Between Tests : When you run tests using Playwright, each test typically operates within its own browser context. This means that: Fresh Environment : Every test starts with a clean slate, ensuring that any changes made by one test (such as cookies or session data) do not affect subsequent tests. Consistent State : Tests are independent of each other in terms of state, which helps in avoiding interference between different test cases. Page Isolation : Within each browser context, individual pages (created using page objects in Playwright) are also i...
Post a Comment
Read more