Visual regression testing with Playwright

Riccardo Corradin on 24 May 2022

Visual bugs can be problematic to the user experience. For example, an image might be resized to the point where it accidentally covers a button. From a developer’s perspective, visual bugs can be hard to detect, because they usually do not break the functionality of an application. Visual regression testing allows you to detect these bugs and to write test code during development that allows you to catch visual bugs early on.

Playwright enables you to do both in an effortless way. By writing a visual regression test as short as 2 lines, you can tackle basic visual bugs. Read on and start coding to do visual regression testing with Playwright.

If you want to dive straight into the solution, you can clone the repository here.


Writing your 2 lines of code visual regression test

As an example, we have a simple React application that has been scaffolded with create react app, running on localhost:3000. It contains some clickable elements and a clock that updates every second.

Simple React application

Let’s create our first test:

visual regression testing first test

This test navigates to the React application running on localhost:3000. The toHaveScreenshot function will compare the visual differences with an existing screenshot, or create a new screenshot when there is none to compare against. You can find more documentation here. When we run this test using Playwright, the following shows:

Failed first visual regression test

Since there is no screenshot available the first time to compare against, the test creates this screenshot and returns an error:

The screenshot is created

Comparing screenshots

Let’s see what happens if we run the Playwright test command again:

Test fails again

From the Call log, we can see that Playwright does various things, that can be summarized as follows:

  1. It checks if there is a golden file (base screenshot to compare against)
  2. It takes a new screenshot
  3. It computes the difference to the golden file by using the Pixelmatch library
  4. If there is a difference detected it will take another screenshot
  5. It computes the difference again
  6. It will write an expected, actual, and diff file to the test-results directory

This set of executions is all hidden behind the toHaveScreenshot() function. When we open the diff file, the following shows:

Visual regression time diff

Although the difference is very small, the images are not the same and the test fails.


Dealing with dynamic visuals

We can increase the diff ratio or ignore specific parts of the application that include dynamic visuals. Increasing the diff ratio can be done as follows:

Visual regression test with a diff ratio

Running the Playwright test will output the following:

Diff ratio test succeeds

If the diff ratio is unknown (for example when you have a banner with rotating advertisements in your web application) you can ignore specific sections.

Adding a random image

We need to adjust the test because we are adding a network request to generate a random image:

Waiting for network idle

This ensures the image has finished loading when the comparison is done. The test fails because the diff is larger than 0.02 and varies depending on the similarity of the image.

To exclude the image from visual regression testing, we can introduce element masking as follows:

Visual regression testing element masking

Now the test will succeed.


Updating the golden file

The golden file is the base file, the truth to compare against. If we are making changes to the application, we may want to update this file. For example, if we add an element to the home page, we want to update the golden file. We can do that with the following Playwright command:

npx playwright test --update-snapshots


Using the HTML Report for visual regression testing

Another way to view the diffs is by using the Playwright HTML reporter. You generate an HTML report with the following command:

npx playwright test --reporter=html

Your tests will run as per normal, with the addition of a report generated in the playwright-report folder. To open the report run the command: npx playwright show-report.

In the case of failure, the report will be opened automatically. It contains a detailed view of the error, the steps that were taken, and a very nice slider to see the differences in one image. Note the masked element as the pink rectangle at the bottom of the page.

Playwright HTML Reporter for your visual regression testing

Conclusion

Visual regression testing in Playwright can be done with just 2 lines of code. Behind the scenes, Playwright will take the necessary steps to store and compare the current screenshot with the golden file. Updating the golden file can be done with a single command and you are able to mask sections of your web application containing dynamically generated content. You are also able to introduce a diff ratio when you want to automatically allow small changes to the UI. Finally, the HTML reporter tool is valuable when debugging your visual regression test.

Any comments are more than welcome! Drop me a message through LinkedIn or Twitter @rscorradin.