Visual regression testing with Playwright
Riccardo Corradin
on 24 May 2022
Visual bugs can be problematic to the user experience. For example, an image might be resized to the point where it accidentally covers a button. From a developer’s perspective, visual bugs can be hard to detect, because they usually do not break the functionality of an application. Visual regression testing allows you to detect these bugs and to write test code during development that allows you to catch visual bugs early on.
Playwright enables you to do both in an effortless way. By writing a visual regression test as short as 2 lines, you can tackle basic visual bugs. Read on and start coding to do visual regression testing with Playwright.
If you want to dive straight into the solution, you can clone the repository here.
As an example, we have a simple React application that has been scaffolded with create react app, running on localhost:3000. It contains some clickable elements and a clock that updates every second.
Let’s create our first test:
This test navigates to the React application running on localhost:3000. The toHaveScreenshot function will compare the visual differences with an existing screenshot, or create a new screenshot when there is none to compare against. You can find more documentation here. When we run this test using Playwright, the following shows:
Since there is no screenshot available the first time to compare against, the test creates this screenshot and returns an error:
Let’s see what happens if we run the Playwright test command again:
From the Call log, we can see that Playwright does various things, that can be summarized as follows:
test-results directoryThis set of executions is all hidden behind the toHaveScreenshot() function. When we open the diff file, the following shows:
Although the difference is very small, the images are not the same and the test fails.
We can increase the diff ratio or ignore specific parts of the application that include dynamic visuals. Increasing the diff ratio can be done as follows:
Running the Playwright test will output the following:
If the diff ratio is unknown (for example when you have a banner with rotating advertisements in your web application) you can ignore specific sections.
We need to adjust the test because we are adding a network request to generate a random image:
This ensures the image has finished loading when the comparison is done. The test fails because the diff is larger than 0.02 and varies depending on the similarity of the image.
To exclude the image from visual regression testing, we can introduce element masking as follows:
Now the test will succeed.
The golden file is the base file, the truth to compare against. If we are making changes to the application, we may want to update this file. For example, if we add an element to the home page, we want to update the golden file. We can do that with the following Playwright command:
npx playwright test --update-snapshots
Another way to view the diffs is by using the Playwright HTML reporter. You generate an HTML report with the following command:
npx playwright test --reporter=html
Your tests will run as per normal, with the addition of a report generated in the playwright-report folder. To open the report run the command: npx playwright show-report.
In the case of failure, the report will be opened automatically. It contains a detailed view of the error, the steps that were taken, and a very nice slider to see the differences in one image. Note the masked element as the pink rectangle at the bottom of the page.
Visual regression testing in Playwright can be done with just 2 lines of code. Behind the scenes, Playwright will take the necessary steps to store and compare the current screenshot with the golden file. Updating the golden file can be done with a single command and you are able to mask sections of your web application containing dynamically generated content. You are also able to introduce a diff ratio when you want to automatically allow small changes to the UI. Finally, the HTML reporter tool is valuable when debugging your visual regression test.
Any comments are more than welcome! Drop me a message through LinkedIn or Twitter @rscorradin.