How It Works: <\/strong>Working within your favorite test runner (Applitools supports more than 40 testing frameworks and languages, so it\u2019s compatible with almost any development environment), you can insert visual checkpoints with just one line of code (generated for you by Applitools) into any step in an automation test script. <\/p>\n\n\n\nLike many other visual regression tools, Applitools takes a screenshot of the entire UI when you write the test, and sets it as the baseline. The next time you run the test, it will take a new screenshot and compare it to the baseline, using an AI-powered image comparison algorithm to filter out the noise. <\/p>\n\n\n\n
What We Like: <\/strong>The AI aspect of Applitools helps it overcome the obvious false positives\/noise problems of tools that just take a screenshot of the entire UI and alert you to ANY changes. It has four layers of sensitivity, called match levels:<\/p>\n\n\n\n\nExact<\/strong> is a pixel-to-pixel comparison, which they don\u2019t recommend because it\u2019s likely to cause false positive results.<\/li>\n\n\n\nStrict <\/strong>compares everything including content (text), fonts, layout, colors, and position of each of the elements. This is the level they recommend for regression tests within one browser or operating system because the algorithm knows to ignore rendering changes that are not visible to the human eye. <\/li>\n\n\n\nContent<\/strong> is similar to strict, but it ignores colors. <\/li>\n\n\n\nLayout <\/strong>compares the layouts (i.e. structure) of the baseline and actual images. It makes sure that elements are properly aligned and positioned relative to each other, but it ignores the actual content and color of those elements. It\u2019s great for checking the UI of apps with user-generated content or other types of dynamic content that would change between each test run. <\/li>\n<\/ul>\n\n\n\nIf those four match levels don\u2019t work for you, you can also carve out sections of the page that you know change too often for the AI to handle (dynamic content, moving content, features in beta, etc.). However, we think it\u2019s more efficient to specify what you DO want to evaluate (as you would with Rainforest\u2019s visual editor), rather than having to carve out the stuff you DON\u2019T. If you carve out too much of the page, your test effectively has a bunch of holes where bugs can slip through. <\/p>\n\n\n\n
What We Don\u2019t Like:<\/strong> Unfortunately, Applitools is extremely limited in functionality if you don\u2019t have programming skills. It\u2019s not an accessible tool for non-technical QA team members. You can use Applitools with the Selenium IDE record-and-replay plugin to add visual checkpoints into tests, but these tests are extremely limited (you can\u2019t do database testing, you don\u2019t get detailed test reports, and it can\u2019t handle conditional statements). <\/p>\n\n\n\nPricing:<\/strong> The Free Forever plan offers one user and 100 visual checkpoints per month. If you want to have multiple users and CI\/CD integration, you\u2019ll need to sign a 1-year contract, and they don\u2019t list pricing on their website, so you\u2019ll have to inquire with the sales team. <\/p>\n\n\n\n<\/span>Best Mobile Visual Regression Testing Tool: Kobiton<\/span><\/h2>\n\n\n\nHow It Works: <\/strong>Kobiton focuses on mobile testing and real-device testing, but when it comes to visual testing, it\u2019s really trying to help you maintain a consistent visual output across all devices and form factors. It offers scriptless manual testing: you can record a manual test and it converts it into an Appium script (Selenium scripts are also available). You\u2019ll get automatic notifications of visual anomalies.<\/p>\n\n\n\nWhat We Like:<\/strong> Kobiton is one of the only options that is built for mobile first, and mobile is hard to do well. Also, they go beyond just pointing out differences to making suggestions for improvements based on their analysis of the Top 50 apps in the world. Additionally, users say the screenshot function makes gathering screenshots for different screen sizes quick and easy, which helps them get apps up on app stores faster.<\/p>\n\n\n\nWhat We Don\u2019t Like:<\/strong> The only option Kobiton has for filtering noise and handling dynamic content is a setting to tell the AI to \u201cnavigate to the best matching element,\u201d but there\u2019s no way to know how consistently it will be able to identify the best match. <\/p>\n\n\n\nPricing<\/strong>: Kobiton\u2019s pricing is based on minutes of test run-time, like Rainforest. However, the cheapest plan starts at $500 a year for 500 minutes\/month.<\/p>\n\n\n\n<\/span>Best Open-Source Visual Validation Tool: PhantomCSS<\/span><\/h2>\n\n\n\nHow It Works:<\/strong> PhantomCSS is built on top of PhantomJS\/CasperJS. It can define the steps of the test script so you can navigate around your app, and then it takes screenshots of each step and uses ResembleJS to compare the images to a baseline. You\u2019ll get notified of every difference. It has no tools for minimizing false positives. <\/p>\n\n\n\nWhat We Like:<\/strong> PhantomCSS has one of the most active and healthy communities on GitHub for any open-source visual regression testing tool. That\u2019s very important for an open-source because that community is your only source for support when you run into trouble using it. <\/p>\n\n\n\nWhat We Don\u2019t Like:<\/strong> Like all open-source visual regression testing tools, there\u2019s too much noise for practical usage because you\u2019ll get notified of every difference between your baseline image and the screenshot of each test. And I mean literally<\/em> every difference. The only way to handle content that you know will have minor changes frequently is to use Javascript to replace content within the elements you\u2019re testing with a set of representative dummy content\u2014so ultimately you\u2019re not testing what the user would see. <\/p>\n\n\n\nPricing<\/strong>: Free<\/p>\n\n\n\n
![\"Google](\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61e87a27efd9fe3da3e48588_visual-regression-testing-1.png\")
<\/figure>\n\n\n\n![\"Google](\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61e87d7dac8952ca6eadfd6e_visual-regression-testing-2.png\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61eb1abd28fe924f9e718894_correct%20margins.png\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61eb1acaeb33ffb257c247ae_bigger%20margins.png\")
<\/figure>\n\n\n\n![\"Google](\"https:\/\/uploads-ssl.webflow.com\/60da68c37e5767dfb65004c0\/61e87dd199be3bb52379a69d_visual-regression-testing-5.png\")
<\/figure>\n\n\n\n