Understanding Comparison Thresholds

Achieving reliable visual regression testing often involves managing minor, acceptable visual variations, such as subtle anti-aliasing differences between browser rendering engines. Comparizen addresses this challenge with configurable Comparison Thresholds, allowing you to precisely fine-tune the sensitivity of the visual comparison engine. This guide explains the available thresholds based on the official Comparizen documentation.

Understanding Comparizen’s Comparison Threshold Types

Comparizen employs two key threshold settings to control how visual differences are detected and reported:

1. Per-Pixel Threshold (Pixel Tolerance):

   • Purpose: This threshold defines the tolerance level when comparing the color value of individual pixels between the baseline image and the new snapshot.
   • Mechanism: Comparizen leverages the sophisticated CIEDE2000 color difference formula. This algorithm calculates the perceptual difference between two colors in a way that closely mimics human vision.
   • Configuration Scale (0-100): The threshold is set on a scale from 0 to 100, where:
     • 0-1: Represents a color difference generally imperceptible to the human eye.
     • 1-2: A difference noticeable only upon close inspection.
     • 2-10: A difference perceptible at a casual glance.
     • 11-49: Indicates colors that are more similar than they are different.
     • 100: Represents the maximum possible color difference (e.g., black vs. white).
   • Impact: Increasing the per-pixel threshold value makes the comparison less sensitive to minor color variations between corresponding pixels.

2. Total Difference Threshold (Overall Image Tolerance):

   • Purpose: This threshold sets the minimum percentage of an image’s total pixels that must be flagged as different (based on the per-pixel threshold) before the entire image comparison is considered a mismatch (DIFF_FOUND).
   • Configuration Scale (0-100%):
     • A value of 0% means even a single pixel difference (exceeding the per-pixel threshold) will result in the image being flagged.
     • A value of 5% requires more than 5% of the image’s pixels to differ before flagging the comparison.
   • Impact: Increasing the total difference threshold makes the overall comparison more tolerant to a small number or percentage of differing pixels scattered across the image.

Configuring Comparizen Thresholds

Comparizen offers flexibility in setting these visual comparison thresholds:

1. Project-Level Configuration (Default): Define default thresholds for all visual comparisons within a specific Comparizen Project through the Project Settings interface in the Comparizen Dashboard. This is the recommended starting point for consistent testing.
2. Per-Snapshot Override (API/Client Library): For specific test cases or UI elements known to have higher variability, you can override the project-level defaults. This is done by passing specific threshold parameters when uploading a screenshot using the Comparizen REST API or one of its client libraries (e.g., comparizen-playwright, comparizen-client).

Mastering comparison thresholds is key to reducing false positives and ensuring your visual regression tests accurately reflect meaningful UI changes. Finding the ideal balance often involves some experimentation; begin with Comparizen’s sensible defaults and adjust incrementally based on the specific needs and characteristics of your application and test results.