The GUI Element Identification Problem

Element identification is a well-known problem for test automation engineers. Both when simulating user interaction (e.g. when clicking a certain button) and when verifying a certain SUT property (e.g. the text value in an input field), it is necessary to reliably identify different elements on the GUI. This is usually done by an unambiguously identifying piece of information, such as the XPath, the label, and internal ID or the x/y coordinates.

Now, if the chosen identifying piece of information changes over time (e.g. the button label is changed) or within different environments (e.g. on a screen with a different resolution), the element in question can either not be identified anymore or is identified falsely (e.g. the wrong text field is used). This typically results in annoying false positives, i.e. failing tests where the tested functionality works as expected and merely the test is broken. Depending on when that problem arises, it can even be hard to detect (e.g. when a text is entered in a wrong field early during test execution).

AI can be applied to either help identify the correct element using a multitude of identification criteria (e.g. XPath, label, ID, class, x/y coordinates), using the "looks" of the element by applying some form of image recognition, or by choosing the historically most stable single identification criterion. All of these approaches have been used in the past, but none addresses the underlying problem: the continuous change of the software. So, if the correct element is identified by a multitude of criteria, the approach is robust against changes to one or even multiple of those criteria. However, if such changes occur, the values of those criteria still need to be updated to reflect those changes. Otherwise, confidence in the identification will decrease after multiple changes occur to different criteria throughout the lifetime of the software. Furthermore, at some point the remaining unchanged criteria will not yield a high enough confidence—thus only postponing the problem, not solving it. The same is true for image recognition: if the past and present image increase in difference, at some point the remaining confidence will not be high enough. Also for choosing the historically most stable single identification criterion—even if it was the most stable one, at some point it might still change.

recheck addresses this problem in an interesting and unique way.