Gray-Box Mobile Accessibility Testing

I hope we can all agree that mobile phones, and their apps, are a critical component in most people’s lives – interestingly, so critical that in some instances people will go hungry instead of running out of phone credit(1); possibly equating their mobile with safety and security as in “Maslow’s Hierarchy of Needs.

Quite frankly as such a critical piece in people’s lives the ability for anyone (regardless of disability; or age) to access, and use, their mobile applications, as and when they need, is critically important.

To date, ensuring the accessibility of a mobile application (in whatever form) has, for some at least, been far from easy. With different Accessibility requirements, mainly set at the lower code level, needed on different platforms; along with different testing methods, skills, knowledge, tools and Assistive Technologies (AT).

Explaining the current issues with mobile accessibility testing

Meet fictional Bill; he’s spent the last 5 years testing native mobile applications. He’s responsible for User-Interface (UI) testing. His is a complicated role. In the past he has mainly coded “whitebox” tests, adding them into larger groups or “testsuites”. Each test-suite coded in specific computer languages (swift; java). Once satisfied, he has then had to then ensure that those test-suites are included along with the application’s own source code in a specially built “test” variant, for execution.

It’s even more complicated when he’s been asked to test the accessibility of an application. As he’s not an accessibility expert, he has had to rely on third-party code libraries for accessibility checking, and ask that they be included in the source code then handle all the expected push- back, as you can imagine!

He’s has also had the additional head-ache of having to deal with several sets of UI requirements for accessibility effectively one per platform. Bill has often complained to colleagues “there’s only one for web; so why can’t there just be one set of requirements for Mobile?

Over the past couple of years Bill, like many testers, has transitioned to tools which allow mobile phones to be operated, controlled, and tested from the outside – “over-the-wire” so to say. Using these tools Bill has dramatically cut his need to code the low-level “whitebox” tests. Instead, he concentrates on writing higher-level tests, which can be used, as Bill describes “amazingly,” on all platforms.

Bill longs for the day he can use these similar higher-level tests for accessibility checking too… To Bill, this just seems like the smart way to proceed.

A solution…

The purpose of this paper is to consider a new method of mobile accessibility testing, applying so-called “graybox” testing, which only uses higher-level, cross-platform, tests; and the widely- used testing frameworks / tools used to run them and does not require any additional code to be “bakedin” a specially built test application!

In addition, this method, being at the higher level, means a single set of higher-level requirements such as WCAG 2.1 can be used to assess the accessibility of any mobile UI (be it a native, hybrid or mobile web application).

The hope is to make Bill, and all other mobile testers, very happy as testing accessibility should now be much easier, and more in-line with the way they work!

Gray-box testing

So, what is gray-box testing? Let’s say you have a box, which is now sitting on the pre-scan conveyor belt of an airport luggage scanner…

The security guard looks at the outside of the box whilst it rolls up to the scanner. What they are doing is black-box testing: testing of a device, program or system whose workings are completely unknown.

The box rolls into the scanner, and the guard looks at an x-ray representation of what is in the box. That guard is now doing gray-box testing – testing of a device, program or system whose workings are now partially known.

When that box is flagged for opening, and the guard opens the box then they are doing white- box testing – testing of a device, program or system whose workings are completely known.

Although, just as an FYI, in the context of software testing white-box testing generally comes before gray-box and black-box testing, as it is done on the source code, rather than via the UI of a compiled application.

Why use gray-box testing over white-box testing…

As we indicated in Bill’s story earlier, in a mobile testing context, white-box solutions generally consist of a set of separate iOS or Android tests which get “baked in” along with the source code during code compilation in order to create a testable version of an application.

Earlier, we also referred to white-box tests as being “lower-level.This was really due to the fact that they are intended for inclusion within the source code itself. At this lower code level, white- box tests can be used to create (instantiate) objects available within the code itself, for testing. As such, white-box tests have access to:

  1. each UI component, and its attribute values, in the language of the UI Framework(s) used to create it; and
  2. specific code-level knowledge about the broader application.

Access to the low level code is of course good, but, when thinking about accessibility testing in particular, the problem is that a good deal of home-grown accessibility testing knowledge / skill is needed in order to obtain meaningful / actionable results.

Developers particularly need to be cognisant of code-level requirements for ensuring the inclusion of mechanisms that expose additional text descriptions, amongst other things, to AT users. As such, developers need to aware of accessibility best practices on iOS(2), and Android(3), which cover platform specific accessibility mechanisms. Things like:

On iOS:

  • accessibilityLabel: A succinct label that identifies the accessibility element, in a localized string.
  • accessibilityHint: A brief description of the result of performing an action on the accessibility element, in a localized string.
  • accessibilityTrait: The combination of accessibility traits that best characterize the accessibility element.
  • accessibilityFrame: The frame of the accessibility element, in screen coordinates.

On Android:

  • android:contentDescription XML attribute when labelling static elements;
  • the setContentDescription()(4) method for labelling dynamic elements. Testers, on the other hand, need to understand methods for testing the as-implemented accessibility features on the different platforms. Generally testing them through platform specific testing frameworks like XCUITest(5) on iOS, and Expresso(6) on Android.

    And, all of this required knowledge for developers and testers also really dictates at least some appreciation of the ways different ATs on different platforms deal with coded accessibility features.

    In teams which do not possess such deep accessibility knowledge, or skills, third-party accessibility testing code libraries are an option.

    However, in more recent times we’ve seen the general market’s appetite for such third-party solutions shrink; driven by factors such as security concerns often associated with the need for these third-party code libraries to be “bakedin” to source code.

    In light of this, a need has emerged for low-entry-bar accessibility testing which can be achieved from outside the mobile-application-under-test; and, here, is where gray-box testing comes into the picture; especially as gray-box accessibility testing has been used successfully for some time in a web context.

So, how does gray-box testing work? Firstly in a web context…

To get a high-level view we’ll look at how Selenium, as a widely used gray-box testing tool, is used when testing web UI content.

Why? This will lay a good foundation for our later discussions on mobile UI testing.

This paper was posted by  Alistair Garrison from Level Access in ICT Accessibility Symposium 2018


Gabriel Weinkauf · October 28, 2019 at 11:29 pm

Wonderful post but I was wondering if you could write a litte more on this subject? I’d be very thankful if you could elaborate a little bit further. Bless you!|

Gianna Puddy · October 30, 2019 at 8:14 pm

constantly i used to read smaller articles that as well clear their motive, and that is also happening with this article which I am reading at this place.|

Mandi Turturro · November 1, 2019 at 7:02 pm

Hello! I simply want to offer you a big thumbs up for your excellent information you’ve got here on this post. I’ll be returning to your web site for more soon.|

Reuben Garacia · November 16, 2019 at 6:10 pm

I do trust all of the concepts you have presented for your post. They’re really convincing and will certainly work. Nonetheless, the posts are too brief for novices. Could you please lengthen them a little from next time? Thank you for the post.|

Hairstyles · November 21, 2019 at 1:10 am

Thanks , I have just been looking for info about this subject for ages and yours is the best I’ve found out till now. But, what concerning the conclusion? Are you positive about the source?

Free Stuff · November 23, 2019 at 12:10 am

My developer is trying to convince me to move to .net from PHP. I have always disliked the idea because of the costs. But he’s tryiong none the less. I’ve been using WordPress on several websites for about a year and am anxious about switching to another platform. I have heard fantastic things about Is there a way I can transfer all my wordpress posts into it? Any help would be really appreciated!

Cherelle Naret · November 24, 2019 at 10:15 pm

This information is invaluable. When can I find out more?|

Freebies · November 26, 2019 at 12:04 am

One important issue is that if you find yourself searching for a student loan you may find that you’ll want a cosigner. There are many circumstances where this is correct because you could find that you do not possess a past credit rating so the loan provider will require you have someone cosign the borrowed funds for you. Great post.

Hairstyles · November 29, 2019 at 1:26 am

An attention-grabbing dialogue is value comment. I think that it is best to write more on this topic, it may not be a taboo topic however typically individuals are not sufficient to speak on such topics. To the next. Cheers

Hairstyles · November 29, 2019 at 5:14 am

This actually answered my drawback, thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *

Social media & sharing icons powered by UltimatelySocial