Accessibility Jumpstart 6: User Testing
Welcome back! Last time we covered how to use assistive technology yourself. But just because we can use it doesn’t make us experts. Here’s how to get real people to give feedback through user testing accessibility in your app or site.
Finding your testers
When I first approached this issue, I had no idea even where to begin. Here are some of the steps I suggest you explore.
Going local
One of the first thing is to see if there are community groups or schools for the blind in your area. This worked out extremely well, and the Carroll Center not only was able to assist, but had lists of many potential testers. They also sent out the email blast on our behalf. Having a connection like this to locals I personally believe is one of the best ways to go, outside of an internal hire. Look for a school, or community groups in your area.
Going remote
While I personally haven’t done so, it seems that testing services such as UserZoom or Fable provides such services. Your mileage may vary, but I can’t recommend this route one way or another as I haven’t used it.
Using a third-party
The reason I haven’t had to go down the remote road is because we now work with a third-party that has on-staff usability testers with disabilities using specific assistive tech. This has really taken the pain out of the exploration part of the process, but of course isn’t cheap. Some of the bigger services include Level Access, Deque, and TPGi (formerly the Paciello Group).
Decide what you want user tested
Once you’ve chosen your direction to gather testers, its time to answer a few questions.
- Which assistive technologies do you need to test?
- On which devices?
- How skilled should your users be with their chosen AT?
Skill-level and comfort with the device being used makes a massive difference in the quality of the test results. I’d always recommend that your testers use their own devices and AT for the sake of comfort and speed of the test. Also, highly-skilled testers may used to working around very badly designed applications or sites and can make your app look like gold. The same goes for testers that are just learning their chosen AT, and can make even a very well-designed app seem broken.
Use a survey to identify your audience ahead of time. At work, we felt it was valuable to have both the aforementioned groups to give us a better sense of what our app was like for a wider range of users, but maybe that’s not valuable for your needs when user testing accessibility.
Preparing for the test
When preparing for user testing accessibility, you should absolutely have in mind what you’d like to test, and be ready for many technical issues to spring up even if you plan ahead. Allow for ample time and minimal tasks to complete. Have additional tasks ready if there is extra time or some simpler flows for users that couldn’t complete the original tasks. I found this last point important especially for users that were less skilled at their AT, and felt badly as they were unable to complete the original task (through no fault of their own).
Additional things that were important was sending over non-disclosure agreements ahead of time, providing directions to the testing facility, greeting and offering assistance to the testing location once they arrive, making sure the site or app was publicly accessible and not behind a VPN, providing water and compensation, having the camera ready to record, and explaining what was in the room and who was in the testing room with them.
The test itself
Don’t be afraid to ask questions or things like “Could you please slow down the voice on your screen reader?” There are so many fascinating things I learned just from doing this and observing tester behavior. Did you know the Windows keyboard is super important for testing Windows AT? I didn’t. Did you know that some iPhone users blank the screen, swipe and listen with the phone next to their ear, not even looking at the device? I didn’t. Did you know that even if you hit 4.5:1 contrast that it’s not enough for some users? I didn’t know that either.
Testing common flows, sites, and apps with a wide group of abilities has been some of the most rewarding, interesting, (and yes, sometimes embarrassing) experiences of my career.
Conclusion
Testing actual users once in a while is awesome and definitely better then never doing so. No one knows AT better than someone who uses it every day. However, strive to include users with various abilities much earlier on in the design process. I know I want to grow this area far more this year.
That’s all for now, in the next article I’ll cover the various methods I have used for automated and manual dev testing. See you then.