Beth Archibald (barchibald@userworks.com), UserWorks, Silver Spring, MD, USA
This position paper is in response to Scenario 1, regarding the software project with software developers from various countries.
A bit of background on our experience first. UserWorks conducts usability evaluations in person and remotely. We have completed testing at users' sites and have completed remote usability evaluations via web conferencing software. We are currently researching the role and value of a variety of online web conferencing software packages in terms of accessibility, cost, and usability concerns. Regardless of the "location" of the usability evaluation, we routinely ask participants to explain what they are thinking about as they explore an application or web site. A drawback, however, of remotely conducted usability evaluations is that it removes our ability, as administrators, to pick up on non-verbal cues to participants' experience, expectation, preference, satisfaction, or understanding. In some remote testing situations, participants must explain what we otherwise take for granted in face-to-face meetings.
In a similar vein, while having obvious advantages, virtual meetings can pose additional burdens upon participants, above and beyond connectivity issues. In Scenario 1, there are two that come to mind: Participants may have varying levels of language ability in the chosen language of communication; and some participants may not have exposure to the special needs and considerations of participants from other cultures or of participants who have disabilities.
As in remote usability testing, our experience led us to develop guidelines to prepare for conducting remotely held meetings via web conferencing software. On our part, as the administrators of the session, it was incumbent upon us to be prepared. We ensured that we had the following on hand: an additional phone line so that the computer could be used simultaneously; confirmation of the meeting details (via email was most convenient in our case); a room or area where we would not disturb others or be disturbed; and written instructions for using the web conferencing software. For online meetings, administrators should also make certain that participant(s) have the same information, not only so that time can be used efficiently, but also to provide participants a means to prepare in the way that best suits their needs.
As usability evaluation administrators, we ask participants to describe what they are doing, seeing, expecting, and feeling. Remote discussions prevent us from seeing facial expressions or other cues we would pick up on in face-to-face meetings. During online meetings, if participants have language or communication barriers, using a think aloud process can help other participants in the discussion confirm their understanding and also to uncover the point at which understanding breaks down. While some linguistic expressions may have to be rephrased, the detailed explanations will help to provide an additional dimension for participants who otherwise may not understand fully what is going on. Sometimes the perception of "accessible" or "understandable" may not be as other experience it.