What kinds of information and insights did it give you about the usability of the prototype? Although one is prototyping and the other is testing, what it has in common with body-storming is that the think-aloud user-testing technique is immediate, inexpensive and effective. It has enormous potential to quickly reveal usability issues and connect these challenges to the emotional impact on users.
As an observer, you are able to record challenges as they unfolding and identify user misconceptions which point to design elements that must be amended (Neilsen 2012). I think that the higher number of tests you do, the more effective and useful the technique becomes as it allows the tester to not only identify usability issues, but also to identify the most consistent user challenges and misconceptions and therefore prioritise which design elements most need to be altered. Three different scenarios and users were tested during the tutorial, and I think our team honed our observational skills and ability to narrate without self-editing our thoughts as we progressed through each exercise.
What aspects of the technique worked well or were frustrating? Speaking generally, an area of improvement for next time is that as invigilators, we gave into the temptation to intervene in the testing process and assist users who were really struggling to complete the task or to run our own commentary via interjections. In other words, at times we needed to “shut up and let the users do the talking” (Neilsen, 2012).
We also could have asked more prompting questions, rather than allowing users to fall silent at times, whilst they worked things out. Valuable data may have been lost as the monologue was internalized.
I think another pitfall when facilitating is learning to prompt neutrally, without asking leading questions that may influence the user’s behavior and skew the data collected. Neilsen called this “biasing user behaviour” and it takes some skill to exercise restraint as a facilitator (Neilsen 2012).
As an observer, when consciously looking for “clues” as to the user’s emotional state and reactions, I started to really question the accuracy of my interpretation of gestures as I became acutely aware of how different individuals express emotions through different expressions. This leaves a lot of room for misinterpretation. For instance, a raised eyebrow in one user seemingly indicated puzzlement, and in another user it had indicated a totally different emotion: self-consciousness.
From the perspective of performing the testing tasks myself, it took some confidence to give an honest and unmediated monologue – the temptation was to fall into silence when challenged, and I also had to push against my ego, and resist the urge to suppress comments that I feared would make me sound stupid. I just had to force myself to get over this, but I think if I had been in a more formal environment with strangers testing me, I would have felt even more self-conscious.
If I were designing the test for my own project, I think I would have also ensured that I scoped the user testing more effectively and communicated key aspects that I wanted to test with the other team members (Greenberg 2012. P.236). I am not sure that we were clear from the brief about what particular aspects of the websites or user behavior we wanted to pay specific attention to, instead we tried to record every single reaction to the set task, which was challenging even with the benefit of reviewing videos.
Neilsen, J. (16 January 2012). Thinking Aloud: The #1 Usability Tool. https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/
visited 19 September 2016
Greenberg, S. et al (2012). ‘Think Aloud’ in Sketching User Experiences; The Workbook, Elsevier, Chapter 6.3, Pp.235-40