Allstate launched a new mobile app feature, but the QA, Dev, and business teams did not agree on what constituted feature completeness. Usability testing identified 13 user issues and got each team on the same page about what needed to be fixed and why it was important to users.
14 participant scenarios
$50 total compensation
10 participant interviews
2 mobile OSes tested
1 happy client
The participant demographics were pretty general since as testing was for the purpose of vetting the interaction patterns.
Our audience was anyone who had downloaded mobile apps to their phone to be our audience.
We had only two weeks to prepare to present our findings, so formal recruitment and usability testing were not feasible.
Additionally, the most current build was only available on test devices so testing was limited to the company campus.
I developed the research plan, wrote the script, interviewed and took notes, analyzed results, and presented findings to the business stakeholders.
Brian provided project oversight, took photos, and led stakeholder discussions.
Given the short project timeline, we were not able to recruit external participants or able to work in a formal usability lab.
Our initial research plan had us using a mailing list to recruit stakeholder colleagues and a conference room as a testing environment where we would capture information via video and with the assistance of a notetaker.
We realized participants would have industry knowledge which could bias their results, but this was mitigated by our focus on interaction patterns and ease of use.
Next, I wrote scenarios which ensured participants came into contact with all the new interaction patterns. After a round of revisions with my team lead, I sent the test plan and scenarios to our stakeholders for approval.
The stakeholders approved of the plan but determined that the mailing list recruitment process would take longer than initially anticipated. Given this additional hurdle, we decided to shift our format from informal usability testing to guerrilla usability testing.
My team lead and I met with the client’s UX research team to discuss strategies for performing guerrilla testing onsite at the client campus. This meeting provided us with excellent information for the best places to find research participants among other details, and confirmed that we were on the right path.
The technique for approaching participants was to identify individuals who had completed lunch and were using their mobile device. I revised our test plan to include these changes and sent it to the business stakeholders for final approval.
As an incentive to participants, we offered a $5 Starbucks gift card for their time and input. I modified our existing consent form and began writing the script I would use when approaching potential participants. I decided to audio record participants to capture think aloud details and revised the scenarios to make room for note-taking both during and immediately following the testing sessions.
I organized my documents and arranged my script, incentive, screener, and consent form to make testing as smooth as possible, and then began approaching potential participants.
My approach to testing – recruit, run the scenario while making notes, review the audio immediately afterward – worked very well. The notes caught most issues and the audio helped me clarify anything I had not fully noted.
I collected all instances of issues or praise into a spreadsheet and coded them for themes. Once I had identified themes, I collaborated with my team lead to provided each issue a rating based on technical difficulty / scope and user impact.
From these two ratings we determined the overall priority of the enhancements from the perspective of the user. My team lead and I developed a presentation to present to business stakeholders and user stories for the development team to consider adding to their backlog.
This project was even more successful than I was expecting and business was happy with the findings.
I gained a lot from the experience, both personally and professionally. I learned how to hone an approach script, manage my paperwork for testing, perform research analysis, and present findings to business stakeholders.