background
I was hired as a UX Researcher (Contract) at Google to work on a Virtual Reality based training experience designed to promote empathy and reduce hostile interactions in our society.
The non-for-profit nature of the project meant the key metrics of success for the project were essentially defined by the degree of positive impact created and empathy generated in the community. Being an integral part of the multi-phased research driven project since its early stages enabled me to deploy a gamut of user research methods. I gained first-hand experience working on complex systems, learning about the cross-functional interdependencies and working efficiently in an agile environment consisting of cross-functional stakeholders, strategic partners and vendor partners.
Note: Due to non-disclosure agreement, the description below will only include my contributions from a user research process and methods standpoint. All sensitive information is currently redacted from the case studies below.
the team
I had the pleasure of working as an embedded UX researcher under the supervision of my UXR Manager, a support UXR and following robust cross functional team:
• Internal Stakeholders: Product Manager, Program Manager, 5 Engineers, 2 UX Designers, Partnerships lead, Product Counsel, Product Inclusion team and UXR Ops.
• External Stakeholders: Strategic partners from Academia and VR Industry, Subject Matter Experts and Vendor partners.
Research roadmap
Duration: March 2019 - July 2020
The phased research approach (from generative research to evaluative, efficacy and overall impact testing research) included three main pillars of research:
People Research - Primary and secondary target users (identity redacted as users belong to a sensitive population group) behavioral research, pedagogical efficacy, short-term and long-term community impact
VR Research - Validating that the VR tech is usable, feasible and new feature exploration
Content Validation Research - Testing the 3D visual assets (VR Scenarios and NPC characteristics), in-scenario storyline and speech for believability (i.e. real-life likeness of the simulated environments)
Case Studies
1. VR Believability and Usability Study (Offsite)
Project Background
Having proven the realism and usability of the 3D elements (VR environment and NPC animations) the VR tech and the conversational AI elements of the simulation against agreed upon gating criteria in an on-campus (NYC headquarters) setting, the team wanted to test the VR system against the believability and usability metrics in off-campus locations with primary and secondary participants from a diverse set of backgrounds across the country. The success of this research was instrumental to green-light the next steps in the product development process.
Research Goals
• Identify VR and general system usability problems for primary and secondary users in an off-site setting
• Test whether previously identified VR and general system usability problems are resolved (location-agnostic)
• Test the NPC responses and validate if the conversational realism metrics pass the gating criteria for the primary user
• Validate if the in-simulation 3D elements (VR environment and NPC animations) are believable and realistic for the primary user
Synthesis and Insights
• I triangulated the subjective survey responses with the post session interview responses, objective system logs data and analyzed A/V data to start building themes for the final insights report.
• For the first time, we successfully tested the VR system with over 45 participants at 3 off-site locations. Thus, green-lighting the Beta stages of product development process.
• The median subjective ratings for General System Usability metrics successfully crossed the gating criteria (SUS score of 78 out of 100)
• The median subjective ratings for VR Environmental Realism metrics successfully satisfied the gating criteria as well (5.5 on a scale of 1 - 7)
• Conversational Realism metrics failed to cross the gating criteria with majority of participants reporting delays and inconsistencies in NPC responses. This prompted a follow-up investigation into the conversational AI model that drives the NPC speech.
• The usability metrics pertaining to the VR controllers failed to satisfy the criteria as well with participants citing form-factor and in-simulation navigation related issues.
• Most participants, with an overall median subjective rating of 6 (on a scale of 1 - 7) reported positive attitudes towards the VR system.
• The key insights noted above were reinforced by participant quotes and gifs highlighting critical interactions.
Participant Info
• N = 15 participants from previously identified 3 primary user archetype groups (Based years of work experience, previous training, location and demographic background)
• 1 secondary user (the human instructor)
Method
• Primary user research methods used: moderated usability testing, pre and post-session surveys and in-context interviews, A/V capture and system logs data analysis.
• Collaborate with Product Manager and project leads to scope out the study, agree on the gating criteria for the research metrics, anticipated stakeholder interdependencies and the end-to-end research timeline.
• Engage with the Partnerships team to build strategic relationships and recruit specialized primary and secondary users at 3 geographically and demographically diverse locations across the nation.
• Engage with the UXR Ops to plan and execute logistics of shipping and setting up proprietary VR setup at 3 test sites.
• Train the human instructor (secondary user) to engage the trainee (the primary user) within the context of the VR simulation
• Write and execute discussion guide including: Team and project intro to participants, project goals, VR controller guide, safety and chaperone lines, brief explanation of the the in-VR scenarios to prepare participants for success and post-stimuli interview prompts.
• Participant session flow:
• Lab set-up:
Measures (Sensitive and proprietary data is redacted)
Pre-stimuli variables:
• Demographics Survey
• Years of previous experience in the field (in-context of the training that the VR simulation is designed to provide)
• Gaming Experience (Likert style)
Post-stimuli variables:
• Standard System Usability Survey completed by the primary user (Combination of Likert style and open-ended questions)
• VR Usability Survey completed by the primary user (Combination of Likert style and open-ended questions)
- Comfort, ease of use, effectiveness and level of satisfaction of VR controllers, Headset and in-simulation 3D elements and navigation
- Agency
- Ease of locomotion
- Degree of simulator sickness
• VR Realism Survey completed by the primary user (Combination of Likert style and open-ended questions)
- Realism of 3D environment (Degree of Presence, embodiment and immersion)
- Realism of NPC graphics (Degree of sentience, co-presence)
- Conversational realism (Real life like-ness of conversational AI based NPC speech)
- User attitude and sentiment
• Performance and Skills Rating Survey completed by the secondary user (Combination of Likert style and open-ended questions)
- Situational awareness
- Communication skills
- Emotional regulation
- Time management
• Post-session open-ended interview questions
• Simulation logs data
Deliverables
• Subjective responses to demographics survey, composite SUS score, VR usability survey and VR Realism survey
• Intent analysis to test the fidelity of conversational AI based NPC speech model
• A/V recordings
• Open-ended responses to post-session interview questions
• Researcher notes
Tools
• 9' x 15' play area
• VR Rig
• Proprietary simulation software
• A/V recording system
• 2X Displays
Stakeholder Interdependencies
• 1 Partnerships team member support to recruit participants
• 1 UXR Ops team member support to plan and execute VR setup shipping and set up
• 2 Eng for tech support
• 1 Product manager for supervision
• Strategic partners
Timeline
• 3 weeks for planning and scoping research
• 2 weeks for recruiting participants
• 1 week of shipping VR setup to off-site testing location (total 3 weeks for 3 locations)
• 3 days per off-site location for testing (total 9 days at 3 off-site locations)
• 2 weeks for data analysis, report writing and final insights report readout
2. New scenario content validation & Perspective gathering Study
Project Background
Once the factors affecting the usability and realism of the VR tech and the conversational AI elements were fixed, the team green-lighted content production of 2 new scenarios for the VR experience. One of the major goals for Q2 2020 was to validate the content for these 2 scenarios with primary users as well as gather perspectives from the community members affected by hostile situations in real life.
COVID-19 pandemic induced in-person testing limitations not only proved to be a severe obstacle to achieve the set goals but this also meant that multi-million dollar contracts with content production vendors were now in a limbo.
I took on the challenge and got creative by designing an all remote telephonic roleplay based content validation and perspective gathering study.
Research Goals
• Validate the content (VR environment, NPC responses and overarching storyline) for the 2 new scenarios in early stages of production with the primary users
• Validate the NPC responses and overarching storyline with community members directly affected by hostile situations in real life to see if the content is realistic and non stereotypical or caricature like.
Synthesis and Insights
• I triangulated the subjective survey responses with the post session interview responses, objective transcription data and analyzed A/V data to start building themes for the final insights report.
• Successfully identified the critical story arc for the 2 new scenarios that help green-light next steps in content production. This helped the team stay on track for the Q2 OKRs despite the pandemic induced roadblocks.
• Identified the missing chunks in the content for 2 new scenario.
• Identified inaccurate and stereotypical NPC responses for replacement.
Participant Info
• N = 7 participants per each scenario from previously identified primary user archetype groups (Based years of work experience, previous training, location and demographic background)
• N = 7 community members per each scenario
Method
• Primary user research methods used: remote moderated roleplay sessions, in-context interviews and post-session surveys, A/V capture, logs data analysis
• Collaborate with Product Manager and project leads to scope out the study, agree on the anticipated stakeholder interdependencies and the end-to-end research timeline.
• Collaborate with Partnerships team and UXR Ops to recruit participants for the study.
• Conducted fun auditions with the larger team at Google to recruit NPC voice actors for the roleplay study.
• Conduct roleplay exercises with primary users, design and execute unmoderated surveys with both the primary users (who participated in the roleplay exercise) and the community members
• We experienced several logistical and compliance related roadblocks due COVID-19 restrictions. So, I explored all possible tech avenues to test the VR system remotely and proposed these options into bare minimum, ideal and ambitious categories based on the cost, complexity of underlying tech, time to setup and compliance limitations involved.
• Furthermore, given the sequence of events happening around the US in summer of 2020 and the sensitive nature of the project, I proposed to postpone the last leg of the study.
• Participant flow for primary user:
• For each of the 2 scenarios, I transcribed the audio sequences captured from the roleplay sessions with primary users, re-formatted it in a dialogue style and designed unmoderated surveys for both the primary users and the community members.
• Primary participant who participated in the roleplay sessions would give survey feedback on their own session transcripts and 2 other primary users' session transcripts for each of the 2 new scenarios
• Each of the 7 community members would give survey feedback on 3 session transcripts obtained from the roleplay sessions with the primary users for both the new scenarios.
• Participant flow for community member participants:
Measures (Sensitive and proprietary data is redacted)
Post-stimuli variables:
• Post-session open-ended interview questions
- Critical story arc
• Post session unmoderated survey completed by primary users and community members (Combination of Likert style and open-ended questions)
- Conversational realism (Real life like-ness of conversational AI based NPC speech)
- Perceived consistency in the NPC responses
- Perceived accuracy in the NPC responses
- Perceived level of agitation in the NPC responses
- Subjective feedback on NPC attitude and emotions
- Subjective feedback on NPC's visual representation and characteristic descriptors
• Audio transcripts
- List of missing intents and corresponding responses
- Typical order of occurrence of intents
- Frequency of intent occurrence
Deliverables
• Subjective responses to post session surveys by primary users and community members
• Critical story arc for each scenario
• Intent analysis of the audio transcripts from the roleplay sessions
• List of missing intents, their typical order of occurrence and frequency
• A/V recordings of the roleplay sessions
• Open-ended responses to post-session interview questions
• Researcher notes
Tools
• 2D Sketches, list of intents and corresponding responses and audio recordings of scene descriptors for each of the 2 new scenarios
• Google Meets
• Google Forms
• Rev manual transcription service
Stakeholder Interdependencies
• 2 voice actors per scenario
• 1 Partnerships team member support to recruit primary users
• UXR Ops team for recruiting community members
• 1 Product manager for supervision
• Strategic partners
Timeline
• 3 weeks for planning and scoping research
• 2 weeks for recruitment
• 3 days of testing per scenario (total 6 days of testing for 2 new scenarios)
• 2 weeks for data analysis, report writing and final insights report readout
Reflections
• Conducted moderated and unmoderated tests to validate disparate components of the VR system with 200+ users, both on-site and off-site locations across the nation.
• Working in a high intensity environment with quick turnaround times helped me devise ways to maximize my work efficiency and exceed expectations by delivering impactful and actionable insights.
• Collaborating with some of the smartest individuals and external strategic partners helped me build a strong network in the industry.
• Worked with Product Counsel and Product Inclusion partners to ensure the research plans I designed were inclusive and met compliance and confidentiality requirements.
• Effectively pivoted to an all-remote setting at the onset of COVID-19 pandemic (Beginning of Q2 2020), to ensure the team stays on track and successfully completes all the research milestones for the quarter.
