Conversational Design Case Study (Project Molly)
Role: Lead Researcher + Conversational AI Designer
Team: Director of Engineering, Senior PM, Senior Engineer (and other contributors as needed).
Stakeholders: Director of Engineering, VP of Customer Experience, Director of Customer Experience, Chief Operating Officer.
Situation:
Vacasa is a national tech-enabled vacation rental property management company. Everyday Vacasa’s Guest Experience Team receives thousands of calls from guests that are staying in the homes that Vacasa manages and sinks massives resources into fielding guest calls.
Guests may call to discuss anything, for example: getting directions to the home, trip cancellation, requesting for additional towels/pillows/coffee filters/etc., wanting to bring a pet cat on the trip, requesting snow to be plowed from their driveway, and getting the home’s dishwasher repaired.
Designing a customer-centric experience is essential because the vacation home rental industry runs on repeat guests and great reviews.
Problem:
Vacasa uses a massive amount of resources on fielding guest calls. Leadership wanted to explore ways to be more efficient and reduce spend.
Phase 1 Approach:
I conducted 15 in-depth-interviews (5 interviews for each target segment), and explored their entire experience, including actions, thoughts and feelings, from the trip inception to booking, planning and preparing to the moments leading up to departure, eventually we discussed the trip itself and anything relevant that happened afterwards.
The phase resulted in a 4 stage guest journey map which was used by executives in strategic planning; product developments resulting from this body of work led to an increase in conversion and engagement. (see below)
Phase 2 Approach:
To create the best possible conversational flows we needed to understand what guests wanted, needed, and expected from a conversational AI bot.
I recruited guests that had called our support line during their most recent stay. Participants called into other conversational AI bots to provide us with feedback on what they liked and didn’t like about each bot/experience.
I had regular discussions with stakeholders and relevant team members, e.g. Guest Experience Specialists (those that answer all of those guest calls).
I mined and analyzed caller data to understand call type and volume, which helped us to prioritize the flows that would be the most impactful for our guests.
Observation: listened to hundreds of calls and did thematic coding to understand the current user experience, conversational journey, and pain points. This helped our team gauge the feasibility of conversational AI solutions.
I executed a substantial amount of secondary research to better understand the conversational AI landscape and earned a Google conversational AI course certificate.
I put together a report and framework for better understanding the phases of a guest call. I presented the initial report to my team and stakeholders. Then worked closely with the Director of Engineering to determine which call dispositions that we wanted to tackle and in what order, then I designed those conversational flows in Figma. The flows were then built in Google’s Dialogflow. We launched and tested the flows with a portion of our users. Our initial focus was on complete fulfillment of individual callers call intent.
Note: In Vacasa’s early days, every agent signed their emails with the name ‘Molly’. We named the conversational bot ‘Molly’ in tribute to the Vacasa’s original fictional CX agent.
(See slides below for more details.)
Phase 1 Outcome:
Flows had some wins in the form of intent fulfillment and call deflection, and got a taste of what conversational AI is all about- but we also learned about about what wasn’t working which set us up for some impactful iterations in phase 2.
We also decided to take smaller steps forward in the next iteration, which focused on getting Molly to correctly identify the caller’s intent instead of achieving complete call fulfillment.
Phase 2 Outcome:
In the second round, the enhanced flows made guests more open to utilizing the conversational AI bots - much more so than in the first round. While talking to human agents is ideal, many users took the attitude that call bots are a reality and they are here to stay. Experience insights helped us to plan further iterations. For example:
The more the bot talks, the more annoying it is.
We used customer feedback to improved the bot persona, and better, more cohesive flows by leveraging design criteria that broke up the call into four phases. We also decided to create a near exact copy of our current IVR in our conversational AI platform so that we would have a benchmark on how much time were were saving by using AI instead of an IVR system. This approach also allowed us to minimize disruption to the callers’ experience while we designed and tested specific branches/intents one at a time.
Efficiency is key. We decided to look up trip info using the caller’s phone number and (if there was a match) we led users through flows depending on how close their trip was because guests that are already on their vacation/in the home are calling about different things and have different (typically more urgent) needs than guests that are still planning their trip. This way we could more quickly identify the caller’s intent which created a more satisfactory user experience. Segmenting callers in this way also gave us the level of caller urgency which helped us automate who should get sent through to the human agent and when.
Work was still in progress when I left the org.
Try out an early version of Molly: (205) 858-2986
Note that Molly behaves as an IVR in the initial interactions, i.e. callers are prompted to select from a menu. This was intentional as we felt that there were many advantages to iterating on our original IVR. Once you make it through an initial menu, Molly will ask, “what else I can help you with?” at this point Molly acts as a traditional conversational AI bot. Try this:
(Molly says If you’re looking to book a new stay press 1…) Press 3
(Molly says If you need the address of your Vacasa rental press 1…) Press 1 (address intent; note: the info presented here would be more personalized, and this intent would be fulfilled/call would be deflected if you were calling from a number that matches a guest number in the Vacasa system.)
(Molly says … what else can I help you with?) Try “Can I bring my cat?” or “Is there a crib in my home?” or “What kind of coffee maker is in the home?” or “Is there a pool?” or “Does the home have ______ (towels, sheets, toilet paper, soap) or should I bring my own?”
Note that Molly will successfully identify your call intent and confirms your intent with you. This is the half of the battle! The confirmation step can be omitted but we decided to keep it in the flow so that we could get clear data on how successful Molly is at identifying various intents. If you were calling from a number that Molly could match with a guest reservation, she would also fulfill some of these intents as well.
Not to worry: when Molly says that the call is being transferred to a human agent, the call is actually not being transferred, so feel free to play around.
Report sample below.
