Intent is a data science company that enables site personalization and smart media based on predicted user behavior. This case studies examines a core product in their travel vertical that was underperforming, and the iterative design process that uncovered why, and what to do about it.
UX Team Manager
Research Analysis, Ideation, Design, Quantitative Research
12 Months (Multiple Iterations)
2019 - 2020
One of the biggest challenges facing the company was the relatively low performance of their hotel media product, which performed at about 60% of the performance of their flight product. This despite the fact that margins on hotels are much higher for US travel websites (30% of the total booking price compared to a fixed $9 commission for each flight booked). Somehow, the value to advertisers wasn’t being created, meaning that users seeing the experiences were not seeing value either.
We were tasked as a squad (engineers, PM’s, designers and a researcher) to explore alternative treatments that might better serve the user, and ultimately provide more value to advertisers. Our Director of Product asked it in this way: “How would you help people compare hotel options if you could start completely from scratch?” The ultimate goal was to create something that was served in the same way (more on that later), but could improve monetization of our current hotel solution by at least 20%.
The particular ad experience in question was a unit called the Exit Unit (XU). When a user visits a partner’s site (Expedia, for example), their first click will open a new window behind their current browsing window. This window can be as big as the parent window, and will be seen once the user is done with their current browsing session, and closes the top browser window. Whatever we created had to use the same mechanism. Otherwise, any other existing technology was fair game.
Our audience was fairly broad, since anyone that traveled and booked accommodations would be a potential user of this product. We decided to target 18 - 64 year olds living in New York City, who had used at least one of our partner’s travel sites to book travel in the past 6 months. We tried to bring in a variety of travellers: business travellers, family travellers, solo travellers, etc., and to draw from a variety of ethnic backgrounds.
Thanks to our continuous research process, we had users in the office every week to be interviewed, and a wealth of qualitative data to draw upon. When contrasting our data about how users shop for flights, and how they shop for hotels, some interesting differences became apparent.
When shopping for flights, users ranked the following factors in order of importance:
Factors mentioned fewer than 10 times have been omitted for simplicity. When shopping for hotels, users ranked the following factors in order of importance:
A hypothesis began to form when we coupled this with qualitative data from our “if you had a magic wand to fix shopping for travel” question during interviews. The most common response was overwhelmingly to display hotel options from various sites in one place while adjusting order based on factors important to the traveller. When shopping for flights, users see it largely as purchasing a commodity. Besides the minority of business travellers who care most about frequent flyer status, travel shoppers care most about price, and see most carriers as interchangeable. There are few considerations when comparing flights: price and flight duration are the two most important. Our existing solution worked well for this worldview, and mirrored how travellers already shop: we were able to display multiple flight options side by side, and the user could quickly scan for the lowest price across several sites.
When it came to hotels, however, the equation changed dramatically. Price is important, sure, but hotels are far from a commodity in a user’s eyes. Pick the wrong hotel, and an otherwise perfect trip will be ruined. Further complicating the decision, accommodation decisions involve many more factors than flights: price, location (how close the hotel is to the activities the traveller wants to do), how well it is reviewed, do the reviews seem trustworthy, what kind of amenities does each hotel have, is it well catered to families? The list goes on, and no two users had the same list of priorities when evaluating a property. Our current solution was failing users for hotels. Simply showing users more hotel websites side by side didn’t simplify the process at all, if anything, it made it worse! Users grew doubtful when exposed to too many choices. How could they know they had seen everything, and that they had settled on the best choice?
Armed with a strong hypothesis, the team decided to use a Google Ventures style sprint to come up with a new approach. Being our second design sprint, we had learned that the full 5 day schedule didn’t work well at our company, so we opted instead for a modified 3-day sprint with everyone present and a separate 2-day prototype and usability testing session run by the UX team. The results were then shared across the group. The sprint team consisted of 1 researcher (facilitating), 1 designer (myself), 1 PM, 1 engineer, and 1 member from the BD team.
Coming out of the sprint, we had two ideas that we wanted to pursue: one called Super Scores, and one called the Predictor. This case study will focus only on Super Scores. The main concept borrowed from the event ticketing world, where upstarts like SeatGeek have added value to users beyond what Ticketmaster provides. When a user visits SeatGeek, they are able to quickly look at the score of each ticket, which SeatGeek calculates based on the price of that particular ticket compared to historical pricing data for that particular seat. In this way, a user can browse thousands of tickets, but narrow their attention to only the top 10 - 20 valued tickets, ensuring they are choosing the best option for them without having to manually review every ticket available. While booking a hotel is more complicated than buying an event ticket, there were powerful ideas here we could borrow to simplify the process for hotel-bookers.
Super Scores combined elements from our favorite meta site experiences (think Kayak or Trivago), but also layered in a concept of a Value Score: a score calculated for each hotel based on factors each user deemed most important. For example, if a hotel shopper was looking for a hotel in Las Vegas, and they cared most about proximity to the Strip, secondly about Free Breakfast, and third, price, our tool would calculate a score out of 10 for each property based on these criteria, and order the hotels based on their Value Score. In this way, the user could look at the top 5 - 10 hotels, and choose among those while being assured that everything else would be a worse choice based on their preferences.
The team then set out to create mocks. Due to the intense nature of a design sprint, wireframes were skipped so we could get a working prototype together within a day. The initial designs featured three columns: one with search filters and settings, a second with hotel cards matching the filters selected, and a third with a map view that corresponded to the cards on display.
The unique features to this experience compared to a typical hotel booking site were the previously discussed value scores, as well as a tag which showed how good of a match the property was compared to the selected filters.
.png)
The screens were connected in an Invision prototype, with enough interaction to attempt to test:
Five users were scheduled to come in on Friday, and were given a simple scenario of searching for a hotel in Los Angeles. Users were asked to find the best hotel for them, given a set budget. Users weren’t given any help in using the prototype, and were frequently encouraged to think out loud. At the end of the session, we asked users about the specific features outlined above if they hadn’t been brought up during the scenario.
The first version of this design scored an average usefulness of 3.6 / 5 by our first 5 users. Not terrible… But lots of room for improvement!
It was clear from user feedback that the experience needed simplification, and that the value of the Value Scores was not being conveyed. The main question was: were we not explaining Value Scores well, or were they just not useful? For our next iteration, we made a few large changes:
.jpg)
Definite progress was made on this iteration! Users not only rated the experience more useful (4 / 5), but everyone noticed the Value Score badge. Surprisingly, though most people admitted to not having read the copy (confirmed by eye tracking), most also correctly assumed how Value Scores worked, even if unclear on the details.
“Some kind of algorithm that considered reviews, price, and location”.
We then partnered with engineering to select an API that would help power many of the dynamic features (pricing, reviews, etc.) The MVP was built, and we began to iterate on UI changes - pairing continued qualitative research through usability testing and quantitative research by running a series of A/B tests.
Through quantitative and qualitative methods, we arrived at a final version which was both inherently understood and found helpful by users in the research lab and which performed exceptionally well using our main measurement KPI’s: interaction rate (using any feature), and click rate (clicking the CTA on a hotel card). The value score metric turned out to be a helpful metric for users, and showing the map by default was the most successful design.
This version of the solution satisfied the initial problem definition, outperforming our original hotel comparison product by more than 25% on average, across multiple publisher types and points of sale (countries).
The final version of out hotel product hit and exceeded our goal of a 20% improvement over the original design. In addition, it provided an important source of liquidity during the COVID-19 pandemic when our own auction slowed down since it leveraged TripAdvisor's API and diversified the types of ad products we offered. Usability testing showed that the design was very useful among users (4.5 / 5), including several requests for how to find this tool at home once they left the lab!
This was the second design sprint carried out under my tenure at Intent, and it ended up not being the last! The format worked well for tackling a large, cross-functional problem such as this one.