Evaluating Amazon's
Brand Metrics Dashboard
Project Summary
Brand Metrics (BM) is an online dashboard that surfaces shopper engagement and competitive benchmarking metrics. This enables Amazon advertisers to understand how shoppers are engaging with their brand at each stage of the purchasing funnel and how they compare against similar brands.
Problem
-
Despite establishing competitive benchmarking as a top priority for advertisers, adoption of BM was unexpectedly low during the closed beta.
Objectives
-
Explore how advertisers incorporate BM data into their existing campaign management process to discover their unmet needs.
-
Evaluate the usability of the BM dashboard to identify any pain points advertisers encounter.
My Role
-
Sole researcher, responsible for recruitment, data collection, and delivering actionable insights with a comprehensive narrative report.
-
Collaborate with the project manager and consult with designers and engineers to align research goals with the vision for BM’s full-scale launch
Research Impact
Insights highlighted missing data as a main concern for the dashboard, which resulted in a 30% increase to data coverage after launch.
The team redirected their strategy and developed reinforced guidance resources with a stronger value proposition.
My Process
Identifying the Scope
In order to establish research objectives, I needed to create a holistic understanding of BM through the perspective of my stakeholders. I started by holding a preliminary kick-off meeting with the PM, designers, and engineers to understand:
-
What do we confidently know vs. What assumptions are we making?
-
What data do they need to support their decisions?
-
What are the deadlines and dependencies?
After additional 1:1 stakeholder interviews with the PM and design lead to gain a more in-depth understanding of the gaps in our knowledge, I consolidated a list of the most salient observations regarding advertisers’ brand building strategies. I transformed the team's questions and blind spots into clear research objectives that best served BM's project roadmap.
Given the depth of answers the team sought out, I decided user interviews would allow enough engagement with the advertisers to dive deep into their behaviors and experiences. Qualitative usability testing would also help us discover whether the areas of friction were due to UI issues or a lack of guidance.
Recruitment
Barriers
I encountered several barriers throughout recruitment. The list of potential participants included over 1,000 advertisers enrolled in the BM beta program. To facilitate recruitment, I attempted to leverage account executives (AEs) who worked directly with these advertisers and could encourage their participation. Over a week, I reached out to AEs through bulk emails and small batches Then, I pivoted to individual requests for assistance with little to no success.
Solutions
Upon realizing cooperation from AEs would continue to slow recruitment, I informed the PM of delays to our established timeline and turned to our internal account tools to unblock my progress. Using our back-end account management tools, I manually retrieved emails for approximately 700 accounts and sent out recruitment emails to advertisers. I was eventually able to recruit 10 participants.
Data Collection
90-minute sessions with each advertiser included user interviews, walking through their brand building strategies, then evaluating the usability of the BM dashboard to understand:
How do they use BM to reach their goals?
How do they keep track of competing brands?
What is missing from BM to enable informed decisions?
What issues do they encounter while using BM?
Qualitative Usability Testing
I selected tasks based on the common actions that advertisers came to the BM dashboard to perform. The main objective was to assess the clarity of section labels and ease of navigation.
1) Identify what percentage of your total sales came from first-time customers of your brand.
2) Determine what percentile you meet at each funnel stage for your top selling category.
3) Change the time period for the data to represent last month's sales.
4) Determine what percentage of detail page traffic was from brand-aware shoppers.
5) Determine what percentage of your shoppers are repeat customers.
Identifying Themes
At the time, transcription software was not available, so I watched each session video and compiled an Excel spreadsheet of notable user actions and quotes. I used inductive coding to create a list of codes to categorize each observation. After iterating on the codes and restructuring categories as new patterns emerged, I conducted an additional round of thematic analysis to ensure observations were thoroughly organized into distinct, well-supported themes:
The dashboard's vague labels and complicated metrics lack clear definitions.
“Honestly, this is an overly complex combination of how this is defined. It needs more simplicity. I'm overwhelmed and this in my day job.”
Competitors are not clearly defined across brand categories.
“I think it is all brands and not peers, even though it says peer brands.”
Metrics are presented without enough context to be actionable.
“Do we have 200,000 shoppers and 13,000 are from the ‘kitchen and dining’ sections? It doesn't mean very much to me if I don't know what our total shopper number is as a brand.”
Missing data hinders the ability to make informed comparisons.
“Is this information missing on purpose or because you don’t have it? How am I supposed to make comparisons if the fields say there's no data available?”
While analyzing each theme, I discovered that advertisers from smaller or newer brands had different needs than savvy advertisers from larger brands.
New advertisers value...
-
More thorough educational content
-
Recommendations on how to apply BM data to driving conversions
Experienced advertisers value...
-
More specific data on customer behaviors and competitor trends
-
Viewing data over recommendations; they already have strategies in mind
Sharing the Results
I consolidated my findings into a narrative report, which I shared with my cross-functional partners on the Brand Funnel Optimization and the director of the Advertising UX org. To ensure that my partners had a report that was informative and actionable, each insight was presented by demonstrating:
-
The customer need
-
Their current behavior
-
Why it matters
-
How we can make improvements
Advocating for our users
Because smaller/newer advertisers are focused on their return on ad spend. they don’t consider upper-funnel advertising as important or effective. The team wanted to use the BM dashboard to change these advertisers’ perceptions of brand advertising by showing them how much their peers are driving awareness and consideration. Presenting data without explicitly showing value is lost on the advertisers who may not have the resources or knowledge to act on this data.
Given my perspective of the advertisers’ needs, I suggested we reframe our approach to the dashboard in the following ways:
Simplify the way we provide guidance.
The purchasing funnel is a marketing concept we are trying to teach advertisers. Instead, we should directly address their goals, e.g., "Reach more customers" vs. "Awareness".
Provide more robust data.
Include narrower segments so advertisers can understand the data in the right context. Address the blind spots that contribute to missing data and provide explanations for unavailable metrics.
Communicate the value prop.
Show advertisers the data they value most. By surfacing how awareness leads to conversions, we can meet them where they are and show what they can accomplish with brand advertising.
Research Impact
Technical Improvements
Given that missing data was an area of concern, the engineers reworked how they calculate brand segments, leading to an increase in data coverage by 30%. They also changed how BM calculates category metrics to remove outliers and provide more precise customer data. Advertisers now see more data with improved precision. Empty fields in the dashboard where data should populate now provide an explanation for why it is unavailable.
Reinforced Guidance
As a result of my recommendations, the team decided to rethink their strategy and reinforce their guidance resources. They developed the Brand Building Playbook, an area of the BM platform that provides education on brand advertising and recommendations for optimizing campaigns. New advertisers can use it to understand how to build their brands, while savvy advertisers can choose to bypass it in favor of their own strategies.
Reflecting on My Experience
Strengths
I am proud of my advocacy for the advertisers' needs, when faced with a team that was committed to their predetermined vision for the product. By providing data-driven recommendations, I created a compelling narrative toward redirecting our approach, instead of attempting to change advertisers' established mental models.
Lessons Learned
Recruiting advertisers consumed valuable research time and caused significant delays. When starting a new project, I now confirm that the stakeholders who make research requests either have direct access to participants or can provide support on recruitment. That way, we can maintain our schedule. Otherwise, I proactively set the expectation that recruitment may extend the timeline.