Amazon's Brand Building Metrics Dashboard

Project Summary

Brand Building Metrics (BBM) is a tool that helps Amazon advertisers understand how shoppers are engaging with their brand on Amazon and shows how they compare to similar brands. The goal of this study was to obtain in-depth feedback on the BBM closed beta and understand how the BBM dashboard helps advertisers optimize their campaigns and reach their brand advertising goals.

My Role

I worked closely with the Project Manager and consulted with members of the Brand Funnel Optimization team to understand how this research would align with the vision for BBM’s full-scale launch. As the sole researcher on this project, I was responsible for end-to-end research support. This included recruitment, conducting user interviews, and delivering actionable insights to the team with a comprehensive narrative report.

While I am able to describe my process and broadly discuss insights, I am unable to share images of the artifacts I used throughout this project due to the confidential nature of the closed beta program.

Objectives

Understand how advertisers incorporate BBM data into their existing advertising process.
Evaluate the usability of the BBM experience and identify any pain points users encounter with the dashboard.

Impact

The insights collected from this study helped the Brand Funnel Optimization team rethink their approach to advertisers’ brand building objectives and guided the direction of the Brand Building Metrics development leading up to general launch.

Insights highlighted missing data as a main concern for the dashboard, which resulted in a 30% improvement to data coverage after launch.

The team created the Brand Building Playbook,

an educational resource for brand awareness advertising with a strong value proposition.

The Purchasing Funnel

Throughout this project, I refer to “the funnel” when referring to advertisers’ goals. The purchasing funnel, known by many other names, is a crucial concept in marketing and advertisement. It represents the most straightforward process shoppers go through as they make a purchasing decision. While there are more detailed iterations of the funnel, I will outline Awareness, Consideration, and Conversion.

Awareness

The shopper becomes familiar with the brand. The brand remains top of mind for the shopper when they consider where to make purchase.

Consideration

The shopper evaluates their options. They educate themselves on the available offerings and compare competing products.

Conversion

The shopper feels confident enough to make the purchase.

Many advertisers prioritize lower-funnel goals (i.e. driving sales) because it is the most effective way to measure success in their brand. More savvy advertisers also invest in upper-funnel advertising to increase awareness and get ahead of their competition. Larger brands can afford to spend money to remain top of mind, even if it doesn’t lead directly to sales.

My Process

Establishing Research Objectives
My first goal in this study was to familiarize myself with Brand Building Metrics and understand how this research would meet the stakeholders’ needs. After collaborating closely with the Project Manager and individual members of the Brand Funnel Optimization team, I consolidated a list of their most salient questions regarding advertisers’ brand building strategies. I then identified clear research objectives by determining how the team’s questions aligned with the BBM project’s roadmap.

Given the number of questions and the depth of answers the team sought out, I decided user interviews would allow enough time and engagement with the advertisers to dive deep into their behaviors and experiences.

Recruitment
Although I only aimed to recruit 10 participants, I encountered several barriers throughout recruitment. I needed to contact individuals from a list of over 1,000 advertisers participating in the BBM beta program. To facilitate recruitment, I reached out to the Amazon account executives who worked directly with these advertisers and had access to their contact information. Over a week, I reached out through bulk emails, smaller batches, then individual messages with little to no success.

Upon realizing cooperation from account executives would continue to slow recruitment, I informed the PM of delays to our established timeline and turned to our internal account tools to unblock my progress. Using our back-end account management tools, I manually retrieved emails for approximately 700 accounts and sent out recruitment emails to advertisers. I was eventually able to recruit 10 participants.

User Interviews
I conducted 90-minute user interviews with each advertiser. To get an understanding of how they currently manage their Sponsored Ads campaigns, I had them walk through their recent interactions with Amazon’s advertising tools. Then, we did a deep dive into their brand building strategies with the goal of understanding:

What tools do advertisers use to build their brand on and off of Amazon?

How do advertisers use BBM to reach their goals?

How do advertisers keep track of competing brands?

What is missing from BBM to help advertisers make informed decisions?

Identifying Themes

At the time, transcribing software was not available to my team, so I returned to the session videos and took notes on each important observation. I manually performed qualitative data analysis on session notes by coding each observation, then organizing codes into higher-level categories to identify themes.

The dashboard's vague labels and complicated metrics lack clear definitions.

Quote: “Honestly, this is an overly complex combination of how this is defined. It needs more simplicity. I'm overwhelmed and this in my day job.”

Competitors are not clearly defined across brands.

Quote: “I think it is all brands and not peers, even though it says peer brands.”

Metrics are presented without enough context to be actionable.

Quote: “Do we have 200,000 shoppers and 13,000 are from the ‘kitchen and dining’ sections? It doesn't mean very much to me if I don't know what our total shopper number is as a brand.”

Missing data hinders the ability to make informed comparisons.

Quote: “Is this information missing on purpose or because you don’t have it? How am I supposed to make comparisons if the fields say there's no data available?”

Overall, the data was too broad to be actionable or was missing important context for making meaningful comparisons. In taking a deep dive into each theme, I also discovered that advertisers from smaller or newer brands had different needs than savvy advertisers from larger brands.

New advertisers value...

More thorough educational content
Recommendations on how to apply BBM data to driving conversions

Experienced advertisers value...

More specific data on customer behaviors and competitor trends
Viewing data over recommendations; they already have strategies in mind

Sharing the Results

I consolidated my findings into a narrative report, which I shared with the team and the director of the Advertising User Experience org. To ensure that my partners had a report that was informative and actionable, each insight was presented by demonstrating:

The customer need
Their current behavior
Why it matters
How we can make improvements

Because smaller/newer advertisers are focused on their return on investment, they don’t consider upper-funnel advertising as important or effective. Brand Funnel Optimization team wanted to use the BBM dashboard to change these advertisers’ perceptions of brand advertising by showing them how much their peers are driving awareness and consideration. Presenting data without explicitly showing value is lost on the advertisers who may not have the resources or knowledge to act on this data.

Given my understanding of the advertisers’ needs, I suggested we reframe our approach to the dashboard in the following ways:

Speak their language.

Simplify the way we provide guidance. The purchasing funnel is a marketing concept we are trying to teach advertisers. Instead, we should directly address their goals, e.g. "Awareness vs Reach more customers".

Provide robust data.

Be more specific. Include more narrow segments so advertisers can understand the data in the right context. Address our blind spots that contribute to missing data and provide explanations whenever data is unavailable.

Communicate value.

Show advertisers the data they care most about. By highlighting how awareness can lead to conversions, we can meet advertisers where they are and show what they can accomplish with brand advertising.

Research Impact

Reinforced Guidance

As a result of my recommendations, the Brand Funnel Optimization decided to reinforce their guidance resources. They developed the Brand Building Playbook, an area of the BBM platform that provides education on brand advertising and recommendations for optimizing campaigns. New advertisers can use it to understand how to build their brands, while savvy advertisers can choose to bypass it in favor of their own strategies.

Technical Improvements

Engineers on the team changed how BBM calculates metrics to remove outliers and provide more precise customer data. Given that missing data was an area of concern, they also reworked how they calculate brand segment metrics. This led to an increase in data coverage by 30%. Advertisers now see more data with more precision. Fields in the dashboard where data is populated now provide an explanation for why it is unavailable.

Reflecting on My Experience

Strengths

In the face of a team with pre-determined ideas for their product, I am proud of my ability to advocate for the advertisers’ needs. By suggesting we change our approach rather than attempt to change the advertisers' established mental models, I was able to help the team reach a solution outside of their limited perspective.

Lessons Learned
Recruiting advertisers can consume valuable research time and cause significant delays. When starting a new project, I now confirm that the stakeholders who make research requests either have direct access to participants or can provide support on recruitment so that we can maintain our schedule. Otherwise, I proactively set the expectation that recruitment may extend the timeline.