Good things happen when a Product Manager pairs with a UX researcher
UX researchers are responsible for producing reliable insights that can guide product teams’ decisions. A good insight can help build useful and resilient products for people. Poor insight can bog down the product development process and result in a bad decision based on incomplete, inconsistent, and inaccurate data and/or conclusions.
Similarly, product managers (PMs) are responsible for properly interpreting and using these insights to put a plan into action. Unfortunately, as UX research findings leave the nest and begin their journey as stand-alone documents or take-aways, the risk of misinterpreting these findings by folks who were not directly involved in the research process grows. Feelings can sometimes swing between extreme doubt (e.g. “How secure do I feel about these findings to implement a solution based on them? I’ll just ask our data team to pull some numbers”) and blind faith (e.g. “UX research slide deck X stated Y, so I’ll just go ahead and base all my decisions on this one piece of data”). Both scenarios are undesirable. Although the latter one can seem flattering to junior UX researchers, in reality it’s a crutch to a development process riddled with bad decisions.
So, how can product managers and UX researchers ensure that insights are translated into the right product? In this article I’ll talk about some of the strategies that can help improve the product development lifecycle and a few UX focused KPIs. But first, let’s talk about the various levels of insights that UX researchers specialize in.
Data => Wisdom Cycle of insights
UX research turns data into information and information into knowledge and plans that guide product decisions. It provides more than just a usability evaluation or a set of interviews – it’s a powerful process that, if properly engaged, can transform a company into learning-based organization that uses fact-based decision making to achieve strategic objectives.
Insights produced by a UX researcher can be in a form of a raw material (data) all the way to actionable items (plans). Let’s unpack these:
Data
Data can be either qualitative (documents with raw interview transcripts, contextual inquires, co-design sessions, or journal studies) or quantitative (results of a survey question or a simple report pulled from the company internal warehouse). The choice of methods depends on whether the intent is to specify the type of information to be collected in advance of the study or allow it to emerge from the participants. The data may be numeric information gathered through a survey (X% of business owners report their age range to be 18–24), Google Analytics (Bounce rate on this page is X%), or text/audio/image information gathered from the interviews and observations.
Information
As a UX researcher gains more context about the company they are working at, they begin noticing consistent meaning, attributes, and hierarchies across multiple products. For example, at Shopify a UX researcher might match and merge data about a particular UI element or workflow from multiple parts of Shopify— order management, inventory, 3rd party apps — turning data into a new output: information.
Knowledge
UX researcher equipped with analytical tools and methods examines the information and identifies trends, patterns, and exceptions in the data. The analysis process enables the researcher to turn information into knowledge.
Plan
Armed with knowledge, the researcher works with PM and designers to create actionable items and prototypes from the trends and patterns they discover. These actionable items can be tactical: “Improve truncated and poorly formatted pop-over labels in graphs” or strategic: “Traditional commission feature is not currently needed. Instead focus on staff management tools and sales reporting by staff.” A well defined plan turns prototypes into a product.
KPIs (Key Performance Indicators)
Once plans are executed into a product, they generate new data that a PM, UX researcher and Data Analyst study, measure, and record starting the process all over again.
Wisdom
Each time the product cycles through the process, the UX researcher and PM learn more about how the product works and what actions they can take to achieve the desired effects. Basically, research-informed product development process creates a continuous feedback loop and a learning team that can respond flexibly and quickly to new behaviours and patterns of its customers.
As UX research and PM skills increase, so does the quality of the collaborative process
A junior UX researcher is most comfortable with the data and information level of insights. As they gain skills and experience they gradually begin working closely with PMs** and the rest of the team to effectively turn information and knowledge into plan, plan into products and ultimately close the feedback loop of insights by learning from past product development cycles.
**Some PMs struggle to see the value of having a UX researcher on their team, especially when they are already working with a data analyst. If you have a similar struggle, please read this post about the mixed methods research approach to get a better grasp on how data analysts and UX researchers can work together to enhance each other’s insights.
As the UX researcher matures they begin viewing their discipline as a systematic process, not a set of methods or tools and, most importantly, invests in relationships. Early on, they look beyond their direct responsibilities, take time to understand the dynamics of the team, and the product’s north star. Rather than focusing on a “what don’t they get” attitude that dismisses a PMs’ viewpoint, mature UX researcher asks themselves “why does this team member think the way they do and what do I need to provide for them to see my point of view?” They learn what other people on the team need to know, when they need it, and how best to tell them. Finally, they ask for the same and don’t avoid difficult conversations, especially if the conflict is persistent.
So, what are some UX research strategies that can help ensure consistently good product development and feedback loops?
Strategies to ensure successful UX research + PM collaboration
Make it easy for PMs to actively participate in the UX research process
Too often UX researchers take too much responsibility for driving the research side of the product, excluding both data analysts and product managers, allowing the team to become dispassionate observers instead of actively involved owners of the product research. Or the UX researcher tries to meet the objectives and needs of too many customer groups at once, which dilutes the priorities. In both cases, the product gets a tepid response from target users, if it is deployed at all. Instead, UX researcher can team up with the PM to focus on the common goal: build great products and systems for the right audience in the context of company goals and capabilities. I’ve learned from many different teams that there is a strong connection between an actively involved and committed to a good user experience PM and a successful product with a strong end user adoption. The following strategies are also corollary to this one and cannot be executed without a good UX research + PM relationship.
Help PMs communicate insights to other stakeholders
As I mentioned earlier in the article, misinterpretation of insights happens all the time. PMs often mention research that resulted in a particular product decision at company and departmental meetings, and they write about it in @team emails and Slack channels. To help avoid the spread of misinformation it’s important to create proper documentation with linked evidence and record conclusions clearly without much room for interpretation. This is where collaboration with a data analyst comes in handy — quantitative insights can provide additional validation for largely qualitative UX research and increase the overall research quality.
Work with PMs to establish key customer perspective KPIs (no, not NPS)
Net Promoter Score is probably one of the most popular UX Key Performance Indicators (KPIs) among PMs, but it’s also one of the most incomplete. It’s a good start for companies that cannot afford to conduct continuous UX research but largely redundant and imprecise in companies that have dedicated UX research. This means conducting rigorous and rich studies that give a much better insight into the company’s performance and can result in a few powerful UX KPIs, two of which are presented here (let me know if you’d like me to expand on this topic):
User experience index: Are we delivering a good user experience to our customers?
Probably the most useful benefit from measuring the user experience index is that it provides PMs with insights into the gap between present product performance and customer expectations. This can help prioritize and launch product improvements.
Both qualitative and quantitative methods are used to collect user experience data in a form of survey responses and ongoing UX interviews. For example, we survey our customers every year to understand their overall satisfaction with Shopify products and services as well as conduct interviews directly after the launch of the product to capture their satisfaction at the point of experience. It’s important to measure this index on the ongoing basis so as to get a broader and more insightful view of customer behaviour and attitudes — a once-a-year survey is simply not enough and the interviews will help identify loyal customers.
Customer Retention: To what extent are we keeping the customers we have acquired?
In his book The Loyalty Effect, Fred Reichfeld makes the point that ‘A 5% improvement in customer retention rates will yield 20–100% increase in profits across a wide range of industries’. The customer retention rate is a powerful indicator of performance because it demonstrates loyalty through real behaviours (customers have actually decided to come back or stay), instead of factors that focus on predicting future loyalty such as NPS. Psychological scientists who study human behavior agree that past behavior is a useful marker for future behavior under specific conditions.
UX research can give PMs a solid ground for the Customer Retention metric through qualitative research by digging into these conditions of the customers they are working with to measure this metric contextually. If the retention is high for a specific customer category, high customer satisfaction levels can be assumed (unless barriers to leaving are very high or the incentives given to the customers outweigh the bad service they might receive). If the retention is low, it’s important to know not just the actual numbers but more importantly the reasons for dissatisfaction (UX research can help here as well). A complementary measure of customer retention is the customer churn rate, which measures the percentage of customers a business loses over a specified time-frame. Similarly to the Customer Retention metric, it’s important to understand the reasons for churn.
Both measures need to be understood in the context of other measures such as Customer Lifetime Value (some customers might be more expensive or harder to serve than others).
___________________________________________________________________
These can also be nicely combined with more quantitative metrics as well as Product Quality metric:
Usage. How many actions X take place? What is the volume of X traffic over time? How many visitors return to X? What percent of our merchants (cohort) uses X?
Customer intent. What are the most frequent actions and clickthroughs/tap-throughs with regard to X?
Product Quality. What is the bounce rate (percent exit rate) or dwell time (time spent on the page) for the most frequent actions? How many complaints come in? What are they? What is the breakdown of feedback by sentiment? What is the content of feedback?
In conclusion
Create a common PM-UX research roadmap that you can loosely refer to throughout the product development cycle (Image 1).
Consider working out a PM-UX research feedback loop to feed data into product development and back into data (Image 2).
Work together to level up and increase your UX research and PM skills.
Develop a set of north-star strategies to ensure successful and consistent UX research + PM collaboration without losing your soul.
Shameless plug: If you’re a UX researcher and the things we do at Shopify sound exciting to you, come work with us!
Big thanks to Carson Brown ✇ and Alaine Mackenzie for helping me with this article ❤
Have your own approach to UX-PM collaboration or would like me to expand on the topic? Leave a note here or reach out to me on Twitter.
