Spinning up a research plan for product designers
How to start facilitating research to validate your design decisions
Early in 2021, product designers at Shopify were asked to take on a new craft skill — research. User research has always been a pillar of UX here at Shopify and leaped forward by getting more folks involved in the process.
The shift meant designers working on new and exciting features for our merchants would also develop, facilitate, and synthesize their research studies. No better way to get closer to the problems to be solved!
However, a change like this isn’t the most straightforward. Product designers have traditionally worked with project-level researchers. In this new world, product designers would need to take the reins on research
Product designers would need to take the reins on research and, in that process, strengthen their research skills.
and, in that process, strengthen their research skills. As a senior product designer at Shopify, I found myself in the same position as others, having completed only a handful of research-focused work in my career to that point, but eager to dig in deeper.
At the end of the day, becoming comfortable with facilitating research is a great way to expand your skill set and level up as a designer. But if research is new for you then what’s the most effective way to get started? This can be different for others, however I find creating a research plan to be most effective.
Creating a research plan
The plan is more like a blueprint and details the goals, expectations, and hopeful success of the intended research. It also serves as a simple way to share context and present stakeholders with a birds eye view of your research focus.
While working on the post-purchase checkout project I reached the point where my design decisions needed to be validated with users, in this case actual buyers. I needed to facilitate moderated usability tests (qualitative insights) while simultaneously conducting A/B experiments (quantitative insights) on my proposed design optimizations. So, to get that work off the ground, I built the foundations of my research plan.
Below in this article, I share my tips on forming a research plan with examples from my own project. Use these tips to help guide your own research plan so you too can collect key insights and information on your designs.
Define your research goal(s)
Research goals communicate the value of research and act as a backbone for the research plan. It’s important to be explicit about what the outcomes are. Try your best to keep them related. What you want to avoid is diluting your purpose of research. So, develop clear research goals with explicit focus, and touch on the area(s) you’d like to address.
- Example: Understand how we can optimize the design of the post-purchase interstitial page to improve the offer conversion rate, without eroding buyer trust.
In the example above, there are two areas of focus: conversion and trust. I felt the research needed both quantitative and qualitative insights to give well-rounded perspectives on the effectiveness of my design changes.
With a research goal in place, we can move on to the next important area.
Develop a hypothesis
A hypothesis is an assumption based on what you know or believe to be true. Every research plan needs a hypothesis, if not, then what’s the purpose of the study? Create at least one, and up to three hypotheses, that are focused and appropriately related to your research goal.
Here are a few hypotheses I created from the research goal above:
- Example: The current layout is not optimized for conversions. Once a buyer realizes their order is confirmed they drop from the offer interstitial.
- Example: Reducing the height of the offer header will increase offer exposure to buyers and subsequently increase the likelihood of conversion.
- Example: Buyers are viewing their confirmed order in the header and are not interested in the offer.
What I’ve done with these hypotheses is touch on both the qualitative and quantitative aspects of the research goal while speaking to the specific areas that I believe are impacting the effectiveness of the post-purchase offer interstitial. With these hypotheses, we can establish what success or good looks like once the results are fully synthesized.
Establish success metrics
Success metrics are an opportunity to create quantitative outcomes based on the hypothesis. In some cases, qualitative observations must be quantified to appropriately determine success — and leave as little room for interpretation as possible.
Keep your success metrics realistic by asking yourself what an acceptable threshold could be. One important reminder: the goal of research is not to prove you were right, though it is wonderful if that happens. Research is supposed to help you determine whether the experience you’ve created is effective in its outcome.
Sometimes your hypothesis is flat-out wrong. If that happens, it’s okay! Being disproved can be the beauty of research. In my own work, I often hope to be disproved as a way to reveal perspectives not otherwise
Being disproved can be the beauty of research. In my own work, I often hope to be disproved as a way to reveal perspectives not otherwise considered.
considered. These moments allow you to step back and observe your work, improving experiences and making the work more resilient.
The success metric examples were created based on the hypotheses above:
- Example: Offer conversion rates improve +20% (from 10% to 12%)
- Example: 75% of interviewees express clarity, comprehension, and control of the design.
The first example is straightforward; data monitoring can determine if conversions increase or decrease over a designated period. Simultaneously, the second example allows us to observe qualitative insights directly from participants providing insights not otherwise possible through data monitoring. The combination of these insights can determine the outcome of research and bring conclusiveness to the findings.
As the study progressed, the research plan allowed me to stay focused on the objectives. I facilitated 8 buyer usability tests and conducted a 4-day A/B experiment (50/50 split) with participating merchants. Having a well-defined research goal, hypotheses and established success metrics made it easy to synthesize the work and assess whether the findings proved or disproved the hypotheses.
Research findings
The research findings brought high confidence that the proposed design optimizations were effective at increasing offer conversion rates just under the 20% goal set, while 80% of buyers expressed full clarity, comprehension, and control when presented a post purchase offer. With that in hand, I was able to make a full recommendation of rolling out the design optimizations for our general availability launch of post-purchase checkout extensions. A win for Shopify, a win for our merchants seeking new methods to sell their products, and a win for buyers whom these changes would directly impact.
Developing research skills are just as attainable as learning a new design tool. Since my first research study, roughly 15% of my total design time is spent focusing on research. That 15% has brought helpful buyer and merchant perspectives to my work, letting me create the most thoughtful experiences for Shopify and Checkout. It goes without saying that any opportunity to validate your design experiences is a good one. So, take the plunge, bring research into your design process and level up yourself as a designer and the experiences for those using your product!
