Retail research during COVID-19 and beyond
4 remote research methods I used to replace “in-person” research during the pandemic
I have a unique job at Shopify. In a company that most people associate with online shopping and digital experiences, I focus on in-store experiences and physical hardware. I conduct research by going into retail locations and understanding the flow of people and items. I observe the processes that retail owners have created so that my team can create products to improve them. I spend my days in close contact with physical stores, physical hardware, and many other humans.
And then COVID-19 brought all of that to a grinding halt. Suddenly, all the traditional methods for hardware research were unsafe. Visiting retail stores and talking to someone for an hour became out of the question. Sharing hardware that I had touched and having someone else interact with it became questionable. In fact, all of Shopify research stopped for a bit as we let our merchants focus on this new change.
But, COVID-19 wasn’t going away any time soon. Gradually, different teams started doing research again as our merchants adjusted to this new world and we again started collecting insights to improve our products. Most teams were able to swap out research that had typically happened in our offices for remote interviews and usability tests with no trouble. However, doing that with retail (and more specifically hardware) wasn’t going to be possible without losing major insights about the physical environment. So, in order to still get this valuable data safely, I had to get creative. I found four tactics that helped me learn what I needed.
Virtual tours
While the world had changed, I had no insight into how our merchants had adapted their retail stores in the wake of COVID-19. Popping into local shops on personal trips gave me some insight into how the community around me was responding, but it was unclear if this was universal and only offered me a glimpse of the surface. Additionally, my team was developing some new hardware and we wanted to get an understanding of how it would fit into these new environments.
A simple interview wasn’t going to cut it. Many of the details that I am looking for are often things that remain unsaid or are things that I didn’t even know I should be looking for. What kind of lights are in the store? Where do they keep the shopping bags? Do they have a calculator behind the counter? These little details often hint at current problems or areas where we could improve the experience. If we can’t discover them, then we are limited in the ways that we can improve.
So, I turned to virtual tours. For merchants who were still able to travel to their retail stores, I asked them to walk around with their laptop or phone and give me a tour with the camera. This way I could look at the real-life placement of objects and see all the tiny details of their store environment. I would ask questions and probe around the different things that I saw, asking them to open drawers and cupboards to show me all the things that helped their businesses run.
Mostly, though, I stayed silent. Google Meet recordings only keep the video feed of the person who is talking, so I was all too aware that every time I spoke I would just see myself in the recording later. This is something to be aware of if you conduct your own video tours. It’s always a good idea to get a test recording before your actual session with the participant to know what you can expect from the final recording.
Bloggers, vloggers, and customer photos
While virtual tours were very powerful, they didn’t work in all situations. Not everyone I wanted to talk to could travel to their stores to virtually show me around. Additionally, sometimes I wanted to know how things had been pre-COVID-19. For this, I turned to the internet.
The fact that everyone is constantly taking photos and documenting things online is honestly a researcher’s dream. For many of the businesses I was interested in, I could find blog posts with photos, videos by vloggers giving tours of their favorite stores, and reviews with photos submitted on Google and Yelp.
Each one of these photos and videos was an opportunity to find clues about the topics I was interested in. I combed through video frames of camera pans that happened to capture the checkout counter in order to understand the setup and flow of customers. I zoomed in on the background of photos to see if employees had pockets in their uniforms. I checked out Google Street View to see if the store had stickers indicating what credit cards they took. In general, I was a first-class internet stalker.
This method is great when you want some real-life, unfiltered information about how things actually work vs. how they are supposed to work. The only thing you really need to watch out for is whether or not the footage you are looking at is posed. Avoid official photos and videos where people often hide aspects of reality to give off a certain image.
In my old office, we had a photo shoot for our website. Before the photos were taken, we moved plants around, hid printouts of data sheets, and repositioned lights. The final product looked beautiful, but it was not a good representation of the real version of the space. You want to make sure you’re not looking at photos like this.
Shipping hardware
When you need to test a physical object you often need to get it in someone’s hands. And, as our projects moved along, I needed to start getting devices in merchants’ hands. There is no substitute for the knowledge you can get when someone can actually feel and manipulate an object. If they put that object into their physical environment, you can see how well it fits and if there are aspects you need to modify to better fit their task flows.
All this was usually done by showing up at the participant’s location with the hardware you wanted to test. But, just because someone needs to interact with something, doesn’t mean you need to be there too.
When you combine shipping with video calls you get a powerful combination that does a good job of simulating the experience of walking in and showing someone a new piece of hardware. You can see their initial reactions and understanding as they unpack your shipment and you can watch them perform tasks with the device.
I was able to see that the fit was too tight in one of our prototypes, but that people had a good understanding of what the device was for without me telling them. By shipping the hardware and recording the sessions, I was able to make a compelling video of the problem so that we could course correct. Without getting the hardware in our customers’ hands, we would have remained unaware of this design flaw.
When shipping, there are two areas that might trip you up. Shipping, and in particular international shipping, can be very complex. You need to make sure you have the right forms so that your prototypes are not stuck at the border and so that you are not inadvertently hit with taxes and fees. Make sure you carefully read up on import and export requirements before you send anything. Whoever you are shipping with will usually have resources to help you determine what these are.
Secondly, we often want to understand our participants’ first reaction and understanding of a device, and in order to do that, we need to witness them seeing it for the first time. Make sure you instruct your participants to not open the package until their call with you and I even recommend writing it on the box a few times for good measure. You are fighting everyone’s natural instinct to want to see what’s in the box, so make it obvious.
Use a global workforce
The next obstacle I had was a little trickier. I needed to see how long it would take U.S. consumers to perform different tasks on our new device. I needed a lot of data for this and we only had a small number of prototypes on my team, so shipping out lots of test devices was out of the question. Additionally, I needed this device to be set up in a particular way before the test was run; it wouldn’t just work out of the box. If I sent devices directly to participants, the setup would invalidate the test results.
Usually, I would just go to the U.S. with one test device and be able to collect a ton of data in a short period of time. However, COVID-19 made this impossible. So, again it was time to turn to more creative methods.
To solve this next problem, I leaned on our global workforce. While I couldn’t physically be in the U.S., there are many Shopify employees who are already there. I repurposed an internal social meetup tool in order to find employees living in specific areas across the United States. I then reached out to these employees to ask if they would be willing to help me. Even though no one worked in research, and I was contacting them out of the blue, every person I messaged agreed to help!
I prepared research kits for each of them with a series of videos showing the contents of the box I was sending them, the proper setup for the test, an example test that I filmed with my sister, and instructions for how to send everything back to me. As everyone who was helping me had no background in research, I made sure to make things as easy as possible.
In addition to printed guides that covered the test protocol, as well as testing basics, I provided a video of an example test session. This video showed how to respond in a real-life session and prepared my helpers for the fact that participants might go off script or ask questions! I also made it clear that they could reach out to me at any time if they ran into trouble or just had questions.
When you’re using this method, you need to think of everything that could go wrong: from shipping the device to the device crashing, your helper must be prepared for it! You need to think of all the basic things that you usually do during a research session — like responding to questions with questions or avoiding priming the participant — and instruct your test facilitator on how to handle these situations.
No matter how prepared you are, you also need to accept that some things will go wrong. Anticipate that you may have to throw out some of the data, no matter how many precautions you take. Your helpers don’t have the same research background that you do, so you need to understand and keep that in mind when reviewing the results.
Life beyond COVID-19
While all these methods are invaluable during the current pandemic, they have a lifespan well beyond our current times. As the world one day goes back to something like normal, these methods will help you when you can’t travel for other reasons.
Even before a pandemic shut the world down, travel wasn’t always possible, whether because of budgets or deadlines. Often we just ended up giving up on collecting this kind of information in favor of other easier methods (or no research at all!). It doesn’t have to be all or nothing though. As the four methods above show, even if you can’t be there physically, all of that data doesn’t have to be lost.
So, when I can travel again, yes, I will go back to in-person research, but I won’t be throwing away all of these new research methods. They still have a place in our post-COVID world and are yet another valuable tool in your research toolkit. Depending on the question that’s thrown at you and the constraints you have, one of these still could be exactly what you need — no plane ticket required.
