Transcript

The Experimentation Edge - Anuradha Tempe ===

Anuradha Tempe: [00:00:00] Not everything needs to be A/B tested, but B2C is being a fairly new consumer platform and it's important to test the features there. But it requires a specific level of expertise and knowledge to understand how to read the A/B tests and and then decide whether to bring that feature on or not.

INTRO: Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.

This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's jump in and get started with our next guest.

Ashley Stirrup: Hello, and welcome to today's episode. I'm excited to have Anuradha Tempe, lead [00:01:00] product manager at Samsung Electronics America. Welcome to the show, Anuradha

Anuradha Tempe: Thanks, Ashley. Thanks for having me

Ashley Stirrup: Yeah, I'm excited to have you on. maybe we start with a little bit about you and your background and how you first got into AB testing.

Anuradha Tempe: Yeah, I have an interesting background. So in my past life, I was a hardware engineer, specifically working on chip design, and then I pivoted to software, and now I'm leading e-commerce platforms. I consider myself a continuous learner and a student for life, and that's how I am here today with managing B2B as well as B2C platforms currently for Samsung.

And that's how I got involved with A/B testing on the front-end feature side and got to learn more about that as well.

Ashley Stirrup: Yeah. Must be just an absolutely enormous [00:02:00] business

Anuradha Tempe: Yeah, we are fairly big and most of the times people are surprised, "Oh, Samsung has their own e-commerce platform." They do. And yeah, we are doing well. And I would say B2C is fairly mature, even though it's young, but still fairly stable. B2B is relatively new, so there are a lot of interesting things happening on the B2B side.

And yeah, always a pleasure to keep working in chaos and learning more.

Ashley Stirrup: Yeah, and it must be two very different sides of the business. Like the-- I'd imagine the average order value is much, much smaller on the consumer side than it is on the business side.

Anuradha Tempe: Yes. So most of the times people mistake B2B as like a sidekick to B2C. It has a completely different consumer base, and that's something that's very important to keep in mind when designing specifically features for B2B customers. And [00:03:00] A/B testing also varies. You have to work on normalizing the overall test results because a particular order on a one particular day can skew your results and inflate it towards a false positive.

It needs a certain level of understanding to also understand how to lead A/B tests on B2B side.

Ashley Stirrup: Yeah, I can imagine that leads to some interesting challenges on just how you scale it across an organization and that not everybody has the same level of expertise when it comes to running tests.

Anuradha Tempe: That's true. Yeah. My goal is to always be more data-driven and not rely too much on data. Sometimes you need to keep that in mind where to prioritize A/B tests because A/B testing also consumes resources, whether it's on the engineering end or design end, and you're trying to loop in all stakeholders.

So it does require a fair [00:04:00] amount of time and resource consumption. Not everything needs to be A/B tested, but B2B being a fairly new consumer platform and it's important to test the features there. But it requires a specific level of expertise and knowledge to understand how to read the A/B tests and and then decide whether to bring that feature on or not.

Ashley Stirrup: And you must work with a lot of different teams across Samsung, is that right?

Anuradha Tempe: Oh yeah, Samsung is such a huge. It's a conglomerate actually. we have counterparts in Korea obviously, so there's a lot of cross-functional and also cross-department alignment with different teams. And yes, obviously geographically spread out culturally spread out. I think it's a huge company, but we work similar to startups, and it's fast-paced, it's chaotic but it's also, there are lots of resources, so you can also build and test things.

So [00:05:00] you enjoy the best of both worlds.

Ashley Stirrup: Yeah, that does sound exciting. And I'd imagine in that kind of environment, you wanna make sure that everybody's testing in a similar way so you can compare experiment results across teams, and that you wanna get best practices. So something that you learn in one group, you wanna make sure all the other groups are aware of it and can possibly take advantage of it.

How do you try to foster that communication and create a consistent set of rigor across the organization?

Anuradha Tempe: Yeah, that's a great question. And my goal is to take whatever I have learned and spread it internally obviously, and then with other stakeholders. For example, if I'm talking to other PMs maybe within America or even in Korea or somewhere else have that session of knowledge transfer and understand that they may or may not have the same level of resources as [00:06:00] you.

So the goal is not to shove things down other people's throats and be like, "You have to do things this way only because this is the best way." I go by w-what I have learned, and then if you share your knowledge with that open-ended mindset that I'm here to learn something from you as well, it goes down better always, and teams end up aligning much faster.

So yes, knowledge transfer is important, but also realizing that teams operate in their own specific way, and you don't want to come across as trying to break their dynamic or culture.

Ashley Stirrup: Yeah. It's so interesting talking to a wide variety of guests and that, particularly smaller ones, it's easier to create that culture when, especially if you've got like founders or, early executives who are "This is the way we do everything." And it can lead to a lot of kind of camaraderie.

It's okay to have a high a [00:07:00] low win rate 'cause you're just trying to learn and you recognize that, not every feature's gonna win. And so it can be like a really powerful thing. But I would imagine at a larger company where you're kinda trying to introduce that a little more after the fact that yeah it requires the right balance, right?

You need to make sure people still feel empowered and things like that.

Anuradha Tempe: Yes, and I was one of the first PMs on the B2B platform, and B2B tended to be much more chaotic and also fast-paced because there were fewer people fewer stakeholders to decide. And when I came in, I brought in my learnings from B2C, and everyone was startled and also surprised that, "Whoa whoa, okay, what are we trying to do here?

we don't want multiple meetings and want to introduce new processes." But I think that was an interesting challenge where it was also, like you said knowledge transfer. But at the [00:08:00] same time, having that humility to learn from people who have been running the B2B business for a while and being open to their suggestions.

And also letting them know that, "Hey, this is how we scale systems. And in order for our B2B business to scale to this level, it's important to introduce these practices that we have learned from B2C." Because B2C is more mature, and A/B testing in terms of feature implementation it's more rigorous on B2C side.

So what can we learn from B2C without losing the soul or essence of our fast-paced B2B way of doing things?

Ashley Stirrup: Yeah, it's interesting 'cause you can certainly see where some people might feel like, "Oh, this is more overhead, more things I need to do before I can ship, and I just wanna go ship." But I guess there's that military phrase like "Go slow to go fast," that, if you're shipping a lot of stuff that isn't making an impact on the business, then it might feel like you're going [00:09:00] fast, but you're not really. Were there any things you did early on that maybe helped, get some early wins or get some buy-in that, oh yeah, this is a better approach?

Anuradha Tempe: Yeah, in terms of feature launches, I worked on improving our buying journey on Samsung.com and especially implementing the add-on features on our hardware products. So in general, at Samsung you have different categories, whether it's mobile devices or refrigerators, dryers, there are different categories, and within those categories it has its own subcategory of products, and each of the products have their own buy pages.

So the experience can be a lot more fragmented, and one of the initiatives I took was to have a seamless add-on experience, especially during our flagship launches where you are launching, say, a new phone, but you want a security feature [00:10:00] or maybe another software package. You want to install WebEx in order to host meetings with all business stakeholders.

So including that and making those software solutions visible on the same web experience led to an increase in our av-average order value during launches, as well as it improved the mobile experience. Most of the times people assume that B2B customers will only buy devices on desktop or they won't really scroll on mobile platforms.

But we have a huge traffic on mobile devices as well, so it's important to make the journey seamless across desktop, mobile app devices and ensure that you, your traffic is getting converted across all these different platforms. So that was one of the initiatives I launched and we A/B tested the add-ons [00:11:00] feature which led to an increased conversion rate and an increase in average order value.

So that was a huge win in terms of B2B launches for-- especially for mobile platforms.

Ashley Stirrup: Yeah, I was gonna ask. It sounds like that's the kind of thing that you really want a bit of a dedicated experience for desktop versus mobile. How did you think about that?

Anuradha Tempe: Yeah, I think it was obviously data-driven, so I would work on looking at our data reports daily to analyze what's our overall traffic. And then also I noticed that stakeholders may not necessarily buy on mobile devices because they have to get code approvals from business and all those logistics happen a lot.

It's a lot more convenient to do it on desktop, in, on your PC. But they will still browse the options on mobile platforms, and [00:12:00] they'll make a decision on mobile device before deciding to go ahead with generating the code. Talking to stakeholders, talking to the sales teams being really open-minded, as I mentioned, and, being on calls with our customer support teams to hear what problems they are having day to day in terms of resolving customer pain points and bringing those into our design discussions, and then aligning with stakeholders to understand, okay, this is what the problem is and what's the best way to come up with a version one solution before we can reach the perfect solution.

So I always like to break things down into chunks and then deploy a phase one and then a phase two, and so on and so forth.

Ashley Stirrup: Yeah. I think that makes a lot of sense, particularly if your long-term vision is really different. If you make massive changes all at once, it's hard to know which parts of the experience worked and which ones [00:13:00] didn't.

Anuradha Tempe: Right

Ashley Stirrup: Yeah. And I believe you said that you got like almost doubled the number of add-on sales through this whole process?

Anuradha Tempe: I did because it solved the problem of visibility of not having those solutions available when customers were browsing mobile devices like phones, laptops, tablets, et cetera. So it definitely solved the issue of visibility, and then also improving the mobile experience helped because then it reduced scroll time, and customers were able to see all the options in one go.

And then there were also follow-up questions on, "Can I bundle this with something else?" So it led to higher engagement, and then that translated to higher sales and conversions.

Ashley Stirrup: Yeah, that's terrific. And I really love that story of how, people might not be buying on their mobile device, but that it's still a really important part of the buyer's journey. I think that's kinda one of [00:14:00] the key things we talk about on this podcast, is understanding, every business, their customers are going on different buyer's journeys.

Could be even games. Maybe that's a, it's a gaming journey. But that kind of understanding the journey the customer is on and then how you can add value to that journey is just so powerful. And in this example, it's like part of the buyer's journey is mobile, and then the rest of it switches to desktop.

So kinda understanding how and where you can add that value on mobile is s- that's just a really powerful example.

Anuradha Tempe: Yeah. It's always a learning experience and yeah, the more you talk eh, I say product management is a combination of being open-minded at-- because you have, you can't please all stakeholders, but at the same time you have to try to empathize and do the best possible job in the available amount of time.

Yeah.

Ashley Stirrup: Yeah. Yeah, experimentation can be such a, a powerful tool for product managers. I remember very vividly, [00:15:00] it wasn't experimentation, it was user testing, but being so proud. I was a product manager on a mobile app and being so proud of that app and so excited for the user testing, and then watching people use it and realize that they weren't finding this really important button that basically unlocked half of the functionality of my product.

It was just so depressing watching that. But, it changed me for life in terms of understanding the importance of usability testing and, that I don't know everything and that what seems obvious while I'm designing a product might not be obvious to an end user. And all those things apply one-for-one with experimentation as well.

Anuradha Tempe: Yeah, I, and user testing is usability testing is great and often oftentimes more easily doable than running a full-on A/B test. So there have been several initiatives where I would just roll it out to our UAT team and get their feedback before [00:16:00] launching a product or a feature instead of engaging A/B testing teams, because that requires its own resources.

So-

Ashley Stirrup: Yeah, you can use them to solve different problems. So yeah, I think they they can be very complementary. So let's say you had a new product manager you were working with, and they had a feature they were so excited about, they couldn't wait to test it 'cause they were just so sure it was gonna win, and maybe they're not thinking about why it's gonna lose or why it might lose, and how do you help them design an experiment so that if it, they do lose, they have as much information as possible for them to learn why it lost and how they might iterate on the design in order to create a better feature?

How would you help them design that experiment?

Anuradha Tempe: I would say I was that product manager. So I learned as I went ahead with the A/B testing and that was an idea brought in by our sales and business teams where we wanted [00:17:00] to launch our buying-- allow users to buy faster and earlier in the purchase journey. It was an idea that was it m- was there on B2C, and then we wanted to test it out on B2B.

And I realized that during the A/B test that customers are actually getting confused by seeing the Buy Now option earlier in the journey, because if you're buying, say, 100 laptops, you don't necessarily want to just buy it at the click of one button. You want to be able to see what are the specs and what color and all the different software packages like I mentioned earlier.

B2B co-consumers value clarity over faster conversions. Not to say that they don't like speed. They w- want to be able to buy faster, but they want to know everything and be clear on what they are purchasing before going ahead, because [00:18:00] returns and cancellations can be even more complicated, e-especially with bulk orders.

So that was something that I learned when I tried to, I would say, biased towards deploying that because I thought it's a great idea. But that's where the A/B testing and experimentation leads you in the right direction. And once I spoke with the data teams and understood especially when it comes to normalizing orders, right?

Especially on, if on a particular day you see a million sales and you think, "Okay, this feature is working well," but it's only with that particular day and for that particular customer. So normalizing that order and making sure that it's not skewing the results and leading you to an incorrect decision is important.

So I would say I was that PM who learned during the process, and that really was a great example to learn from. And I just say this again [00:19:00] and again, I love to make mistakes and learn from them, and that's the best way you learn. You humble yourself in the process. But I've noticed that everyone does that, and the more you do it and the more vulnerable you are and share results with teams, The right leaders do understand where you're coming from, and you do get the support because you're not trying to withhold any information from anyone.

So yeah

Ashley Stirrup: 100% agree. It's another example of the power of a growth mindset.

Anuradha Tempe: Right

Ashley Stirrup: And coming back to your example there where people needed additional information, though, just such, such a great example of really understanding the buyer's journey and that what are-- it's one thing if you're buying soap on Amazon, right?

You can probably do that pretty quickly. But if you're buying 100 or 1,000 laptops, you wanna measure twice and cut once. You wanna make sure you've got that order right. And so obviously you want that additional information, and you wanna [00:20:00] make sure you haven't missed anything, and you've thought about the warranty and all the other things that go along with a more of an enterprise purchase.

So yeah, understanding that buyer's journey and then aligning what you're doing to fit that need super interesting.

Anuradha Tempe: Yeah.

Yeah, and then I was just going to add that maybe instead of buying, probably adding another feature for engagement would have been a better feature launch, right? Yes, to your point, we are not buying soaps, right? So it's not something that you can undo quickly.

So it's important to ensure that you give the clarity and the confidence to the customer to be able to make that decision to buy in bulk.

Ashley Stirrup: Yeah. And I'd imagine that kinda understanding your different user segments, l- like there might be somebody who's buying the same thing every week, so they'd get new hires or something, and they know exactly what they want and it's fairly straightforward. Whereas somebody else is doing a bigger [00:21:00] purchase and it's maybe their first time, they need more of that hand-holding.

So I would imagine segmenting your user base and then optimizing for those different segments is a really big opportunity for you.

Anuradha Tempe: Yeah, and we are e-exploring so many new avenues of how we can onboard enterprise customers or small to medium-sized businesses because they operate completely... both of them are B2B, but they are completely different from each other. Having that unique customer journey and having segmenting your customers even within the same platform is extremely important.

Ashley Stirrup: Yeah, do you have-- K- how do you think about having like a, a North Star metric and other kind of evaluation criteria, which guardrails and, just how you design experiments to maximize the business?

Anuradha Tempe: I think it's important to understand why we are launching a particular feature. So oftentimes, [00:22:00] we are caught up in all these new ideas and competition analysis, and someone from marketing will come in and say, "Let's do this thing," and, "This is going to lead to high engagement." So that's great.

if the goal is high engagement but not high revenue, then as long as we are aligned on the goal, we can proceed with that. But it's important to understand why we are doing what we are doing especially in this age of AI and personalization. It's very important to understand what's the North Star to your point.

I would align with our... especially if it's like a business initiative, with our business stakeholders to see what's the r-return on investment we are expecting on this. Is it conversions or is it overall revenue? And oftentimes conversions higher conversions means higher revenue, so they are interlinked.

And if it's NPS, because some features are not necessarily going to bring [00:23:00] in revenue, but they will soothe customers who are agitated with like a confusing experience online. So having that North Star especially w- and I would give an example of one feature where the goal was to reduce our support tickets because users were getting confused with a discount that w-they were not receiving, but they were expecting to receive it.

I aligned on the North Star that this is going to reduce our overall support tickets, and then came up with the pre- and post-messaging. And we didn't necessarily A/B test it. It was more of a pre- and post- alignment and comparison of support tickets, and they did go down after launching that feature.

So yes, to your point, it's important to understand what's the North Star. Is it going to bring in revenue? If not, what is this going to solve? And then what are the guardrails? And most of the times, The major guardrails are [00:24:00] that you don't expect your sales to go up with this launch, especially if it's an engagement feature.

But you want to make sure that they are not going down as a result of launching it. Having those guardrail metrics in terms of conversions or NPS, CSAT. In B2B, it's important to have direct channel open with our customers because there may not be a day-to-day NPS dashboard that you can review similar to B2C.

But actively taking those calls with sales stakeholders and hearing, "Oh, our customer is not liking this input," or, "They want this changed," or, "They're not able to buy these many phones," or, "They're out of stock." And keeping those things in mind and ensuring that your guardrails are not impacting the overall revenue negatively.

I would keep those things in mind.

Ashley Stirrup: Yeah. Yeah, and I couldn't help but thinking of that example where you have an issue around a discount [00:25:00] and people thinking they should be getting it and they're not. Designing that experiment and the hypothesis and picking which metrics to track when that must have been a pretty interesting one.

Anuradha Tempe: Yeah. I think that was a very challenging feature launch because we were also migrating our platform to a new platform. And so it was a parallel technological migration as well as a new feature launch. And yes to your point having that active feedback from sales teams and our NPS teams is important because those particular set of customers, we call them EPP, which is our employee partner program segment, and they always receive a discount.

So they were expecting to get a discount on a particular product, but they weren't seeing any in cart and checkout. And instead of getting a frustrated customer in cart and checkout, it's better to have them receive that message early on that, "Hey, this product is not [00:26:00] open to EPP discounts," is something that helped reduce the volume of support tickets that were going up exponentially because of that absent messaging.

Ashley Stirrup: Yeah. such a great example. So as you think about kind of the future of experimentation for Samsung, what do you s- what do you think of as some of the priorities?

Anuradha Tempe: I think in terms of there are lots of there are new features that I'm working on new products that are being launched. We also are trying to integrate a lot of generative AI solutions in each and every product launch that is going out this year and in future. In my mind, making sure that we are not hung up on writing documents or specs and as that is the ongoing trend i-include generative AI everywhere.

So I have been running a lot of experiments to see where I can improve operational [00:27:00] efficiency, and I came up with an A/B testing copilot. So if I just input my roadmap now, it will generate a hypothesis and an initial plan of A/B tests that I don't need to spend 45 minutes drafting. So coming up with those kind of operational efficiencies to decide which features need A/B tests and then having guardrails for that.

Is this low confidence, medium confidence, or definitely requires A/B tests, so high confidence? And then give it to the correct teams in order to roll that out. So automating processes is obviously important, but also having that factor of trust is important. So I like to have a human touchpoint to ensure that we are not the model is not hallucinating or re- leading to a result that we are not inclined towards or keep us honest there with some kind of human [00:28:00] intervention in everyday or day-to-day activities.

But yeah, and I've been reading up on sequential testing, so maybe that's something that I'm trying to educate myself on how we can personalize A/B testing and come up with AI-driven A/B tests that minimize human intervention.

Ashley Stirrup: Yeah. Yeah, I'm convinced at this point that AI can be a massive turbocharger for most businesses, but we haven't reached the stage yet where, you're gonna be replacing a lot of people. It's more you're turbocharging the people that are using it. Obviously, it's always a possibility, and then you've got the question of are we gonna reach AGI and suddenly AI is better at designing an experiment than you are.

But in the meantime, I think it can really help automate things. It can help you make sure that you're, designing experiments with rigor and help you accelerate the analytics process, but you still need the human in the loop to really, get [00:29:00] the most value out of it.

Anuradha Tempe: We may not need as many humans as we do today, but we still need a human touchpoint to make sure that we are not steering away from the result we actually intended to have from the get-go. So yes, completely aligned on that.

Ashley Stirrup: Anuradha, this has just been a fascinating discussion. I think your business is just so interesting. Obviously, Samsung is such a, a leader in the modern technology world, so it's been really exciting listening to all of your stories

Anuradha Tempe: Yeah, and thank you for inviting me and to have talk as a guest on your podcast. And always it's fun to learn more about different trends in the industry, and that's the only way you grow. Thanks for having me here.

Ashley Stirrup: Awesome. Thank you so much

Anuradha Tempe: All right. Thanks. Bye

[00:30:00]