Good product managers are customer-obsessed. They build products and features that will solve their customers' problems. But Peter Yang from Reddit argues that great product managers focus on how products make their customers feel. Dive into this episode with us as Peter teaches us all about the emotional wheel and how PMs can learn a thing or two from gaming.

Today's Guest – Peter Yang

Peter Yang is an experienced product manager with a wealth of experience from tech giants such as Twitter, Twitch, and CreditKarma. Currently, Peter serves as Reddit's Senior Group Product Manager for the platform's audio product, Talk. When he's not building great products, Peter is teaching aspiring PMs how to do the same. Peter has sold over 5,000 copies of his book, "Principles of Product Management."

Highlights

How have you defined product sense for yourself? How have you developed it?

Peter: Product sense is kind of like a fancy word for just understanding what the customer wants and what problem you're solving.

There’s two experiences that come to mind that have had a big impact on me. The first experience was when I was working at Twitch/Amazon. I really liked Amazon’s “working backwards” process. To summarize it, though, I learned the importance of asking a few, crucial questions. When you think about any product; number one, who is the customer, number two, what is the customer problem?

And how do we know that this problem actually exists? What is the most important customer benefit that you want to provide? A lot of products want to provide multiple benefits. But it's very important to focus on the most important benefit. Another thing Amazon does is that they make you leap forward in time and write a press release as if the product has already gone to market.

This practice forces you to talk about a product using customer language. This helps us align multiple stakeholders.

Another experience that had a big impact on me is during my time at Reddit. Reddit is the original community platform. During my time there I started becoming more active on Twitter. There, I came across a concept called “community-led product development.”

Many PMs just ask questions to their customers. But actually building a customer community is a really powerful way to understand what product the customers want, but also to get early adopters to champion your product for you.

Can you tell us more about the benefits of community? What are some of the pitfalls that we should be aware of?

Peter: With community-led growth, you have this almost daily feedback loop with your customers and, usually, it's the people who are most excited about your product that are willing to spend the time to give you feedback and be part of this community.

If they feel like they're part of the journey with you in building the product, once you actually launch it, they will be your strongest champion., right? So For example, as we’re building this audio product at Reddit, the moderators who are involved in this community are the strongest champions for this product.

And it's much better for another moderator to hear the product pitch from a fellow moderator rather than a Reddit PM.

I’ve also noticed that as companies scale and hire more PMs, they become much more metric-obsessed, as opposed to customer-obsessed.

PMs are measured based on things like how many A/B tests they push forward. So, they start to lose touch with the customer a little bit.

Part of that is also due to all the internal stakeholders that they have to talk to, and also even get approval from, to even be able to talk to the customer. For me, the reason that I do product is to talk to the customer and to build what the customer wants.

For founders that have done a few million in ARR already but are just starting to think about community– what tips would you give them when starting out?

Peter: The real magic from community comes from your customers being able to talk to each other. That's the key difference between the community and just doing something like a conference call with your customers.

So, to setup a community, you can use something like Discord or forum software like Reddit. Invite your customers into that community to build your product with you.

You don’t need to have that many people in the community. You can just start with 20 or so and slowly grow it from there. And once you have people in the community, make sure you are active in the community.

Just have authentic dialogue with the customer. You don’t need to be too formal. It’s fine to just talk about Squid Games with your customers.

The important this is to make them feel like they’re part of your team. And I Once you do this, all of a sudden you realize that you’re not limited to your engineers and designers to think for your customer. You’ve extended your team by getting your customers involved. They want to see your product succeed, too. It’s super motivating.

In the past, you’ve said that good PMs are obsessed about customer problems, while great PMs are obsessed about how they want their customers to feel while using the product. What inspired this?

Peter: I did an interview with Rahul who is the CEO and co-founder of Superhuman. They want their product to be the fastest email platform ever made.

But I first reached out to Rahul to just get help with their product. He was more than happy to spend time to walk me through their product and streamline my workflow. It was very impressive that the CEO did this for a random customer.

So that goes back to the point of customer and community obsession. But the real answer to your question comes from video games.

Sure, video games don’t solve any particular customer problem. They’re there for entertainment. But the best games, specifically RPG games, pull on emotions in players. I remember how I felt playing games like StarCraft of World of Warcraft and getting that feeling of joy and accomplishment through them.

People don’t necessarily remember your product for what it helped them do. They remember more about how it made them feel.

So, PMs that really care about shipping something great should care about how their product makes their users feel.

Superhuman has this whole emotional wheel that they use to precisely define what kind of feeling you want people to have while they’re using the product. That’s a good north star to have.

Can you double-click on the concept of an emotional wheel? What kind of emotions are they trying to optimize for?

Peter: The point of the emotional wheel is to get really precise about the emotions your product pulls on in your users.

For Superhuman, they are optimizing for joy. They want to help people get to inbox zero and feel a sense of joy. They don’t explicitly say “joy,” though. They go down the wheel and define joy as enthusiasm, excitement, hopefulness, optimism, accomplishment– basically more nuanced emotions like that.

You wrote a book called Principles of Product Management. We’d love to hear a little bit more about that.

Peter: The reason I wrote that book was because I struggled to become a PM. I failed the Facebook PM interview loop three times in a row.

It was very difficult and eventually I was able to finally become a PM. But I didn't want other people who are trying to become a PM to feel like they were alone in the journey. So, I wanted to share the advice that I learned along the way.

The book is split into three sections. First section is just about the core principles you should have as a PM leader. Things like taking ownership, having high empathy, and more. The second section is all about building a mission, vision, strategy, and a roadmap. This is where we dive into the actual process to build a great product.

And then the third section is about acing the PM interview. So, it's really targeted for people who are either transitioning to product management or new in their career as a PM and helping them get started off on the right foot.

Connect With Our Guest – Peter Yang

Big thank you to our guest, Peter Yang, for joining us on this episode of Product to Product! To learn more about his product thinking, follow Peter on LinkedIn.

Highlights

Can you tell us about your role at Webflow and what your company does?

Frank Ramirez: Webflow is a company that builds, what we like to think of as, a visual software development environment for no-code creators. You could think of it as a website builder, but our ambitions are much broader than just websites in the future.

We’re on a mission to enable anyone to create software on the web. I’ve been a PM at Webflow for the last two years. Recently, I’ve stepped into a group leadership role.

Now, I’m building out a team of exceptional PMs focused on, what we call, our ecosystem. We’re really thinking about how we can really create an economy of creators on top of Webflow. We want creators selling things like templates, design assets, design systems, and more to other no-code builders. Additionally, we’re thinking about how third party developers can really extend the core experience of Webflow and fill in all those use case gaps that we're never going to be able to get to ourselves to create an amazing experience for our customers.

Tell us a bit about your team and the PMs you’re working with.

Frank: Currently I’m working with three PMs. I'm looking to hire a few more as well. These PMs are all focused on different areas of the product– whether it's the core product or thinking of our content management system, which is what a lot of people think of as the core piece of what's slow in Webflow.

Some of my team is working on upcoming feature launches. We have PMs thinking about our ecosystem and thinking about how Webflow can be extended via APIs, SDK, and more.

Can you talk about the product sense skills that you look to nurture across your team

Frank: Product sense is a really important component of decision-making. At it’s best, product sense is a deeply informed, educated guess.

At it’s worst, it's just an unchecked assumption. So, as a leader, how can I look for that in PMs? To me, it’s all about uncovering the assumptions at any stage of product development. Especially in the beginning where taking a wrong assumption early on that is not built into a process of validation or checking in can have a larger and compounding impact later on down the line.

Early in their careers, product managers need to be encouraged to think in processes. They need to be explicit and look for ways to de-risk their testing. Tactically, those things can look like templates.

For instance, at Webflow, as we continue to grow our headcount across product, we’re thinking about elevating and really standardizing our processes. We need to ensure that every PM can follow best practices.

How do you know which bets you can make as a team?

Frank: A lot of it's going to come down to how strong your perspective is on the market, the customer, and the product that you're building.

I think that the stronger your perspective is on each of those, you’re in better shape for a risky bet. Especially if it's been validated through a bit of time. So, this is hard for a brand new PMs to do if they’re fresh to a market or a certain customer segment, or even working with a new product.

But if you've been working for a while in a certain area, I think you can leverage your expertise a bit more. And I think that speaks to the idea of pattern recognition. I can give you an example where I've leveraged it a lot and really leaned in on my product sense.

In visual design, any time you’re working with a canvas, you don’t want to bound your customer too much. You want your customer to be able to build whatever they like. So, naturally, there's a huge long tail of bets, use cases, and edge cases that customers will try to do or try to build that you won't be able to necessarily plan for. They’ll look like isolated cases.

If you have a deep conviction of the type of experience you want to deliver to your customers, then you can look at the commonalities across these disparate use cases and come up with abstractions that cover many of them.

You don't want to solve just one customer's problem. You want to generalize it.

Validating your tech stack is trickier than validating a simple design. How do you go about this?

Frank: We had a scenario where many of the engineering leaders were coming from a gaming background. So, they ended up choosing a gaming engine as a rendering environment for the UI.

No question that gaming engines can create great graphics. But what we didn't really think through was the product development process we were going to go through. So, we had to come back to that.

I think we can really illustrate two key points. Number one, what are the customer outcomes we're really striving to enable with whatever tech stack that we use? It should be a means to an end. People don't buy the tech stack, they're buying the product and the experiences we create for them.

Creating those hero flows to illustrate the grand vision really challenges our team to compare tech stacks that can help us serve this user well.

Bringing in that customer context is number one. Number two– how quickly of a release and iteration cycle do we really want? What expertise do we have on the team and how do we bring in other cross-functional members like product design or QA into this kind of development process?

Thinking of that as a little bit of a life cycle would have helped out a lot. We would have found that there's a high barrier to entrance for designers when it comes to gaming engines as opposed to a web engine.

Can you tell us about how you deal with inheriting old roadmaps?

Frank: I inherited a roadmap and went along with it without really questioning it. In retrospect, that was good and bed. Doing this definitely taught us a lot about our own development processes. We learned about the limits of our product that will inform future roadmap items.

But generally speaking, this is a common situation for any PM to come into. When you come into a new organization and there’s a lot of existing thinking done, you ask yourself, “do I just use it?” I think in some cases it's okay to quickly leverage someone else who was from that era.

If you can, however, I encourage first principles thinking. Spend time thinking about the key motivators and drivers.

What are the key outcomes for the product that you're owning? And then come up with a plan based off of that. But that can take weeks, if not months, especially if you're onboarding to a brand new space.

That's where a little bit of intuition comes in. That's where you're talking to other members of the team. That's where maybe you're leveraging more of your engineering lead or design lead, who may have more shared context getting up to this point.

You’ve described PMs as the centre of a spider web holding everything together. Can you talk more about why you use this analogy?

Frank: This is how I quickly visualize the role of a product manager. There's a lot of times when people talk about the role of a product manager, they tend to say there's user experience, then business management, and then there's engineering.

In the middle of that Venn diagram, that's where you are. That’s the PM. But going beyond the simple description, PMs need to have a bit of an ear to the ground with customer support and sales. They need to have a self-serve relationship with data.

Being a PM is really about having a central context point. Why and what is the organization doing?

I heard that you believe PMs need to find their “superpower.” What superpower do you think you might have?

Frank: I started my career in engineering. Did that for quite a few years and then decided that engineering was not for me. I realized that product management was what I wanted to do. So, I had to find a product management role. That’s a very difficult transition for anyone to make.

In those cases, what I recommend to people is to find their superpower. What is it about your background that you can really leverage and position yourself for? Whether it's company type, technology type, or stage of company. Where is that your unique background can be amplified by a PM role?

I think my superpower is communication, especially asynchronous communication. As well as a technical bias having been an engineer. I know that generally speaking, on every development team PMs are going to be generally technical, relative to the regular population. But you also have some PM's that are definitely technical because they were engineers themselves.

About Our Guest – Frank Ramirez

Big thank you to our guest, Frank Ramirez, for joining us on this episode of Product to Product! To learn more about his product thinking, follow Frank on LinkedIn.

Highlights

Why community-led growth? Why now?

Pulkit Agrawal: It's not a new concept, it's been around for awhile. There's been the early SaaS-style lots like Salesforce and HubSpot that have done a really great job at building community. Hubspot with inbound.org.

Salesforce is really driven by the whole ecosystem of solutions. But also a lot of recently IPO'd tech companies have been driven by communities. The trend of individuals buying from other individuals is continuing to grow.

So, we're kind of moving away from this world where there's top-down decision-making when purchasing software, especially in B2B. It's a much more dispersed software buying process. People are looking to buy from other people. We’re seeing the rise of influencers in B2C.

So this is really the right time for teams to be thinking about community and what the role of individuals within their community is and how they can be driving success through that. It's a really efficient way to scale growth rather than relying on organic or paid. If you've got a strong word of mouth distribution, then that really helps.

Are we early in community-led growth?

Latif Nanji: It's a yesterday thing. The importance of it has really to do with the nature of your business, especially if you're in B2C, to be able to grab that many eyeballs and attention to direct the right information and facilitate that growth is imperative.

When you think about some of the most successful communities, they help limit the lead gen cost for these brands. Whereas in the early days of a company like Salesforce, so much of their money was put into marketing.

And now, because of communities like Discord and Slack, you're able to leverage them. I think people should be starting to think about what it actually means in their own context.

Gabriel Fraga: About 10 or 20 years ago, community-led growth was a “nice to have.” There weren't a lot of companies that were doing it.

Companies had to be brave to start building communities. Today, though, because of changes in consumer behaviour, it’s a must-have. People don’t want to just buy a product. They expect more from the brand.

I think COVID has also played a big part in that, as well. Everything is happening online now. I was reading today that in 2030, a hundred percent of the world population will have internet access. So, imagine much growth we're still yet to see. Community is a must-have. But today is the right moment to start thinking about what it means to your brand.

How should startups think about communities when it comes to their go-to-market strategy?

Bob Moore: Community is a major part of the brand equity of companies like Shopify and Webflow, but I would also bet if you went back in time and looked at their seed pitch decks, none of them probably have the exact phrase “community-led growth” in there.

What’s been happening in recent years is that there's just been a bit of pattern recognition that's kicked in when you look back on a lot of the companies that have been the fastest growers over time.

It’s about creating dimensions of value for your customers and folks that are in your pipeline. Even all the way up to the discovery phase who have never engaged, or who are really far away from buying. Part of your growth strategy should include allowing these groups to create value for each other, together.but doing things that kind of allow for the existence of a group of those people that have that same common interest to actually.

Partnerships is a perfect example of one dimension where you have a different kind of community at multiple tiers.

There's the community of people that are your end consumers, but also a general audience in the specific industry you’re targeting which is their own ecosystem of other technology companies that their product integrates with. And that may have data flowing back and forth in between. The way in which you work with those companies creates this kind of meta-ecosystem effect, where if you have the same ideal customer profile, but you're not a competitive product.

You can actually multiply the size of the community that you can access and that you can cultivate by working really closely with those partners. And it turns out that one of the fastest and most effective ways to build community.

And the more and more you see of that, I think the more you'll find that you can then tap into the downstream effects of having everyone in one place.

What is the difference between being community-led versus product-led? Do you have to pick one or can you do both?

Bob: There's no answer you can give here that's not going to make somebody angry in some way about the definition being slightly off. But I think in my mind, when I think about product led growth, I think about the way in which actual user experiences that exist within the course of using a product, drive them to become more engaged or, ideally, to actually drive other people into the product and create these feedback loops.

These are viral growth mechanics where the core value proposition of what people are doing in the product is something that ends up extending value out to other people which creates these virtuous effects. To me, with the community-led, you can almost draw a bigger circle around that. Product-led growth you could argue is part of the greater community-led strategy, which is really just saying how, how come.

This is really looking to accomplish the same thing as a positive feedback loop. And it can be done before people are necessarily in the product. You can do it just by virtue of affiliating and associating and following our brand. Make it such that education becomes lower friction. Making people feel like an inside helps to start developing loyalty before they actually open their wallet and pay for a product.

So, I think they're related. I think you could make an argument that one sits inside of the other. But ultimately it's going to be fairly rare that a company can get benefits of one and not also figure out ways to implement the other.

Latif: I might be a little bit at risk of disagreeing here, although I do think that one is a subset of the other, I believe it's the inverse relationship. I’m happy to be wrong here, but I think when a company is starting the most important thing is product-market fit.

The first thing you're actually doing is going to be related to elements of the product, like growth, which are really intrinsic to the product. But after, there's that unique experience with those first subset of users, then you're going out and finding some aspects of the flywheel.

And with product-led growth, I actually see the community as a subset of that, to be able to generate the users that are coming. So that's part of that flywheel, and community-led growth reduces the friction to enable more product-led growth. When you go to that zero-to-one journey, it's almost always about just getting the product and then surrounding it with users to continue leveraging the growth that you.

I look at it as both. It can be a product or a marketing-led function where you want to figure out if you want to be doing a lot of stuff, synchronously or asynchronously.

Once you’ve got that product-market fit, how would you drive growth?

Bob: It has a lot to do with market entry and the sequencing of how the business gets built. We're very much running what I would consider to be like a category creation playbook.

We almost have to go and define what our product is and what the best practices are in how to use it before people are able to actually use it and engage with it. Because we're kind of changing the paradigm for how people use technology in a certain user persona. And because of that, my bias is probably where it's saying the community stuff comes first.

We were writing deep, thought leadership blog posts before we wrote our first line of code because it was very much about that “creation motion.” Whereas I think probably for the majority of businesses it is the inverse. Differentiation is going to come from seeing evidence of that product-market fit. Which by definition requires having a product there.

So, make sure you invest in product-led growth mechanics first, and then it kind of builds up around that engagement. It kind of goes both ways, but it may not be that every company is going through that cycle. Companies start at different spots in the circle based on what their vision is and where they're coming from.

Latif: I love that adaptation to the definition. I agree. We came to a better definition because of those different types of businesses and where you're creating an entirely new category, you're going to have to do those things prior to building the product.

So, I really appreciate that point. And I think the takeaway from the audience is not worrying so much about the definition, but what actually matters to the business and which order you need to actually do them because you've got a perfect exit where, maybe you don't have to go build the product. You have to actually educate the market and segue into it while you're building.

We’ve had user-generated content (UGC) on social media for a while. What’s different about communities today?

Gabriel: The concept of community hasn't really changed. But how community is applied to a business has changed massively. I think social media had a big boom pre-financial crisis in 2008.

And in 2010, there was a lot of heavy investment into social media. That's when I started building community, around 2010 at Sony.

Social is big and everyone is in there and it's noisy. It's also affected by politics. It's difficult to find a space where you can have meaningful and valuable conversations with your customers.

And this has led, in my view, to the micro-communities that we are starting to see where every company must have as their own community in different, closed spaces.

It could be a Facebook group, it could be a Slack channel. It could be their own technology. It really depends on the company. But we're shifting from this massive, Facebook style social platform to micro communities. These are meaningful communities where you can have focused conversations about your company, your product, and what your customers are trying to achieve.

And they don't have to be millions of people. I think this is something that I like to stress as well, because you think of numbers, how big does my community need to be?

There isn't really a right or wrong answer, though. It's just about the community and what your mission is with your customers.

How do you think about which platform to choose for a community?

Gabriel: Back in the day you didn’t have a lot of options. Today you have a broader spectrum and it really depends on what you want to do. Webinars, for example, are a great way of building community. And now you have online event platforms that have community at the core of it, as well. It really depends on what you want to do, but there are lots of different choices.

How do you decide on the kind of community you want to build?

Bob: This reminds me of the work of Alex Hillman. He's a really great thinker on community building. One of the distinctions that he always makes is he says that communities can fall into one or two buckets. You can have a community of interest, or you can have a community of practice.

In a community of interest, people all joined together because they all like Star Wars, for example. A community of practice is where people are all actually working on something together or in the same way and learning how to become better at that thing.

His thesis is that the communities of practice are the real communities that work and that really communities of interest can, ultimately, fall victim to not being a source of real social connection and engagement. This happens when the communities get too wide and note deep enough.

So I think about it like this: how effective are we being at building a community of practice and what does that look like?

Even when there’s a common interest, what we really care about is the fact that they're getting really good at using Crossbeam by virtue of participating in that community. They're becoming better practitioners of the thing that they're there to learn more about.

What are the challenges in building an effective community?

Latif: When thinking about a community and the responsibility of it, your brand needs to be the first consideration. What is the brand mission and what are you trying to contribute to that mission? Second, determining how wide and horizontal you want to start, or if you want to go deep vertically.

So when I think about the challenges of creating that community, it’s starting with where is your business at?

If you're in a company that's like 25 to 50 people, you want to be able to start to build out the tentacles of interest. Start making people aware that these potential communities of practice could exist in the future. And so one of the things, as an example, we did, as we looked around and said, “what right now are people gravitating for but people aren’t paying attention to?”

And that’s how we decided to build out podcast community. We decided, okay, the types of users that we're serving are PMs and they love async information because they're really busy people and they want the highest quality so that they can download best practices and use them as data points.

We wanted to make sure that it was in line with our brand and we wanted to get the top and best people we thought that could speak to specific areas of interest. So they were very topical subjects throughout the themes of the year. Then we also had to make sure that we continued to maintain that level of quality of content over a period of time, because the bar always continues to get higher and higher.

What are some tips you have in attracting members to a community and then engaging them?

Gabriel: The main challenge is understanding the purpose of your community. After you build a following, the community members want to understand their role here. What are they here to do? Once you nail that then you can figure out which technology platform is right for you. Podcast? Newsletter?

Typeform actually didn’t have a house for our community. But that community exists. So you have to go out there and attract that community and bring it home. And that was our main challenge. Where's our community? How do we bring it to one home? And we didn’t have all the answers at the beginning. We had to go through a bit of trial and error.

We also humbled ourselves. Although we built our product, we allowed Typeform users to educate each other and the best ways to use the tool in practice.

Bob: I do think there are these different tiers of community within your product. If you're doing product-led growth, you'd look at something called a user maturity curve. In anybody's product, they might be different things, but you know, for us, it's a signal of like, have they added other users to the platform?

Have they connected a data source? Have they connected with a partner? You can kind of do a similar version of that as like a community maturity framework. For example, have they subscribed to your newsletter? Okay, if they’ve subscribed, have they opened the newsletter?

Looking at the engagement and activity allows you to go through this process of micro-funnels.

How do you measure the success of your community building efforts?

Gabriel: There are two things that you need to look. One is activity ratio. Are they attending webinars? Are they messaging other community members? Are they creating their own posts?

This helps you understand how helpful your community is for your customers. Sometimes you might find that communities are actually faster at helping your customers than your own support teams might be.

And then you have engagement. This is what we’re looking at at Typeform right. We want our community to drive more product engagement. Is our community helping our customers do more with our tool?

Sometimes measuring the success of community can be hard, but we know that if we build one then our customers will be more loyal to the brand.

How do you market a community pre-launch, especially if you have a small following?

Bob: This is when you’ll have to do that classic Paul Graham thing of doing what won’t scale. Very early on, we latched on to anybody that we could have a conversation with. Anytime we saw them get fired up, we’d turn them into micro-celebrities in our own little world. Whether it was interviewing them for our blog or putting them on the website, we’d use them to build up credibility and thought leadership.

We really tried to make the first dozen or so people in our community feel like as if they were founding this company with us. We weren’t shy about sharing the spotlight. We wanted them to feel a certain sense of ownership in this.

The more people feel invested, the more likely they are to post or tweet about you.

Learn more about our panelists

Pulkit Agrawal
Co-Founder & CEO at Chameleon
Connect with Pulkit on LinkedIn

Latif Nanji
Co-Founder & CEO at Roadmunk
Connect with Latif on LinkedIn

Bob Moore
Co-Founder & CEO at Crossbeam
Connect with Bob on LinkedIn

Gabriel Fraga
Sr. Community Strategist at Typeform
Connect with Gabriel on LinkedIn

Highlights

Before you got into product, it seems like you had some experience in high school science teaching. You had said that teaching exposed you to outcome based thinking. Can you talk a little bit about that experience and what outcome-based thinking was like in that environment?

Michelle: I didn't have a typical path into product and I think that's something you hear more and more these days. The path is always fairly windy.

I started out actually as a pre-med student. I was really focused on becoming a doctor. I did everything that I possibly could to get the experiences that would lead me to this path of one day becoming a physician.

So, I spent some time in my senior year of college working on a health mission to El Salvador. One of the things that I was doing there was organizing and leading a group of students to do both onsite one-day, medic brigades in different rural communities across El Salvador, as well as public health workshops. These workshops would focus on things like dental hygiene and feminine care.

However, what I started realizing when I was in that kind of environment, is that I was actually never in the room with the doctor. My goal was to be in the room with doctors talking to patients so I could spend my time hearing, absorbing learning. This wasn’t happening. I’ve had a couple of other experiences too that weren’t quite what I wanted. And so, I actually stepped back and I said, you know, I'm not sure if going straight into medicine is the right thing for me.

That led me to believe that maybe I just needed to take a pause and reevaluate. Medicine is also very cost prohibitive. Considering my options, I also was an education undergrad minor. So I ended up joining Teach for America.

I knew that I could do science. And so I said, “Hey, I would love to teach high school science.” I was looking to go back to my home state of Texas and teach ninth and tenth grade chemistry and biology.

I started to really get the whole curriculum down. I knew how to balance chemical equations, the cycle of life– but I was still doubtful if I could really teach students effectively. How can I measure if I’m getting the students to where they need to be?

At that time, everything was really oriented and organized around an end of year test. It was a state standardized test. In order to progress to the next grade you had to achieve a score of 70 or higher on this test. So as a new teacher, I'm like, “okay, well, let's take a step backwards.”

If my end goal is to get these kids to be able to pass a test at 70% proficiency or higher, how do I organize an entire quarter or a year around the outcomes that are needed to show that they actually understand? Teach for America had a really systematic way to approach teaching using objectives. They also made sure you were thinking about classroom management.

So, one of the things that I really started then leaning into was objective-based thinking. If your objective is, “What are the cycles of photosynthesis?” You have to be able to break those steps down to their components and then be able to align the content and the exercise around those individual steps.

What I would end up doing is creating this tracker that could break down every single objective, align every single objective to a specific point in the text or curriculum that talked about that particular topic.

And then at the end of the day when I had either small quizzes or tests, I can align every question back to a specific objective. Then I can look at my color-coded Excel and see whether or not the students actually got the objective. What percentage of my students understood and mastered that objective? How many were either in yellow or red?

With that, I could actually create different cohorts or sections within my classroom that really paired up people who are stronger with people who are weaker in a particular area or objective. I also could use that to say, “wow, every single one of my students are in the red. They did not mess with this objective.”

It would help me look and target areas where I might need to either rejig the way that I was communicating something or I could go back in and self-reflect.

I was really able to leverage this approach to actually move the outcomes of our students to some of the highest in our district.

Can you tell me more about how you relayed your experience from inside the classroom to a career in product management?

Michelle: The most surprising thing about my career is that when I reflect, I contribute a lot of my success to those formative moments in the classroom.

In the classroom, I wasn’t just looking at an aggregate of data but I was also using it to experiment. I had an A/B testing perspective. It gave me the ability to look at the different tools I used, what prevented the outcomes, or what accelerated them. I could reflect on different tactics I used and evaluate how students responded to them. Now, I’d ask myself, “how can I integrate that into seemingly less obvious objectives or pieces of the curriculum?

Through your career, were there any defining moments where you really started to gain your product sense? And what were those pieces of information you were really soaking in to become the product leader you are today?

Michelle: My background in science set me up for a very outcome-based hypothesis-driven framework to approach a lot of problems. What data do I have that leads me to believe XYZ hypothesis? How am I going about evaluating that? What experiments might I come up with?

In my world of education technology at Pearson and Cengage, we really didn't have the ability to A/B tests. But we did have the ability to talk to users. We had the ability to come up with different solutions based on these hypotheses and then go and put them in front of users, ask them questions, and see how they react.

It wasn't necessarily about what users were saying. It was really about how they were engaging and where we saw friction in their experience. That really helped me build that intuition that I needed.

But it wasn't just about what users are saying or what they're asking for. It was more about digging past what they were saying to understand how they were actually engaging with the product and that flow. Where are they coming up to friction points?

In my earliest years, it was really about talking to users and deeply developing that skill of empathy. I was putting myself directly in their shoes. There were some times where I would flip the persona. I would take the lens of my 60 year old mother. How is she going to interact with this? How do we get her to become an active user? That’s where I started.

Once I was able to join an organization like Kayak that had A/B testing built into its DNA, pairing t user generated insights with actual data and the ability to test those hypotheses really unlocked my ability to move quickly, but also to orient on the right problems.

Is there an example that you experienced where the data just spoke the language for everybody in the room, but your intuition felt like they were being misguided by this data. How did you handle that?

Michelle: At Netflix I worked in the kids and family area where our audience were kids and their parents/caregivers. Generally, if you think about the audience of Netflix, it's a very broad population. Different devices, different demographics, different locations.

But what keeps bringing people back to Netflix is the library of content that continues to be refreshed. Users discover new content, new favorites, and it's exciting. There's always something new to discover. The company has built itself around the “binge” that they’ve created. Netflix is oriented around “how do we ensure that there's always something new for a user to discover?”

For kids, that's not really what the data is saying. The research is saying that kids want their favourites. They don't want the next new thing. They want to rewatch, over and over and over and over again, their favourites.

It doesn't matter if it's the 10th time or the millionth time. They want to see that thing over and over again. And there's a variety of reasons why this is true. From a developmental standpoint, you're still learning the language at that age. You're learning interactions. You want comfort, you want safety, you want something predictable and known.

We could see that in the research.When I first joined Netflix, though, the organization was really trying to continuously push on helping kids find their next favorite.

There’s a number of reasons why that was important to the business at the time. But my intuition was saying that we really need to help build the trust with our youngest members– the kids and their parents.

We came up with this hypothesis that kids aren’t asking their parents for Netflix, they're asking for Pokemon. Kids aren’t turning Netflix on to browse. They know what they want.

And if we can help build up that trust and that comfort– and really that self-sufficiency that a kid can come in and navigate on their own– then it’s better for the business. We’re helping the kids build a system of comfort.

We can then use that comfort to fulfill our business and content goals of helping them discover. That was the big learning for me there. I came into the common belief that we needed to get kids to discover new content in order to get them to come back to Netflix.

But the data was saying something different, the research was saying something slightly different, and I think we proved that out with experimentation.

Can you tell more about those experiences where you’re doing a user interview but your observations of the user are more useful than their actual responses?

Michelle: That ability is one of the things that sets a good and great product manager apart. Being able to move through and move past what users are saying and interpret deeply. Really examining how they're saying it or what they're doing in conjunction with what they're saying.

Here at Hinge, one of our goals is to build a dating app designed to be deleted. We want to help users find their match, their pair.

We are the intentional dating app. Which means that we primarily attract a population of users who are interested in longer-term relationships and not necessarily hookups.

So one of the things that users always come and write to us about is that they want to understand what the other user is looking for. What is their intention?

Are they looking for something casual? Are they looking for something long-term, something serious? Having these answers is going to help them find the cohort of users that best represent their intentions, as well.

Well, it turns out it's not as cut and dry as that. If we just put a feature on the app that says something like, “are you looking for a long-term relationship or for something casual?” that is going to be impacted by who you actually meet on the app.

Do I meet this person or that person? And then how do I respond to them? It’s really not as straightforward as saying yes or no. It’s much more nuanced than that.

When we deliver features like this, it never works out in the way users expect it to be. It's either too restrictive or it’s not specific enough and we're not meeting their exact needs.

So, the question then isn’t “are you looking for a long-term relationship or are you looking for something casual?” It's more, “how do I understand how much you're going to invest in this experience?”

Whether or not it turns out to be a long-term or short-term relationship doesn’t matter as much here. This is the real question users are asking themselves: Do I put a lot of effort into my profile? Do I have longer answers to my prompts when I send it? Do I send it with a comment or not? And so there are other things around intentionality as a topic that are actually more meaningful than me saying “I want a long-term relationship or I'm looking for something casual.”

How are you thinking about the future of what tools you use to gain information? What are the tools you use to help get your team up to 11?

Michelle: I actually think that this is one of the single most important things that product teams can do that will determine whether they are really successful or just meeting the standards. Product teams are a team. They need to be symbiotic and have close relationships with one another– data scientists, researchers, designers, product managers, engineers, QA.

We're all working together to facilitate some end goal and verbal and written communication are the two most important things to ensure that there is alignment. We all need to understand what is happening, when it's happening, and why it's happening. That’s why it’s really important to have written documentation.

Did we really set up the narrative of what's going on? Hey, here's some background context and some data that we have that lends itself to support the idea that if we do X, Y, Z thing, that these outcomes will be the result of them. Beyond just having the framework, this is why documentation is important. The other thing that I think is really important upfront is the decision-making criteria around whether something will or will not be successful.

That's where you really start to dig in with your data scientists to understand how we are going to measure this. What things do we need to ensure we have to make decisions? And then based on this we’ll respond by making that decision. Aligning on that upfront saves a lot of time back and forth in the debate of how we are going to proceed.

That's number one. Before we actually get to the actual execution of a product spec, you need to gain alignment on the roadmap of work.

Then, all of that is aligned to what you as a product team are actually trying to deliver both for your users and then for your business. How are you going to measure that?

I really work my teams to say, “Hey, what research and data do we have? What have we seen being successful? What gaps do we currently have within our current structure that is preventing us from serving those users? Or if we don't do this, are we at risk of major competitors coming in?”

It’s all about enabling product teams to do a lot of that discovery work for themselves. As a leader, it’s important to be clear on those outcomes.

I do think that people oftentimes will jump straight to solutions, right? If we see a 30% drop in a certain metric then people will want to find a solution right away.

In fact, I think it's more important to understand first the variety of reasons that might be contributing to that. And which of those are the biggest areas for us to go and investigate?

Do we have enough information about any one of these opportunities? So I encourage my teams to go very deep in opportunity assessment before they even get down to solutions, because everybody can have solutions. The question is, are you working on the most important areas and do those then actually align to those important opportunities?

Or are you working on a solution that is actually only impacting 10% of users when there's this other big opportunity area that impacts 90% of users. So a test in that area versus the other area would be that much more effective and have higher potential.

Can you talk a bit more about opportunity trees? What does that look like?

Michelle: A friend of mine actually, a few years back, shared an article from Theresa Torres where she talks a little bit about opportunity assessment trees and why we need to do more discovery.

We spend so much time brainstorming what features to build and not a lot of time talking about what area we could go and do research in and explore to potentially build features to solve within.

In this framework you start with your objective, your outcome. For a particular team, it could be to increase the rate of profile completion.

So we want people to complete their profiles at a higher rate than they are today. And there could be a variety of reasons why people aren't completing their profiles. The onboarding process might be too long. The onboarding process may be unclear. Maybe users don't have time.

There are then a bunch of big areas that could be causing this friction. Without a proper discovery process you might assume that onboarding is too long and then run off to fix that.

There are all these gaps that could present themselves, but you would not have explored them. And so the way that this works is rather than jumping to solutions, you actually start with what are all the things that could be potentially contributing to this problem.

That's where we start. We start with OKRs. And then we're going to talk about all the ways that users might be impeded or success might be facilitated through. Next, we'll have to look at the data we have on this.

From there, that’s when you can assign some opportunity sizes to those different areas. Here you can work with your team and let your imaginations run wild with solutions.

Finally, it's just really about executing on different A/B tests and having your hypotheses pretty tight and your experiments frameworks clearly articulated and I found that with this kind of framework, my teams are able to do 30 to 40% more effective tests and actually get way more learnings out because you're not just building to fundamentally change something. You're really building to learn here and to improve.

Latif: Yeah, I see that as a great lateral thinking methodology where someone might fixate on a problem and maybe the problem adjacent to it on each side. But this tree just says every possible problem in that relative domain needs to be uncovered. That sort of a brainstorming experience that does require some creativity can really have an impact.

Michelle: The nice thing about that is that now you have a whole roadmap, right? So you're not having to go back and say, “alright, well, this thing didn't work. Now we have to go through this process all over again.” You kind of did all of that work upfront.

If this particular feature or test that you aligned on doesn't produce the results that you were interested in you have a very clear way to iterate. You're not stuck.

About Our Guest – Michelle Parsons

Big thank you to our guest, Michelle Parsons, for joining us on this episode of Product to Product! To learn more about her product thinking, follow Michelle on LinkedIn.

No matter how you choose to represent your agile product roadmap, there are certain standards you just gotta follow.

1. Treat an agile product roadmap as a statement of intent

First thing’s first: approach your agile product roadmap as a statement of intent. (We’ve covered this already.) Treat your agile product roadmap as a living document that can (and 100% will) change.

“From just a mechanic standpoint, making sure you have a roadmap that can actually be updated is really important,” says Ahsan Nanji, Director of Product Management at Electronic Arts. “I’ve seen PMs struggle with creating a plan that doesn’t build any flexibility into it. So you want to — with either the software or whatever process you’re using — be very adaptable to change.”

If you’re too hung up on dates or milestones, you restrict your product roadmap and ultimately set yourself up for failure. Give yourself breathing room, so that way you can sync your product roadmap nicely with the flexibility and changes you’ll experience in an agile environment.

Download our ebook to see some real-world examples of agile product roadmaps.

2. Determine if time is right for your agile product roadmap

When we asked Casey McKinnon, VP of Product at ecobee, about his agile product roadmap best practices, the very first thing he brought up was evaluating the role of time for your roadmap.

“Well, first of all I would consider if time’s a factor,” he says. “If the idea is that you need to hit a specific date, you need to compare that to something where you can put your product out into the world and there are no consequences for when it’s launched.”

Determine how time plays a role (if at all) in your day-to-day function. If it doesn’t play a role: awesome, your product roadmap won’t incorporate time.

Maybe a timeline is for you. Maybe you’re not that concerned with dates, so you want to employ fuzzy time buckets. Or maybe you have a million sprints you need to show on your roadmap. Whichever way you approach time, knowing whether it applies and its particular role in your environment will lead to a more effective agile roadmap.

3. Remember who’s seeing your agile product roadmap and tailor to them

Creating a product roadmap without an audience in mind is pretty useless. The whole point of a roadmap is to communicate and align with your stakeholders. One of the very first questions you should be asking is: who’s going to see this?

“You want to understand your audience first, because they want to see different things,” says Latif Nanji (our CEO here at Roadmunk) when we asked him about his agile product roadmap best practices. “If you’re building this agile roadmap for engineers you can get a lot more granular about information, whereas sales and marketing are more concerned about the ‘what’ and ‘when.’”

Knowing your audiences means knowing what expectations to set; you know who you’re catering to. Imagine a mystery novelist showing up to a romance novel convention, expecting to sell out. It’s not going to happen. Romance novel fanatics are looking for very specific things (i.e. scandalous cover art and tales of unrequited love) that a mystery novelist (most likely) won’t offer.

The same goes for your product roadmap. If you’re expecting to get buy-in from C-level executives, but you’ve created an agile roadmap specifically for developers, don’t expect a very productive meeting.

“One of the worst things you can do is have everyone see the same roadmap,” says Latif.

If you know each stakeholder has individual goals and concerns, you can’t show up to each meeting with the same exact plan. Offer something relevant to them. The result: expectations are managed and stakeholders can feel assured that their needs are being addressed.

4. Know your product market. Like really know it.

This seems obvious. Of course a PM should know their product market! But we’re talking really knowing what upcoming features users want, beforehand. In agile this will change — and quickly. But a strong market knowledge makes your product roadmap much more adaptable to inevitable changes.

Sarah Pyo, Product Manager at Expedia, puts it this way:

“Understanding how your product works technically will help with prioritization, because you’ll know different levers to pull to drive your product. What new, futuristic features can we push out there and what is the demand?”

Recommending a user-centric method to collect market knowledge for your product roadmap, Rahul Kulkarni, Product Manager at Shopify, says:

“Validate low-fidelity mockups with customers as often as possible. Get real feedback and parse out what’s relevant.”

Chances are your market will be disrupted (many times), and as a result, so will your product roadmap. Keeping tabs on users’ wants and needs will prepare you to be just as agile; allowing you to update your roadmap swiftly to reflect any market shifts.

5. Check-in with your agile product roadmap (as a team)

This should be the easiest guideline to remember because it’s a natural outcome of agile. Working in short cycles, you’re immersed in a constant feedback loop. You’re regularly forced to stop and ask yourself, “So what worked and didn’t work this last cycle?”

This introspective (and zen) behavior pushes you to check in with your roadmap often. Your document should always be up-to-date with your product strategy and if you’ve created (and you should have) a living document that can change, it’ll be easy to update to reflect recent changes.

Casey McKinnon agrees with this practice, but emphasizes incorporating the whole team:

“You should continue to revisit your roadmap with your whole team. Don’t let it be an artifact of product, but an artifact of the whole company. This way we all agree, we all bought in, and everyone’s really excited about the roadmap.”

As a PM it’s your responsibility to ensure you’re sitting with your roadmap consistently, but also making sure you’re roping in your team. This way alignment is not only easier, but way more consistent.

Once your plan is planned, use our agile roadmap template to hit the ground runnng.

Behind every effective agile product roadmap is a well-thought out plan. As true (and cheesy) as that statement is, it’s also a bit of a weird statement since a product roadmap is meant to be a (flexible and transparent) plan that aligns stakeholders across your organization.

So what we’re saying is you need to plan for your roadmap plan. A truly effective agile roadmap (i.e. plan) is one that people will have confidence executing. And without proper thought put into that roadmap plan, you’ll be hard-pressed to find individuals willing to stake their time, money and energy on your product roadmap.

You might be asking: what’s the point in planning for an agile product roadmap when it will ultimately change? You’re not wrong — your roadmap should be a statement of intent that you update frequently. But you should be updating your agile roadmap to reflect changes brought on by your agile environment, organization and market — NOT because you poorly planned your roadmap. Prep work for your agile product roadmap can prevent unnecessary changes and even complete overhauls of your plan.

For all the PMs we’ve spoken to, putting together a roadmap plan is a key step in their workflow — particularly in agile. So before you open up your roadmapping tool (or app or whiteboard or spreadsheet... shudder) to create your agile roadmap, here are five steps you should take for effective agile product roadmap planning.

Real-world ways to build agile roadmaps. Get the helpful, non-boring guide. Download it here.

Step 1: Set the goals for your agile product roadmap plan

Your agile roadmap communicates with and aligns stakeholders, but what are you addressing? Even if you have a list of impressive features to release, the big picture must be considered. Your objective might be achieving product-market fit, or launching “sticky” features. Whatever the goal, your roadmap should convey the plan for achieving your high-level strategy.

Without established objectives, expect stakeholders to walk away from your roadmap with different priorities (the TOTAL opposite of what you want). Your roadmap should always tie directly back to your primary goals so that stakeholders are clear about your product strategy.

Ask yourself if the document speaks to your company and product vision

Bonus question: If you want to get reallyyy specific, also ask yourself what key performance indicators you want to focus on. Planning your main KPIs will allow you to create a roadmap that explains how you’ll hit those targets.

Step 2: Evaluate your resources for the roadmap plan

If you have ten developers, but two are on vacation and one is shared by marketing, how much can you ship in your next sprint? The answer: probably not as much as you could with a fully available team of ten developers. Planning for your agile roadmap should address your available resources and their constraints.

Another thing to plan against: the speed at which resources operate. You’re going to get pushback on how fast (or slow) your team is delivering on your plan—even in agile. Plan a cycle length that works best for your team — especially if you’re representing time on your roadmap.

Determining your team’s speed will let you develop a plan grounded in reality.

Step 3: Check-in with your users

It boggles our minds when organizations build roadmaps based solely on quantitative market data. User needs go much deeper than statistical research. Speaking first-hand to customers during your planning provides insights into disregarded investments. Before committing to anything on your roadmap, it makes sense to understand if your users, you know, actually want these things.

Additionally, user research makes a really, really strong case when getting buy-in.

User research shows stakeholders that your agile roadmap isn’t just a random, aimless plan you pulled out of thin air, but one developed from authentic user needs.

It’s pretty hard to refute real user needs.

No matter your agile style, check out our free agile roadmap template and make it your own.

Step 4: Determine your agile product roadmap plan's building blocks

Next up: establish your major themes, and the features that fall under each of them. Then break it down into epics and user stories. From there develop high-confidence estimates on your features, epics and stories, and identify any dependencies that could hold you back.

Hack away until you have all the building blocks you need before assembling your roadmap.

If this seems like a lot of work before creating an agile roadmap, we say: planning for these elements will prepare you to address (and defend) your choices when changes do pop up. And if changes are needed, you can speak to impacts and trade-offs that must be made.

Step 5: Hold planning sessions

This step isn’t so much isolated, but more ongoing and works in conjunction with the previous steps. Make agile roadmap planning a team effort by putting your whole product team in one room. This is your opportunity to really understand together your use cases, discovery work and objectives before making any roadmap investments.

Think of it as a way to get buy-in for the plan for your plan (again—strange, we know).

Early buy-in during your planning stage will be beneficial when getting official buy-in and public affirmations from your stakeholders.

Want to see real-world examples of well thought-out agile product roadmaps? Check out our ebook (and also get a handy checklist to add structure to your planning process).

We've got even more agile content for you. Get the real-world guide to agile roadmaps here.

Now that we’ve established that the idea of an agile product roadmap isn’t counterintuitive, we wanted to know how real-world agile teams were using product roadmaps.

We picked the brains of five super cool product managers to find out the reality of agile roadmaps. How are real-world, goal-crushing (eternally stressed out) PMs approaching their agile roadmap strategies? Sharing their roadmapping processes, as well as irksome agile roadmap myths and challenges, these PMs clear any confusion around agile product roadmaps. Here’s who we chatted with:

• Breanna Hughes, Senior Product Manager at League Inc.

• Catherine Shyu, Product Manager at FullContact Inc.

• Ahsan Nanji, Director of Product Management at Electronic Arts (EA)

• Sarah Pyo, Product Manager at Expedia

• Jessica Johnson, Senior Product Manager at Goodreads

Get our real-world guide to creating any style of agile roadmap. Download it here.

Breanna Hughes

Title: Senior Product Manager
Company: League Inc.
Industry: FinTech
Tweet at her: @unbrelievable

What do you think of when we say agile product roadmap? Go!

I guess it would be like a theme-based roadmap. At a high-level, it’s a problem-based roadmap that outlines the central problem. It’s not a roadmap of features and building. It’s a direction of where you want to go, but being comfortable to iterate as you go.

What’s your approach to your agile roadmap?

Based on business goals and KPIs we define problems we want to solve for the quarter through discovery. From those discoveries comes a list of features to build or tests to try.

Throughout the quarter we evaluate and reconvene. We take a look on a weekly basis with the product team and a monthly basis with the leadership team and ask, “Hey, do we even bother doing this?” That’s where the agile nature comes in.

What’s the biggest misconception around agile and roadmapping that you’ve heard?

That timelines don’t exist. There’s always a timeline. There needs to be! You need some sort of frame of reference to say how much you’re willing to invest in this thing — at least in startups. The reality for most startups is you need to know when this is good enough; when do we move on to a different strategic initiative.

What do PMs need to know before creating an agile roadmap?

KPIs. Definitely need to know why you’re doing it. What is the business impact and your goal for the quarter? You should also know what other teams’ goals are. Make sure you’re not in conflict of one another.

Also, gather tons of feedback. Based on the goals and the areas of the business I want to impact, I’ll talk to stakeholders and customers. Talking to your team is really important too — like your designers and developers. They have great ideas too. Talking to people is the best thing you can do.

What is the biggest challenge when creating an agile roadmap?

The biggest challenge I think is knowing the potential impact of what you release. You can have all the complex formulas and data scientists in the world helping you, but it’s an educated guess at best. Build small first and then build on top of that. It allows you to better predict what the impact is going to be.

Catherine Shyu

Title: Product Manager
Company: FullContact Inc.
Industry: Software
Tweet at her: @cthrin

What do you imagine when we say agile product roadmap?

I think of a loosely planned roadmap for the future of the product. Ideally it has an underlying structure and product principles the company wants to follow, but the exact items (and timing) have the flexibility to change.

Walk us through your own agile roadmap. How do you approach this process?

I’m a visual person, so my roadmap has different swimlanes for each platform. It functions more as a peek into the product vision and where it’s going, so it doesn’t have dates or sizes attached to it.

Once we start planning for the next quarter, the appropriate roadmap items will move into a new document where engineering can size them. We’ll end up with an overarching “Q# List” of items that product and engineering commit to together.

What’s the biggest misconception you’ve heard about agile roadmaps?

The biggest misconception I hear is that the words “agile” and “roadmap” shouldn’t even go together. Working in agile doesn’t mean you no longer have to plan ahead. It just signifies a mindset that lets you change direction faster based on incoming feedback from the customers, the market, and so on.

What does a PM need to know before starting an agile roadmap?

An agile roadmap should be directional, and not so detailed that it goes out of date as soon as you share it with people. Unplanned work will always pop up, and there will be projects that other departments need done too. In practice, adding dates to a roadmap often means you’ll have to update it anytime a date or scope changes. It can quickly become a burden.

What do you find is the biggest challenge when you’re creating an agile roadmap?

Trying to marry a vision for the future with the reality of time—while not having to update it every day. Roadmaps communicate:

• Where a product is going
• Timing of features
• Level of effort required for features
• How features are interconnected and more

We’re likely overloading this one document and trying to fit too much detail in there, which then can become out-of-date. The roadmap tries to do so much, and is consumed by so many different stakeholders that every company’s roadmap is a little different based on their needs.

Ahsan Nanji

Title: Director of Product Management
Company: Electronic Arts (EA)
Industry: Gaming
Tweet at him: @DarthEsso

When we say the term agile product roadmap, what comes to mind?

To me an agile product roadmap is one where the actual timeline and ordering of things aren’t necessarily set in stone, but are oriented around customer responses and requests. An agile product roadmap is kind of done with the intention of change being planned into the process.

Tell us about your own agile roadmap. How does it look?

So what we have is a timeline view of the product that gets broken down into a few different views. One view shows which internal scrum team is working on each feature and which client team adopts which features.

At a high-level, we always have a clear plan two quarters out, with a detailed release schedule over the next 12 weeks. And then we have a vision that goes at least two fiscal years out from today. We’re really just looking at one single, big roadmap with detailed deliverables and feature specs for one to two quarters. Everything that’s beyond is more aspirational.

What’s the biggest myth you’ve heard about agile and roadmapping?

I think a lot of people assume that agile means there’s not much planning involved. I would say there’s more planning and more communication involved.

People think that roadmaps handle communication by themselves, and that’s not the case because you’re assuming everyone is monitoring your roadmap as closely as you might be. So you need to actively communicate any roadmap changes and track who might be affected.

What does a PM need to know before starting an agile roadmap?

There are so many ways to answer that! They should definitely know what the dependencies are for work to be completed. They should have a good sense of the estimates for the work that they’re committing to (for anything being delivered in the short-term). They should also really understand what their customers want or need, and in what order.

What’s your biggest challenge when creating an agile roadmap?

Getting the estimate fidelity and certainty of the schedule, because I know that tends to be the one thing in agile versus waterfall that is a difficult balance to strike. Also, having clear workback plans that give you the confidence that you’re going to actually meet the commitments that you’re making.

No matter your agile style, check out our free agile roadmap template and make it your own.

Sarah Pyo

Title: Product Manager
Company: Expedia
Industry: Travel & Hospitality
Tweet at her: @spyo

When we say agile product roadmap, what do you think of?

I immediately think of timelines and deliveries. Because to me an agile methodology is about getting a team together to continuously deliver in certain timelines. So I really just think of a timeline that can communicate back to my stakeholders within that timeframe.

Tell us about your own agile roadmap. How do you approach it?

My team’s approach is more of a hybrid. In order to communicate to my non-technical stakeholders, I need to break it down into swimlanes that are more simplified — and not so engineering-focused. So we break up our swimlanes by themes that we’re targeting. These are like customer, marketing and product feature goals we have.

We obviously don’t plan for a full year with agile, so when I put together my roadmap I think 120 days ahead. So tackling what we can deliver in four months and breaking that up by themes and overall goals for our customers. That’s how I break it up.

How do you plan for your agile roadmap?

I’m really proud of our process over here. Our teams do the daily stand-ups of course. But we also do grooming sessions with just the product team and our technical product manager. (We’ll also have our UX leads in the room.)

I really rely on those meetings to articulate my thoughts and problems to our leads. Before I can even make an investment, I need to see if it’s something our customers want. We do a lot of discovery before we ship to ensure our use cases are all understood by product.

What is the biggest obstacle when creating an agile roadmap?

I think it comes down to setting expectations for your stakeholders outside of your engineering or UX teams. I know it’ll always be challenging to evangelize what agile is to non-technical teams (there’s a knowledge gap), but it comes down to incorporating those other teams to understand agile processes.

Do you find having an agile roadmap there helps align those psychologies of time better?

Yeah, it does. We do a few steps when we do our roadmap planning. We do high-level numbers and goals for our target months — that 120 days. Which cycle will we hit this? This is what we’re aiming to do. So we set expectations early-on.

Having that visual of the high-level of what we need to do sets that tone for the team so that they know what they want to complete—the goals they want to achieve.

Jessica Johnson

Title: Senior Product Manager
Company: Goodreads
Industry: Social “cataloguing”
Tweet at her: She doesn’t tweet, so connect on LinkedIn!

What do you think of when we say the term agile product roadmap?

Having your cake and eating it too. Everyone wants the speed of agile work but the predictive visibility of a roadmap. While agile on its own is great, I’ve yet to see a larger company be comfortable running totally agile—so you definitely need to find middle ground.

Tell us about your own agile roadmap. How does it look?

I’m a big fan of hybrid approaches. I like roadmaps that use swimlanes to talk more about allocation to a problem than predicted results.

Sure, some major projects with set requirements end up more waterfall, but when you have a problem with no clear end goal, it’s useful to know how much time you can devote. Saying, “Let’s see what we can do in two sprints” is way more accurate than, “It will take us two sprints to get to x outcome.”

Do you think there are best practices in place for creating an agile roadmap?

I think there’s constantly a healthy tension about how much clarity you can create versus how much flexibility you need. On a perfect product team it should adapt. Sometimes becoming more predictable and sometimes giving more slack depending on the scope of the problem.

What’s the biggest myth you’ve heard about agile roadmaps?

That you don’t need a roadmap if you’re agile! The larger your team gets and the more coordination you need across teams, the more important a roadmap becomes. Sure roadmaps are imperfect, but without them coordination is impossible.

How do you plan for your agile roadmap?

There’s a lot of background work in figuring out the priority and dependencies of initiatives. It’s unseen work, but it’s so critical. Digging into data and research; understanding how much time problems might take; coordinating and negotiating with stakeholders. If you don’t do this in advance, you won’t have the right roadmap. After this, it’s just a matter of making the pieces fit with your team’s resources.

What do you find is the biggest challenge when you’re creating a roadmap in agile?

Knowing that even the best laid plans go awry! Despite knowing in advance, it’s hard to be honest and reconcile when you don’t have the time you thought for that pet project.

We created a helpful, actionable guide on all-things agile roadmaps. Get it here.

As a methodology, agile says: forget about planning on timelines — they’re unhelpful and hard to meet! On the other hand, roadmaps help you visualize plans, often on a timeline. So can you build an agile product roadmap? Or is this idea counterintuitive?

Let’s look at the facts. Agile indicates that teams are very flexible and willing to change their plans on a daily or weekly basis. That means project deadlines and dates are pretty insignificant. Yeah, you can add dates to your plan, but if you’re being truly agile (i.e. flexible), expect those dates to change.

Meanwhile, a roadmap usually asks: “Where the heck are we going to be in x amount of time?” Not everyone roadmaps along a timeline, but a roadmap almost always looks to the future beyond the day-to-day and week-to-week — even if not visually presented on a timeline.

So one concept (agile) sheds the constraints of time and another (roadmapping) often appears inherently time-based. Seems pretty counterintuitive. But we have to respectfully disagree. An agile product roadmap is a valuable tool and quite frankly, the concept of roadmapping fits rather nicely in agile environments. Here are some reasons why a roadmap does work for agile.

We created an actionable guide to creating any style of agile roadmap. Download it here.

Product roadmaps change as often as the agile world

Agile is an umbrella term. You’ve got your scrum shops, your extreme programming teams, your kanban processes. They all fall under “agile.” Each of these methodologies have their own nuances and ways of operating, but ultimately they all share a few major themes:

• You’re operating in short cycles
• Change comes quickly, so you always have to be prepared for it
• There is a ton of feedback because you’re operating in cycles that include checkpoints for evaluating your work

These themes are all rooted in the core idea that when functioning in agile, you must be flexible.

A product roadmap is a statement of intent. It’s not a literal roadmap. Your roadmap should not be an amalgamation of hard deadlines that must be met. It should be an incremental plan that openly embraces change. Plans can and will alter, and as a result so will your roadmap.

This flexibility of a product roadmap pairs extremely well with the nature of agile methodologies.

Roadmaps change frequently, allowing them to easily fit into agile settings — which also get reoriented often.

Roadmapping and agile both cut the bullshit

Agile is all about the hard truth — whether you like it or not. Agile provides realistic versions of how projects will play out. If a project is derailing, you’re told straight up it’s derailing. This transparency is part of why agile has picked up so much steam: it cuts the bullshit about what can or cannot be delivered.

Just as agile environments function on transparency, product roadmaps create transparency. You build a roadmap to clarify where the hell you’re headed. It’s a communication asset designed specifically to align various parties on the what, when and how of your project. And when that plan changes, you present a shiny, updated version.

A product roadmap’s function (showing transparency) directly ties into one of the major motivations of adopting an agile methodology (operating with transparency). They’re not in conflict; they fit nicely together.

You can’t escape long-term planning

One thing we’ve noticed when speaking to various PMs is that many of those who claim to operate in agile actually aren’t. Okay, they're operating in some form of agile, but waterfall elements are still mixed in — so it’s not purely agile. The reason: as your company grows, the harder it gets to ignore long-term planning.

Plenty of startups have successfully adopted an agile mentality that dismisses long-term timelines. But as these companies expand, stakeholders increase, user needs intensify and communication needs arise, PMs start juggling way more variables — which requires a deeper degree of planning.

Your dev and product teams may function in speedy cycles, but the rest of your growing company probably doesn’t. Other stakeholders don’t give a damn about your daily or weekly psychology. They want longer-term visions to plan hiring, budgeting and more. A roadmap addresses long-term goals and tackles planning pressures that inevitably creep up.

What does an agile product roadmap look like?

There are many different ways to structure an agile product roadmap. But having spoken to countless product managers, We’ve noticed a few trends. Here are a few quick agile product roadmap examples. (You can find and customize all these examples in our template library.)

Theme-Based Roadmap

Swimlane-style product roadmaps are especially effective for agile PMs. For the teams operating in pure agile (i.e. the ones that aren’t concerned with any dates), it’s common to organize a roadmap by theme. This means the headers might relate to different areas of product development.

Sprint Roadmap

If you work in a more structured agile environment, another option is to structure your product roadmap by cycle or sprint—and not link those sprints to specific dates.

Fuzzy Time Roadmap

Many agile teams also organize their product roadmaps according to “fuzzy time.” This means that rather than stating explicit dates, their product roadmap might include loose time buckets like In Progress, Future and Completed.

Agile-ish Roadmap

Whatever your agile style, take Roadmunk's agile roadmap template and make it your own.

And if you’re an agile-ish team that’s still incorporating waterfall elements, your product roadmap tends to have dates with a caveat. Your dates closer to the present tend to be more granular, whereas the further along you go the more abstract (i.e. fuzzy) the time headers become. It wouldn’t be agile if the dates were the be-all-end-all.

So we debunked the myth that product roadmaps don’t belong in agile environments, but why do you actually need one? Download our ebook for some reasons for the not-yet-convinced and real-world examples of these four powerful ways to visualize your agile product roadmap.

Need more agile content? Check out this guide on creating roadmaps, no matter your agile style.

Highlights

Wealthsimple recently became the first broker in Canada to offer fractional share purchases. What was it like for your team to work on such an exciting product?

Avrum: I think fractional shares are actually a feature that's really aligned with our mission of democratizing finance. Sometimes buying a single stock of Amazon or of Tesla is actually cost prohibitive.
And fractional shares allows others to participate in the potential upside of those companies and invest in the companies they love and that they use. But with a minimal investment that ultimately aligns with how much they're prepared to risk.
It was a pretty exciting launch. I can tell you that I use it because I'm way too cheap to buy a full share of Tesla. But I want to be part of the journey.

It seems like you’re always delivering exciting things have a strong sense of your customers’ needs. Can you explain how your team thinks of product sense and synthetic thinking?

Avrum: It's interesting because if I contrast my experience at Wealthsimple with my experience at Freshbook– Freshbooks is basically accounting and billing for small service-based business.

I've never run a small service-based business. And so, my intimacy with that problem and that sort of lived experience was always a bit once removed. Obviously, we did tons of research and customer interviews to really deeply understand that customer profile. But because I hadn’t ever worked in that context I was never able to be fully intimate with that space.

Contrasting that with Wealthsimple, I use all the products like a regular client. I think that gives me a little bit of a cheat code when it comes to product management. I’ve learned that if you actually use the products regularly that you’re building, you have an ability to be intimate with it in a way that’s hard to replicate.

Your familiarity with and empathy for the customer is stronger. Now I will caveat that really heavily though, because I think the one danger is that you start to confuse yourself as being the target client.

And I'm certainly not that and recognize the incredibly privileged position that I'm in. We're not the be-all and end-all when it comes to customers.
That said, using your own product is by no means a substitute for talking to customers.

Could you elaborate on the term “synthetic thinking”? What does that term mean to you?

Avrum: Product managers who are early in their career often think that the role of the product manager is to be the idea person, person who sits at their desk and pontificates about the future.

You quickly realize that is not your job. That doesn't mean that it doesn't take a sharp mind to be a product manager. Rather, I think that the real challenge, and where the real genius of some of the best product managers that I've ever worked with shines through, is this ability to synthesize from a myriad of different inputs.

Structured data, unstructured data, customer interviews, market data, business cases, app reviews. This large corpus of basically infinite data at your disposal and synthesize that into a point of view. That synthesis is what I refer to as synthetic thinking.

That is far more difficult than sitting at your desk and pontificating about the future of the world. But you’ll notice that you have even better ideas when you are recognizing patterns in these overwhelming data sets and then synthesizing those patterns into a point of view. This becomes a point of view for your products and your product strategy that you can then pursue.
When you see this done well, it's incredibly impressive. But it also takes a lot of practice to get good at. Anything that requires pattern recognition takes many, many kicks at the can in order to be good at it.

Just start recognizing the common themes and patterns. This is also something that is a sort of a lifelong journey for a product manager. Synthetic thinking and the ability to filter noise into a signal takes time.

How can PMs teach their team about synthetic thinking?

Avrum: When I coach others on synthetic thinking I always encourage them to employ a rigorous scientific methodology. That is something that you never innovate on because it's been around for thousands of years.

It's very, very well-defined process. From this large corpus of data that you have you synthesize the hypothesis, and the hypothesis needs to be falsifiable.

It needs to be built on an educated assumption about how things work or how things could work. And it's falsifiable in so far as you have to be able to prove or disprove it using an experiment. So that's the first thing and that never changes.

Following a rigorous scientific structure at least insures that you're following a systematic process of problem solving. The second piece that I think is really important is how you communicate that hypothesis and that approach in a way that is persuasive, compelling, and really tells a clear narrative. This is really challenging.

And that's where maybe the scientific method is an incomplete process and has to be coupled with some of the communication practices and storytelling practices that product managers are often really good at. You don’t want to be articulating this as a dry scientific hypothesis.

You're trying to tell a story of a customer problem informed by all the data and insight that you have. That ultimately informs a point of view. Getting really good at telling that story and letting others engage in it in a deep and meaningful way is hard.

Ultimately it is really worthwhile because it gets everyone to buy into the approach that you've set out. So, storytelling is another piece of the puzzle that I think is crucially important for product managers.

Can this level of thinking be used by a PM at a startup building a zero-to-one type of product?

Avrum: I actually think it's even more critically important in a zero-to-one context that you're following this rigorous scientific approach.
Now, I’m assuming this is pre-product market fit. You need to have a hypothesis for what you’re building. Being scientific here means that you're open to the idea that you're wrong and that, at a certain point, if you are not succeeding, you actually have the opportunity to pivot.

Be ready to try something else informed by what you've seen in the market. Make sure you don't keep running at the same wall over and over again and expecting different results. This is very important at an early stage context because you only have so much runway.

You need to be really scientific, rigorous, and objective about how you're evaluating your success and your failures. Don’t get precious about an idea or approach. Let the data from your in-market experience dictate what you do next.

Remember– scientific method doesn't mean overly bureaucratic or overly process driven. It just means being rigorous and systematic.
And that means only using the minimum amount of process in order to preserve that rigor is what you need. The most important thing for an early stage company to do is ship and ship a lot. Getting bogged down in cumbersome processes is not necessarily going to help you.

I’ll use Wealthsimple Trade as an example of this process.

We started out as a way to do automated investing. It's automated, passive investing. We realized there was a gap in the market for a simple, client-friendly, self-directed investment platform. There wasn't one in Canada that was really serving the needs of new investors.

Even established investors just didn't have that much in the way of product offerings, especially when you contrast with markets like the U S.
We actually had a brokerage. We had all the backend services, infrastructure, and the platform that would allow us to build a self-directed experience. We could do this without a tremendous amount of additional work. We could actually leverage the asset that we already had and build a really super lean product layer on top of it.

The beginnings of Wealthsimple Trade were you could only open a single account type. There were, admittedly, limited socks even available on the platform. But the fact that we were able to leverage our existing brokerage infrastructure and platform, and relatively quickly add a product layer on top of it allowed us to validate that there a need for this to happen in the Canadian market.

Customers were interested in something simpler, something lower cost, and something that ultimately solves the problem of self-directed investing in a modern way. Our hypothesis was that this product would have a market and have interest. It was validated fairly quickly. Right out of the gate, we had massive interest. Our waitlist was hundreds of thousands of customers long.
The nice thing is that we didn't have to invest that much early in the early days to quickly validate that there was a business here that there was a real opportunity here.

If this hadn't been successful, we had built a layer on top of our existing brokerage. So it didn't jeopardize our existing business.

At that early stage, could you have done an even smaller version of that experiment to test against the scientific method?

Avrum: Wait-lists are a great tool. But they have their limits in terms of what they can demonstrate. And I think one of the key challenges with a wait list is that everything's smoke and mirrors and you basically set up a website.

You haven't actually written a line of code, other than the website, in order to validate whether people are interested in this concept or idea. The problem is less if there's actually a product and a meaningful value exchange between the customer or the potential customer and your business.

The validation only goes skin deep. It's like if you ask the person what features they want, people just keep saying yes to everything. But if they have to pay for that feature, if there has to be a value exchange in order to secure that feature, then suddenly they become a lot more selective about what they want.

And then suddenly, they have to prioritize because you've inserted a constraint. This constraint is something you're going to have to give up. And that isn't always money. It could be your time. It could be like the switching cost of moving from an existing player to you.

But there has to be a value exchange in order for that validation to be really meaningful and start to demonstrate product market fit. Wait-lists are a great tool in the arsenal, but they often don't don't imply enough value such that you can really be super confident in the results.

It's a great vote of confidence. But you need enough of a product where people are exchanging value at scale. And then you can say, you’re on the right track. I’ll also say, I think I’ve misused wait-lists in the past in my career.

If you build this grand vision on a landing page, get people super excited, and then you go dark for months as you build the product, you can actually burn some credibility and some trust with those customers because you showed them a glimpse of something. It wasn't really necessarily even close to being ready.

That excitement kind of fizzles in a way that is ultimately not helpful to your business. And you erode a bit of trust. But you can leverage wait-lists properly. You want to be close enough to the actual realization of that product that it doesn't feel like you're misleading your customers. It’s obviously never your intention, but it can certainly feel that way if there's a big gap.

Tell me about some moments in your career when you really started to connect the dots on this type of thinking.

Avrum: I went through somewhat of a typical product journey that a lot of product managers go through. I came out of university thinking I was supposed to be this “idea person.” But for a little bit in your early career your definitely do play that role.

It’s really humbling for product managers when you've shipped things that you think are great but people don't adopt or they don't have the impact on the business that you assumed that they would. There's a very real, humbling experience when you realize that there has to be a better way.

There is an unavoidable amount of product failure that you need to go through where you learn lessons, the hard way. You want to make sure that you're not putting a business in existential jeopardy to learn some lessons as a product manager.

That was actually one of the great benefits of joining a company like Microsoft. It was bureaucratic enough that there was no danger that my junior PM was going to torpedo the various businesses that I worked on.

Those experiences are humbling in two different ways. One is that the project or the thing you deliver ultimately does not have the intended impact. In fact, it has a null result.

It doesn't achieve anything. And that sucks. What's worse is when it has the opposite of its intended impact or it causes a degradation of the user experience and some unintended, unanticipated way. And that's even more painful. But it's perhaps also more impactful to you as a product practitioner because you never want to feel that ever again– to try to to ship them what you believe has value to the world and have it be just thrown right back at you and saying, no, thanks.

This is terrible. There is no more humbling experience for a product manager, but it's also probably, there's no more elemental experience for a product manager to learn that. Like, well, maybe there's a better way of doing this.

Highlights

You’ve worked at a lot of big organizations like LinkedIn, Pinterest, and Soundcloud. What did product sense look like in those organizations?

Kristin: People at all different stages of their product career have asked me about product sense and how to develop it. And although metrics and data-driven decision-making are really emphasized, those are only showing you one piece of the puzzle. I think the other half is having a very strong sense of your users. It’s like a gut feeling. You need to be very in tune with the community of people you’re building for.

What are their biggest pain points? What are the challenges they're facing every day? That is how I would define product sense. Keeping a pulse on the market is also very important because to develop that intuition. You need to be familiar with where the landscape is changing.

The problems that your users face are changing over time. At the core, product sense is being in tune with your users’ deepest pain points and also having a sense of how to solve them.

You don't necessarily want to do everything your users say, but you want to listen and dig deep to understand their problems and then work with your team to find innovative and simple solutions to solve them.

How do you ensure your connection with the user stays deep over a long period of time? What tactics, for example, would you give a junior product manager?

Kristin: I’ll start with my first product role at Twitch. If I was talking to a junior PM, I would ask them to think about their first 30, 60, and 90 days on the job. Your goal is to understand your customers. Try to talk to them. At Twitch, for example, I set up many calls with developers, streamers and viewers. In the days before COVID, there were also conferences– TwitchCon, BlizzCon, etc.

Think about where your customers aggregate and go where they go. Whether it's online or in person,go there to listen and learn. You can get a great sense of product sense on Twitter, but take that with a grain of salt. Sometimes the loudest users are the ones praising you or giving their feedback on forums like Reddit.

But I do think it's really important to monitor all of these sources of feedback.If you’re just starting up, though, you might want to use a user research team. Sometimes you might not have this support. In one of my early roles, I did not get any support from user research because I think there were four people across the entire company.

They didn't have time. And so I did sort of my own user research just to get a pulse on the market and what people were looking for from an insights perspective. If you work in a B2B environment, you could also spend some time with customer-facing teams like business development or customer success. Maybe you even go into the help center and read Zendesk logs. This can help you to really get in tune with the problems of your user base.

When you’re bringing on a new team member, do you embed content like Zendesk logs as part of their onboarding process or do you let it naturally occur over time?

Kristin: For my teams, I like to use a blend. When we had someone new join us at SoundCloud, their onboarding packet would have a mix of past research, key takeaways, or recordings of different interviews with creators or listeners.

One thing that I had implemented at SoundCloud was a guest speaker series event where every month we'd host creators from different platforms. So I invited some Pinterest creators, we had Twitch streamers, musicians who use YouTube, and we had the SoundCloud DJ of the month.

All of this would be part of their onboarding experience. But in that onboarding, having these conversations live is also really important. But it doesn't just stop with onboarding.

Some great advice I received early on in my career is that it doesn’t matter if you’re the CPO or entry-level, you should be interacting with your customers in some way a couple times a month. At Pinterest, for example, our CEO would often have lunch with a “Pinner” on a bi-weekly cadence to keep that product pulse.

He would read and present to us letters that he received from our users, as well. I think that product sense is something you never stopped working on, no matter what level of product you are, no matter what maturity product is in.

Latif: In the B2C world, you may be able to call on the millions of users that are using your product. Whether it's for SoundCloud or LinkedIn or Pinterest.

But for the B2B companies out there– if you have 500 companies as customers, it can seem a little bit weird to ask for their time because they can be really busy. But it's so important. And I remember in 2017 when we brought on a very notable bank in Canada as a customer, they came into our office and we took a full 90 minutes and we actually asked them to present how they use our software.

What was their onboarding experience like? What changed for them in their organization? And there were so many insights in there. We had to spend 45 minutes on Q&A because every single person in the company was throwing up their hands.

It was the craziest experience. So that story really resonates with me. And I don't think you have a huge large user base to do that. You could do that at any company, at any size. And I think that's such a great example that you've provided of how people can continue to engage with the customer if they're not on the front lines or in product management.

Sometimes PMs will try to build something their users are talking about and it just doesn’t work. Can you talk about a time that feedback or product sense didn’t work out?

Kristin: I learned both good and bad things from the first product I launched. The biggest lesson learned was that you have to be careful about the loudest voices or maybe your highest paying customers. In B2B, for example, higher paying customers can sometimes drown out the other voices in the room. Or sometimes people will ask a lot for something and then you launch it but no one uses it.

I learned this at Twitch. If I could go back in time, when I launched Twitch insight, I wish that I would have only accessed the data through our portal and an easy spreadsheet download versus spinning up an API. The API had a ton of requirements and it was really difficult for us. Anytime we made a change to the API, we had to let people know. And we couldn't make a ton of changes at once.

At my first role, I spent a lot of time doing a limited preview with a selection of larger game developers, indie game developers, and international developers. That helped us catch a lot of mistakes and made sure the product was welcomed when it went to the market. Those developers then turned into big champions for us.

But I will say I felt so much pressure from the larger game developers and even from the BD team to push out the API. I did it for that first release. Then I also realized that of course they're going to use it, but at the end of the day, it was more important for them to have more of the right data than have access to the API so early.

So if I could go back in time, I would have pushed back even more strongly and said, okay, you can wait for the API. It's hard because sometimes you are in a tough position as a PM. If you're working on a B2B product, you need to keep your largest customers happy. Sometimes those are the people paying your bills.

It's kind of a push and pull, but I do wish I had negotiated a bit harder. It all worked out in the end, but things were slower than I would have liked them to be.

How do you strike a balance between being data-driven or data-informed?

Kristin: My background before PM was all data analytics and insights. So I have an interesting perspective on this. I would say typically it's a blend. When I work with our teams, we try to start out with a list of questions that we're trying to find the answers to.

We'll sit down and try to understand if we can find this out through data. Can we find this out through research? Do we need to even run an experiment? When we’re asking those questions we're trying to maximize our resources.

Data is very important for any team because it helps you understand how your product is being used. It's important to set hypotheses. I'm a big fan of OKRs to really understand and have visibility into what's happening with your product and your customers.

Data can often tell you where a fire is, but sometimes it can't tell you why. I think that's also where the qualitative piece comes in. Let's say if likes drop at Pinterest, we’ll do a data analysis to figure out what's happening
But sometimes there could be something else happening in our product that triggered that.

You’ll need to do a deeper investigation because data is going to tell your numbers but even then, your data's only as good as your tracking.
Sometimes you don't even track the right thing. So I think it's important to build from the bottom up. Make sure you're tracking the things you want to track before you launch. But I do think there's such power in just having a simple conversation with someone who might use the product.
That may tell you so much more and give you that product sense versus looking at all of your dashboards and spreadsheets.

Another big launch we had at Pinterest was Reactions. One thing we did before we even went to market was just show some really early prototypes to a couple of creators that I knew closely just to get their own reactions to.
And I think the insights from just a few key conversations were so powerful and that saved us so much time. No amount of data analysis could have shown us that insight.

How do you think about product sense in a smaller company versus larger organizations?

Kristin: In both big and small companies you almost need to prove yourself before getting more investment by finding that product market fit.
In a bigger company, you’re going to have a lot of historical data available. But Top.gg is a small team so we don’t have all that information.

Right now we are laying the data foundation, but there are some opportunities where we're looking beyond the market and we have to trust our gut and make that call to take the company to where we want to be in five years.
Sometimes that does mean you do take a hit on your initial metrics. But you have faith that you're going to build those back up later. I will say, this is a much different approach than what I've traditionally done at larger companies.

Sometimes at larger companies, you're in a state where it's so hard to move your monthly active users or weekly active users. And so, you're running a lot of experiments to get those small percentage incremental wins. But I think where Top.gg is right now, we're really in a 0.1 to one stage. And we're going to be making some bold moves to figure out the future of our company. I'll give an example.

Traditionally, at a larger company, if you're doing a redesign you would typically remove that from core product functionality changes. But I've been in situations where we’ve had resource limitations and couldn’t wait. We just had to go for it.

We have to launch the redesign and change the core product to help get us into the future.

Latif: That's amazing that there is such a correlation between the zero to one type projects or the zero to one atmospheres in a company like SoundCloud. I'm a huge fan of their product and what they've done, but having to make changes that large, to get the needle to move is important.

I suspect in some of those decisions you made or were part of– how did the decisions get made? For example, there could be someone who thinks the product should go in one way or another, and product sense at those levels could be just a bunch of people yelling at each other. And that’s just the reality of business, sometimes.

Have you seen a situation where information you've had gets thrown out the window when there's a large bet to be made on product direction? And what have you learned in terms of the realities of business around product sense and in that regard?

Kristin: One thing they did a fantastic job of at Pinterest was transparency and communication.

When I first joined Pinterest, I joined to build out the foundation of creators. And it was actually quite controversial at the time because there were a lot of debates around Pinterest’s positioning.

Do we want it to be more of an individual experience or a social experience? If you looked at Pinterest, at the time you didn’t see many kinds of people on the platform. It was more about the images. But bringing in creators and bringing them to the forefront was really what changed the dynamic.

So when I joined, it was almost controversial adding reactions to the platform. Pinterest wasn't really regarded, at least internally, as a social network. And they wanted more differentiation. I bring that up because while I was there the leadership and I was of course advocating for creators and community. Finally, leadership made the call to make it our focus.

And when this decision was made, our CEO and leadership were very transparent with the company on their thinking and why they pivoted the entire company to focus on creators and shopping. At the time, this was a big pivot for a big company.

The transparency paid off for Pinterest. They were just named number one platform for creators by AdWeek. I was so proud to be part of that foundation, but living through it was very controversial at the time.

I will contrast it to another example from SoundCloud. If you know the history of the company, we’d joke that SoundCloud was a startup with 12 or 13 years of tech debt. But the company is great. It’s been around for so long that it’s gone through several transformations. There was a previous back and forth between focusing on creators and then listeners, but that episode didn’t work out.

But now, they're really focused back on creators. One tough thing that many startups often have to make when you have limited resources is where are you going to invest in? What's going to fall off the table? And while I was there, one hot topic was trying to prioritize how much we want to lean into creators versus becoming like a digital service provider.

The latter meant focusing on building and selling creator tools. But there's a listener experience, too. Where are we going to make the call? This was the decision that I as a director of product has to make. One value I remember from Twitch and Amazon that I loved was “disagree and commit.”

When those changes are happening, as a product person you might not like it. But I do believe in having a culture where you can speak your mind. At the end of the day, leadership needs to make that call and say, “Hey, can we disagree and commit? And can we do this and go all in?”

One thing that was tough at SoundCloud was finding a focus.

There was so much we wanted to do. But we didn't have the resources to do 80% of that. Accepting that was difficult, we wasted a lot of cycles because of this denial. Whereas we could have done something more similar to Pinterest. Sometimes leadership does have to make the call and say, “Hey, let's focus here and let's move on.” Those are some of the learnings that I want to apply to Top.gg because we need to move fast and we have limited resources.

Latif: That’s the quality of great leaders. There could be difficult decisions and directions in the product, and nothing can set the stage where people can disagree and say, here's the direction we're going.

This means that the sensibilities around the product need to be in that direction. It creates a focus. It creates an alignment. So, sometimes product sense needs to actually be informed by leadership, especially if your whole company is surrounded by a singular product. Which in today's age seems to be a very common trend as opposed to a collection of them.

So, I really love those examples because sometimes when things are not working for a product manager, often it's about just understanding what the executives company's goals are and, or strategy, and that should come out with a bullet point sometimes around the product.

How have you transferred some of the product knowledge and senses into being a CEO now?

Kristin: One project that I always bring up for my time at Pinterest was a product called the Engagement Tab. This product actually came out of a hackathon that myself and an incredible engineer worked on outside of our regular jobs.

Product sense was such a key role in building this out. We took it from zero to one and it's one of the most loved products by creators today. It lets them respond to every single comment on their pin in one place on the platform.
Again, at the time that was a very controversial product because we weren't sure if we were going to invest in creators. So, working on that as a little side project and taking that to scale, taught me so much about what I want to bring to Top.gg

Part of that was having a really clear vision for it and also building directly with the community and with the creators themselves. That's one of the reasons I joined Top.gg, they have a really vibrant vibrant community that's heavily, heavily engaged. I think our Discord servers have almost 200,000 people. There’s generally 40,000 people online at any given time.
I think this applies actually to every product, but especially in the creator and passione economy– bring the community along and build for them. This is a value I’m living as a CEO.

You also need to be okay wearing many different hats over the years. At Pinterest, when our product analyst left I had to do product analytics until we hired someone new.

Or when something's not working you have to be willing to go dig through that spreadsheet or do something manually. That's something that I've brought to Top.gg and that's a big part of our culture. It doesn’t matter what level you are or what your function is.

We're all willing to get our hands dirty. That is a really successful trait I've seen in product managers.

About Our Guest – Kristin Chen

Big thank you to our guest, Kristin Chen, for joining us on this episode of Product to Product! To learn more about her product thinking, follow Kristin on LinkedIn.

During this week’s Product to Product episode, Amy Chyan, Product to Product podcast producer and host, chatted with Monica Dabaghi, Senior Manager of Product Management at Sony Interactive Entertainment.

Monica works on Sony Interactive Entertainment’s PlayStation product on the Commerce Team. As a leader on the team, Monica develops the team’s strategy and also ensures that it is delivered. Monica shared her team’s delivery framework for implementing a strategy and also shared how she supports other Product Managers within the organization through a mentorship program she helped create.

Monica’s talk is filled with great insights and helpful tips for delivering a strategy and we highly recommend watching the full talk. If you’re tight on time though, we’ve pulled out some highlights below.

Highlights

(The highlights have been condensed and edited for clarity)

Monica’s journey into product (1:07)

Amy: Monica, can you tell me about yourself and how you got involved with product?

Monica: Yeah, so I have been in product management for about 10 years now. My early career, I was in sales and operations, working for a payments company, while I was in college. I really enjoyed understanding the nuances of how payments worked. This was in a time when websites were really starting to take off and businesses were taking their stores online. Something that I got really passionate about was helping small businesses go online and utilize e-commerce systems. That was something that got me into product. My first job in product was for a medium sized company, iPayment. I was really helping manage online gateways, POS systems, those types of products. It was really, really fun learning a lot about commerce systems and payment platforms.

About five years ago, I joined PlayStation. My key role at PlayStation at the time was to help manage the payment systems within the PlayStation Network platform. If you’re playing a game online, or if you want to buy a digital game from the PlayStation Network, you’re utilizing the PlayStation Store. That store obviously has commerce systems integrated, and those are where I started working. Over time, we kind of grew that platform and grew the functionality so much that we ended up expanding our team, expanding the Scrum teams that we were able to work with and a lot of the development folks in that group.

Monica’s product team (2:47)

Amy: You talked a little bit about expanding your team. What does your team look like right now?

Monica: So right now there’s four of us that work specifically on the payment and fraud platform, and then another group of four folks who work with us on commerce. Within the commerce platform at PlayStation, we have somebody who’s overseeing our subscriptions platforms, that’s helping manage how we bill or manage our PlayStation Plus product or PlayStation Now product. We have somebody who manages our offers platform. The offers platform, that’s how we provide banners and promotions, discounts, and coupon codes in the store for our players. Then we oversee our payment optimization work stream, which is how we add the payment methods our players want to use. PlayStation’s a global brand. We’ve got stores in over 70 countries, and players in all of these countries don’t necessarily use credit cards as their primary payment instrument.

In these different countries, we want to really understand what payment methods make sense and how we can get those integrated into the platform. We also oversee the actual transaction routing. That’s a bit of a deeper layer into commerce where we’re managing all of the data, how we ledger these transactions, recognizing revenue. Then another layer of that is fraud management. When you’re dealing with platforms of this size and scope, especially PlayStation, which is a brand that draws a lot of attention, you have to really manage your risk and exploits that we see in the field. That’s what our team does. It’s pretty fun stuff.

Monica’s delivery framework (6:59)

Amy: In our pre-interview, you mentioned that a chunk of your role is to develop your team strategy and then deliver that strategy. To help your team deliver that strategy, you mentioned you have a delivery framework in place. Can you expand on that framework and how it shapes the way your team works?

Monica: At PlayStation, in product management, we kind of have a philosophy, that we focus on our customers and the customer outcomes that we want to drive with everything that we develop. First, we always think about how we can make our customers’ jobs easier. There’s a great book, I’m going to totally space on the author, Jobs to Be Done, Anthony Ulwick? But yes, a really good book, definitely recommend it. That’s a philosophy that we at PlayStation really like to follow when we’re putting together a strategy for how we deliver products. We’re starting with the customer outcomes in mind. What does the customer want, need? What job are they trying to do that we can make easier? Finding that out is kind of difficult, because you can’t just say, “Hey, customer, what do you want?” If we could do that, all of our jobs would be really easy.

Going through an analysis, research, spending time with our consumers, spending time on the platform, using our product, is something that we all spend a ton of time doing. We pull a bunch of data from the platform and we see like, “Oh, like a lot of people are abandoning their journey here.” We can find places where that happens. What we’ll do is we’ll then start to form a hypothesis. Why are they abandoning their journey here? What is something that I think, as a product manager, from my research or from something that I’ve learned, might be able to help our customer continue that journey or make it through that gap?

We start to think a little bit about impacts. If I was to confirm my hypothesis and say like, “Yes, I can close this gap,” what impact would I have on my customer? It might be, “Oh, I can help reduce 10% abandonment at this stage in the customer journey.” Reducing that abandonment is going to get a customer to the content that they want to play faster or easier. They don’t have to spend as much time waiting for a download, whatever that might be. That’s kind of a really key first step in our framework is understanding and defining the outcomes that we’re trying to drive with a product or a feature. That’s those hypotheses and impacts that we’re predicting.

The next part of the framework is what we call research and analysis. This is where we’re doing proof of concepts. We’re talking to our engineering folks, our partners that are going to help us understand feasibility level of efforts. How much would it take for us to test this hypothesis? How can we try to understand if those impacts are accurate or if we’re on the right track with those impacts that we’ve put forward. We hopefully have some indication from a proof of concept. We love to do proof of concepts, because that’s something we can prototype really quickly, get it out there and see how our players are actually interacting with whatever the feature might be. There’s some cases, especially in commerce systems, where you can’t, because if we’re dealing with payments, there’s tax and legal kind of parts of these systems that you just kind of have to send it out there, make sure it works and let it go, but most of the time we’re testing our hypotheses and we’re doing kind of rapid prototyping.

Once we have the results from our POCs, from our LOE discussions with our engineering partners, we go into the kind of the last part of this framework, which we call solutioning. Solutioning is where we’re putting together a full plan of key deliverables, when we think those are going to go out. That’s when we’re talking to our legal, our PR teams if we’re going to be releasing a major feature to the platform, and then we get a release timeline kind of solid. That’s it. It seems like a lot, but depending on the scope of the feature, that framework can go really quickly, so something that you could get done in like a month, or if it’s something massive, it could take a year, more than one year, but yeah, that’s pretty much the framework we use for almost everything.

Creating a growth environment for product managers (14:20)

Amy: Also in our pre-interview we learned that one of your passions is helping other product managers. At Sony Interactive Entertainment, you’ve helped create a framework for growth in the product management career. Can you share what you’ve done to create this growth environment? How do you provide mentorship as well?

Monica: Yeah, so something that has been super important for me and for folks on my team is that when you move into product management, I mean, 100% of us I think, are coming from various backgrounds that are not product management. We went to business school, we got degrees in finance or marketing. There’s not really a product management degree that people are giving. Folks come into product management from a lot of different places, sales and operations, I’ve got some folks on my team that came from engineering, some folks on my team that came from data science, so these various backgrounds. Product management as a craft is something that I think can be really easily taught. That’s something that I’ve taken up, especially within my team, as something that we do and we spend time on every year. It’s one of our key goals every year is that something should be around the development of your craft of product management.

I’m going to shout out a couple. Again, that Jobs to Be Done book, really a good book that we evangelize at PlayStation, especially within my team. Some folks within PlayStation and myself, we get together and do like product type TED talks. We have a product camp that we like to do. Product camp is something that I think is super fun. It’s with a lot of other product managers. What you’ll do is you’ll take a feature idea or a business case that you want to try to kind of get moving and present it to other product managers first. Maybe before you take it to your stakeholder group before you take it up to executives. You’re shooting it out with other product managers to get their input, to get maybe some ideas that you didn’t think about.

I encourage some folks on my team to go take some courses with Pragmatic Marketing. Really they do really great courses. I think they are moving them now virtually, so it might be even easier to take some of their courses, but they’ve got really great product to product learning and development courses about different parts of product management. How are you gathering requirements from various stakeholders? How do you negotiate priorities, competing priorities with stakeholders? Some of the key challenges that I think product managers have across every industry, every company, is how can you really manage not just the strategy of what you’re going to do for your work stream or for your product, but how can you manage the expectations that your stakeholders have, that your leadership has? Those are some of the things that we spend a lot of time on, and it’s something that I think is super important, especially because again, none of us came from a background of product. We all come from something else.

Audience Q&A period

(17:45) One of our questions says, how do you track voice of customer/their feedback and the requests that come in through internal employees and stakeholders? Do you offer a view to those who submit feedback so they can see where the request is in the pipeline or the roadmap?

(22:29) Would you say you spend more time in the problem space and hypothesis of your framework or less time on the solution phase or vice versa or the opposite?

(24:04) Besides the products at PlayStation and Sony, what are your favorite products?

Speakers

A product leader with a belief in outcome driven innovation, and the importance of diverse teams in delivering amazing products. Over 10 years of experience practicing the craft of Product Management, and leading teams of product managers; from small fintech startups, to one of the world’s largest brands. Together we successfully introduce new products, features, and services that bring joy to over 100M customers worldwide.

Amy’s a Content Marketing Specialist at Roadmunk on the Marketing team. She produces Recess, the Product to Product podcast and video content. Prior to Roadmunk, Amy worked as a journalist in various Canadian newsrooms and wrote for publications like NBC, CBC, Vice and more.

You can follow her via her website, Twitter, and LinkedIn.

During this week’s episode, Amy Chyan, Product to Product podcast producer and host, chatted with David Cutler, Product Director at Spotify.

David is a data-driven PM. He turns murky data into knowledge and insights for his teams. During this Recess break, David shared some of the features he’s launched at Spotify and how he uses data to make informed product decisions. He also shared how important he thinks data teams are to product orgs.

David shared a lot of great insights into using data to make product decisions. We highly recommend watching the full talk but if you’re tight on time, we’ve pulled out some highlights below.

Highlights

(Highlights have been condensed and edited for clarity)

David’s path to product management (1:29)

Amy: Can you tell us a little bit more about yourself and how you got involved with product?

David: I’ve been at Spotify for almost three years at this point. So at the end of the summer, it’ll be my three year anniversary, but yeah, I’ve kind of bounced around in my career before I actually came to Spotify. I went to school in Washington, DC, and I’ve always been kind of a tech nerd at heart, I call myself, but I had a major in information technology, which is sort of a mix of business and computer science in school. And when I came out of school, I wasn’t entirely sure what I wanted to do. But Accenture was a consulting company that was popular in terms of software consulting and tech consulting. And they approached me and I sort of liked what I heard about the opportunity, and I joined Accenture for three years.

And the fun story I have about consulting is at least when I was working there, it was before the term of product manager was really popular. It wasn’t really known yet at the time, and when I think back to what my role was as part of Accenture for three years there, it really was what we think about being a product manager is today, right? So we went into different companies. Our role was essentially to understand what the employees and what the company was trying to achieve through technology and software. So they had a particular business challenge. They knew they needed to innovate and enhance their current processes and operations. And they knew tech was the way to do that, but they weren’t sure how to bridge those two things. And they relied on us as consultants to come in and understand what they were trying to achieve, what their metrics were, what are their KPIs? How do you understand how to be successful? And then we can offer different options of software that can really help them reach their goals, which essentially, if you’re going to boil down what a product manager does, it’s kind of it.

And from there, I moved into a data analytics company. I was there for around six years where I did a bit of consulting work, similar to what I did at Accenture, but also that’s where I got my feet wet and to kind of more pure product management, where we’re using a set of tools and products and capabilities to build more of a platform product for our customers, that then we took to market and actually sold as a separate product. And that is where it sort of clicked for me as I really enjoy the concept of product management. I love innovating. I love creating, but I also love collaborating with not only the user, but also the engineering team and others that are involved in actually developing that.

So from there, I went to Bloomberg where I worked on a product that built out as a data analytics platform that was servicing compliance organizations within financial services firms, where we were, I’m not going to say fraud detection, but we were trying to get ahead of some of the illegitimate, I would say, behavior that traders would sometimes engage in. So insider trading and front running and all that fun stuff that people think are cool until the SEC comes knocking at your door. But that again was kind of your typical product management type of role. We had a significantly sized product team and then a big engineering group. And we had to work together with them plus our clients to build the right products.

And then from there, I had a couple of opportunities I was considering, and I realized one of the things I haven’t done yet in my career is work on a product that had millions of end users, right? And I think there’s a fundamental difference in building products for millions versus hundreds or thousands or even teens, right? Depending on what type of product. So I wanted to move into that B to C world, a business to consumer world, rather than the B to B that I was in before.

David’s role at Spotify (7:44)

David: In terms of my particular role, so Spotify, a large company, a big part of our employee base is the engineering organization. And the engineering organization has multiple different competencies, right? You have product management, obviously, we have 250 plus PMs at this point, we have data science, we have machine learning engineers, we have data engineers, infrastructure, app designers, and essentially the engineering organization as a whole is a partnership of all of these different competencies that are coming together to really build different features, products, capabilities. And my particular role as a Product Director is I am responsible for leading the product strategy or, not necessarily the roadmap, but more the direction and the strategy of a particular product area, which is a collection of teams that are coming together for a common goal.

But I’m also responsible for helping to support product managers on my team, their particular career, make sure they’re aligned, try to make sure that it’s a healthy environment for PM’s to work and grow and innovate. And a lot of my role is making sure I understand what their interests are and what their passions are, but also how that fits with what other products within the company are being built out, so that we can provide the greatest experience, full stack, right? It starts out with infrastructure, and then data, and then the app itself, and then that experience that users have with Spotify.

Importance of data teams to product (9:22)

Amy: I mean, you’re touching a lot on analytics, data science, machine learning, and Spotify obviously uses that to understand user behavior. We get the year end list, “This is your most listened to, these are top 10,” or whatever. How important, then, is data teams to product works?

David: Yeah. So this is something I learned throughout my career and through different paths that I took and different opportunities I had working with different companies, which, again, was actually an amazing part of consulting, is you get to learn what are the needs and considerations and interests of different companies in different parts of the world, which gives you a better holistic perspective of what people really care about. But as I kind of went through my career, each year, I noticed more and more that data itself was becoming more of a primary asset of a company that could help them make multiple different types of decisions, whether these are big business decisions of, “Should I acquire this company? Should I make this content exclusive?” Particular feature decisions of even really granular stuff, like, “Should I put a sleep timer on my app itself? Or should I offer podcasts in the primary app? Or should I build a separate app? I have different user cohorts. Should I create a new app for that, or just allow that to go within the flow of the primary application?”

And what I really noticed and what I’ve seen, especially at Spotify, is data has become the most important asset that a company can actually maintain and develop to help through all of these processes within a decision making framework, right? So you mentioned user behavior. That’s a very big part of one of the things that my team does on a day to day basis, which is manage a handful of data sets that really look at user behavior, and not for anything much more than understanding what is that experience for our users? What do they like to listen to? What are some trends and kinds of behavioral signals that allow us to understand how to provide them a better experience?

And that is actually a good example, where if you’re going to consider data as a primary asset and a primary product, then you need to respect that and hire teams that focus just on data, right? And this didn’t exist, I would say five plus years ago, in a lot of the companies that I worked with. It was always a commoditized component of what you’re building and business intelligence teams started being built out, and reporting was a big part of it. And then analytics came into play and then more predictive analytics as you work your way up that chain, which became important, but essentially, I’ve never seen data used the way it is now. And even if you look at other tech companies and you look at some of the roles that they’re offering in some of their career sites, there’s a lot of data science, data product managers, data infrastructure roles, because companies are really focusing on building out teams for the sole reason of maintaining data as an asset.

Can data lead PMs astray? (12:41)

Amy: Do you think data can actually lead PMs astray if they are slicing the data the way they want to see it?

David: Yeah. So we talked about that a lot, is yes, we want to be a data driven company, right? We want to make sure that we’re using data in any decision that we’re making and help basically create evidence or backup for arguments. Something that I always think about and talk to my team about is I would rather see us become data informed, where we use data as one of the input mechanisms that help us make a decision. So it’s almost unfair to product managers, if we say we’re just going to use the data to help us make a decision, and it will tell us we don’t have to worry about a thing. It almost commoditizes us as product managers. I think the value that we bring is we can offer contextual evidence or a story to what the data is telling us. And yes, there is a risk that we could spin that in our own way to try to drive whatever decision we think should be made. But there’s a bit of social responsibility there and trust that you have.

So one of the things we talked about as a team is, yes, there’s quantitative evidence and analysis that we need to consider. There are certain KPIs that we want to define and track progress against and see if we’re successful. But there’s also a qualitative component to what are we trying to achieve here, and do those line up? And that’s what helps us reach a decision or reach a way to figure out what direction we want to go in. So I always say it’s kind of a mix of quantitative and qualitative, but again, I like the term, “data informed” rather than, “data driven.”

Features David has launched at Spotify and how the data informed them (14:36)

Amy: Can you talk about some of the features you helped launch at Spotify in a product lead role? I mean, in our pre-interview you mentioned Spotify for kids, how did you come to the conclusion that you needed to launch that, and what was sort of the data or the dashboard that your team’s like, “This is it, we need Spotify for kids.” And now it’s like the Spotify duo as well?

David: Yeah. So one of the advantages of being in the data world within a company like Spotify, is that you have many users and there’s kind of a broad cohort that you can look at, we’re relatively horizontal. So a data team within an organization, they’re not necessarily making the decision of, “Yes, we are going to launch Spotify for kids.” I think there’s a lot of different teams that are involved in that. I would do a disservice to other people if I said that our team was one who did it, it’s more of, when you’re looking at some of the core data sets within your company of user behavior, for example, one of the inputs you would actually look at to say, “Do we even need Spotify for kids?” Right? “Do we need Spotify for artists,” which is an app that artists have access to, to understand the behavior of their fans. And I mentioned before, do you need to create your own podcast app or just build it in?

A lot of times you do start with the data that is created based on the behavior of the user. So one of the things that happens with Spotify for kids is one of the use cases where parents would use Spotify to entertain their children and they would use their own profile, and they would play children’s music whether it’s at the dinner table or just to keep them occupied, which is what my sister does. But what ultimately happened is, yes, that was great for the parents, but we also noticed that it started to, I don’t want to say balloon, but intermix the children’s music with their own music that they like to listen to on their own time. And if a big part of our value prop is our suggestions and our playlists, like Discover Weekly, then we started to devalue those products that we are offering.

And so you can actually look at the data to see, “Well, is the user behavior that’s occurring, actually having an impact on other data products that we are providing that gets surfaced in the app itself?” So not only do we look at what is our user base, what do they look like in terms of location, geography, what are their tendencies and their behaviors? And then we look at what’s the impact that it has of not creating a separate app. And so then you can actually experiment and test. You could say, “Well, if we were able to keep this separate, how much more improved would their Discover Weekly be, and how much more of a better experience would it be for the kids themselves?” so it’s hard to say that any one team is the one responsible for launching, but again, being part of the data teams, you get exposure to all of these different initiatives that are happening across the company, whether it’s something for the artists, whether it’s something for the kids. Whether we launched Spotify stations, which was more of an experience for somebody who wanted to interact with Spotify more like a radio. I’m not going to say that’s my parents, but my parents love it. I said, “Hey, dad, here’s stations. And press a couple buttons.”

And you’re always looking at how the data can inform you, of if you should actually go in a particular direction, such as launching a separate app, or maybe launching a new feature within the app itself. Not to mention when you do launch a new product or feature, you could determine if it’s being utilized efficiently, and if people are in favor of it, just by looking at the interaction data itself on a day to day basis. And so there’s constant back and forth and input, and there’s a hypothesis that you make, and then you see the results of that experiment. And that all comes from the core data infrastructure and data feed that a lot of product managers at Spotify are involved with, right?

A lot of our role is understanding who are the different stakeholders in the company that care about this data? And usually you learn pretty quickly that there’s hundreds of use cases, hundreds of people in teams that really become super important stakeholders, too. And it’s a weird way to think about it because when you join Spotify, you think, “My end user is the end consumer, right? The person who downloads Spotify and they’re listening to music and I’m building a product for them,” but you quickly learn a lot of your primary stakeholders are actually other product teams within the company. And that’s also where your value as a PM comes in, you need to understand what are different initiatives? What are dependencies that you have on them? What do they have on you? Who are the different roles of individuals that we care about serving as a product team that has data as our primary product?

Audience Q&A

(20:05) Is the data team set up like a shared service at Spotify?

(22:08) Can you give an example of a feature that you were surprised was priority based on user data, or conversely, a feature you guys launched and you’re just like, “Let’s sunset this. They don’t like it at all.”

(27:50) What competencies would you recommend current PMs develop in the next 10 years?

Speakers

David is a Product Director at Spotify and resides in Brooklyn (in non-quarantine times). He works with various teams across the organization that use data and insights to understand how user behavior and App feature/functionality can be attributed to the performance of Spotify as a product and company. He previously worked at Bloomberg, Teradata and Accenture in various product management roles, many of which focused on how data can be used as a competitive and operational advantage. David graduated George Washington University with a dual major of Information Technology and Finance, and has picked up various hobbies that are solely focused on keeping busy over the previous 4 months.

Amy’s a Content Marketing Specialist at Roadmunk on the Marketing team. She produces Recess, the Product to Product podcast and video content. Prior to Roadmunk, Amy worked as a journalist in various Canadian newsrooms and wrote for publications like NBC, CBC, Vice and more.

You can follow her via her website, Twitter, and LinkedIn.

During this week’s Product to Product episode, Liz Papierz, Event Manager at Roadmunk, chatted with Jason Costa, Director of Product at Reddit.

Reddit is an app used by millions of users globally. Jason currently works on Content & Communities and is in charge of developing a strategy for that area. In this conversation, he dove into how he develops that strategy and some of the metrics he uses to measure success.

Jason shared a lot of great insights about strategy development and we highly recommend watching the full talk. If you’re tight on time though, we’ve pulled out some highlights below.

To attend a live Recess session and have the opportunity to ask questions of our guests, follow our Meetup page for future events.

Highlights

(Highlights have been condensed and edited for clarity)

Jason’s path to product management (1:34)

Liz: Jason is currently the Director of Product at Reddit. Jason, to kick off the conversation, can you tell us a little bit more about yourself, and how you got involved with product?

Jason: I studied computer science in undergrad, and then in 2005, I’m dating myself here, but in 2005 when I graduated, I got a job offer at a little company in Mountain View called Google. I ended up joining to work on a variety of different products there, from Google Analytics to Google Checkout, to the open social standard, primarily in technical roles. And I remember sort of being out on the customer frontline, and meeting with a lot of engineers and product managers at client companies, and kind of getting feedback, and hearing from them as touch points. Like, “Hey, we’re missing this”, or “If you did this, it could enable x, y, z for us”, and I remember sort of always going home from work, and playing that game of like, well, what if we did this instead? And could that be more beneficial than this other thing that they suggested? Or what if we did these things instead?

And that was sort of my first sort of foray into the products sphere. And I stayed largely in technical roles at Google, but I did notice that there was an ilk, or a profile of product manager, at least at Google, and it was very common to see someone who had CS background and then went to business school, so I ended up going to business school, did a PM internship at Facebook the summer that I was in business school. Ended up joining Twitter when it was about 350 employees to work on the platform, their APIs. Was there for a couple years, went to go join Pinterest as one of the first hundred employees, actually one of the first two product managers on the ads team. Got to help build that business from zero to about a hundred million annualized revenue run rate, and did that for a couple of years, which was wonderful. Got to work on Pinterest analytics for publishers, their campaign management UI, their ads API, all of their billing workflows. And then transitioned over to the consumer team, and worked on basically leading the pins team, so everything from creating rich pins, where we could add additional data to the particular pins, recipe information, product data, like what’s the stock, what is the price, to actually crafting video pins, which ended up being like, hey, how do we deliver sites on the motion end of the pins?

And then after about three and a half years, I ended up transitioning into Venture. I got a really neat opportunity to go work at a firm based out of China, and I was always intrigued with what was happening with product in East Asia. It was very different, the population there had skipped the desktop revolution, they’d gone straight to the mobile device. It was largely cashless society, they were using the mobile device for basically the wallet. You had lots of interesting apps, like YY and Inke, that actually weren’t based on ads at all, they were based on digital goods and virtual gifting as sort of the ads, or I’m sorry, not the non-ads backbone, right, in terms of how the economics of the services worked.

So I went and did that for a couple of years, and it was a wonderful experience, but I started to get a little bored, in truth. I was like, man, I really miss building, I miss being in the thick of the action, and creating real value for users. And so I ended up leaving Venture to go back and get in the game, and so I’ve been at Reddit now for about a year and a half, working on content and communities.

Jason’s role at Reddit (7:01)

Jason: My particular role is driving the content and communities team, which is focused really on two important engagement vectors on the service. One is just the very sort of concept of community, right, how do we actually manifest a sense of belonging in the service, so when users come to Reddit, they feel like, yeah, these are my digital people, right? This is my digital tribe. And that might be folks coming together to converse and discuss around politics. It might be just your favorite NFL team, or your favorite band. It might be about a TV show that you love, like Game of Thrones. So being able to establish that sense of community on Reddit, incredibly important.

And then also figuring out, as you can imagine, on the content front, how do we get more people sharing and producing content through posts, through comments, and making sure there’s a really healthy stock and flow of content on the service. Because I do like to say, on Reddit, people come for the content, and then they stay for the communities.

How Jason develops his strategy (8:31)

Liz: What factors do you consider when developing your strategy? Do you use any specific processes? And also, I really liked how you used the word belonging. How do you work that feeling, almost, and that concept around your strategy?

Jason: So in terms of general process, and this is sort of Reddit agnostic, and just how I like to operate as a product person, I really, really am a big believer in having what I like to call the four legged stool, where you have a product person, a tech lead, design lead, and then a data lead. And data and products, from a process perspective, I like to say should be working together to very tightly and crisply define what the product, or rather, what the user problem is that you’re setting up to solve. And so, that’s the very first step in the process, is how do you make sure you’re identifying and framing up very acutely what the problem is to solve for the user. And at that point, once you have a pretty good sense of like, yeah, this is it, this is the bounded problem that we’re setting out to go solve, bring design and engineering in very, very early into the process.

And frankly, let them off the leash. Let them, if you trust your counterparts, and you should, let them go do their craft, and so design and engineering sort of embark, with you there as a sounding board, of course, on setting out to craft a solution to user problems. And oftentimes I like to say, design sort of brings the concept to life through visual and interaction design, and then engineering breathes life into it, right? Makes the heart beat. And then you get to a place where you have a concept, and your role all the way through is to be that sounding board, but also to be evaluating the trade offs, right?

Solutions don’t come in one shape or size, you can go many different paths. And so you want to be evaluating what the pros and cons are of each path, you want to make sure that the quality of the decision making remains really high, because one little compromise to make everybody happy can actually end up having disastrous impact later on down the line on the user experience, and on the product’s chance at success. You want to be time boxing the decisions. It’s very easy to kind of wander out in the forest and naval gaze, instead of focusing on building something, and getting it into the hands of users.

And then all the while, you certainly want to come back at the end, and kind of ask yourself, amongst the group, “Hey, did we actually solve the problem that we set out to go tackle at the beginning?” And to ensure that the answer is yes, you want to be self auditing many, many steps along the way, right? You want to be making sure that you’re often checking in with the team, and saying, “Hey, we’re still tackling problem x, right? We haven’t moved on to problem y or z?” Because it’s very easy to tack in different directions, and it tends to happen in very small increments, on a daily basis. But then you get a month out, and you’re like, wait, what are we building again, here? Right? What are we solving for?

So that’s like the general process that I like to use, in terms of product development. In terms of setting the strategy, I’m always sort of looking at what’s the goals of the company, what’s the mission of the company? And how do we enable that through the experiences that we’re putting into the hands of users? And so for me, I’m always thinking about how do we make sure that we have a very, very large compelling, high quality, fresh corpus of content, hence the content part of the name. And then how do we make sure that we have active and vibrant communities, where people are coming together as a digital collection, and conversing about a particular topic that tends to drive a range of emotions. Like I’ve generally found that people have an affinity to a particular community because it touches on some emotion for them. That might be fear, it might be hope, it might be pain, it might be pleasure, it might be social inclusion. I just joined a fitness group on Reddit, because I’m out of shape, given the shelter in place orders.

And it’s funny, I’ve been going back to it to learn about these exercises you can do inside your home, and chatting with people, and one, I’m like, wow, I’m not alone in this, right? There’s a lot of people who are feeling that pain. But then the other thing I realized was like, I’m gravitating towards this sub because it gives me a sense of hope that in the future, I’ll be healthier again.

So it’s sort of like thinking more strategically, like how do we connect people to experiences that enable the mission of the company? And that’s sort of how I try to set up the strategic scaffolding, for what we focus on.

Utilizing metrics (14:09)

Liz: You mentioned that you have to be careful about compromising on certain decisions. How do you ensure that when you’ve had, say a strategy or a plan put in place, that you do stay close to it, and you compromise when it actually might be beneficial, but then also know when to say, “Whoa, whoa, whoa, we have this due in three months is our timeline, this is going to drive us right off the highway”. Do you have any specific metrics that you pull your team back to, or how do you keep on track?

Jason: Yeah, that’s a great question. One, I make sure that I carve out an hour every day, typically at the end of the day, to kind of be going over, more from an editorialized perspective, what we’re working on. And then checking in to see what the actual status of it is, right? So like are we doing the due diligence and practicing good hygiene around sending notes out for meetings where we’re making decisions that might change the path of the product? Are we constantly updating the product specs, and making sure that we’re capturing, because things naturally evolve, right, as you get more voices in the process.

It’s going to evolve. And then I always try to, where I can, sit in on discussions where things are chatted about. I try not to get in the way, I try to kind of let the respective leads of the team usher in their own destiny, but pressure test, right? Like I like to get involved in sort of the pressure testing process, and hey, can we do x, y, z? Have you thought about this instead? But the big thing that tends to act as the scalable guard rails are the metrics, right? And so I always will look at what is the behavior that we’re trying to incentivize, or manufacture, or just generate more of? And are we measuring success correctly against that particular behavior.

And once I feel like we have the right metric, then I’m constantly pointing the respective product teams back to hey, whatever you’re doing, is it driving this metric, right? Now there are some things, like you mentioned, that you just have to kind of take into account, like you’re down on resourcing, right, and there’s just no way that you’re going to be able to hit this. Or you do realize, well, we made a mistake in the planning process, and actually this would be the better decision to go forward on. And you do have to course correct.

So there are situations like that, it’s case dependent, but I generally find that just making sure that the goals are hyper clear, and over communicated, so that everybody is on the same page, that tends to be the right set of guard rails that keeps people on track.

How Jason measures the feeling of community and belonging (17:17)

Liz: Just out of curiosity, for the feeling of creating community and belonging, do you have a way to measure that at Reddit? I feel like that’s one thing that product managers talk about is like they want their product to be sticky. I feel like Reddit’s very addictive, by nature, you get on a scrolling spree. But I was wondering, do you have a metric to measure that?

Jason: Yeah, so we do have a metric that we look at, and we call that active communities. And it is a threshold by which we measure, on a rolling time basis, how many events or engagements are happening in this particular community. And we’ve done a ton of analyses around, without getting into the details too much, we’ve done a ton of analyses around when does a community tip and become what we call self-growing and self-sustaining, right, where it gets to a place where we don’t even need to be involved in recommendations, or discover surface areas. Like this thing has taken off, and it is snowballing, in terms of acquiring new users, generating more posts, generating more comments on a daily basis, and so it kind of hits escape velocity. And at that point, we know that it’s almost never going to churn, right, like the retention bar is so high that it has basically become, as I mentioned, self-sustaining and self-growing. And our team is responsible for growing that particular corpus of communities.

So we want to activate as many as we can, and then retain virtually all of them. And so that’s the way that we measure, hey, are we at a good place with regards to the total breadth of communities on the site? And then we apply some additional filtering layers, like Reddit is increasingly becoming a global service, and that means you have to be accessible to people from very different backgrounds. They might have cultural orientations, different religious backgrounds, different political beliefs, and when that starts to play in, you have to recognize those differences, and accommodate those differences. And so something we’re starting to think about is do we have enough sort of age appropriate communities? Do we have enough topical diversity, right? Like are we doing enough to sort of make the service accessible to everyone? And so that’s sort of the next layer of filtering that we start to apply, like okay, we think we have a good pulse on what belonging looks like, now can we scale that to everyone, right? Now can we make a sense of belonging available to everybody in the world?

Audience Q&A

(20:20) How do you define and cultivate healthy communities that generate meaningful content? How do you define and cull “bad actors”?

(24:51) What languages or geographies is Reddit most focused on growing in, and what sorts of strategies have you developed tailored to those contexts?

(27:07) What’s your favorite subreddit?

Speakers

Jason Costa is currently working on product strategy at Reddit. Previously, he was a Venture Partner at GGV Capital where he led six investments and spent significant time advising the fund’s portfolio on product strategy.

Prior to joining the GGV fund, he was one of the first members of the monetization team at Pinterest, where he helped launch their ads API, analytics tool for businesses, billing infrastructure, and the ads campaign management interface. Additionally, he led consumer efforts on the Pins team, including product managing the Rich Pins and Video products.

Jason also worked on the platform team at Twitter, managing an ecosystem of more than a million apps, and at Google where he helped  the Google Checkout API and the OpenSocial schema. He holds a B.S. in Computer Science from USC, and an MBA from MIT.

Liz is the Event Manager at Roadmunk on the Marketing team. She produces Recess and in non-pandemic times, organizes Roadmunk’s Product to Product in-person events held in Toronto, New York and Chicago.

Prior to Roadmunk, Liz produced events for a renewable energy company and a non-profit. You can follow her on LinkedIn.

During this week’s Product to Product episode, Liz Papierz, Event Manager at Roadmunk, chatted with Ralf Herzel, Product Manager Mobile at Universal Music Group.

Ralf is based out of Universal Music Group’s Berlin office. Ralf’s previous product management experience includes working for a gaming company. He shared how his background as a gaming PM influences him in his current role. He also dove into how he conducts user discovery to increase engagement and retention.

Ralf shared a lot of great information and we highly recommend watching the full talk. If you’re tight on time we’ve pulled out some highlights below.

Highlights

(Highlights have been condensed and edited for clarity)

Ralf’s journey to product management (1:39)

Liz: Can you tell us a little bit more about yourself and how you got involved with product?

Ralf: Sure. I studied business administration and after that I got my first full-time job as a category manager in an e-commerce company, because I always wanted to be in the digital space, obviously. And I did this for three years. But I know something was missing at some point. And then I got an offer from a gaming company in Berlin and I wasn’t sure about, okay, Berlin, I don’t know. And gaming, I don’t know. And I thought, “Okay, but I want to try it. I want to take the adventure.” And that was seven years ago.

And then I started basically as a product manager. And gaming industry means free to play gaming, mobile games, PC games, and it’s extremely fast. It’s very dynamic. You learn a lot. You learn a lot about how you use data, how you gain insights, how you basically create items in the game, how you create new features and basically how you run your own little world because you’re kind of the CEO of the game, right? So it’s a stressful world, but also it’s great for learning. It’s really something interesting to do once in your life. And a few years ago, then, I did the switch and moved to the music industry where I am right now. And yeah, so that’s my current job as a product manager for Universal Music.

Ralf’s role at Universal Music Group (3:55)

Ralf: For Universal Music, it’s really important to take part of this transformation of the music industry and we do this obviously. And one way we do this is the marketing that we have here in Berlin. So we help to support artists in their way to reach out to their friends, to their fans better, but also to generate more streams on Spotify, Apple music, Deezer, and so on. And in the end just to use these new possibilities better.

I come in, in two ways. So I have two projects. My main project is an app. It’s a music discovery app where users should discover new music in a very active way. Because if you go on Spotify, it’s more passive discovering. But we try to make this more active, more interactive, more engaging. And for us obviously the benefits which are obvious. And the other project is like, yeah, maybe 10%, 20% of my time that the web tool. And it’s about curating Spotify playlists. So it’s an internal tool. And for that, I only have two engineers. For the app project I have a cross functional team, the classical setup. And yeah, so that’s my basic contribution to the company. I try to generate more insight and also more activity for the artists that we have.

How Ralf’s gaming background influences him in his current role (5:14)

Liz: Earlier when you were talking about starting with your career path to Universal, you mentioned working as a gaming product manager, which is, I think a very unique area of product management. And when we were originally talking, we talked a bit about how this influences you in your current role. And one thing I found really interesting is you brought up those three big pillars of gamification and how you apply them now. Can you tell us a little bit more about how that past experience is influencing you now?

Ralf: Well, so first of all gamification is like a password, right? Everybody wants to use it. And I think sometimes it’s a bit overused. Maybe you don’t want to open your insurance app and then it’s gamified and you can reach a new level. Right? That seems weird. But my app is highly engaging. It’s user facing. It’s for a younger audience. And so that you don’t get lost in the small things sometimes you need to have a bigger picture. And the gamification, I would say there are three big pillars that help to basic to hook your users, to retain your users, to help engagement. The first one is being part of a community. So whatever you do, you want to be part of a community.

And this I noticed in my gaming time that basically no matter how good the quality of the game was, because it also was on games that say they weren’t the highest quality. But the community was strong and people always log back into the game to enjoy the community. And there are many examples out there of games. Also, ones I didn’t work on that are not young, they’re not fresh, and if you would start nowadays you wouldn’t have fun. But people are still playing them because of the community. And for my app, I want to achieve the same that people can connect to each other, that they can play each other’s music. And that they have this kind of feeling of the community that should help.

So the other pillar is basically this, seems very old school, like when we still lived in caves, it’s like this hunter mentality. And we all notice, “Oh, I found a great deal on Amazon.” Or, “I found a movie I didn’t know before was great on Netflix.” And it’s like, boom. It’s like cool for us. And yeah. Then as a hunter, we help you. And so I also want to give this achievement to users as they find and discover new music. So that’s basically the core that we have for the users, what should basically give them joy and personal gratification in a way.

And there I’m basically at the last pillar, which is this personal gratification. It’s something that you have a challenge or task where you’re in control of, and that you can finish. It’s tiny things like imagine you have a to-do list and it’s really gratifying if you do the press and it’s like the ping. And then it’s like, “Oh, the task is done.” And it’s super gratifying. And so we try to do the same. People can finish music and finish sets and vote for music. And they should get this personal gratification feeling. And in the end this all combines to a big reward for yourself. All these three pillars, they combine in one, big, personal reward. And that’s what we’re trying to achieve. And that’s, I think, where gamification is helpful and not harmful.

How Ralf conducts user discovery (9:31)

Liz: As part of your role at Universal is to increase engagement metrics and retention through user discovery. So how do you conduct user discovery at Universal? Do you have a specific process? What tools do you maybe use?

Ralf: So first of all, quantitative is basically what we use most of the time. And of course you also see what the users are doing in the end. So we use Mixpanel and there you basically check the user events, user behavior, and it’s extremely helpful. And also we do a lot of A/B testing for new features and we see how people are behaving. But in the end, sometimes you just don’t know why something is working or something doesn’t work. If retention goes down and you don’t know why because you didn’t implement something new, sometimes it’s weird. So then you go into the direct user discovery.

And then we do several things. So the easiest thing you could do basically is go to the app store and see what the reviews are saying. People are sometimes very vocal about stuff. And one star, worst app ever, and I hate German rap and so on. Right? But then a bit more complex, you go into real user interviews, like you put up a camera and you talk to them, which needs a lot of preparation, but it’s also very giving and very rewarding for us. User surveys online of course. Also very interesting. Can be sometimes a bit misleading, but also extremely helpful. And yeah, so that’s more or less what we were looking for.

Sometimes we also try to get direct contact with people in the app. Community management is also something I learned from the gaming industry. There you have usually like customer service people, you also call them gamers. And they are basically the line between me and the direct end user. And so that’s very helpful to see what’s going on, what do they want to improve. Why do they use the app so much? What is a pain point? And it’s also quite helpful. And yeah, I would say standard ways that we do user discovery. And, oh, one thing I forgot also inside the company I talk of course, because they’re also stakeholders and they also have ideas. They also have requirements. And so that’s another way of finding interesting new possibilities to improve the app.

Implementing a new feature based on user discovery (11:46)

Liz: Can you share a specific feature that came about because of doing user discovery that you maybe thought like, “Oh, people really want X?” And then you made it and how did it go?

Ralf: Yeah. So actually what is not about a feature that was developed from scratch, but we already had and we thought like, “Oh, it’s great. It’s working. It’s intuitive.” So it was just like the playback of a song and how you would vote for it. And we thought, “Ah, it’s so easy. Right?” But we lost the outsider view, the fresh eyes. And then we did these series of user interviews. I don’t know how much they were, like 12 or something. I just got a bunch of young people who would be the target group of the app in the end. And then we did some very exploratory testing of the app. And then they came to this part of the app and they didn’t get it. and I was like, “Oh, what? How can they not get it? It’s so easy, right? And obviously the mistake is not with them. It’s with me or with the team because it just didn’t work.

And so from there, basically, we got, “Okay, we need to change it.” And so we changed two things. First of all, we made it easier and faster to actually do this playback and do the voting. And we added tutorials. So that was basically the start. And this then took many iterations. Like for example, tutorial. And we started off first just with static images. Then we moved to a video. Then we moved to interaction. Then we changed with tools and so on. And so it took many iterations until we were at a point where we were like, “Okay, the funnel is great.” And the retention increased and also engagement metrics increased. So in the end, it’s all about testing and then iterations. When you’re at a point that you’re happy with and then actually you need to iterate more because you’re never really happy. Right? So that’s in the end it’s like how we did it there.

Advice on stakeholder management (15:02)

Liz: What are some tips that you have for managing stakeholders?

Ralf: So I’ve been lucky that, at least currently, I didn’t have really issues with stakeholders. In gaming there were a few harder ones I have to say. And in the end, always base your decisions on data so that helps. And also if you do your forecast, do your homework, and base it on that. And then try to tell a story. When you present something to them, try to tell a story. And from there basically take them, try to motivate them to get on board. Because if they are motivated to and they like your product, then they also will commit to this. Because I don’t want to motivate people, I just want to tell a story and say I have this product that works and we can achieve something with this. Just listen to me. And so in the end it’s just like tell a nice story and base it on data. And then you are usually in a good spot.

And always be open also for other feedback. Sometimes, you’ll be maybe not anymore in the music industry. So in the beginning was extremely new for me. And I didn’t know what was working. What were the big players? I mean, of course Spotify and so on. But for example, how big YouTube is in the scene. And so what kind of genre’s working and how different the territories are. And so there’s also some things that I need to be open to and to understand and always improve my horizon.

Audience Q&A

(17:04) When you were testing these ideas, what metrics are you using for retention?

(19:39) Have you tried to utilize allies for your project? And if yes, how successful has this been?

(20:55) Do you work in an agile or waterfall environment?

(29:33) What are some challenges of working on an international team and how do you overcome them?

(25:09) Could you tell me how you shifted careers and what you did to familiarize yourself with the product management role at your first stint in the gaming industry?”

Speakers

Ralf is a Product Manager at Universal Music in Berlin. He works with two teams on a music discovery app and on an internal playlist tool. He uses a data-driven approach to further strengthen the understanding of the listeners and improve the impact of the UMG artists in the streaming realm. Previously he worked in data-critical Product Management roles in the gaming industry, where he was managing several F2P mobile and PC games.

Liz is the Event Manager at Roadmunk on the Marketing team. She produces Recess and in non-pandemic times, organizes Roadmunk’s Product to Product in-person events held in Toronto, New York and Chicago.

Prior to Roadmunk, Liz produced events for a renewable energy company and a non-profit.

You can follow her on LinkedIn (https://www.linkedin.com/in/lizpapierz/).

We had three wonderful product leaders speak to our Toronto Product to Product community about gathering and prioritizing user feedback to create customer-centric products.

Karen Stephen from the Canadian Broadcasting Corporation (CBC), Swapna Malekar from the Royal Bank of Canada (RBC), and Martin Kuplens-Ewart from Coinsquare, shared their insights on this topic and the ways each of their companies approach connecting with their users to collect feedback and the trials and errors of then prioritizing that feedback.

We’ve selected some of our favorite highlights from the event below but be sure to watch the video, as this is not a talk you want to miss.

Highlights

(The highlights have been condensed and edited for clarity.)

The “Hell of Feedback” (2:50)

Martin: Ok, so quick show of hands in the room — how many of you have had to deal with that big bucket of ideas and feature requests — it could be a set of spreadsheets that go back a few years or it could be an unmanageable list of stuff?

It looks like about 60% or 70% of [the audience] have had that experience. So there are better ways — you don’t have to deal with that bucket — but that’s the hell of feedback. The tragic thing about dealing with that bucket of gloop is that it drains our empathy, it drains our ability to really care about what’s behind those asks and about those inquiries and it leaves us accountable for managing and dealing with and reporting on stuff that is actually not going to help.

What is the problem? (14:14)

Karen: For feature requests, we tend to look at, what is the problem? And we’ll often do that through our OKRs. We might not be addressing the feedback as it’s coming in but we will mine it. When we know we’re going to be doing some kind of feature work, we will then often go and look at the feedback we’ve received and see what people have said. We’ll try and understand what the actual problem is and then we’ll try to validate it. We like to make sure that we go back and do some user research and confirm with the analytics — is this actually a problem? Is that feature not getting used? Is there something else that we can do? Is this really a problem that needs to be fixed for everyone or is it a three people that wrote in kind of problem?

Choosing which customers to listen to (22:37)

Karen: People feel passionately about the CBC and they have this deep connection that they might not have to other brands. One of the things that we get are people who are writing in because they just can’t use the technology — they haven’t kept up. We can only support so many people on so many devices and we do an awesome job but if you’re still running Netscape on the computer that you first got in 1995, that’s not going to work so well. So we tend to disregard that kind of stuff. [With] a [lot of the feedback], we try to look at what they’re saying, rather than how they’re saying it. I found that you really have to learn how to kind of separate what they’re saying from how they’re saying it and then go beyond that and have some empathy and to try and really understand what their problem is and how they’re experiencing it.

Managing feedback from internal stakeholders (23:52)

Swapna: We have stakeholders who have spent 20+ years in the bank. They have been financial planners or have spent their entire careers in the financial industry, so they know the bank in and out and they know the customers in and out. I think these internal stakeholders are the most passionate people — they have a solution in mind, they have a vision and they are not afraid to share it with us. We have tried to change the mindset of our stakeholders from a pure banking professional or business professional sense into more of a customer mindset.

We had to make them understand that we first start with the customers’ job to be done, and then we understand the gaps or the pain points that we have as of today that can be filled in to enable those jobs to be done or to solve those jobs. We then move onto defining personas, the user profiles, the different use cases and then we drill down deeper into the success metrics, etc. We had to sort of educate them on how digital product management works…and that has helped us a lot because they feel involved in the entire product management process. If they feel as if they are building the product, and that’s how in my experience, they have started to change their mindset — to change from a solution based mindset into more of a customer pain point kind of experience.

Nothing beats time spent with a customer (47:38)

Martin: Nothing beats two hours spent with your customer, with your user in their environment watching them work, interact with your product. Or more importantly watching them do the thing before and the thing after your product — that’s going to be the greatest source of filtering of value, understanding, and empathy that you can get, and the more of your teammates you can bring to have those experiences, the better. It’s maximum exposure — lots of focused face time. You can present concepts, prototypes, play, sketch, draw, give them index cards to sketch new mobile interfaces and quickly iterate. That face time, that really experiencing your product through someone else’s eyes and with someone else’s hands and them driving, you can’t beat that, you really can’t.