Stefania Olafsdottir is a mathematician, philosopher and genetic researcher turned data scientist. She was the head of data at QuizUp, a mobile game app, before co-founding Avo, a company changing the way organizations use data to make better decisions for their customers.
In this Hoss Hangout, Stefania joins us from her home in Iceland and talks to us about the importance of customer obsession and feedback, her upcoming podcast and the dreaded curse of knowledge.
- Stefania talks about the importance of direct feedback from developers and says she especially likes to understand when a developer decides not to use a feature. She says it highlights a potential problem and the opportunity to fix it.
- Stefania describes Avo’s method for collecting customer feedback and why it’s important to understand the motivation behind a customer’s question.
- Are you familiar with the curse of knowledge? Stefania describes it here.
- Stefania talks about the philosophy behind Avo’s upcoming podcast, which will share both inspiring and frustrating data stories to help people and companies learn valuable lessons about data and decision-making.
- “I think you really need to never stop sitting down with your customers and feeling their pain.”
- “Never stop obsessing about your users and their experience, and how your product solves their problems.”
- “We encourage good and bad thoughts and feedback and criticism, because both of them will help us grow and make better decisions for them in the future.”
Matt Hawkins: Our guest today is Stefania Olafsdottir, the co-founder and CEO of Avo, which is a company changing the way organizations use data to make better decisions for their customers. Previously, Stefania was a mathematician, philosopher and head of data at QuizUp. Stefania, it's great hanging out with you.
Stefania Olafsdottir: Sweet, likewise.
Matt: We have a lot to cover. I'd like to just jump right in, into metrics and insights. So first question for you - at Avo, you are making it easier for PMs, devs, data scientists to track and manage product analytics. I'm curious, how do you approach the developer experience with your product?
Stefania: Yeah, that's a great question. I think it's a multifaceted one, actually. One aspect of that question is what is our - how do we prioritize it? How much do we care about it? And then the other aspect of this also, is sort of like, what is our attitude towards it? And I think the other aspect of that question is, how do we ensure that it’s good? And I think, how do we approach it? We obsess over it.
So, we make it also very clear for all of our customers and for the developers that we work with, that we do obsess over it. And when we get feedback about developer experience, we respond to it, as soon as we can, ideally. Obviously we always have to do this in sensible ways, we can’t always do everything immediately, but this matters a lot to us. And then I would also say there are two paths - are two ways - through which we hold ourselves accountable to do that. We have the qualitative parts and then we have the quantitative parts. The qualitative is very related to what I just shared - we just take our customer feedback very seriously. We want to know how developers experience our products, the pain points, in their own words, the reasons why they like Avo, anything that could make their lives easier. One of my favorite things to do is dig into the decision-making when a developer decides to not use the feature we have. And like, why not? What’s blocking you from doing that? Because that I think can be super educational. Because it either highlights something about their situation, their product, their workflow, their tools, or it highlights something about the documentation in Avo could be better, to make the value proposition clearer or the how clearer. Or, it highlights that they hate the developer experience about what we have built. And highlights the opportunity to fix it.
And this really guides a lot of our feature development. This is our core persona. Surprisingly, because Avo, ultimately, we’re accelerating self-serve analytics culture in companies. But the reason why we focus so much on developer experience is that is part of our fundamental belief and what will unlock self-serve analytics culture in companies - where every person can make a decision on their own. One of our fundamental beliefs there is to unlock self-serve analytics culture, we need to make the lives of the data consumers easy - the PMs and the analysts that go and look up information to make decisions about the company. And the prerequisite of doing that - to make that job easier - is actually to make the job of the data producer easier. The person that decides what should be implemented, what tracking should be implemented, what data should be gathered. And then ultimately, the developer that writes the code that gets the analytics event into a database or into Amplitude or Mixpanel or Segment or whatever it is.
And, you know, 65% of our user base is developers.
Matt: Oh wow.
Stefania: Those are the folks that really touch this implementation process, making sure that data is accessible to the rest of the company. And so that's why we see that group as a fundamental part of unlocking self-serve analytics culture, and the world effectively.
Matt: Yeah, that's awesome. Talk to me about kind of that feedback loop. You mentioned, you know, responding to feedback as quickly as possible. There's a lot of insights and learnings there. I'm just curious, kind of from a very tactical perspective, when you, when a developer mentions something, whether it's on your community, on Twitter, chatting with, you know, somebody out there, how, what does that look like in terms of, you know, prioritizing their feedback, getting that into the roadmap, or otherwise?
Stefania: Yeah. So basically how do we, where do we fetch developer feedback to decide what to build next? Is that, am I understanding that correctly?
Matt: Yeah. Yeah. And when you hear something like that, what does that feedback look like? Is that something that, you know, you just kind of throw into a spreadsheet and track it? Is it like, I'm just curious, your very tactically - you know, because I think there is so much to learn from the developer community. And they're often very vocal on various communities. I'm just kind of curious, tactically, how you get that kind of into your - almost back to your product team and how you prioritize what to take action on.
Stefania: Yeah. Okay. That's a good question. Um, so I think it's twofold. It's like, how, how do we make sure we are thinking about all of our developer feedback and documenting it in a way that we can understand how dominant each feedback is or prioritize it. And then the other part is how do we take that data and use that to, as an input into our roadmap.
So, for how we gather feedback, we, number one, we try to work closely with our customers at any opportunity that we get. So that's the customer feedback, but it sounds like you're talking also about like, just communities like Twitter and things like that. Is that right?
Matt: Yeah. Yeah. Whether it's, I just know, like on social and, you know, forums and whatnot, there might be feedback that comes through there, but even, yeah, just even if it's just, as you're talking with your customers, one-on-one, you know, you're probably getting a lot of feedback, a lot of requests, just kind of curious, you know, I think a lot of teams are not great at that product feedback loop. It seems like you guys are, you know, you must be good at it, given your success. Just kind of curious if you had any tips to share, you know, for other teams, how they could build a great product feedback loop.
Stefania: Yeah. Great. Um, so yeah, so let me stitch it back together then with what I already started talking about.
Number one, we try any chance we get to actually hear a developer speak or say something. We try to use that opportunity really well and whatever they say, and whatever the context is, we encourage good and bad thoughts and feedback and criticism, because both of them will help us grow and make better decisions for them in the future and for any of their colleagues or anyone in this industry.
So that's number one, like our company-wide attitude is just celebrate any information that we can get, and try to get the difficult feedback. Just really aim for getting that difficult feedback. So, set the stage - like we want to hear the good things and the bad things, so it's really important.
And then the other part is we have, uh, we take inspiration from a book called The Mom Test.
Matt: Yeah, I’ve read it. It’s a good one.
Stefania: Exactly. Yeah. For, for listeners or viewers or whoever is here with us - to set the stage, it’s this book that talks about how can you get customer or user or market feedback?
How can you approach a conversation to get such feedback where you would get honest answers even from your mom who really just doesn't want to hurt your feelings. So that's, like, the fundamental concept of the book. Would you agree with that, Matt?
Matt: Yeah. Yeah, absolutely. Because yeah, most people just want to be agreeable.
Matt: Even outside of your mom, you know, most people are asking their friends for feedback on projects, features, whatever, even strangers, just - they don’t want to - they feel like it's a confrontation if they’re not, uh - outside of my German friends. They will be very honest with me, so I like to survey them first.
Stefania: Ah, that's good. Yeah. Anyway, so that’s like a general attitude. Another one also is like, another inspiration to us is the usual interview sections and the design sprint book and the concept, um, they cover that fairly well. So we like to do this. And, for example, when we were doing Y Combinator, we tried to set up, uh, or when we were in Y Combinator, which there was three to four months where we would have the opportunity to meet a lot of people. Every week, many times a week, all of our YC friends. And we would just use every opportunity that we could get to just ask them to walk through the product and think out loud about what they hated and what they liked. So, that's what we like.
We also like to sometimes be with people when they're installing things or doing things, um, to just like experience their frustration. Typically and often, unfortunately, developers. Don't necessarily like that experience. They just want to be there and hack through whatever they want to do.But when we get the opportunity, we grab that opportunity. So that's like how we obsess over this and what, how we use that opportunity.
For collecting feedback, we generally try to write up a qualitative analysis on feedback from our customers. We maintain it in Asana mostly, but we also have, um, like feature requests and thoughts through a product called Canny, which allows for like a change log management feature request management, bug report management, where customers and people can upvote and downvote and comment and add thoughts.
But in general, something generic like that, I don't think that is - it's a good, like source of truth, it’s a good way to have some list of things, but I don’t really feel like that gives you the in-depth conversation and questions that we like to have.
Matt: Yeah, there’s almost no substitute for - I love that you mentioned sitting down with a developer, sitting down with your customers. Obviously that’s a little bit harder in this COVID world, but even doing a screenshare on Zoom, even that is so much more powerful than using a tool like FullStory. Now, it’s hard to do that at scale, but it’s just the things that you pick up on, especially if they’re talking out loud, are incredibly insightful.
Stefania: Yeah. And like, just because you mentioned this, it doesn't scale. Okay, sure. But there are two things on that that I think are really important. Number one, it's so important to do things that don't scale in the beginning. And then eventually, you'll find a way to have a lot of things scale so that it can be doable, but in the beginning you can't rely on things just working out, you just really have to be there and feel the pain. The other aspect of that, that I wanted to mention is, um, even as you start doing things in a more scalable way, I still think you really need to never stop sitting down with your customers and feeling their pain and experience.
Stefania: That's one of those things that, I can see myself - like when we're a $100 billion company, I'm still gonna want to be doing that. Right?
Matt: Absolutely. Still put your email in the Twitter feed and say emai me, CEO at Avo dot com.
Stefania: Yeah, I just can’t stop obsessing over our customers’ success. It’s my main passion.
I mean, hey are trying to solve a problem that I deeply have a passion for helping others solve, basically. And so, yeah, it's fun.
Matt: Yeah, absolutely. So -
Stefania: I actually have more, but maybe you want -
Matt: Yes, no keep going. Sorry, not to interrupt you.
Stefania: So like we have, as for communication channels, like sometimes you get these questions that are just vague questions and you don't really understand the intent of the person from the question.
They ask a question like, “I'm looking for a way to press this button,” or something. But there's something completely different that they're eventually trying to do, and this button was how they imagined being able to do it. And so we get questions like that through our Slack community, through our private channels, with a Slack channel with our customers, and through, um, Intercom, for example, and email. And we always then focus on asking questions back and trying to get as much insight into, like, what were you actually trying to accomplish and then try to understand, like, why weren’t they successful in doing that? So those are important channels for us as well and how we respond to that.
And then I also want to mention like, through our customer success function and through our just general customer check-ins, particularly for the, I guess like the higher touch ones or during the onboarding process, we have frequent meetings, like, weekly or bi-weekly or something like that, where the customer success person meets the stakeholder and stuff like that. And we really like to try to - the developers at Avo and the product folks at Avo - they love to join these conversations. Sometimes with an agenda to get feedback, but sometimes just like to listen in and hear what they’re talking about, hear their frustrations. And if there’s no one on the call and they start talking about product feedback or asking questions, our customer success person is really good at just starting a Zoom recording, with their permission, to share with our product team. And then we set the stage and ask them to try to be very explicit about like what was the frustrating thing about that? Just like, be vocal, share your frustrating thoughts and your happy thoughts while you’re going through whatever you’re asking.
Matt: Yeah. So for your customer success folks, I’m curious, how - I mean, you have a fairly technical product. A significant percentage of your users are developers. How technical do they need to be? And I’m just curious how do you build up a function like that with a product like yours - whether it’s that function or just across your organization. How do you think about that?
Stefania: That is a really great question. And I’ve thought about this quite a lot. Both for the success functions and the customer-facing functions - but as well as for like marketing functions and then even for just like operational functions - like the person who is managing the team funds or like the budgets or hiring. And from my perspective, what I think I’ve learned is, when you are building a category, of a very technical product with a very technical audience, I actually think that it’s really important that the Avo team in its entirety - and particularly for anyone that really works closely with the customer - is at least very technically savvy, and ideally has experienced the role of the main stakeholder before.
I think there are a few things behind this. Number one, it’s about just like, adopting the vision of the company. And I want to talk about that more. And then the other thing is how critical and sort of detail-oriented our audience - a technical audience - is. And I want to talk about that as well. So, for the first one, for incorporating the vision and the passion, for executing a huge mission of a very technical product, you need to have - or in general, not a technical product - you need to have passion to go there. You need to have willingness - at least the best teams - they are willing to have passion for that. And for something like Avo, where, we’re not curing cancer, we’re helping the companies of the world that are curing cancer be more efficient in doing that. We’re also helping people be more happy and productive in their work. And less frustrated and have less job churns. So, there are different drivers that need to be there, but I think it’s really important that anyone who works on the Avo mission needs to be able to somehow relate to the mission and the vision of the company. And when you’re building a technical product, it’s very difficult to be able to get there if you aren’t technical. So that’s that.
And then the other part about the success part - I want to talk about that as well, but I’m curious to hear what you’re thinking.
Matt: No, yeah, I mean I think passion is - you absolutely have to have that. That is, for me certainly, if you’re not passionate about what we’re doing, it’s a dealbreaker. Because I think passion is one of those things that - it drives you that extra mile, it’s infectious, your customers can feel it, people thrive off that, it attracts them. And so you’re going to, whether you’re in sales or customer success or whatever function, that bleeds through to your customers. So I think, yeah, that’s absolutely critical.
Stefania: Yeah. I love the concept of the talent density from the No Rules Rules book by Netflix. We want every person in the company to be making really great decisions about the future of our customers. So, I think that matters because of that. And then the other aspect of that, the other reason for why I think this matters, is the audience. So, when you’re building a product that has a technical audience, like Avo does - it’s developers, it’s product developers - that’s the 65% of our user base. The others are data scientists, data analysts, date engineers and product managers. All of those folks are detail-oriented people, they have high standards for quality, and they won’t just let go and forget like maybe typos in text or a link that doesn’t work or like a misused concept when you’re overusing the term “platform.” All of those things that indicate that you don’t have - that the Avo brand - what they are maybe considering adopting or working with, all of those could be indicators that the Avo team doesn’t really know what they’re talking about. So it undermines their trust, a lot.
Matt: You lose credibility. You know, you have to maintain that every individual is a representative of your brand and if they don’t have to technical chops to talk the talk so to speak, then it’s not a great situation.
Stefania: Yeah, and I think and that doesn’t end - this is like, customer success - really important, support - really important, all of the content that we create, our online social media presence, just like every touch point that we potentially might have, needs to fulfill the quality standard of a technical audience in the content that we’re creating.
Matt: Yeah. So one follow up question on that actually, do you achieve that by having - kind of going the Netflix model of everybody, you know, you trust everybody on the team wholeheartedly to deliver that, or is it the case that you also have - and maybe you trust everybody but there’s sort of a trust but verify - do you have a - for things on social or content that you post, do you have a final review? Do you review it? Or do you have somebody on the team that everything passes through?
Stefania: Oh, that’s such a good question. Because it sounds like two major aspects, which is, like, number one, it touches on the general concept of peer review, which has sort of become an industry standard for software development. It’s not weird at all that you expect to get and give great and thoughtful and critical feedback on pull requests. That’s exactly what everyone - both the pull request creator and the pull request reviewer - they assume that their role is to shape the best stuff. And it’s not personal when folks are giving good feedback. I don’t feel like this is necessarily adopted as a concept or accepted as a concept in all other aspects of a company. The town hall presentation or the budget or the marketing blog post, or whatever.
I remember at QuizUp, for example - so, my background is I’m a data scientist. I’ve been there for 10 years, I’m a mathematician and a philosopher turned data scientist in sort of the academic world, basically, and genetics. And then I went into being the founding analyst of a gaming company called QuizUp, which was like, at that time, a rocketship and we were scaling super quickly. We eventually hired 10 or 16 or 20 folks in data roles or growth roles or things like that. And one of my core beliefs in building that culture was to make peer review and code quality and reporting quality a fundamental aspect of the data culture. Which typically and historically in most other organizations is like a cowboy type of role and there’s no code sharing. So, that’s a concept that I like and we have that also for our content that goes out. The other part that I wanted to mention is that I, obviously, obsess over the positioning and marketing and like the image of Avo. You know, you want it to be perfect. But it’s also my role to make sure I’m never a blocker. And when you have a very tight schedule, it’s easy to accidentally become a blocker. So I have actually, unfortunately removed myself for example from reviewing all of the content, but I love to take every opportunity that I get to give extremely thoughtful and extended feedback to really share my assumptions on why I’m giving specific feedback. And that’s sort of how I try to make it more scalable. Instead of reviewing every blog post, then like, if I see a weird word, I might write like a two-page summary on like, okay, I don’t like when people say “your developers” because it implies that you own the developers and they’re yours to allocate as resources, and I don’t think that is a part of the culture that I want to create in the world, so I think we should never use that phrase. So that’s how I try to scale that, basically.
Matt: Yeah, you instill your philosophy with the team. And it’s a sign of growth, too, that you’re able to kind of hand that off. Growth - you as a leader, Avo as a company, where your team is empowered and they know what you’re looking for. You’ve gone through enough iterations with them that they know those phrases not to use.
Stefania: Yeah, it’s an endless alignment mission but yeah it’s a really beautiful thing because particularly like you’re saying, they know the phrases that you’re doing - I love this analogy of the curse of knowledge, have you heard about this concept?
Matt: I have not.
Stefania: It’s this - when you don’t understand how - that you know more than the folks that you’re talking to. You assume that they have the same understanding that you do. Like I would say acronyms are the ultimate symptom of the curse of knowledge. When folks join a new company and everyone is just talking in acronyms and they forget that you don’t know what a MAU is. Like, why would you know that? Monthly active user. But so, my favorite visualization of how intense the curse of knowledge is, is like this great anecdotal experiment that I read about, which is a group of folks that were given the assignment to tap along with a song they were given, and then an equally big group was given the assignment to guess the song that they were tapping. The tappers were then asked to estimate whether the guessers would accurately guess the song. And in 50% of the cases, they estimated that the guessers would in fact guess the song. However, the guessers guessed the song in 4% of the cases. And this really brings home the point that when you know the song, you hear it in your brain, it’s so obvious what you’re doing, right?
Matt: Right, right.
Stefania: And this is something I try to remember in everything that I say to most people I talk to - like do they understand what I’m talking about or am I under the curse of knowledge?
Matt: Yeah, that’s fascinating, I mean there could be a breakdown with your team, there could be a breakdown with customers, with prospects, and yeah, huh - I had not heard that concept before but that’s pretty interesting.
So, jumping around a little bit, on the theme of content - word on the street is you have an upcoming podcast. If you’re open to it, I would love to know more. I’d love to know more about what it’s about, what you’re diving into and if there’s any learnings, too. We talked about the importance of content and how important that is, you know, for all teams really. And for others that are thinking about maybe starting a podcast, a YouTube channel, doing webinars, I don’t know if there’s a couple lessons you’ve learned from doing that?
Stefania: Yeah, I mean so we are, like I said a little bit earlier, I am very passionate about this challenge that we’re working on at Avo. And it really goes back to when I was part of building that culture, self-serve analytics culture that would empower the entire company to make good and great data decisions or basically, good and great business decisions using data instead of your gut. Although your gut can never be removed from it, data never tells the whole story, but it’s such an important part of it. My role in that organization was really twofold, it was the cultural and the technical role. So, the cultural one being you know, get people curious to use data to answer questions. And that means doing things like some internal marketing of exciting findings that can drive some changes because that is the trigger that gets people excited to want to then do that for what they’re deciding on. And it’s also, the other culture part is just supporting people and asking the best kind of questions, not naive questions, and using the best data to make their decisions and really thoughtfully estimating things. So, that was so much fun. The philosopher and the mathematician in me really loved that role. Then, the other part is the technical part. It’s like, making sure the infrastructure is in place for people to actually be able to access the data. And that we have the right data in place and that the quality is reliable so people aren’t making incorrect decisions that throttle the company growth because they were missing a data point somewhere. And this is something that was so early when I was doing it - I was doing this in 2013. Amplitude didn’t exist already as a concept - we adopted it a little bit later - it still had the beta label on it. Mixpanel had recently surfaced. Zynga had been pioneering ways of doing, you know, customer retention and cohort analysis for game user success and all those things. So all of these concepts were so early, and they still are. In the world, we’re getting to the early majority or a mainstream adoption of really thoughtfully making product and business decisions based on user experience because the end user is now like the ultimate success of the company because everything is SaaS and end user related. So, there are all of these major shifts in how software is being created and how business is being made successful, and they all stitch back to end user experience and the data that explains the end user experience. But the tools to really thoughtfully make decisions based on this data is very lacking, and even when you have the tools to visualize and view the data, the paths and the processes and the tools to actually make sure that data is reliable are very, very clunky. It touches so much on like everything from how you build your data culture, whether your data person is built into every product team or whether you have these isolated pods of people that act as a supporting function, it’s these organizational decisions that really matter in how you’re successful and making your business more successful using data.
And that’s something that I feel like that content - I just, at least, still have so many conversations with folks everywhere around the world where they’re seeking information and advice on how to build their organization. So, you know, I have my data points - my own experience, I also consulted a bunch of folks, we have our Avo customers - but our mission here with this podcast and the interviews that I want to make there is just talk to product and data leaders basically and growth leaders, the folks that have been making big business decisions based on user experience data and hear their learnings about how to really structure your organization to make this successful. And give the world inspiring data stories, but also really frustrating data stories so they can learn from them. So, I’m just really excited about that. First and foremost maybe because I love those conversations, but also because I think that’s something we still are missing. Certainly was when I was there - we’ve seen some versions of this pop up, you know, there’s some content that’s growing.
I remember the first time I heard a data podcast episode that I was like, “Oh my god, yes!”
Matt: These are my people, this is it!
Stefania: Yeah - some confirmation that I wasn’t like flying solo in how I made decisions about building the culture. But yeah, helping people make better decisions about their organizational structure and data culture is something that I’m very passionate about.
Matt: It sounds absolutely fantastic. And I think you’re right, I think you really are filling a gap, and you can be the center of that conversation around making better product decisions through data. I love it. So anyway, I look forward to subscribing myself.
On that note, Stefania, this has been absolutely amazing. Really appreciate you hanging out today. One final question - for those other entrepreneurs out there, other data scientists, those passionate creating an amazing developer experience - any parting thoughts, words of wisdom? And also, where can people learn more about Avo?
Stefania: Okay, great question. Parting words. Huge. I think my main message for this, particularly for the early journey, is probably never stop obsessing about your users and their experience, and how your product solves their problems. Just obsess about it. I really took a lot of inspiration from Marty Cagan’s Inspired book, which he just released another book as well, so it’s great, where you have these four pillars that you need to obsess over if you’re going to build a successful product. It’s the user - the usability, what’s the experience? - it’s the desirability, the market, who actually wants this, it’s the engineering feasibility, does it make sense to build it? And it’s the business viability. Is there any way we can make it sustainable to keep building this to solve these problems and make money off it. These are the four pillars I recommend obsessing over and never stopping to.
And where they can find Avo? Avo.app is our website, and you can find us on social media. I’m there as well somewhere, you can find me. Stefania.io I think is my website, I might have an email address there somewhere. Yeah.
Matt: Fantastic. Well thank you, this has been tremendous, I really appreciate you doing it.
Stefania: Likewise. Thank you so much for having me, Matt, it was so much fun.