Unlock the Power of Product Analytics for Transformational Success!

Unlock the Power of Product Analytics for Transformational Success!

• Thank you very much.
• My name is Renee de Leon.
• I'm the founder of Verada Product Lab, and I basically have a product, a fractional product practice.
• I specialize in product instrumentation, which means tracking what you're doing within your app with product analytics.
• And that's kind of what we're going to be talking about today.
• The title of today's talk is enabling transformation, measuring product opportunities and impact with analytics.
• Of course, this whole event is about analytics and data, so we will have an opportunity to see how it all ties together here and just to get started.
• How many of you have heard the.

Have heard about transformation?

• It's a buzzword.
• It's like it's all over the place.
• One of those words in tech and in just business that we hear so much that at some point it just kind of loses its meaning.

And you're like, what is this?

Right?

• It's announced to talk about reorgs, to talk about better ways of doing things, or to bring someone new into the team.
• And at the end of the day, we're kind of left mystified.
• So let's hope, too, that in this presentation, you get a better sense of what transformation is and how it can truly help your teams do better work.
• Beyond the buzzwords, beyond what you may have heard before, a little bit about me.
• Like I already mentioned about my company, but before that, I've spent five years working in product building products.
• I'm originally from Puerto Rico, and have always been sort of in the digital space, working first in ed tech, then e-commerce, and I've worked in other industries as well, from government systems to life sciences.
• And now I'm working with clients in IoT.
• So really, I have run the gamut.
• But what I wanted to kind of mention a little bit about my experience is that even though I've been in product for 15 years, the reality is that I have not always done well.
• You know, yes, we can build products.
• Anyone can build a product.
• But to truly understand what brings value to a customer, to truly understand what is worth building, requires a set of skills that I wasn't born with.
• Nobody is perfect.
• So at the beginning of my career, I may have relied too much on just gut feeling, which sometimes is good, and sometimes, you know, it's terribly bad.
• You may launch something that nobody really wants to use.
• I really wish that some of the techniques and tools, which, to my own defense, some of them were not even available at the beginning of my career.
• I really wish I had known about those sooner.
• I think it would have saved me from launching things that people weren't really interested in.
• And in general, it would have brought better business results.
• So with that, let me go ahead and talk.
• I actually wanted to.
• My apologies.
• I have some worksheets here for you if you want to pass these around.
• So, yours.
• My hope is that we will make this into an interactive session.
• I just talked about how I really want everyone to feel like this is something that you can incorporate into your current business, or maybe into that product idea that you have, something you want to launch, or just a business that you like.
• But we need to exercise those muscles of getting to work with data and use data to make better business decisions.
• So with that, let's talk about transformation.

What is it?

As we often hear, all tech companies, all companies are tech companies now, right?

• All companies have to manage data.
• Even if you're an old-school company, chances are you're managing things like payroll or transactions by integrating with systems that generate data, that consume data, and we're all immersed in it.
• And that has really, that has produced and accelerated over the past decade is that company.
• Even old-school companies, steel manufacturers, have had to adapt their business processes to the reality of having to have these transactions online, to having data about what they're producing online, particularly in the cloud.
• And that is what we normally talk about when we talk about business or digital transformation.
• It's the adoption of technology to translate business operations, whatever those are.
• Translate those operations to a digital format.

And you think, well, that is, you know, isn't everyone doing that already?

• Well, you'd be surprised.
• Last, like they would say, two weeks ago, we just heard that tartan restaurants finally decided to allow delivery and integrate with Uber Eats and other delivery applications.
• Imagine they went through all of COVID in all of this time without enabling delivery.
• And you think, at this point, a restaurant like that would have entered.
• So that's just an example of, there's a lot of transformation to go around.
• You know, there's still a lot of work to be done for companies like this to adapt.
• And to the extent that new technologies come in, like right now, we're seeing it with AI, we're seeing it with augmented reality and virtual reality.
• To the extent that these edge technologies come into companies, we're going to have to continue their process of transformation and adaptation into those new technologies.
• So it is really an ongoing process of digitalizing all of your operations.
• Now, oftentimes, along with transformation, we hear another term, another buzzy term that has to do with transformation, which is transformation into the product operating model.

Have people heard about that?

• The product operating model was popularized in Silicon Valley by many of the major tech companies there, but really distilled into some principles by this guy named Martin King.
• And what he's describing is a way of operating in which you are not just developing products.
• You're saying, oh, executives come up with an idea.
• They then say to a business analyst, 'Put together some requirements,' and then those requirements get sent to the engineers, and voila!
• Now, everyone has to create this product that may or may not be wanted by customers, but there is a better way of empowering people who are on the ground - the engineers, the stakeholders, the product people - to come up with creative solutions to problems.
• So, in this new model, instead of having these feature teams that were basically just receiving instructions and producing things, you had what is known as empowered product teams - people who were informed and who understood what the company's problems were well enough to say, 'I know how I can solve that.'
• 'I know a better way to do that.'
• And instead of prioritizing solutions, what you would have to do as a product, as a leader, as an executive of a company, would be to prioritize problems and to tell your people, 'Look, the problem that we're having this quarter is that we're facing a high cost of acquisition and we're acquiring a lot of customers, but it's just costing us a lot to get them to that first point where they pay and get on board.'
• And then your teams can say, 'Aha, I know exactly where the bottlenecks are, I know where we can streamline that process.'
• What you'll hear is the need for strategy narratives from executives, the need for what is known as OKRs, which you may have used in your career, and as well from the engineering side and the development side, the need for continuous releases so that you have the flexibility to change directions in a product.
• If you were thinking of releasing or developing a feature and then midway through it, it looks like the market's no longer doing that, you can go ahead and change directions or pivot.
• Now that sounds fine and dandy.
• Sounds fun, sort of, but many times it fails.
• In fact, according to productplan.com, 80% of digital transformations fail.

Any ideas why people are triggered?

• Yeah, go for it.
• And then we'll go.

Lack of resources, disorganization?

Yes, yes, definitely, definitely.

• There are a lot of cultural reasons for that.
• You can have cultural resistance or, as you said, disorganization, inconsistency.
• So that means, like we said, OKRs.
• But then we forget next quarter.
• It's like we only do them, you know, once a year and we don't really revise them.
• So we don't really have a cadence of OKR remissions and OKRs, which are, of course, objectives and key results, in business silos. Everyone is doing the OK architecture in their own way, and the operations group is doing their own thing, and technology is doing their own thing, but they're not really in line with one another.
• But today we're going to talk about what I believe is really one of the biggest challenges to adopting a truly successful transformation process, which is metrics. There is a lack of a common language to describe operations and what success means, and transparency is lacking such that everybody understands the same reality.
• When I was in EdTech at Kaplan, I remember one of the important metrics over there.
• I'm going to talk a little bit about retention and graduation.
• But one of the first things that they ran into when they attempted this transformation was that each department had a different definition and calculation for retention, for student retention, and what they measured as a student staying versus dropping, leaving, or flunking.
• So, some people were only, you know, measuring drops versus others that were measuring drops in and fails.
• Now, that is a problem.
• You can think about situations in your own industry, in your own company, where it's like, yeah, we're not, we think that we're talking about the same thing, but actually when you go down into the data, the definitions of these metrics are different by different departments.
• And we can't have a common understanding of reality when that happens to the other thing.
• The other situation other than that common understanding is that the key results, those KRs and OKRs, are defining output in terms instead of outcomes.
• And of course, if your job is to measure developer productivity, yeah, that's
• That you're measuring outputs, that's perfectly fine.
• But too many companies launch OKRs only to end up just measuring whether something got released or not.
• In fact, when I was at Kroger, we were also going through this sort of process of transformation, and we didn't know. I mean, we were trying to transform.
• We're all sort of new to this.
• And I remember that when we got this software so that we could all standardize our OKRs, you could only enter, you know, it was very constrained so that people would only be able to enter key results as numbers.

Right?

• So you had to pick your metric and the number, you know, whether it was a total, a percentage, or whatever, but we didn't have clear metrics.
• So in the end, a lot of people, including myself, ended up just saying, 'so feature to display coupons on checkout screen, 50% complete.'
• You know, that's not a key result.
• That's not a key result.
• What you truly want to measure is the actual impact that it is having.

So if you are releasing that coupon feature, what you want to know is, is it making people add more things to their Cart that one they originally had?

• Is it actually increasing the order value that is a sure result, not whether something got, you know, 50%, whatever.

What is 50%?

• Release like that in itself is excellent.
• Wonderful.
• Good for you.
• Exactly.
• So, of course, this wouldn't be a talk about transformation if I didn't throw a transformer in there somewhere.
• So I tried to fit it in like, transformers, finally.
• Check.
• So this is the point where we're going to say, let's go, let's see how we can do this the right way, and let's see how we can create key results or metrics that reflect a common understanding of what is going on in the business.

What is the bottom line?

• And what you'll normally see is that this process involves different types of data.
• Right.
• And we're going to talk a little bit later on.
• We have some presenters here that are going to delve deep into business intelligence, and I don't think we have a speaker today who will cover marketing analytics, which may or may not be within the company, but I'm going to cover product analytics.
• So, to kind of picture the relationship here.
• What you have is business intelligence that basically collects all the operational data that the company owns.
• And that data will come from systems like customer relationship management, CMS, CRM, or ERP for resource planning.
• Or it may come from the app's own database, but it's really gathering all the data the business is generating and managing it in a way that it can be served, integrated, and displayed.

When we talk about product analytics, what we're saying is, can we examine the things that happen within our app?

Can we track the user behaviors?

Can we classify those behaviors by types of users?

Are the users that are coming into, let's say, an education app?

• The users are coming in as instructors, behaving differently than the students, or those that are coming as tutors, behaving differently than the full-time faculty instructors, to really understand the product experience and work.
• People are getting confused or they're getting stuck.

In order to do that, we're going to see a little bit later how we split user attributes, meaning how can we describe our users from events?

How can we describe what they do within the app?

• And like I mentioned in the previous slide, one thing that happens is that some of these technologies, platforms, tools that we use for product analytics also serve campaign management for marketing.
• Not all of them, but there's a little bit of a blurry, you know, middle ground.
• So, it is important to talk about campaign analytics, which you may or may not have seen, because sometimes those exist within an internal marketing department, and in that case, it may be well integrated into the BI system.
• Like if your company is like a big retailer, chances are they want to know very intricately how their online campaigns are actually affecting the bottom line versus maybe another company that is starting out.
• They may be selling some things.
• They may have separate, what they call a CRO, like an agency that manages their campaigns and sends them leads, but those leads and that information, that data may not be super integrated into the BI system.
• So, all important to the ecosystem of data that companies are dealing with, but different levels of information.
• So, this is sounding already a little bit too much, so maybe we should go with our instincts, like the kitty.
• The kitty may have really good business instincts, kind of like I thought I had.
• But let's talk a little bit about a bit of basic implementation of product analytics.

Like, what does that entail?

• My company has BI.
• My company has a campaign, but let's say I do want to understand better how people are behaving within my app.

How do I get started?

What, what, you know, what are the tools?

What are the things that I need to know in order to launch a program?

• Product analytics.
• So we're going to go and examine these four different aspects of it.
• We're going to talk about business metrics, North Star metrics, events, and user attributes.
• This is where you have the worksheet that I share with you.
• My hope is that you can follow along with each one of these and not just absorb what's here, but also think about how this applies to my business or the company that I work with, my clients, or come up maybe with, you know, an idea of a startup if that doesn't apply.
• But it's definitely a muscle to exercise.
• So I encourage you to take some notes and give it a try.
• I like that picture, by the way.
• You can do it.
• All right.
• So when we talk about implementing product analytics, the first thing that we need to understand is business metrics, what I like to call the industry levers.

When I come into a job, one of the first things that I like to know, given that if it's a new industry, if it's a new market that I'm approaching, is what are the business metrics that move this industry?

What does success mean in this particular business model?

• And that may be different for retail communication to business, to business SaaS software.
• But it is important for you to understand what those levers are such that you can release things that are actually solving problems.
• Let's start with retail and e-commerce.

Any ideas there?

• Sales.
• Sales, of course.
• Of course.
• So there are some very specific business levers that you need to understand if you work for any retailer, such as a major retailer in the US or abroad, like I did.
• One of them is what is known as AOV, which is defined as average order value.
• They define as average order value.
• And that is exactly what it sounds.

So take all the orders that have been generated in a given, everything is time box, right?

• So in a given month, water.

And what is the average amount of orders?

• Of course you want that to go up.
• You want that to be maximized over time.
• Another one that they, that they track very closely is card conversion percentage, or the reverse of it, the other way of seeing it, which is card abandonment, of course.
• You want to minimize card abandonment.

You want to maximize card conversion, which means people who actually, and we've done it, I mean, we put something on a cart and they're like, do I really need it?

• You abandon the car, and you forget about it.
• You never come back to it again.
• So many online retailers do not, say all of them are tracking those, are tracking those abandoned cars, and they want to know what happened for you to make that decision, to just not check out after you put something in.
• And of course, total orders.
• Now compare that to education.
• I already mentioned my experience when I was at Kaplan, but I'm sure here, if I don't know where Eddie is, but like, anyone who works in, oh, there is, there she is.
• Anyone who is in a higher education environment will likely be familiar with these metrics because they're likely publicized to everyone that works there.
• One of them is enrollment.

Right?

How many people are enrolled first term retention?

• Because over the years, there has been a very strong correlation with passing your first term and staying as opposed to dropping.
• So that is why universities, and you know, junior colleges, community colleges, spend so many resources to help students who are just starting out with orientation, with remediation courses, with all sorts of mentoring, tutoring.
• That first year is chock-full of resources for these students because their ongoing success is so correlated, so dependent on their success in the first term, and of course, graduation.
• We want people to actually finish the course, to earn the degree, earn the certification, and that is a measure of business success.
• Most recently, I was working at a B, two B SaaS company.
• I continue to consult with several of them, and you'll notice another set of business levers there, which, depending on the industry and the focus you may have, more or less.
• But what you normally see are subscriptions, upgrades, renewals, and one that venture capital investors really like to know.
• Many times the annual recurring revenue for ARR.
• And of course, there are other metrics that are more related to costs, like the cost of acquisition, cash, all those things.
• But I'm mostly focusing here on the growth metrics.
• So before I flip to that, take a moment to think about your industry.
• I have a space there for you to think through.

What are the business levers in your organization after you understand your business levers?

• One metric that is particularly tricky, that will come, that will be able to distill after.
• And you've probably heard this because another buzzword is the North Star metric.

Have you heard about North Star Metrics?

• No.
• So people get super.
• The first time I heard about the North Star Metric, it took me a while to understand what it was because I was in that mentality of just business metrics.
• That's all I need.
• But the Northstar metric is not just a business metric.
• It's not a business metric.
• It's actually something that happens within your product experience that is a signal of growth, an early signal of growth.
• So you're not talking about enrollments themselves, you're not talking about order value.
• That's not what a North Star metric is.
• It is something that happens.
• So.
• So, to give you an example, in my last company, because I struggled so much to come up with this for a while, I had a product leader who asked us, we don't have a North Star metric.
• It's a problem.
• We don't know how we can predict whether we're on the right track when we launch products or not.
• So this company was a B, two B SaaS company.
• It was in the life sciences, had this, this basically system that allows you to organize your documents for auditing and to keep a quality control, quality management system.
• And finally, after much discussion, what we figured out was that this signal that our customers were actually progressing towards their preparedness for the audit, which is what they wanted.
• They needed to be prepared at all times for an FDA auditor to come through the door and say, 'Show me your quality, your complaints, show me your, you know, your actual training records for your employees, or show me this, or show me that.'
• What they needed to have were a series of signed processes, and actually, those signatures were tracked electronically. They were official signals that they were producing these documents, and that these documents were actually passing from the floor manager to the quality control supervisor.
• So signatures or sign-offs, as we call them, were actually the North Star metric.
• It wasn't just because the more sign-offs you had indicated collaboration within the company and things actually getting done, but also if the number of customers from the company grew, what would happen to Cynos? They would go up. You would have more customers. Yep.
• They would go up.
• You would have more customers.
• Yep.
• So I have a question for the North Star metric.
• Yeah.
• Is it like the macro mission and then the.
• Well, the macro mission is the North Star metric, and then the tangible things like the sign-offs to get to that.
• Exactly.
• Exactly.
• It is a signal that tells you that you are on track for growth.
• And in this example, it was like sign-ups indicate that there are people actually using the product and collaborating within the product, or there are simply more users, more customers with the product.
• So it's a signal within the product that tells you that.
• Tells you that.

To give you other examples here, as you can see, for a company like Facebook, what is their data attention?

• Right.
• They're selling ads.
• They're selling ads.

Right?

• That's how they grow.
• That's how they grow.
• So they need people to be logging in, actually go scrolling, scrolling, scrolling forever, for better or worse.
• So their actual North Star metric is daily active users.
• Now, because Facebook has been.
• I was actually one of the first to track daily active users.
• That metric became super popular, and companies that were even in the business-to-business software industry, which had nothing to do with attention, were using daily active users when that didn't mean anything.

Why?

• Because if you're using business-to-business software, chances are you're kind of forced to do it.
• Think about it.
• Yeah, think about the last time that you filled out a performance assessment. Like rolling eyes, you didn't want to do it.
• So it really doesn't matter how many people logged in to fill out the performance assessment because you were forced to do it again.
• You have to think about what your game is.
• For example, for Uber, the game is transaction.
• The game is transaction.
• So their north Star metrics is the number of rides per week.
• That is what indicates growth.
• It's not revenue itself.
• Revenue will follow from that signal.
• So, take a moment.
• Now, look at these examples.
• Yeah, go ahead.
• I'm going to ask for your help.

Since you're a member of the coin, what would be an all-star metric?

- Great.

• I'm gonna have an AI tool that helps you with this pretty soon, but resourcefulness.
• But if you think about our program, a large part of it is attendance.
• That is the signal, what happens within the experience that leads to events growth of the program.
• So if you say attendance for, you know, and every.
• By the way, I can't stress this enough, I'm saying attendance, I'm saying sign offs.
• All those metrics by themselves need a time box.
• They need a metric.
• So you need to say attendance per month or sign ups per quarter or per week.
• So you need to determine depending on how often you want to review this metric and what makes sense with the volume.
• Right.
• So if you were to say weekly attendance as a total, maybe it wouldn't make much sense, but if you say as a percentage of people enrolled.
• Right.

So like presentations too, right?

• Like the number.
• Yeah, I was thinking more of a how many, how many members spoke.
• But I think that's a lagging indicator.
• I love that you brought that, that term.
• I haven't used that terminal in this presentation, but that is what we're trying to avoid.

Right?

• Like the lagging indicator.
• We're always gonna have lagging indicators, meaning metrics that by the time you get to see them, it's too late.
• You can't really intervene.
• It's important to know those.
• It's important to have that data.
• But ideally, in order to make real time product decisions, you need to have what is known as leading indicators.
• And that is where the North Star metrics comes in, because it is something that you can see.
• Oh, you know, for example, Airbnb nights booked is tanking.
• What.

What is happening?

Right?

• Like, you can see that night, 9th book probably for a month.
• Or actually Airbnb is a weird case because it's just nice booked.
• They don't, they don't have a time box, but Tinder matches made.
• You know, you can act on those signals.
• You can act if that is falling, you can order value or you, if people are leaving the cart, you can act on those leading indicators.
• So take a moment again to think about what would be a good Northstar metric for your business.
• And we'll move on.
• But, yes, this one's a tricky one.
• So I have another question.
• Sure, go for it.
• So it's good to put, I think you said putting a certain order, a certain time frame.

What's the time frame to check if you're going in the right direction?

• I know it varies.
• So if you.
• Depending on the volume.
• So if you have a company, let's see, they're like Fitbit.
• Ideally, people are putting on their fitbit every day, so ideally, you should have enough data every day.
• You could even check that daily.
• But you may have an app that, just by the nature of it, you're not going to have daily.
• Maybe it's a once a month check in thing.

So you have to think about what is the natural flow of work within the app?

What is the expectation in a performance assessment type of app?

• You're not going to have daily something to track daily.
• You might have something, you know, every quarter or something else based on just a natural cycle of how people engage with that.
• Like Tinder.
• Like you could find a date, and then if you end up with a date, they can stop using it until you get.
• You break up.
• Exactly.
• And by the way, by the way, it's funny because those matching apps are having a lot of trouble, in part because their business growth is at all with people's, you know, results.
• Yeah.
• With their own goals.
• Like, people want to be matched forever, like, some of them not, but some people are looking for that long term match.
• I'm helping lead this app.
• The app doesn't have that same, you know, that same goal for you.
• They want to keep you for a longer time.
• So that's when the churn rate is necessary.
• Yeah, that's another.
• That's another.
• Yeah.
• Retention metric that.
• That you can have as well.

But what within the product is a good indicator of churn?

• Because you're not going to get churn.
• It's very abstract.
• So you need to find what behavior within the product is an indicator of churn.
• When I was at Kaplan, one of the early machine learning products that we created, I was really cool.
• I'm actually very proud of that.
• When I went, got used, got expanded was a faculty dashboard that we use machine learning algorithms and studies to figure out the behaviors within the classroom that indicated early from week one whether a student would likely fail that course.
• Wow.
• And we were able to send that data to the advisors and to the instructors, such as to red flags, so that they could intervene early and say, look, this one's likely going to struggle.

So make sure you keep the close eye, you help them, you know?

• Yeah, I think every product should have its own North Star metric.
• You're a company that has a lot of products, you know, like, let's say meta.
• Now they have Instagram, now they have Facebook.
• Each one is going to have maybe a different North Star metric, and that's normal, but within a self contained product, you should probably have one that is your indicator, because otherwise it's just too much noise.
• And we run into that problem of different sizes, different departments measuring success differently.
• And what you want to aim for is for the entire business to have the same concept of success, to generate that alignment so that.
• Such that everyone's going towards the same thing now with great questions.
• All right, so work through that, and then we're going to go to.
• To events.

- So.

• So it is possible that you may need to actually do a little bit of study of your map, your user journey, before you have a true good understanding of what your North Star metric is.
• It is possible you're going to have to experiment a little bit before you get a good sense of what are those signals.
• And you do that by segmenting users, types of users, so that you can get a sense of how each type of user behaves differently from what is known as events.
• And events are things people that do.
• People do within the app.
• And the Northstar metric may come from those events.
• Right.
• If we say sign offs, then that's an event in itself.

Now, how do you identify events?

• Let's do a little exercise here together.

Have you guys done user stories?

You're familiar with that?

Or narratives, perhaps, where you kind of talk in general about something a user does?

• So here we have.
• We have a kind of a narrative for a user.
• We say, I'm talking as a user, and say, I am an online student who is currently employed and using my company's business subscription from my laptop in our Miami online office.
• As soon as I log in, I review my course progress and select my next lesson.
• When I finish a course, I get a batch, which I choose to share on social media as well as download it from my personal website.
• Now, that's a simple narrative.
• You might have longer narratives, so to talk, but you can think of narratives for your business.

Can you identify there where the events are?

• Basically, yeah.
• There you go.
• Let's take a look.
• As soon as I log in, that's an event that you need to track.
• Logins.

I review my course clicks on course review, or whatever page you have to review your course progress, select my next lesson, how many people of those that logged in actually went to their next lesson after reviewing their progress, then to share it on social media?

How many people, after they finished the course, shared it on social media, how many people downloaded the batch that you offered them?

• All those things are things that you want to track in order to understand your level of engagement, of retention, how people are progressing through your app.
• There are some events that are not user generated.
• These are systems events.
• These are things that the system does by a given algorithm.
• And they can be like notifications, you know, that's something that the system blasts with a given set of conditions, or in this case like a batch.
• The user didn't necessarily procure the batch.
• The system said, we're sending her a badge because you finished the course.
• So you can also track those.
• There are certain situations in which just having that data of how many badges have been awarded may be important to you in your analysis of what's going on.
• All right, so let's look at the reverse now.
• Let's look at the user attributes.

What do we need to know in order to get a picture of how a type of user is interacting with the different events in the platform?

• Same paragraph, so we don't need to read it again, but let's go through the attributes.
• Student, right.
• As opposed to saying, I am a faculty member or I am a tutor or I'm an administrator.
• If you have different roles within your software, you know, whether it's super years wherever you want to track how those roles and interact with the application, or if you have different, not just roles but tiers, you're going to have some users that are premium users that are paying.
• Maybe you're going to have some free users that are not paying, and the behavior of those is going to be different.

So you want to make sure that in the attributes of your system, you have in that object, you have the attribute of saying free user versus paid user, right?

• So all that needs to be, you know, within the data itself, who is employed.
• So you may have a user profile.

Why do we have user profiles?

• And companies like, ask us to like, fill it out.
• Chances are they want to know this.

They want to know, are you taking this course as a student or are you employed already?

Are you unemployed?

What's your stats?

• Because that's data that's important to them in understanding, is my course valuable to people who are employed, or is it more popular with people that are unemployed or people who are starting their career and all those things they're going to grab from the user account and pair it to the events of what you're doing on the app.

Business subscription.

• Again, your software may have different tiers, levers, you know, levels of subscriptions.
• They may have a business enterprise subscription or an individual user subscription.
• You want to make sure that that attribute is captured from my laptop.
• You can actually know whether people are interacting with your app, experience your app, mostly from the web or from mobile.
• And many times those patterns of behavior change and are different for people who use web versus laptop.
• There's a joke that millennials, we don't purchase anything important through mobile.
• Have you people who are millennials, that above is like, we have our roadblock.
• If something is above $100, we purchase from a laptop.
• But that's something that other generations may not have, necessarily.
• So it would be interesting to see that other generations like, yeah, I'm going to purchase a flight from my cell phone.
• But we're like, no flight.
• I need my life.
• Make sure we don't miss anything.
• Yeah, yeah, yeah.
• Make sure.

I see the full chart of, where is this going?

Where's the layover?

• You know, you want to go to North Carolina.
• Yeah, yeah, yeah.
• To stretch, you know, too much stress.
• You know, I need to see the full.
• The full screen.
• But you'd be surprised.
• I mean, people do hopper all the time, like, oh, yeah, that's good enough.
• And of course, location.
• Location is another one that may be important to you, particularly as it relates to language, the culture, or just the growth of your regions.

If you separate your sales by region, you may want to know, you know, are you getting more engagement from certain regions than others?

• And all of that you can get from product analytics tool.
• So I know you probably work through your worksheet.
• You've seen that there is a lot of thinking that goes into it.
• So this is the kind of area where AI can help us think a little bit more, in a more detached way about our business.
• And sometimes we can.
• So what I did is, inspired by Barca, I created.
• Yeah, thank you.
• I created an app, a tool that leverages AI to give you the main metrics of your business, things that you need to pay attention to.
• So I don't know if this is your business model from critical events, user attitudes.
• Send it frequent for Ada to ask, and maybe you can compare the metrics that you wrote down yourself thinking about your business from your worksheet.
• Go to Loretta and I'll give you the signup links in a moment.
• But try it out.

Try it out and see, is there a difference?

• You know, and give me some feedback as well, because we want to make sure that we have something that is of value, which again, is very much in line with product.
• I'm going to go ahead because this is, of course, a live demo and I'm going to show you how this works.
• So, actually, Barca, I don't know how I predicted that you were going to ask me about your business, what I did.
• So I have.
• I have a little blurb here that I'm going to feed into Verata.
• Into Veronica AI.
• It's called Veronica Vereta.
• You get it.
• And as you know, it just starts always with the greeting.
• It says, hi, this is Veronica Radar Products Lab AI agent would love to help you set up your, set up meaningful product metrics for your business.
• So tell me about your product.
• And I'm going to tell Veronica that I have an app that provides courses for women speakers and helps them search for speaking opportunities.
• I'm adding functionality to your app, so that's
• I'm taking an artistic license.
• It helps them search product.
• Yeah.
• For speaking opportunities and events across different web platforms.
• So kind of the kayak for speaking.
• And then the basic paid subscription is self guided, but users can upgrade to a premium subscription to receive live instruction in personalized coaching sessions.
• Let's see what Veronica says.

- Quite a few there.

• Let's see what she said.
• She said.
• Thank you.
• All right, here are some suggesting metrics to track user engagement.
• Metrics.
• Number of courses completed by each user, time spent on the platform per session, frequency of logins, and then conversion metric.
• Conversion rate from free to paid subscribers, retention rate over time, and then event metrics.
• Number of speaking opportunities viewed by users, which right now you can't see because we search, you know, but if you kind of had that all within your platform, you'd be able to see is Renita slacking off and not applying to anything.
• And the answer is yes.
• So, yeah, the number of speaking opportunities users apply for and then success rate, insecure and speaking engagements.
• So again, if we went deeper, you could actually have even more metrics if you use an actual product analytics tool.
• But this gets you started.
• This definitely gets you started.

So, any questions so far?

• Yeah.
• So you're assuming that people understand their customer journey.
• That's right.
• That's part of.

That's part of the assignment?

• That's part of the assignment.
• Yes, it is.
• And it's kind of like a chicken and egg problem.
• You have to start with something.
• You have to start with a basic understanding of your journey in order to kind of have some basic metrics to start with.
• But once you start using a product analytics software, there's actually functionality now with AI and everything that traces the user journey for you.
• So you're able to, you have to start with something abstractly, but then you'll get the software to kind of pinpoint.

- I'm going to show you one pretty soon.

• Yeah, no, that makes perfect sense because Twitter didn't start out as a tweet platform.
• They pivoted.
• And there's so many companies that pivot it because they want something.
• Realize people are using it for something completely different.
• Yeah, there we go.
• That is the address, so take note of that.
• Or if you actually just go to beret dot productlab.com comma, you'll eventually be able to make your way to it.
• Just hit sign up.
• But you will be able to sign up and start using Veronica for free.

All right.

• Now, the market for product analytics is, has been expanding over the years.
• I mean, the number of tools that are out there is amazing.
• You know, they vary by price, which by the way, like, if you just want to get started with an app that doesn't have a lot of pressure, chances are that you're going to have a good premium model that you can go for.
• So you can probably even start for free and they'll start charging you once you reach a certain number of traffic.
• They all have sort of the same model.
• I have done this analysis.
• We don't need to cover all this, but contact me if you're considering, if you're considering making a purchase, you know, purchasing a license for product analytics, I can definitely guide you to the one that best matches your needs.
• The two market leaders are Pendo and amplitude.
• With these other ones trailing content square is an emerging one that is getting a lot of cool things going on lately.
• But really, this is a race.
• You know, it's an arms race.
• And they're adding more and more AI functionality where you can actually ask them questions like what event correlates with retention and it actually just tells you so.
• It's really amazing what's going on.
• The things that you have to keep in mind is the functionality around GTM, which stands for go to market.
• So these are like the little guides, you know, the little pop ups to rate your experience or, you know, Pendo has that, but you could integrate with Walkme or with even open source things that are out there.
• Like I mentioned, there's inbound marketing, so, so there is campaign management software and these different apps integrate to different degrees with those campaigns.
• So you have an online store, and you want to see how the journey goes from the campaign to the actual checkout.
• You can do so with some of these apps, and it integrates really well.
• A b testing, which is when you release an experience only to a few customers.
• So kind of like a pilot in a way, only to a subset of customers, and then you compare it with the rest that didn't get the new functionality, and you see if the one that you released the needle, whatever metric you were trying to solve.
• So if you had a new functionality in the hopes that people would add more items to their cart, you can see if that functionality actually moves the needle or it actually nosedives and people end up checking, like leaving the cart.
• And so before you expose the whole set of your customers to a functionality, you get a chance to test it.
• You should also think about session replays.
• I think somebody asked about session replays, but it's the ability to actually see recordings of sessions, user sessions.

So you can see if a user got stuck in Epwee, it actually shows you, oh, who's here?

• You mentioned about replays.
• So you can actually kind a little bit voyeuristic, but you can actually see what.
• More data.
• More data, exactly.
• Exactly.
• You can see what people experience and empathize and get a sense of how people are actually experiencing your product.
• And there's also heat maps, so some of them actually show you where people are looking at in the screen where they're pointing as they're navigating.

• And before I close, I just want to give you a view of amplitude, so you actually get to see what these tools look like.
• So.

So this is amplitude.

• If they're getting ready, I just got a sneak peek, and they're getting ready to release a 2.0 version that is gonna look even better than this, and it's gonna have AI.
• But this is a demo for what they use right now.
• And you can see it's just a home.
• But once you're here, you can create charts, and you can create funnel charts, for example.
• And in the funnel chart, I can.
• I think this demo is for, like, a music streaming app.
• So I can say, show me how many people signed up and went from signing up after that to playing a song.
• And from there, how many people went to.

Let's see.

• Let's see.
• Friend.
• Yes, to add friends.
• So this gives you a funnel of how people are trickling through the app.
• And you can see that a good number of people who sign up end up playing songs, but then nobody's adding friends.
• Nobody really cares about adding friends.
• So if that's something that you really think is important, maybe you make a decision to add better functionality there or improve that experience.
• Or maybe you say, you know, I don't want to invest in this because people are not really consuming this with friends.
• It's not what they want to do.
• But you have the data, you have the ability to see that you can also create experiments.
• So that is the a b testing that I was showing you.
• I'm not going to create an experiment right now.
• Let me see if they have one that's already done.
• I'll check it out later.
• But you can create also just retention graphs.
• So again, you would say new user.
• You can add user properties so you can add, see, let's say, playing a song.
• And what is the retention of users who play the song so you can play here forever.
• It is.
• It is really a very complete platform.

Let me see.

• They have what they have here.

Let's see, they have.

You can create dashboards to share with your team.

• So with dashboards, you'll have different things that you can, different charts that you can add, and it's really a complete solution.
• But there are others.

That's just kind of to give you a mental view of how this actually works, because it's not until you see it that you get a sense of, what am I getting?

All right, so with that, let's connect.

• Let's connect.
• I offer consulting in this area for anyone who is deciding to instrument their products.
• And hint you should.
• And I also offer some fractional expertise as well, in particular around product transformation.
• You can connect with me on LinkedIn.
• So that's, that QR code will not go to a malware kind of thing.
• It actually goes to my LinkedIn profile.
• And you can also follow Vereta product lab on LinkedIn as well, if you like.
• At this point, I will close and open it up to final questions.
• I don't have a question, but that was a great, phenomenal presentation.
• Thank you.
• Yeah.
• You're already talking about a defined product and a defined market to go earlier.
• R and D.

Yes.

Oh, yeah.

• Yeah.
• So when you are earlier in your journey with a product, you know, your main issue as a founder is not even this stuff.
• You're like, thinking about product market Fitzhen.
• You're thinking people will even want to, you know, fry it out.
• So at that point, you are probably going to be better served at first with more qualitative types of, types of information.
• So you're not going to have a company with a bi, you know, team helping you out.
• You're not going to have probably even marketing campaigns, paid marketing campaigns and search engine.
• You just need to know who would possibly, could possibly want my product and is it valuable to them.
• So that's where user interviews would be, are important, you know, whether you're using surveys or better yet, just sitting with people.
• And there's some really good techniques.
• I recommend always this book called the mom test for that you've probably heard about it.
• And it's called the mom test because when you are in that process that you just mentioned that you try, you're trying to find that product market fit and whether people want to use it.
• There is a, there is a tendency we all have of showing our product to people and looking for acceptance.

Like, do you like it?

• And people want to please you.
• People want to make you feel good, even if you bomb and you have something I've never tried.
• So people will say, oh, yeah, that's nice.
• Oh, that's great.
• I'm going to go home and I'll try it out.
• And never happens.
• Right.
• So what you, there's a way of engaging with people, with strangers with the market to ask questions that will give you that answer of whether people would want to use your product or not.
• Yeah.
• So I want to share about 15 years ago, like, in another life, I was a Pilates instructor slash personal trainer.
• And back then, or it was a long time ago, but, like, lululemon was not a thing.
• Like nobody.
• It wasn't sort of like the name that it is today.
• And as a trainer, I remember the Lululemon team went to, I used to work at Equinox in Miami beach.
• So obviously, like a, you know, big sort of fitness community.
• And they invited us for sushi, and we went to, like, this tiny little somebody's apartment, like super low key.
• And they had us try on the clothes.
• At the time, they only had, like, sort of like a concept store in South beach.
• And they, you know, for you being a trainer, they, they gave us each sort of like a piece of their clothing, and then they had us wear it.
• And then, you know, later that we met with them and they wanted feedback.

So, like, hey, like, what did you think about the pants?

Like, was it comfortable?

Like, did you like this?

What about the material?

Did it let you run?

• And it was so, again, like, back then, you know, a pair of pants was like a $100 for like a pair of yoga pants.
• Like, there wasn't, nobody was doing that.

Like, yeah, you would buy fitness gear, but it's not what it is today, you know?

• So I think that they did it right.
• Right.
• Like, you know, that was sort of like a really good grassroots approach.
• Obviously, fast forward.
• It's been, you know, 15 plus years, and I still buy Lululemon because it was, like, my experience was so positive.
• Right.
• Like, they gave me free sushi.

They cared about my experience, you know?

• Like, I think.
• I think it goes a long way for you to try to create sort of, like, a following.
• Right.
• And then the other thing that they did right, back then is, like, they had a program that if you are a fitness instructor, if you do, you could enter this, like, group of elite people that you get, like, 25% off your purchases.
• Right.
• So if you are.
• If you teach yoga, you teach pilates, whatever, you're gonna be wearing their clothes.
• You have a phenomenal body for the most part, if you're teaching something.
• So people are gonna wanna.
• They're gonna see you and they're gonna wanna buy that product.
• Right.
• So, you know, obviously fast forward.
• Like, Lululemon is a very successful company that made a ton of money, and it's
• Like, it made it for exercise gear to cost over $100 for, like, a pair of yoga pants, which was so.
• I mean, I think, like, to your point, I think that it's gonna be mostly sales.
• Yeah, it's gonna be mostly.
• Yeah.
• Just.
• Just one on one kind of interaction with people.
• There's a.
• There's a saying, get out of the building.
• Get out of the building.
• It's like, when you're creating a product, at some point, just hit the road and start showing it to people and see if they're willing to sign up for it.
• And that will teach you a lot about the value that you're providing.
• If you've ever been in a situation where you go into, like, a site or an app and it says, sign up for the beta, what they're trying to gauge from you, it's like, they don't have the product yet.
• Maybe they haven't even developed it, but they want to know if you're.
• If you're willing to give them their email, you're willing to sign up for the beta.
• It means that there's early demand for it.
• And if they put that sign up for the beta and only two people sign up, it's like, well, maybe this is not a good.
• A good product.
• It's not a good feature.
• There's something missing here, because if we were to spend a lot of money developing it, nobody would want to, you know, nobody want to consume it.

Yeah, no, no, that's fine.

I think that's a hard thing to figure out.

• Absolutely.
• Yeah.

How would Alex help out with this?

• Or how would you go about.

How would you go about bridging that?

Those two different concepts?

• Yeah.
• So if you have an existing customer base and what you're trying to release is another feature, something to sign up.
• In addition, some of these, particularly Pando, has a lot of functionality for you to promote, to create, to create promotional pages, to create even just fly outs for betas and sign ups.
• So.
• Or just lead people with, you know, with any kind of fly out that says NP's for just lets them.
• Lets them interact with it and say, yes, I'm interested.
• If you, again, if you don't have a gift like already, a yemenite established market of people using your product and you're very early on, your go to market strategy is going to be very much sales driven.
• You have to be in founder mode.
• In sales mode.
• Yes.

I threw that to see who would be triggered.

• So basically you're trying to add different instagrams, different steps to call cost action.
• Exactly.

To engage.

• To engage with.
• Exactly, exactly.
• This is more about it's squarely marketing and sales.
• That's what go to market activities are.
• And of course you want to track those campaigns.
• You want to track with this software how many people responded to those campaigns and do those calls to action because it will give you that data of whether it's working or not.
• But it's largely a product marketing sales function.
• Yeah, yeah.
• And Tatiana is amazing at that.

I'll send those links higher.

• Yeah.

Thank you very much.

But you do run out of use.