Data in a Business

The Essential Questions

What is the business about?

Understand What is its core mission, what value it offers to users.

Also, who are the competition, how are they doing, and how is this business better than the competition ?

What data is required?

Define the necessary infrastructure to allow measure the business:

• Choosing and defining the metrics that allow to measure the business (the KPIs).
• Create / Define the frameworks and standards, for:
  • Measuring (telemetry guidelines): What are the events we need capture? and how? Document it.
  • Collecting and processing data: What aggregations need to be done? Document it.
  • Testing hypotheses of the impact of new activities (AB testing, pre/post launch analysis)
  • Standard data definitions and methods (Churn, Survival, ARPU, Stickiness)
• Reporting Infrastructure.

How is the business doing?

Answer this, By understanding the top-level reality of the business. This is very tied to the previous step, but also involves:

• Know what are the business goals. Be part of the discussion on defining the business goals if needed. These are key, this is how we evaluate the business health, each business activity and even each individual contribution.
• Setting up, analyzing, digesting and sharing frequently the top metrics.

The Essential Questions, part 2: day-to-day activities

Once is clear what is the business about, how is it doing and where it should be going there’s the need to accompany the business on its daily activities.

What should the business be doing?

And equally important, what it should not be doing. Includes:

• Understand what drives the current KPIs volumes? How can we get more of that ? What opportunities are we missing? What kind of activities do not drive KPIs further?
• Advocate that we should be working on the right things, aka the activities that potentially contribute the most to the set goals. And avoid wasting resources on the activities that don’t contribute as much.
  • Guide teams on creating the hypothesis of the ROI for each planned activity (guessing the ROI of an hypothesis is a hard thing, but it should leverage the learnings from the previous activities).
• To be able to fairly and scientifically evaluate the activities, there’s the need to be a constant data advocate and educate business on measuring and being data driven. For example argue that every activity should include an AB test.

Did the activity help?

On each activity answer: did it (the activity) help achieve the business goals? And by how much?

Is a quantification exercise.

• Here often the activity owners ask from the analytics team data the results:
  • The resulting analysis should be tackled in a structured and consistent way, across every activity (as defined in the “What data is required?”), this will allow to compare activities directly.
  • Activity owners might not be asking for the right metrics that best measure the business impact. So advocate instead for the right metrics to use.
  • Naturally, activity owners are highly biased about their activity success and might not acknowledge bad results. But data analysis ethics require the truth.
• “And by how much?”, is essentially the contribution of the individual activity towards the end goal, often hard to measure, but key even if just an estimation.
• When activity is not helping (towards the goal) then data analysts are responsible to raise it with adequate communication, to allow for timely course correction.

What did we learn?

Once we have the numbers on by how much did the activity help, create an opinion of what worked what didn’t and why? and create hypothesis for the next activities.

• When activity is unsuccessful ask why, was the problem in the hypothesis or in the execution?
• Build and curate the history of what worked and what didn’t as it helps crystallize the mental model for ROI estimations of new activities in future.
• Communicate the learning to rest of business organization, so that all organization becomes smarter.
  • Communicate using best practices: tell stories instead of dumping numbers, be straight to the point (see minto pyramid).