Data Analysis Techniques

Techniques to transform and interpret data in a useful way.

Quantify Impact

I want to, Understand how big is a change.

(i.e. calculate the relative change)


Lets say a bank raised a rate from 3% to 5%, difference is 2% but is ambiguous to say “the rate was increased by 2%”. It looks like a very small increase, from a +2 increase from an original of 3.

Relative calculation general formula: (new - reference) / abs(reference)

For negative numbers needs that denominator to have abs(), to force negative number into a positive one.

I want to, Understand how much is one number bigger than other

You looking at a table of numbers, and comparing 2 numbers, wondering by how much one is bigger than the other ?

I want to, quickly size up something to see how important it might be

The art of guess estimating can be very useful to quickly run numbers on what is going to happen, or impact, etc…

Data Quality: confidence interval

I want to

Give out some certainty on the average number i am giving out, by adding a range to where is this number is expected to fall into, most of the time.


When estimating a parameter like a mean from random samples, confidence interval helps quantify uncertainty, giving an interval in which we can expect the value to be most of the time.

Prepared Excel: confidence_interval.xlsm


def confidence_interval (s):
    from scipy import stats
    import scipy as sp
    import numpy as np
    import math
    n, min_max, mean, var, skew, kurt = stats.describe(np.array(s))
    #The location (loc) keyword specifies the mean.
    #The scale (scale) keyword specifies the standard deviation.
    # We will assume a normal distribution
    R = stats.norm.interval(0.975,loc=mean,scale=std/math.sqrt(len(s)))
    return R

Data Quality: hypothesis testing

I want to, Make sure the results i got are statistically significant (and not due to chance).

Test of Proportion

For comparing percentages metrics like Conversion Rate, Click-through Rate etc, use a proportion test.

Finding Sample Size

To find what sample size is needed for a test, use:

In R stats:

To go from a 1.5% to a 2.5%, thus a 1% increase rate, at a 5% significance level and 80% power of test:

> power.prop.test(p1=0.015, p2=0.025, sig.level=0.05, power=0.8)
 Two-sample comparison of proportions power calculation 
          n = 3075.582
         p1 = 0.015
         p2 = 0.025
  sig.level = 0.05
      power = 0.8
alternative = two.sided

We need at least 3076 samples in each group.

In Python:

To go from a 1.5% to a 2.5%, thus a 1% increase rate, at a 5% significance level and 80% power of test:

> import statsmodels.stats.api as sms
> es = sms.proportion_effectsize(0.015, 0.025)
> sms.NormalIndPower().solve_power(es, power=0.8, alpha=0.05, ratio=1)

We need at least 3029 samples in each group. (not sure why they not exactly same as in R, might use slightly different calculation approximations)

Test if statistically significant

Chi-Squared Test

In R:

For example, to test two campaigns each with a 1000 displays, 32 and 54 conversions:

> prop.test(c(32, 54), c(1000,1000))
    2-sample test for equality of proportions with continuity correction
data:  c(32, 54) out of c(1000, 1000)
X-squared = 5.3583, df = 1, p-value = 0.02062
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.040754721 -0.003245279
sample estimates:
prop 1 prop 2 
 0.032  0.054 

The p-value is less than 0.05, so we can reject the hypothesis that conversion rates are equal and assume the second group has a higher rate.

Prepared Excel: hypothesis_testing_proportion.xlsm

Prepared Excel: hypothesis testing proportion_Chi-squared.xlsm


Test of Means

Comparing non-fractional values that follow a normal distribution (e.g. Average Order Value, Time Spent on Page etc.) is done with a Two-sample unpaired t-test.

Welch Two Sample t-test Two-Sample T-Test

Find Sample size

In R:

The recommended values for h are: 0.2 for small effects, 0.5 for medium and 0.8 for big effects. Sample call for computing the test sample size for a campaign with an estimated medium effect and 10000 customers in the control group:

> pwr.t2n.test(n1=10000, d=0.5, sig.level=0.05, power=0.90)
 t test power calculation 
         n1 = 10000
         n2 = 42.21519
          d = 0.5
  sig.level = 0.05
      power = 0.9
alternative = two.sided

Only around 43 customers are needed in the test group.

Test for significance

In R:

> t.test(group1, group2)
    Welch Two Sample t-test
data:  group1 and group2
t = -1.5631, df = 99.423, p-value = 0.1212
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -7.0219603  0.8334419
sample estimates:
mean of x mean of y 
 125.0789  128.1731 

Our p-value is greater than 5% so we cannot reject the hypothesis that the values are different. More tests or a larger sample size may be necessary.


Data Quality: Get an idea on the dispersion of the data

I want to

Get an idea on the dispersion of the data, what values are more common, what values are outliers.

See chart type Histogram and Boxplot from:


This is a great way to test out models / hypothesis / behavior by generating random and (on purpose) biased data to test, evaluate formulas, algorithms, theories. And this is something that computers are great at, and that a few years ago was not possible to do.

This also allows for optimization, plot the convergence over time, see if possible to make the algol converge faster, plot the simulation progression over time.

Specifically, can be useful for:

Sometimes in paper is enough. Paper is costly (especially in time) to run many simulations. To get for example the distribution of the solutions. For more than a few executions, use a computer (Excel, Python, Ruby, R, etc…).


Python Code:

Analysis: Reveal impacting features in a specific goal.

Lets say i am looking at a web site new user registration flow, and i want to find areas that can be improved, for example i might find that users in mobile devices in Portugal are having a hard time to register and that consequently might lead to the finding that a page in that flow is incorrectly translated.

For then we need to break down the data by the dimensions available and look at them combined, to find strong signals.

This is also useful to pin down performance problems or even outliers.

How to

When doing breakdowns:


Create a classification, so that we enrich the data set with new useful features to analyze the rest of the data with. Also to use as modeling parameters.

This could include classifying for example: user behaviors, user value, etc…

Clustering and Similarity matching

Find what characteristics users that churned have (a churn model), then look for similar users cohorts from your user base that have similar characteristics and thus potentially at the risk of churning, create hypotheses on activities to help reduce churning.


Regression as an estimation / prediction technique

What results are we getting in 3 months if this trend continues? When are we getting to result XYZ?

Regression as a causal modeling technique (aka correlation finder)

Regression can be used in explaining how one or many (using multivariate regression) factors affect an outcome of interest.

For example, what contributes more to a movie final revenue is it the movie budget size or the viewers rating? Likely both things influence it… For example a movie with a very low budget can’t compete with a huge budget movie in general, but a big budget movie can also end up not being liked by viewers. So which of those 2 variables contribute the most to the final movie revenue ?

Here we can build a regression model that exactly quantifies how much these contribute to the revenue, given a set of historical data.

Data Cleaning: Fill in missing values

Python and Pandas:

You may use reindex() method of DataFrame:

x = pd.date_range('2013-01-01','2013-01-07',freq='D')
y = range(7)

To add missing days (like holidays) you need to reindex it:

x2= pd.date_range('2013-01-01','2013-01-07',freq='4H')

Then you may fill the gaps in values using interpolate() method of Series (different interpolation methods are available):




Optimization Process

Baseline - where you are now?

Formulate hypothesis on what to change to get to the goal

Test (A/B test)


AB testing

Because correlation does not (always) imply causation, this is the way to test an hypothesis created from an observed data correlation. (output from a previous data science method).

This also, for example, the approach that the medical community uses to tests whether a new medication is effective or not. And is a fairly common solid scientific method.

Hypothesize how to make to make it better, run an experiment to validate it. Assist product changes by ab testing it, and measuring its consequences…

The setup of an experiment is to compare 2 groups where the variable we want to test is different for each group, while keeping all the other variables (possible confounding factors) the exact same. (see also

We essentially need 4 data points, these 2 metrics for each group: - Test group size (at the start of experiment) - Success group size (at the end of experiment)

We need to assure the experiment runs for enough amount of time so that we can be sure of the results (statistical significance). Apply a proportion test for example (

references: sample sizes

Intelligent Agents Model

Amazingly, Optimization, AB & Multivariate Testing, Behavioral Targeting, Attribution, Predictive Analytics, … can all be recast as components of a simple, yet powerful framework borrowed from the field of Artificial Intelligence, the intelligent agent.

Artificial intelligence Optimization Process

The goals are what the agent wants to achieve, what it is striving to do.

When the agent achieves a goal, it gets a reward based on the value of the goal.

Given that the agent has a set of goals and allowable actions, the agents task is to learn what actions to take given its observations of the environment – so what it ‘sees’, ‘hears’, ‘feels’, etc… Assuming the agent is trying to maximize the total value of its goals over time, then it needs to select the action that maximizes this value, based on its observations.

So how does the agent determine how to act based on what it observes? The agent accomplishes this by taking the following basic steps:

  1. Observe the environment to determine its current situation. You can think of this as data collection.

  2. Refer to its internal model of the environment to select an action from the collection of allowable actions.

  3. Take an action.

  4. Observe of the environment again to determine its new situation. So, another round of data collection.

  5. Evaluate the ‘goodness’ of its new situation – did it reach a goal, if not, does it seem closer or further away from reaching a goal then before it took the past action.

  6. Update its internal model on how taking that action ‘moved’ it in the environment and if it helped it get or get closer to a goal. This is the learning step.

By repeating this process, the agent’s internal model of how the environment responses to each action continuously improves and better approximates each actions actual impact.



Tool is just the means to an end choose the best tool for the job at hand.

Excel is the notepad for data, great for a quick look, quick chart, prototype, final UI for a ad-hoc analysis, mostly used for EDA phase. Handles well small dashboards and reports, and some data manipulation, but from a certain size/complexity starts getting too time consuming and better to move on to next tool.

Python, is great on data ETL, data automation, handles building a data product with increasing complexity and size (as opposed to Excel) but also not too bad for EDA, especially with IPython. Fully automated dashboard that creates and html report, sends email to people, runs in a server by itself, with very low maintenance. Allows reproducible data analysis. Might miss all the absolute latest most complex machine learning algorithms.

R is the most complete statistical and machine learning algorithm collection available (?). Better for running algorithms that do a specific thing, not to good to build complex automations and logic on top (compared to python).

comments powered by Disqus