Exploratory Data Analysis (EDA) Cookbook — Starter Notebook

Welcome — get from curiosity to testable leads

This notebook is a practical scaffold for rapid exploratory data analysis (EDA). Its purpose is to help you move from raw tables to useful hypotheses and experiment ideas you can hand off to product, research, or operations. Keep the scope tight: look for surprising patterns, plausible causal leads, and clear next-step experiments rather than exhaustive modeling at this stage.

How to use this notebook

• Run cells in order.
• Keep observations short and record candidate hypotheses in the hypothesis block near the end.
• Capture any data quality issues as immediate actions (missingness, duplicates, evident errors).

Starter cell-by-cell scaffold

1) Setup

Import standard libraries and set display options.

# Python imports (run in a Jupyter notebook)
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
pd.options.display.max_columns = 120

2) Data loading & quick profiling

Load a sample (or the full dataset if small). Show shape, dtypes, and a quick summary.

# Load data
df = pd.read_csv('your_data.csv')
# Quick profile
print(df.shape)
print(df.dtypes)
df.head()

3) Missingness and basic quality checks

Measure missing values, duplicates, and obvious out-of-range values.

# Missingness
missing = df.isna().mean().sort_values(ascending=False)
print(missing[missing>0])
# Duplicates
print('duplicates:', df.duplicated().sum())

4) Distributions and visual checks

Plot distributions for numeric variables and counts for categoricals. Look for skew, multimodality, and unexpected zeros.

# Example distribution plot
num_cols = df.select_dtypes(include=['number']).columns[:6]
for c in num_cols:
    sns.histplot(df[c].dropna(), kde=True)
    plt.title(c)
    plt.show()

5) Cohorts and quick segmentation

Compare distributions across meaningful groups (time buckets, user cohorts, geography, plan type).

# Example cohort comparison
cohort = 'user_type'
metric = 'outcome_metric'
pd.concat([df.groupby(cohort)[metric].describe(),
           df.groupby(cohort)[metric].median().rename('median')], axis=1)

6) Correlation matrix and pairwise checks

Use a correlation heatmap for numeric features to surface strong linear relationships and possible collinearity.

# Correlation
corr = df.select_dtypes(include=['number']).corr()
sns.heatmap(corr, cmap='vlag', center=0)
plt.title('Correlation matrix')
plt.show()

7) Simple anomaly detection

Flag extreme values with a z-score or IQR rule. Note whether anomalies are data errors or meaningful rare events.

from scipy import stats
z = np.abs(stats.zscore(df[num_cols].dropna()))
anom = (z > 3).any(axis=1)
print('anomalies:', anom.sum())

8) Quick feature transforms & checks

Try simple transformations (log, binning, rates) when distributions are skewed or when ratios make intuitive sense.

# Example transform: log
df['log_value'] = np.log1p(df['value'])
# Example rate: per-session
df['rate'] = df['successes'] / df['attempts']

9) Rapid cohort-based experiments to try

Sketch simple A/B-style or before/after comparisons you could run based on observed differences. Keep designs minimal and measurable.

Hypothesis capture template (use and save)

Recording candidate hypotheses prevents useful leads from being lost. Use this simple template for each idea:

1. Observation (one sentence): e.g., "Conversion drops 40% for users joining after 8pm."
2. Hypothesis (one sentence): e.g., "Users who sign up after 8pm encounter a slower onboarding job that increases drop-off."
3. Evidence from EDA (bullets): List the specific charts or stats that support the observation.
4. Immediate data quality checks to run: e.g., verify timezone, check sample size, detect bots.
5. Suggested experiment or next analytical step: e.g., instrument timing of each onboarding step, run a targeted UX test, or holdout group A/B test.
6. Priority & owner: e.g., High — Product Analyst

# Example hypothesis record (as a dict you can append to a list or DataFrame)
hyp = {
  'observation': 'Conversion drops after 8pm',
  'hypothesis': 'Onboarding step X times out for late users',
  'evidence': ['conversion_by_hour plot', 'session_length_by_hour median'],
  'checks': ['verify timezone mapping', 'remove suspicious sessions'],
  'experiment': 'A/B test simplified onboarding flow for late-hour users',
  'priority': 'High',
  'owner': 'ana'
}

Documentation & sharing tips

When you share findings with stakeholders, provide a one-slide summary with: the key observation, a one-line hypothesis, the evidence (one chart + a short table), recommended next step, and estimated effort & owner. Keep the narrative focused on what can be measured and changed.

Quick EDA checklist

• Confirm dataset boundaries, time range, and units
• Summarize missingness and duplicates
• Plot distributions and detect skew/multimodality
• Compare cohorts that matter to the business
• Surface strong correlations and obvious confounders
• Capture 3–5 candidate hypotheses with next actions

Next steps and experiments

Prioritize hypotheses that are high-impact, low-effort, and quickly testable. For each prioritized item, define a minimal measurement plan (metric, baseline, success threshold, length, and ownership).

Where this fits in the Discovery & Innovation Hub

This starter notebook is designed to turn data curiosity into actionable experiments. Use it as a repeatable ritual in discovery sprints, research huddles, or regular analytics reviews.

Notes & safe practices

• Do not share identifiable personal data without appropriate approvals.
• Record sample sizes and confidence when proposing experiments.
• Treat anomalies as either data issues to fix or rare-but-valuable signals to study further.