Exploratory Data Analysis Workflow
A repeatable EDA routine in pandas: check shape and gaps, summarise, look at skew, compare groups, test correlations. Shown on a 500-row dataset where the mean misleads and a log transform fixes it.
Exploratory data analysis (EDA) is a short, repeatable routine run before modelling: check size and types, missing values, summary statistics, distributions, group comparisons and relationships between columns. Interviewers expect the key interpretation: when the mean and median differ a lot, the column is skewed, so report the median and consider a log transform.
Exploratory data analysis is a short, repeatable routine you run before any modelling: size and types, missing data, summary statistics, distributions, group comparisons and relationships. The goal is to find problems and form questions. A typical discovery is that the mean and median differ a lot, which signals skew and changes how you summarise the column.
- In an EDA example of 500 users, monthly spend had a mean of 53.6 but a median of 35.5, a sign of a right-skewed column.
- A log transform reduced the skewness of the spend column in the EDA example from 2.07, strongly right-skewed, to 0.09, nearly symmetric.
- Grouping by plan in the EDA example showed pro users spend about 3.6 times as much as free users, a gap that overall averages hide.
- A correlation near zero, such as 0.036 between age and spend, means no linear relationship, not no relationship, so the data should also be plotted.
- A high correlation found during exploratory data analysis does not show causation, because another variable may drive both or the direction may be reversed.
01What EDA is for
EDA is the stage where you look at the data before trusting it. It answers practical questions. How big is it? What is missing? What does each column look like? Do groups differ? Are any columns related? You are hunting for surprises: wrong ranges, impossible values, skew, and groups that behave differently.
It is not about producing many charts. A short fixed routine run every time catches more than a long exploration done ad hoc. The routine below uses a made-up dataset of 500 users with age, plan type and monthly spend.
02The dataset
import numpy as np, pandas as pd
rng = np.random.default_rng(7)
n = 500
plan = rng.choice(['free', 'pro'], n, p=[0.7, 0.3])
age = rng.integers(18, 60, n)
spend = np.where(plan == 'pro',
rng.lognormal(4.6, 0.5, n),
rng.lognormal(3.2, 0.6, n)).round(2)
df = pd.DataFrame({'age': age, 'plan': plan, 'spend': spend})
df.loc[rng.choice(n, 25, replace=False), 'age'] = np.nanThe seed is fixed, so you can reproduce every number in this article. The spend values follow a lognormal shape, which is how money, time and counts often behave in real life: many small values and a few very large ones.
03Step 1: shape and gaps
print(df.shape) # (500, 3)
print(df.isna().sum().to_dict()) # {'age': 25, 'plan': 0, 'spend': 0}Five hundred rows, three columns, and 25 missing ages, which is 5% of the column. Ask why. Missing at random is easy to live with. Missing because of a pattern (say only free-plan users skip age) would bias anything that uses age. Also check dtypes with df.dtypes to catch numbers stored as text.
04Step 2: summarise and spot skew
print(df.describe().round(1).loc[['count', 'mean', '50%', 'max']])The output shows 475 ages counted (the missing ones are skipped), mean age 38.2 against median 38.0, and for spend a mean of 53.6, a median of 35.5 and a maximum of 339.1.
Age looks symmetric because mean and median agree. Spend does not. The mean is much higher than the median because a few large spenders pull it up. Skewness confirms it:
print(round(df.spend.skew(), 2)) # 2.07
print(round(np.log(df.spend).skew(), 2)) # 0.09A skew of 2.07 is strongly right-skewed. Taking the log brings it to 0.09, nearly symmetric. For a skewed column, report the median, and consider a log transform before modelling or plotting. Quoting "average spend is 53.6" would describe almost nobody.
05Step 3: compare groups
print(df.groupby('plan').spend.agg(['count', 'mean', 'median']).round(1))The table shows 343 free users with mean spend 29.6 and median 24.2, and 157 pro users with mean 106.2 and median 94.2. The two groups differ a lot, but mixing them is not what causes the skew: each group is itself right-skewed (about 2.1 for free and 1.8 for pro), so the log transform is still worth doing within groups. Pro users spend about 3.6 times as much as free users. Overall averages hide this. Grouping by every sensible category is often the single most useful EDA step.
06Step 4: look at relationships
print(df[['age', 'spend']].corr().round(3).loc['age', 'spend']) # 0.036A correlation of 0.036 between age and spend is essentially zero. In this data age tells you nothing about spend. Correlation ranges from -1 to 1, and it only captures straight-line relationships. A U-shaped relationship can give a correlation near zero, so plot a scatter chart as well. And a correlation, even a strong one, does not show that one variable causes the other.
07What to write down
End the routine with notes you could hand to someone else: 25 ages missing (5%), spend is right-skewed so report medians and use a log transform, two plan groups behave very differently and should be modelled or reported separately, age is unrelated to spend. These notes drive the next steps: cleaning, feature choices, and which questions are worth testing.
Charts help once you know what to look for. A histogram for each numeric column, a box plot per group, and a scatter plot for pairs of columns cover most needs.
08Common interview questions
Q: What are your first steps on a new dataset?
A: Shape, dtypes, null counts, a few rows, then describe(). After that, distributions and group comparisons.
Q: Mean and median differ a lot. What does that tell you?
A: The distribution is skewed or has outliers. The median is the better summary of a typical value.
Q: How do you handle a right-skewed column?
A: Report medians, consider a log transform, and check whether it is a mix of different groups.
Q: Does zero correlation mean no relationship?
A: No. It means no linear relationship. Plot the data.
Q: Does high correlation mean causation?
A: No. Another variable may drive both, or the direction may be reversed.
Q: How do you spot outliers?
A: Box plots or the 1.5 times IQR rule, then investigate whether they are errors or real extreme values before removing anything.