NumPy & Pandas: The Data Workhorses

Nearly every AI and data science pipeline in Python passes through two libraries: NumPy, which gives us fast arrays of numbers, and Pandas, which gives us labeled tables we can slice, group, and reshape. This chapter teaches the core skills you will reach for every single day: building arrays, manipulating tables, cleaning messy data, and loading data from common file formats.

NumPy Arrays: The Foundation of Numerical Computing

A NumPy array is a dense grid of values of the same type. Unlike a Python list, operations on an array run inside compiled C code, so loops over millions of numbers are dramatically faster. You can create arrays from lists, from ranges, or from special functions.

Creating Arrays

import numpy as np

a = np.array([1, 2, 3, 4])          # from a list
b = np.arange(0, 10, 2)             # 0, 2, 4, 6, 8
c = np.linspace(0, 1, 5)            # 5 evenly spaced values from 0 to 1
d = np.zeros((2, 3))                # 2x3 matrix of zeros
e = np.ones((2, 2))                 # 2x2 matrix of ones
f = np.eye(3)                       # 3x3 identity matrix
g = np.random.randint(0, 10, size=(3, 3))  # random integers

Shapes, Indexing, and Slicing

Every array reports its shape (a tuple of sizes per dimension) and its ndim (number of dimensions). Indexing a 2D array uses two coordinates, arr[0] selects the first row, and slicing keeps ranges using start:stop:step.

m = np.array([[1, 2, 3],
              [4, 5, 6]])

print(m.shape)       # (2, 3)
print(m[0])          # first row: [1 2 3]
print(m[1, 2])       # element at row 1, col 2: 6
print(m[:, 1])       # second column: [2 5]
print(m[0:2, 0:2])   # top-left 2x2 block

Vectorized Operations and Broadcasting

Instead of looping, you apply operations to the whole array at once. Broadcasting lets NumPy combine arrays of different shapes by stretching the smaller one across the larger one automatically.

v = np.array([1, 2, 3])
print(v * 2)          # element-wise multiply: [2 4 6]
print(v + np.array([10, 20, 30]))
print(np.exp(v), np.sqrt(v))

w = np.array([[1, 2, 3],
              [4, 5, 6]])
print(w + v)          # broadcast v across both rows

Pandas Series and DataFrame

Pandas wraps NumPy arrays with labels. A Series is a one-dimensional labeled column; a DataFrame is a two-dimensional table of Series sharing an index.

Creating and Inspecting

import pandas as pd

s = pd.Series([10, 20, 30], index=["a", "b", "c"])

df = pd.DataFrame({
    "name": ["Aya", "Ben", "Cam", "Dee"],
    "hours": [38, 42, 35, 40],
    "pay":   [22.0, 25.5, 21.0, 24.0]
})
print(df.head(2))
print(df.shape)      # (4, 3)
print(df.info())
print(df.describe())

Selecting Data

Use df["col"] or df.col for a column, df.loc for label-based rows, and df.iloc for position-based rows.

print(df["hours"])          # a Series
print(df[["name", "pay"]])  # a sub-DataFrame
print(df.loc[2])            # row with label 2
print(df.iloc[1:3])         # rows by position
print(df[df["pay"] > 23])   # boolean filtering

Data Manipulation with Pandas and NumPy

Filtering and Grouping

Boolean masks select rows that meet a condition. groupby splits the table, applies a function per group, and combines the results — the classic split-apply-combine pattern.

staff = pd.DataFrame({
    "dept": ["sales", "eng", "sales", "eng", "eng"],
    "salary": [55, 70, 48, 80, 65]
})

print(staff[staff["salary"] >= 60])
print(staff.groupby("dept")["salary"].mean())
print(staff.groupby("dept")["salary"].agg(["mean", "max", "count"]))

Aggregating and Merging

a = pd.DataFrame({"id": [1, 2, 3], "score": [88, 91, 79]})
b = pd.DataFrame({"id": [1, 2, 3], "grade": ["B", "A", "C"]})

merged = a.merge(b, on="id")
print(merged)

print(a["score"].sum(), a["score"].mean(), a["score"].std())

Handling Missing Data

messy = pd.DataFrame({"x": [1, None, 3, None],
                       "y": [4, 5, None, 7]})
print(messy.isna())
print(messy.dropna())          # remove rows with any NaN
print(messy.fillna(0))         # replace NaN with 0
print(messy.fillna(messy.mean()))  # fill with column mean

Reading Data from Many Sources

Pandas reads tabular data from almost anywhere with one-line readers.

import sqlite3

csv_df   = pd.read_csv("sales.csv")
excel_df = pd.read_excel("sales.xlsx", sheet_name="Q1")
json_df  = pd.read_json("sales.json")

conn = sqlite3.connect("shop.db")
sql_df = pd.read_sql("SELECT * FROM orders", conn)
conn.close()

html_df = pd.read_html("https://example.com/table.html")[0]
print(csv_df.shape, excel_df.shape, json_df.shape, sql_df.shape)

For a URL or a CSV hosted online you can pass the URL directly to pd.read_csv. Writing data back is just as easy: df.to_csv("out.csv"), df.to_excel("out.xlsx"), or df.to_sql("table", conn).

Assignments and Solutions

Exercise A: Array Arithmetic

1. Build a 4x4 array of random integers from 0 to 100.
2. Print its shape, the mean of each column, and the maximum of each row.
3. Normalize it so every column has mean 0 and standard deviation 1 (use broadcasting).

import numpy as np

arr = np.random.randint(0, 100, size=(4, 4))
print(arr.shape)
print(arr.mean(axis=0))
print(arr.max(axis=1))

norm = (arr - arr.mean(axis=0)) / arr.std(axis=0)
print(norm.mean(axis=0), norm.std(axis=0))

Exercise B: Employee Table

1. Create a DataFrame of 6 employees with name, department, and salary.
2. Filter to employees earning above the mean salary.
3. Group by department and report the average salary per department.

import pandas as pd

employees = pd.DataFrame({
    "name": ["Ana", "Bob", "Cid", "Dee", "Eli", "Fay"],
    "dept": ["sales", "eng", "sales", "eng", "eng", "sales"],
    "salary": [52, 68, 47, 74, 70, 55]
})

print(employees[employees["salary"] > employees["salary"].mean()])
print(employees.groupby("dept")["salary"].mean())

Exercise C: Loading Real Data

Download any CSV you have (or the classic "titanic.csv"), load it with pd.read_csv, then answer: how many rows, how many missing values per column, and the mean age of passengers whose age is known.