Linear Algebra Cheat Sheet — Vectors, Matrices, Eigenvalues, Dot Product | Dataplexa

Linear Algebra

vectors · matrices · dot product · determinants · eigenvalues & eigenvectors · transformations

Sheet 3 of 6 Stats & Math Intermediate Printable

Vectors

must know first

Vector Notation

v = [ v₁ v₂ v_n ] ∈ ℝⁿ

A vector is an ordered list of numbers. Can represent direction + magnitude in n-dimensional space.

Vector Magnitude (Norm)

‖v‖ = √(v₁² + v₂² + ··· + v_n²)

Euclidean norm (L2). Unit vector: û = v / ‖v‖

Dot Product

a · b = Σ aᵢbᵢ = ‖a‖‖b‖cosθ

Result is a scalar.
a · b = 0 → vectors are orthogonal (perpendicular).
Used in projections, similarity, neural nets.

Cross Product (3D only)

a × b = det(i j k
a₁ a₂ a₃
b₁ b₂ b₃)

Result is a vector perpendicular to both a and b. Magnitude = ‖a‖‖b‖sinθ.

Vector Operations — Example

a = [1,2,3]   b = [4,5,6]

Addition:
a+b = [5,7,9]

Scalar mult (×2):
2*a = [2,4,6]

Dot product:
a·b = 1×4+2×5+3×6 = 32

Magnitude:
‖a‖ = √(1+4+9) = √14 ≈ 3.74

Unit vector:
û = a/‖a‖ ≈ [0.27,0.53,0.80]

Cosine similarity = dot product of unit vectors = cos θ. Range −1 to 1. Used everywhere in ML: recommendation systems, NLP embeddings, document similarity.

Matrices

m × n · notation · types

Matrix Notation

A = [a_ij] ∈ ℝ^m×n
m rows × n columns · a_ij = element row i, col j

Matrix Type	Property
Square	m = n (rows = cols)
Identity (I)	1s on diagonal, 0s elsewhere. A·I = A
Zero (0)	All elements = 0
Diagonal	Non-zero only on main diagonal
Symmetric	Aᵀ = A (a_ij = a_ji)
Orthogonal	AᵀA = I (Aᵀ = A⁻¹)
Positive Definite	All eigenvalues > 0. xᵀAx > 0 ∀ x ≠ 0

Matrix Operations

add · multiply · transpose · inverse

Matrix Multiplication

(AB)_ij = Σ_k a_ik b_kj
A is m×k, B is k×n → AB is m×n

Not commutative: AB ≠ BA in general.

2×2 Example

A = [[1,2],[3,4]]
B = [[5,6],[7,8]]

AB[0][0] = 1×5+2×7 = 19
AB[0][1] = 1×6+2×8 = 22
AB     = [[19,22],[43,50]]

Transpose — flip rows↔cols:
Aᵀ     = [[1,3],[2,4]]

Inverse (2×2 formula):
A⁻¹    = (1/det(A)) × [[4,-2],[-3,1]]

Determinant & Inverse

det · invertibility · singular

2×2 Determinant

det(A) = |a b|
|c d| = ad − bc

det(A) = 0 → matrix is singular (not invertible, columns are linearly dependent).

3×3 Determinant (Cofactor)

det(A) = a(detM₁₁) − b(detM₁₂) + c(detM₁₃)

Expand along first row. M_ij = minor matrix after removing row i, col j.

Matrix Inverse

A⁻¹ = 1 det(A) adj(A)

Only exists when det(A) ≠ 0. A·A⁻¹ = A⁻¹·A = I.

Property	Rule
(AB)⁻¹	B⁻¹A⁻¹
(Aᵀ)⁻¹	(A⁻¹)ᵀ
det(AB)	det(A)·det(B)
det(Aᵀ)	det(A)
det(A⁻¹)	1/det(A)

Solving Ax = b

If A is invertible:
x = A⁻¹b

Example: A = [[2,1],[5,3]], b = [8,13]
det(A) = 2×3-1×5 = 1
A⁻¹    = [[3,-1],[-5,2]]
x      = [[3,-1],[-5,2]] × [8,13]
       = [11,-14]

In practice: use Gaussian elimination,
LU decomposition, or numpy.linalg.solve

Eigenvalues & Eigenvectors

characteristic equation · PCA

Definition

Av = λv

λ = eigenvalue (scalar) · v = eigenvector (non-zero). Matrix only scales the eigenvector — direction unchanged.

Finding Eigenvalues — Characteristic Equation

det(A − λI) = 0

Solve this polynomial equation for λ. For each λ, solve (A − λI)v = 0 for eigenvector v.

2×2 Example

A = [[4,1],[2,3]]

det(A−λI) = (4−λ)(3−λ)−2 = 0
→ λ²−7λ+10 = 0
→ λ₁ = 5, λ₂ = 2

For λ=5: (A−5I)v = 0 → v = [1,1]
For λ=2: (A−2I)v = 0 → v = [1,−2]

PCA connection: Principal Component Analysis finds eigenvectors of the covariance matrix. Eigenvalues tell you how much variance each component explains.

Rank, Span & Vector Spaces

rank · null space · linear independence

Concept	Meaning
Span	All linear combinations of a set of vectors
Linear independence	No vector is a combination of the others
Basis	Linearly independent set that spans a space
Rank(A)	Number of linearly independent rows (= cols)
Null space	All x such that Ax = 0 (kernel)
Column space	Span of columns of A (range/image)
Rank–Nullity	rank(A) + nullity(A) = n (cols)

Full Rank Condition

rank(A) = min(m, n)
Square matrix: full rank ↔ invertible ↔ det ≠ 0

Matrix Decompositions

SVD · LU · QR · Cholesky

SVD — Singular Value Decomposition

A = U Σ V^T

U = left singular vectors (m×m orthogonal)
Σ = diagonal of singular values σ₁ ≥ σ₂ ≥ ··· ≥ 0
Vᵀ = right singular vectors (n×n orthogonal)
Works for any m×n matrix. Foundation of PCA, LSA, recommender systems.

LU Decomposition

A = L U

L = lower triangular matrix
U = upper triangular matrix
Used to efficiently solve Ax = b: solve Ly = b, then Ux = y.

QR Decomposition

A = Q R

Q = orthogonal matrix · R = upper triangular. Used in least squares, eigenvalue algorithms.

Decomposition	Use case
SVD	PCA, LSA, pseudo-inverse, low-rank approx
LU	Solving linear systems, determinants
QR	Least squares, eigenvalue problems
Cholesky	Symmetric PD matrices, sampling
Eigendecomp	Square matrices — A = QΛQ⁻¹

Low-rank approximation: Keep top k singular values in SVD to compress data. Used in image compression and collaborative filtering.

Linear Algebra in ML

applications · numpy

NumPy — Common Operations

import numpy as np

A = np.array([[1,2],[3,4]])
b = np.array([1,2])

A.T                    # transpose
A @ A                   # matrix multiply
np.linalg.inv(A)       # inverse
np.linalg.det(A)       # determinant
np.linalg.solve(A,b)  # Ax = b
np.linalg.eig(A)       # eigenvalues,vecs
np.linalg.svd(A)       # U, Σ, Vᵀ
np.linalg.norm(b)      # ‖b‖
np.dot(b,b)             # dot product

ML concept	Linear Algebra behind it
Linear regression	β = (XᵀX)⁻¹Xᵀy — normal equations
PCA	Eigenvectors of covariance matrix
Neural net forward	z = Wx + b (matrix-vector product)
Cosine similarity	a·b / (‖a‖‖b‖)
Recommender system	SVD / matrix factorization

Norms & Distances

L1 · L2 · Frobenius

Norm	Formula	Use
L0	count of non-zeros	Sparsity
L1 (Manhattan)	Σ\|vᵢ\|	Lasso regularization
L2 (Euclidean)	√(Σvᵢ²)	Ridge, distances, norms
L∞ (Max)	max\|vᵢ\|	Worst-case error
Frobenius	√(Σᵢⱼ aᵢⱼ²)	Matrix norm (like L2 for vectors)

Euclidean Distance between two vectors

d(a,b) = ‖a−b‖₂ = √Σ(aᵢ−bᵢ)²

Linear Algebra Mastery Checklist

sheet 3 complete

Vectors	Key point
Compute dot product	Σ aᵢbᵢ
Find magnitude / norm	√(Σvᵢ²)
Create unit vector	v / ‖v‖
Cosine similarity	a·b / (‖a‖‖b‖)

Matrices	Key point
Matrix multiply	(AB)ᵢⱼ = Σ aᵢₖbₖⱼ
Compute det (2×2)	ad − bc
Find inverse	(1/det)·adj(A)
Solve Ax = b	x = A⁻¹b

Eigenvalues & Decomp	Key point
Find eigenvalues	det(A−λI) = 0
Eigenvector for λ	(A−λI)v = 0
SVD form	A = UΣVᵀ
PCA uses eigenvectors of	covariance matrix

Next up → Sheet 4: Calculus for ML · derivatives · gradients · chain rule · partial derivatives · gradient descent · Jacobian · Hessian

Back to Cheat Sheets