Everything in linear algebra is built from one object: the vector. Before we can multiply matrices or diagonalize transformations, we need to be completely comfortable with what a vector is, how to combine vectors, and how to measure them. This chapter builds that foundation and — crucially — two ways of seeing it: the geometric picture (arrows in space) and the data picture (lists of numbers).
What is a vector?
A vector in \(\R^n\) is an ordered list of \(n\) real numbers, written as a column: \[\vv = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}, \qquad v_i \in \R.\] The numbers \(v_i\) are the components (or entries) of the vector, and \(n\) is its dimension. The set of all such vectors is the space \(\R^n\).
There are two complementary ways to interpret this list, and switching between them fluently is the first real skill of linear algebra.
The geometric picture. A vector in \(\R^2\) or \(\R^3\) is an arrow from the origin to the point with those coordinates. It has a length and a direction. This is how we reason about angles, distance, and projection.
The data picture. A vector is simply a record: one house is \([1800,\,3,\,1995]\tp\) (square feet, bedrooms, year). Here there is no “arrow”; the vector is a point in an abstract 3-dimensional feature space. Machine learning lives almost entirely in this picture — but it borrows all the geometric tools (distance, angle, projection) from the first.
Vector operations
Two operations generate all of linear algebra: adding vectors, and scaling them by a number.
For \(\vu,\vv\in\R^n\) and a scalar \(c\in\R\): \[\vu+\vv = \begin{bmatrix} u_1+v_1 \\ \vdots \\ u_n+v_n\end{bmatrix}, \qquad c\,\vv = \begin{bmatrix} c\,v_1 \\ \vdots \\ c\,v_n\end{bmatrix}.\] Addition is componentwise; scalar multiplication stretches (\(|c|>1\)), shrinks (\(|c|<1\)), or flips (\(c<0\)) the arrow.
Geometrically, \(\vu+\vv\) is the tip-to-tail rule: walk along \(\vu\), then along \(\vv\); you land at \(\vu+\vv\) (the diagonal of the parallelogram). Scalar multiplication slides a vector along its own line. These two pictures — the parallelogram and the line — underlie every later idea about spans and subspaces.
Linear combinations — the central concept
A linear combination of vectors \(\vv_1,\dots,\vv_k\) is any vector of the form \[c_1\vv_1 + c_2\vv_2 + \cdots + c_k\vv_k, \qquad c_i\in\R.\]
The linear combination is the single most important idea in the subject. Almost every question in linear algebra is a disguised version of: “Can vector \(\vb\) be written as a linear combination of these other vectors, and if so, how?” Solving \(\mA\vx=\vb\), finding a basis, computing rank — all are linear-combination questions.
The set of all linear combinations of a set of vectors is called their span; we study it carefully in Chapter [ch:vectorspaces]. For now, hold this picture: two non-parallel vectors in \(\R^3\) span a plane through the origin; their linear combinations sweep out that entire plane.
The dot product
The dot product takes two vectors and returns a single number that measures how much they point the same way. It is the bridge between algebra and geometry.
The dot product (or inner product) of \(\vu,\vv\in\R^n\) is \[\vu\cdot\vv \;=\; \vu\tp\vv \;=\; \sum_{i=1}^n u_i v_i \;=\; u_1v_1 + \cdots + u_nv_n.\] It is symmetric (\(\vu\cdot\vv=\vv\cdot\vu\)), linear in each argument, and connects to geometry through \[\vu\cdot\vv = \norm{\vu}\,\norm{\vv}\cos\theta,\] where \(\theta\) is the angle between the two vectors.
Let \(\vu=[2,1,3]\tp\) and \(\vv=[1,0,-1]\tp\). Then \(\vu\cdot\vv = (2)(1)+(1)(0)+(3)(-1) = 2+0-3 = -1.\) The result is negative, so the vectors point in broadly opposing directions (the angle between them exceeds \(90^\circ\)).
The sign of the dot product is a quick compass: positive means the vectors broadly agree in direction, zero means they are perpendicular, negative means they broadly oppose. “Alignment” is the word to remember — and it is exactly the notion that reappears as similarity between embeddings in AI.
A single artificial neuron computes \(\vw\tp\vx + b\) — a dot product of a weight vector with an input, plus a bias. “Similarity” between two pieces of text, two images, or two users in a recommender system is almost always a (normalized) dot product. When a vector database returns “nearest” results by cosine similarity, it is ranking by \(\frac{\vu\cdot\vv}{\norm{\vu}\norm{\vv}}=\cos\theta\). The dot product is the most-executed operation in all of machine learning.
Length, distance, and norms
The Euclidean norm (length) of \(\vv\in\R^n\) is \[\norm{\vv}_2 = \sqrt{\vv\cdot\vv} = \sqrt{v_1^2 + \cdots + v_n^2}.\] The distance between two vectors is the length of their difference, \(\norm{\vu-\vv}_2\). A vector with \(\norm{\vv}=1\) is a unit vector; we normalize any nonzero \(\vv\) by \(\vv/\norm{\vv}\).
Different ways of measuring “size” are useful in different settings. The general family is the \(p\)-norms: \[\norm{\vv}_1 = \sum_i |v_i| \ \text{(``Manhattan'')},\qquad \norm{\vv}_2 = \sqrt{\textstyle\sum_i v_i^2}\ \text{(Euclidean)},\qquad \norm{\vv}_\infty = \max_i |v_i|.\]
These norms are exactly the regularizers of machine learning. Penalizing \(\norm{\vw}_2^2\) is Ridge regression (smooth shrinkage of weights); penalizing \(\norm{\vw}_1\) is Lasso, which drives some weights to exactly zero and so performs feature selection. The geometric difference — a round ball versus a pointed diamond — is why L1 produces sparsity, a fact we will make precise later.
Angle, orthogonality, and projection
Rearranging the geometric form of the dot product gives the angle directly: \[\cos\theta = \frac{\vu\cdot\vv}{\norm{\vu}\,\norm{\vv}}.\]
Two vectors are orthogonal (perpendicular) when \(\vu\cdot\vv=0\). Orthogonality is the most important special relationship between vectors; entire chapters (projections, least squares, the SVD) are built on it.
Projection onto a line
How much of \(\vv\) lies along the direction of \(\vu\)? Drop a perpendicular from the tip of \(\vv\) onto the line through \(\vu\); the shadow is the projection: \[\proj_{\vu}(\vv) = \frac{\vu\cdot\vv}{\vu\cdot\vu}\,\vu.\]
The projection is the point on the line closest to \(\vv\), and the error \(\vv-\proj_{\vu}(\vv)\) is orthogonal to \(\vu\). “Closest point = orthogonal error” is the seed of least-squares regression (Chapter [ch:orthogonality]): fitting a model is projecting the data onto the space of possible predictions.
import numpy as np
u = np.array([2.0, 1.0, 3.0]); v = np.array([1.0, 0.0, -1.0])
dot = u @ v # -1.0
norm = np.linalg.norm(u) # sqrt(14)
cos = (u @ v) / (np.linalg.norm(u) * np.linalg.norm(v))
proj = (u @ v) / (u @ u) * u # projection of v onto u
print(dot, norm, cos, proj)Standard basis vectors
The vectors \(\ve_1=[1,0,\dots,0]\tp,\ \ve_2=[0,1,0,\dots]\tp,\dots,\ve_n\) are the standard basis of \(\R^n\). Every vector is a unique linear combination of them: \(\vv = v_1\ve_1+\cdots+v_n\ve_n\). They are mutually orthogonal unit vectors — the cleanest possible coordinate system, and the prototype for the orthonormal bases that make the SVD so powerful.
Chapter summary
A vector is a list of numbers, seen either as an arrow (geometry) or a record (data).
Vector addition and scalar multiplication generate linear combinations, the core concept.
The dot product \(\vu\tp\vv=\norm{\vu}\norm{\vv}\cos\theta\) measures alignment; it underlies neurons and similarity.
Norms measure size; L2/L1 norms are Ridge/Lasso regularizers.
Orthogonality (\(\vu\cdot\vv=0\)) and projection (closest point, orthogonal error) seed least squares.