Fundamentals of Vectors and Matrices

1 Vector Basics

A vector is a quantity that has both magnitude and direction.

1.1 Directed Line Segment and Vector

Given two points point A and point B , the directed line segment from A to B is called a directed line segment . A is the starting point and B is the ending point .

The directed segment AB has three attributes: the position of the start point A, the direction towards B, and the length (size) of AB.

A vector is a quantity with direction and size, usually represented by an arrow.

The vector represented by the directed segment AB can be drawn as an arrow, a bold italic letter a , or a single letter with an arrow.

1.2 Coordinate Representation of Vectors

Place the tail of the arrow at the origin; the coordinates of the head give the vector’s coordinate representation . In two dimensions a vector a = (a₁, a₂). Example: a = (3, 2).

In three dimensions the same idea applies, e.g., a = (1, 2, 2).

1.3 Magnitude of a Vector

The length of the arrow representing a vector is called its magnitude . For a vector a, the magnitude is denoted |a|. Using the Pythagorean theorem, |a| = √(a₁² + a₂²) in 2‑D and |a| = √(a₁² + a₂² + a₃²) in 3‑D.

1.4 Inner Product (Dot Product)

To define a product of two vectors that captures both magnitude and direction, we use the inner product (dot product): a·b = |a|·|b|·cosθ, where θ is the angle between a and b.

The inner product also measures directional similarity; larger dot products indicate more aligned vectors, which underlies the cosine similarity algorithm.

1.5 Cauchy‑Schwarz Inequality

From the dot‑product formula we obtain the famous Cauchy‑Schwarz inequality : -|a||b| ≤ a·b ≤ |a||b|.

2 Matrix Basics

In neural networks matrices are used to simplify mathematical expressions.

2.1 What Is a Matrix?

A matrix is a rectangular array of numbers. Rows run horizontally, columns run vertically. A 3×3 matrix has three rows and three columns.

When the number of rows equals the number of columns the matrix is a square matrix . Column vectors and row vectors are special cases of matrices.

2.2 Matrix Equality

Two matrices A and B are equal (A = B) if all corresponding elements are equal.

2.3 Matrix Addition, Subtraction, and Scalar Multiplication

These operations require matrices of the same shape. Addition and subtraction are performed element‑wise; scalar multiplication multiplies each element by the scalar.

2.4 Matrix Product

For matrices A (m×n) and B (n×p), the product AB is defined. Each element of AB is the inner product of a row of A with a column of B. Matrix multiplication is generally not commutative (AB ≠ BA), but the identity matrix E satisfies AE = EA = A.

2.5 Hadamard (Element‑wise) Product

For matrices A and B of the same shape, the Hadamard product A ⨀ B multiplies corresponding elements.

2.6 Transpose of a Matrix

The transpose of matrix A swaps rows and columns: element (i, j) becomes element (j, i). It is denoted Aᵀ.

This article ends here; the next article will discuss derivatives.

- END -