3.2 Inner Products

Inner products provide the foundation for geometric concepts such as length, angle, and orthogonality.

Definition 3.2 An inner product is a function that takes two vectors and gives a real number. It must satisfy three key properties:

  1. Symmetry:
    The order of the vectors doesn’t matter. \[ \langle \mathbf{x}, \mathbf{y} \rangle = \langle \mathbf{y}, \mathbf{x} \rangle \]

  2. Linearity:
    It behaves nicely with addition and scaling:
    \[ \langle a \mathbf{x} + b \mathbf{y}, \mathbf{z} \rangle = a \langle \mathbf{x}, \mathbf{z} \rangle + b \langle \mathbf{y}, \mathbf{z} \rangle \]

  3. Positive Definiteness:
    The inner product of a vector with itself is always positive unless the vector is zero:
    \[ \langle \mathbf{x}, \mathbf{x} \rangle > 0 \quad \text{if } \mathbf{x} \ne 0 \]

Example 3.4 The dot product in \(\mathbb{R}^n\) is: \[ \mathbf{x}^\top \mathbf{y} = \sum_{i=1}^n x_i y_i. \] This is a specific example of an inner product.

The pair \((V, \langle \cdot, \cdot \rangle)\) is an inner product space. If the dot product is used, it is a Euclidean vector space.

Example 3.5 For \(V = \mathbb{R}^2\), \[ \langle \mathbf{x}, \mathbf{y} \rangle = x_1 y_1 - (x_1 y_2 + x_2 y_1) + 2x_2 y_2. \] This satisfies the properties of an inner product but differs from the dot product.

3.2.1 Symmetric, Positive Definite Matrices

Symmetric, positive definite (SPD) matrices are closely related to inner products and are central in machine learning and matrix decompositions.

Let \(V\) be an \(n\)-dimensional inner product space with basis \(\mathbf{B} = (\mathbf{b}_1, ..., \mathbf{b}_n)\). For \(\mathbf{x}, \mathbf{y} \in V\), we have \(\mathbf{x} = \sum_{i=1}^n \psi_i \mathbf{b}_i\), and \(\mathbf{y} = \sum_{j=1}^n \lambda_j \mathbf{b}_j\) for some \(\psi_i\) and \(\lambda_j\) values. Linearity of the inner product gives us \[ \langle \mathbf{x}, \mathbf{y} \rangle = \sum_{i=1}^n \sum_{j=1}^n \psi_i \langle \mathbf{b}_i, \mathbf{b}_j \rangle \lambda_j = \hat{\mathbf{x}}^\top \mathbf{A} \hat{\mathbf{y}}, \] where \(\mathbf{A}_{ij} = \langle \mathbf{b}_i, \mathbf{b}_j \rangle\). Thus, the inner product is fully determined by a symmetric matrix \(\mathbf{A}\), and if it is positive definite, then: \[ \mathbf{x}^\top A \mathbf{x} > 0 \quad \forall \mathbf{x} \ne 0. \]

Example 3.6 Let \(V\) be a 2-dimensional inner product space with ordered basis \(\mathbf{B} = \left( \begin{bmatrix} 1\\1 \end{bmatrix}, \begin{bmatrix}-1\\1 \end{bmatrix} \right)\). Let \(\mathbf{x} = \begin{bmatrix} 3\\5 \end{bmatrix}\) and \(\mathbf{y} = \begin{bmatrix} -2\\0 \end{bmatrix}\) be two vectors in \(V\). Then \[\begin{align*} \mathbf{x} &= 3\begin{bmatrix} 1\\1 \end{bmatrix} 5 \begin{bmatrix}-1\\1 \end{bmatrix}\\ \mathbf{y} &= -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix}. \end{align*}\] The inner product is therefore \[\begin{align*} \langle \mathbf{x}, \mathbf{y} \rangle &= \left\langle 3\begin{bmatrix} 1\\1 \end{bmatrix} + 5\begin{bmatrix}-1\\1 \end{bmatrix}, -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle \\ &= 3\left\langle \begin{bmatrix} 1\\1 \end{bmatrix}, -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle + 5\left\langle \begin{bmatrix}-1\\1 \end{bmatrix}, -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle \\ &= 3\left\langle -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix} 1\\1 \end{bmatrix} \right\rangle + 5\left\langle -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle \\ &= 3\left\langle \begin{bmatrix} 1\\1 \end{bmatrix} , \begin{bmatrix} 1\\1 \end{bmatrix} \right\rangle(-2) + 3\left\langle \begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix} 1\\1 \end{bmatrix} \right\rangle (0) + \\ & \quad + 5\left\langle \begin{bmatrix} 1\\1 \end{bmatrix} , \begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle(-2) + 5\left\langle \begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle(0)\\ &= \psi_1\left\langle \begin{bmatrix} 1\\1 \end{bmatrix} , \begin{bmatrix} 1\\1 \end{bmatrix} \right\rangle \lambda_1 + \psi_1\left\langle \begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix} 1\\1 \end{bmatrix} \right\rangle \lambda_2 + \\ &\quad + \psi_2\left\langle \begin{bmatrix} 1\\1 \end{bmatrix} , \begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle \lambda_1 + \psi_2\left\langle \begin{bmatrix}-1\\1 \end{bmatrix}, \begin{bmatrix}-1\\1 \end{bmatrix} \right\rangle \lambda_2\\ &= \sum_{i=1}^n \sum_{j=1}^n \psi_i \langle \mathbf{b}_i, \mathbf{b}_j \rangle \lambda_j. \end{align*}\] Now, since \(\langle \mathbf{b}_i, \mathbf{b}_j \rangle = 2\) when \(i = j\) and 0 otherwise, we can see that \[\langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{x}^\top \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} \mathbf{y} = -12.\] If we look at \(\mathbf{x}\) and \(\mathbf{y}\) in the standard basis, they are \(\begin{bmatrix}-2\\8 \end{bmatrix}\) and \(\begin{bmatrix}-2\\-2 \end{bmatrix}\) and we can confirm that \(\langle \mathbf{x}, \mathbf{y} \rangle = -12\).

Example 3.7 Let \(V\) be a 2-dimensional inner product space with ordered basis \(\mathbf{B} = \left( \begin{bmatrix} 1\\1 \end{bmatrix}, \begin{bmatrix}0\\1 \end{bmatrix} \right)\). Let \(\mathbf{x} = \begin{bmatrix} 3\\5 \end{bmatrix}\) and \(\mathbf{y} = \begin{bmatrix} -2\\0 \end{bmatrix}\) be two vectors in \(V\). Then \[\begin{align*} \mathbf{x} &= 3\begin{bmatrix} 1\\1 \end{bmatrix} 5 \begin{bmatrix}0\\1 \end{bmatrix}\\ \mathbf{y} &= -2\begin{bmatrix} 1\\1 \end{bmatrix} + 0\begin{bmatrix}0\\1 \end{bmatrix}. \end{align*}\] Keeping in mind that \[\begin{align*} \left\langle\begin{bmatrix} 1\\1 \end{bmatrix}, \begin{bmatrix}0\\1 \end{bmatrix} \right \rangle &= 1\\ \left\langle\begin{bmatrix} 1\\1 \end{bmatrix}, \begin{bmatrix} 1\\1 \end{bmatrix} \right \rangle &= 2\\ \left\langle\begin{bmatrix}0\\1 \end{bmatrix}, \begin{bmatrix}0\\1 \end{bmatrix} \right \rangle &= 1, \end{align*}\] we have \[\langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{x}^\top \begin{bmatrix} 2 & 1 \\ 1 & 1 \end{bmatrix} \mathbf{y} = -22.\] If we look at \(\mathbf{x}\) and \(\mathbf{y}\) in the standard basis, they are \(\begin{bmatrix} 3\\ 8 \end{bmatrix}\) and \(\begin{bmatrix}-2\\-2 \end{bmatrix}\) and we can confirm that \(\langle \mathbf{x}, \mathbf{y} \rangle = -22\).

Definition 3.3 A symmetric matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) is positive definite if: \[ \mathbf{x}^\top \mathbf{A} \mathbf{x} > 0 \quad \forall \mathbf{x} \ne 0 \] If \(\mathbf{x}^\top \mathbf{A} \mathbf{x} \ge 0\), \(\mathbf{A}\) is positive semidefinite.

Example 3.8 \[ A_1 = \begin{bmatrix} 9 & 6 \\ 6 & 5 \end{bmatrix}, \quad A_2 = \begin{bmatrix} 9 & 6 \\ 6 & 3 \end{bmatrix} \]

  • \(A_1\) is positive definite, since: \[ \mathbf{x}^\top A_1 \mathbf{x} = (3\mathbf{x}_1 + 2\mathbf{x}_2)^2 + \mathbf{x}_2^2 > 0 \]
  • \(A_2\) is not positive definite, since for \(\mathbf{x} = [2, -3]^\top\), \[ \mathbf{x}^\top A_2 \mathbf{x} < 0 \]

Theorem 3.1 For a real, finite-dimensional vector space \(V\) with basis \(\mathbf{B}\): \[ \langle \mathbf{x}, \mathbf{y} \rangle = \hat{\mathbf{x}}^\top A \hat{\mathbf{y}} \] is an inner product iff \(\mathbf{A}\) is symmetric and positive definite.

Exercises

Exercise 3.9

  • The kernel (null space) of \(A\) is \(\{0\}\).
  • All diagonal entries \(a_{ii} > 0\).

Exercise 3.10

Show that an inner product is homogeneous in the second spot.

Exercise 3.11

Show that \(\left\langle \begin{bmatrix}x_1\\x_2\end{bmatrix}, \begin{bmatrix}y_1\\ y_2\end{bmatrix} \right\rangle = |x_1y_1| + |x_2y_2|\) is not an inner product.

Exercise 3.12

Show that \(\langle f,g \rangle = \int_{0}^1 f(x)g(x) \, dx\) is an inner product for functions \(f\) and \(g\) that are continuous on \([0,1]\).

Exercise 3.13

Show that \(\langle f,g \rangle = \int_{0}^1 f(x)g(x) + f^\prime(x) g^{\prime}(x) \, dx\) is an inner product for functions \(f\) and \(g\) that are differentiable on \([0,1]\).

Exercise 3.14

Determine if \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 2 & 0 \end{bmatrix}\) is symmetric positive definite.

Exercise 3.15

Find a value \(k\) that makes \(\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 2 & k \end{bmatrix}\) symmetric positive semi-definite.

Exercise 3.16