3.4 Angles and Orthogonality

Inner products also define angles between vectors.

Lemma 3.1 For nonzero \(\mathbf{x}, \mathbf{y}\): \[ \cos \omega = \frac{\langle \mathbf{x}, \mathbf{y} \rangle}{\|\mathbf{x}\| \|\mathbf{y}\|}, \] where \(\omega \in [0, \pi]\) is the angle between \(\mathbf{x}\) and \(\mathbf{y}\).

When \(\omega = 0\), the vectors point in the same direction. When \(\omega = \pi/2\), the vectors are orthogonal (perpendicular).

Example 3.12 Let \(\mathbf{u} = [1, 2]^\top\) and \(\mathbf{v} = [2, 1]^\top\). The angle \(\theta\) between them is given by \[ \cos \theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\|\mathbf{u}\| \, \|\mathbf{v}\|}. \]

Since \[ \langle \mathbf{u}, \mathbf{v} \rangle = 1\cdot 2 + 2 \cdot 1 = 4, \] and \[ \|\mathbf{u}\| = \sqrt{1^2 + 2^2} = \sqrt{5}, \quad \|\mathbf{v}\| = \sqrt{2^2 + 1^2} = \sqrt{5}, \] the angle between them is given by \[ \cos \theta = \frac{4}{\sqrt{5}\cdot \sqrt{5}} = \frac{4}{5}. \] Therefore, the angle is \[ \theta = \arccos\left(\frac{4}{5}\right) \approx 36.87^\circ. \]

Definition 3.6 Two vectors \(\mathbf{x}\) and \(\mathbf{y}\) are orthogonal if: \[ \langle \mathbf{x}, \mathbf{y} \rangle = 0. \] If both vectors also have unit length (\(\|\mathbf{x}\| = \|\mathbf{y}\| = 1\)), they are orthonormal.

Example 3.13 Consider the vectors: \[ \mathbf{u} = [1, 0, 0]^\top, \quad \mathbf{v} = [0, 1, 0]^\top. \]

The formula for the angle \(\theta\) between two vectors is: \[ \cos \theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\|\mathbf{u}\| \, \|\mathbf{v}\|}. \]

The inner product is \[ \langle \mathbf{u}, \mathbf{v} \rangle = 1\cdot 0 + 0 \cdot 1 + 0 \cdot 0 = 0, \] and the norms are \[ \|\mathbf{u}\| = \sqrt{1^2 + 0^2 + 0^2} = 1, \quad \|\mathbf{v}\| = \sqrt{0^2 + 1^2 + 0^2} = 1. \] Thus: \[ \cos \theta = \frac{0}{1 \cdot 1} = 0 \quad \implies \quad \theta = \arccos(0) = 90^\circ. \] Hence, the vectors are orthogonal (orthonormal, actually).

Orthogonality generalizes the geometric idea of perpendicularity to arbitrary inner products. Vectors orthogonal under one inner product may not be orthogonal under another.

Example 3.14 Consider the same vectors: \[ \mathbf{u} = [1, 0, 0]^\top, \quad \mathbf{v} = [0, 1, 0]^\top. \] Define a weighted inner product: \[ \langle \mathbf{x}, \mathbf{y} \rangle_A = \mathbf{x}^\top \mathbf{A} \mathbf{y}, \quad \mathbf{A} = \begin{bmatrix} 2 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 1 \end{bmatrix}. \]

The inner product is \[ \langle \mathbf{u}, \mathbf{v} \rangle_A = [1,0,0] \begin{bmatrix} 2 & 1 & 0 \\ 1 & 2 & 0 \\ 0 & 0 & 1 \end{bmatrix} [0,1,0]^\top = [1,0,0] [1,2,0]^\top = 1. \] and the norms are \[ \|\mathbf{u}\|_A = \sqrt{\mathbf{u}^\top \mathbf{A} \mathbf{u}} = \sqrt{[1,0,0] [2,1,0]^\top} = \sqrt{2}, \] \[ \|\mathbf{v}\|_A = \sqrt{\mathbf{v}^\top \mathbf{A} \mathbf{v}} = \sqrt{[0,1,0] [1,2,0]^\top} = \sqrt{2}. \] With this norm, the angle is given by: \[ \cos \theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle_A}{\|\mathbf{u}\|_A \, \|\mathbf{v}\|_A} = \frac{1}{\sqrt{2} \cdot \sqrt{2}} = \frac{1}{2}. \] So, \[ \theta = \arccos\left(\frac{1}{2}\right) = 60^\circ. \]

Hence, under this inner product, the vectors form a \(60^\circ\) angle instead of \(90^\circ\).

3.4.1 Orthogonal Matrices

Definition 3.7 A square matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) is orthogonal if: \[ \mathbf{A}^\top \mathbf{A} = \mathbf{A}\mathbf{A}^\top = \mathbf{I}. \] This implies: \[ \mathbf{A}^{-1} = \mathbf{A}^\top. \]

Orthogonal matrices preserve both lengths and angles: \[ \|\mathbf{A}\mathbf{x}\| = \|\mathbf{x}\|, \quad \text{and} \quad \langle \mathbf{A}\mathbf{x}, \mathbf{A}\mathbf{y} \rangle = \langle \mathbf{x}, \mathbf{y} \rangle. \] Hence, they represent rotations and reflections in space.

Example 3.15 Consider the matrix \[ \mathbf{R} = \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}. \]

  • This matrix is orthogonal because \(\mathbf{R}^\top \mathbf{R} = \mathbf{I}\).
  • It represents a rotation in 2D by angle \(\theta\) counterclockwise.

Example with \(\theta = 90^\circ\): \[ \mathbf{R} = \begin{bmatrix}0 & -1 \\ 1 & 0\end{bmatrix}. \]

Example 3.16 Consider the matrix \[ \mathbf{F} = \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix}. \]

  • This matrix is orthogonal because \(\mathbf{F}^\top \mathbf{F} = \mathbf{I}\).
  • It represents a reflection across the x-axis in 2D.

Exercises

Exercise 3.25 Prove that \(|\langle \mathbf{x}, \mathbf{y} \rangle| \leq \|\mathbf{x}\| \|\mathbf{y} \|\)

Exercise 3.26 Find the angle between \(\begin{bmatrix}1\\3\end{bmatrix}\) and \(\begin{bmatrix}-1\\4\end{bmatrix}\).

Exercise 3.27 Find the angle between \(\begin{bmatrix}-2\\-2\end{bmatrix}\) and \(\begin{bmatrix}-1\\3\end{bmatrix}\).

Exercise 3.28 Find the angle between \(\begin{bmatrix}0\\2\end{bmatrix}\) and \(\begin{bmatrix}-2\\1\end{bmatrix}\).

Exercise 3.29

Use the law of cosines to show that \[\cos \theta = \dfrac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}.\]

Exercise 3.30 Consider the orthogonal matrix \(\mathbf{A} = \begin{bmatrix}-1 & 0 \\ 0 & 1 \end{bmatrix}\). Show that \(\|\mathbf{A} \mathbf{x}\| = \|\mathbf{x}\|\) for \(\mathbf{x} \in \mathbb{R}^2\).

Exercise 3.31

Prove that \(\begin{bmatrix} \cos z & -\sin z\\ \sin z & \cos z \end{bmatrix}\) is an orthogonal matrix. This is the rotation matrix - it rotates vectors counterclockwise by \(z\) degrees.

Exercise 3.32

Select any orthogonal matrices in \(\mathbb{R}^2\) and \(\mathbb{R}^3\) and show that \(\mathbf{A}^T\mathbf{A} = \mathbf{I}\).

Exercise 3.33

Let \(\mathbf{A}\) be an orthogonal matrix. Show that \(\|\mathbf{A} \mathbf{x}\|^2 = \|\mathbf{x}\|^2\) and that the angle between \(\mathbf{A}\mathbf{x}\) and \(\mathbf{A}\mathbf{y}\) is the same as the angle between \(\mathbf{x}\) and \(\mathbf{y}\).