4.2 Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors provide a way to characterize a matrix \(\mathbf{A} \in \mathbb{R}^{n \times n}\) and its associated linear mapping.

Definition 4.3 Eigenvalue & Eigenvector:
A scalar \(\lambda \in \mathbb{R}\) is an eigenvalue of \(\mathbf{A}\) and a nonzero vector \(\mathbf{x} \in \mathbb{R}^n\) is a corresponding eigenvector if
\[ \mathbf{A}\mathbf{x} = \lambda \mathbf{x}. \]
This is called the eigenvalue equation.

Definition 4.4 The span of the set of eigenvectors associated with \(\lambda\) spans a subspace \(E_\lambda \subset \mathbb{R}^n\) known as the eigenspace.

Definition 4.5 The set of all eigenvalues of \(\mathbf{A}\) is called the eigenspectrum.

Definition 4.6 Number of times \(\lambda\) appears as a root of the characteristic polynomial \(p_ \mathbf{A}(\lambda) = \det(\mathbf{A} - \lambda I)\) is called the algebraic multiplicity of the eigenvalue.

Definition 4.7 The dimension of the eigenspace associated with \(\lambda\) is called the geometric multiplicity.

Example 4.10 Find the eigenspace(s) of the matrix \[ \mathbf{A} = \begin{pmatrix} 2 & 1\\ 0 & 2 \end{pmatrix}. \]

Step 1: Find the eigenvalues

Compute the characteristic polynomial: \[ \det(\mathbf{A} - \lambda \mathbf{I}) = \det \begin{pmatrix} 2-\lambda & 1\\ 0 & 2-\lambda \end{pmatrix} = (2-\lambda)^2. \]

So the only eigenvalue is
\[ \lambda = 2 \quad \text{(with algebraic multiplicity 2).} \]

Step 2: Find the eigenspace for \(\lambda = 2\)

Form the matrix: \[ \mathbf{A} - 2\mathbf{I} = \begin{pmatrix} 2-2 & 1\\ 0 & 2-2 \end{pmatrix} = \begin{pmatrix} 0 & 1\\ 0 & 0 \end{pmatrix}. \]

Solve: \[ (\mathbf{A} - 2\mathbf{I})\mathbf{x} = 0. \]

This gives: \[ x_2 = 0. \]

So \(x_1\) is free, and the solution vectors have the form: \[ \mathbf{x} = \begin{pmatrix}x_1\\ 0\end{pmatrix} = x_1 \begin{pmatrix}1\\ 0\end{pmatrix}. \]

Step 3: Write the eigenspace

The eigenspace is: \[ E_2 = \text{span}\left\{\begin{pmatrix}1\\ 0\end{pmatrix}\right\}. \]

So the eigenspace is one-dimensional, even though the eigenvalue has algebraic multiplicity 2.

4.2.1 Key Properties

Eigenvalues and eigenvectors have several key properties:

  • \(\mathbf{A}\) and \(\mathbf{A}^\top\) have the same eigenvalues, not necessarily the same eigenvectors.
  • Similar matrices have identical eigenvalues. Matrices \(\mathbf{A}\) and \(\mathbf{D}\) are similar if \(\mathbf{A} = \mathbf{P} \mathbf{D} \mathbf{P}^{-1}\) for some matrix \(\mathbf{P}\).
  • Symmetric, positive definite matrices have real, positive eigenvalues.
  • A matrix with \(n\) distinct eigenvalues has linearly independent eigenvectors forming a basis of \(\mathbb{R}^n\).
  • Defective matrices: Have fewer than \(n\) linearly independent eigenvectors.

Example 4.11 Consider the matrix

\[ \mathbf{A} = \begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix} \]

and the invertible matrix

\[ \mathbf{P} = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}. \]

We can compute a matrix \(\mathbf{B}\) that is similar to \(\mathbf{A}\) using the formula:

\[ \mathbf{B} = \mathbf{P}^{-1} \mathbf{A} \mathbf{P} \]

First, find \(\mathbf{P}^{-1}\):

\[ \mathbf{P}^{-1} = \begin{bmatrix} 1 & -1 \\ 0 & 1 \end{bmatrix}. \]

Then,

\[ \mathbf{B} = \begin{bmatrix} 1 & -1 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}. \]

Thus, \(\mathbf{A}\) and \(\mathbf{B}\) are similar matrices, since there exists an invertible matrix \(\mathbf{P}\) such that \(\mathbf{B} = \mathbf{P}^{-1} \mathbf{A} \mathbf{P}\). One can verify that matrices \(\mathbf{A}\) and \(\mathbf{B}\) have the same eigenvalues (\(\lambda = 2\) and 3).

Theorem 4.4 Spectral Theorem: If \(\mathbf{A}\) is symmetric, there exists an orthonormal basis of \(\mathbb{R}^n\) consisting of eigenvectors of \(\mathbf{A}\) (and all eigenvalues are real). Furthermore, \(\mathbf{A}\) can be decomposed as
\[ \mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^\top \]
where \(\mathbf{P}\) contains eigenvectors and \(\mathbf{D}\) is diagonal with eigenvalues.

Example 4.12 We illustrate the decomposition of \[ \mathbf{A} = \begin{pmatrix} 2 & 1\\ 1 & 2 \end{pmatrix}. \]

First, we need to find the eigenvalues/ eigenvectors for \(\mathbf{A}\).

\[ \det(\mathbf{A} - \lambda \mathbf{I}) = \det\begin{pmatrix} 2-\lambda & 1\\ 1 & 2-\lambda \end{pmatrix} = (2-\lambda)^2 - 1. \]

So the eigenvalues are: \[ \lambda_1 = 3, \qquad \lambda_2 = 1. \]

For \(\lambda_1 = 3\), \[ (\mathbf{A} - 3\mathbf{I})\mathbf{v}_1 = \begin{pmatrix} -1 & 1\\ 1 & -1 \end{pmatrix} \begin{pmatrix}x\\y\end{pmatrix} = 0 \Rightarrow x = y. \] Choose: \[ \mathbf{v}_1 = \frac{1}{\sqrt{2}} \begin{pmatrix}1\\1\end{pmatrix}. \] For \(\lambda_2 = 1\), \[ (\mathbf{A}-\mathbf{I})\mathbf{v}_2 = \begin{pmatrix} 1 & 1\\ 1 & 1 \end{pmatrix} \begin{pmatrix}x\\y\end{pmatrix} = 0 \Rightarrow x = -y. \] Choose: \[ \mathbf{v}_2 = \frac{1}{\sqrt{2}} \begin{pmatrix}1\\-1\end{pmatrix}. \]

To form \(\mathbf{P}\) and \(\mathbf{D}\), take the normalized eigenvectors to form the columns of \(\mathbf{P}\) and the eigenvalues to form the main diagonal of \(\mathbf{D}\), \[ \mathbf{P} = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\[4pt] \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix}, \qquad \mathbf{D} = \begin{pmatrix} 3 & 0\\ 0 & 1 \end{pmatrix}. \]

To check, \[ \mathbf{PDP}^\top = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} \begin{pmatrix} 3 & 0\\ 0 & 1 \end{pmatrix} \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} \end{pmatrix} = \begin{pmatrix} 2 & 1\\ 1 & 2 \end{pmatrix} = \mathbf{A}. \]

4.2.2 Relations to Determinant and Trace

Eigenvalues and eigenvectors are related to the determinant and trace of a matrix. For example, \[\det(\mathbf{A}) = \prod_{i=1}^n \lambda_i.\] Furthermore, \[\text{tr}(\mathbf{A}) = \sum_{i=1}^n \lambda_i.\] Geometrically, eigenvectors are directions stretched by \(\lambda_i\); determinant gives volume scaling, trace gives scaling of perimeter.

Example 4.13 Let \(\mathbf{A} = \begin{pmatrix}4 & 2 \\ 1 & 3\end{pmatrix}\). Then

  • Eigenvalues: \(\lambda_1 = 2, \;\;\; \lambda_2 = 5\)
  • Eigenspaces: \(E_2 = \text{span}\{[1, -1]^\top\}, \;\;\; E_5 = \text{span}\{[2,1]^\top\}\)
  • Trace is \(\text{sum of diagonals of the matrix} = 4 + 3 = 7 = 5 + 2 = \text{sum of the eigenvalues}\)
  • Determinant is \(\text{by the formula} = 4(3) - 2(1) = 10 = 5 \times 2 = \text{product of the eigenvalues}\).

Google’s PageRank Algorithm uses the eigenvector of the maximal eigenvalue (\(\lambda = 1\)) of the web connectivity matrix to rank web pages.

Exercises

Exercise 4.13 Find the characteristic equation, eigenvalues, eigenspace, determinant, and trace corresponding to \[\mathbf{A} = \begin{bmatrix} 1 & 4\\3 & 2 \end{bmatrix}.\]

Exercise 4.14 Find the characteristic equation, eigenvalues, eigenspace, determinant, and trace corresponding to \[\mathbf{A} = \begin{bmatrix} 2 & 2\\1 & 3 \end{bmatrix}.\]

Exercise 4.15 Find the characteristic equation, eigenvalues, eigenspace, determinant, and trace corresponding to \[\mathbf{A} = \begin{bmatrix}1&0&0\\0&1&2\\0&0&0\end{bmatrix}.\]

Exercise 4.16 Find the characteristic equation, eigenvalues, eigenspace, determinant, and trace corresponding to \[\mathbf{A} = \begin{bmatrix}2&0&4\\0&3&0\\0&1&2\end{bmatrix}.\]

Exercise 4.17 Given a matrix \(\mathbf{A} \in \mathbb{R}^{m \times n}\), we can always obtain a symmetric positive semidefinite matrix \(\mathbf{S} \in \mathbb{R}^{n \times n}\) by defining \(\mathbf{S} = \mathbf{A}^T\mathbf{A}\). Prove this statement for \(2 \times 2\), \(3 \times 2\) and \(2 \times 3\) matrices.

Exercise 4.18 Explain why geometrically, an eigenvector is a vector whose direction is unchanged by multiplying by matrix \(\mathbf{A}\).

Exercise 4.19