Matrix Theory
A matrix is a two-dimensional table with columns, rows, and values. We use the notation $A_{n,p}$ for a matrix of $n$ lines and $p$ columns. The value at the $i$-nth line and the $j$-nth column is written $a_{ij}=a_{i,j}$.
⚠️ The coefficients $i$ and $j$ are starting from 1.
📚 The number of columns and lines is called the dimension.
A linear system is a set of equations that can be solved together. Matrices are often used to represent and solve linear systems of equations.
We create a matrix $A$ for the variables coefficients, a vector $b$ for the right-hand side. The matrix $S$ is made by concatenating the two.
Basic Operations
Addition between matrices
To add two matrices, they must have the same dimensions. The addition is defined by the sum coefficient by coefficient.
Subtraction
The subtraction is not defined, but we can do $A + -1 * B$.
Scalar multiplication
Multiply each coefficient by the scalar.
Matrix multiplication
The required is for both matrices to share a common row/column value c so that we have: $M1_{n1, \ {\color{red}c}} * M2_{\ {\color{red}c}, p2}$. The result is $M_{n1,p2}$.
For each coefficient, we are doing the following operation:
Matrix division
$A / B$ is not possible, but you can do $A * B^{-1}$ (inverse of a matrix).
Special matrices
| Identity matrix $Id_n$ or $I_n$ |
|---|
| This is a diagonal matrix having $1$ on the diagonal, which means we have $A_n * Id_n = Id_n * A_n = A_n$. |
| Zero matrix $O_{np}$ | Square matrix $A_{n}$ | Diagonal matrix $D_{n}$ | Conjugate transpose $A^*$ |
|---|---|---|---|
|
All values are zero.
\[
O_{1,2}=\ \begin{pmatrix}
0&0
\end{pmatrix}
\]
|
We must have $i=j$.
\[
A_{2}=\ \begin{pmatrix}
1&2\\
3&4\\
\end{pmatrix}
\]
|
Zero-Matrix with non-zero values on the diagonal.
\[
D_{2}=\ \begin{pmatrix}
1&0\\
0&4\\
\end{pmatrix}
\]
|
Change the sign before the complex value $i$. |
| Transpose matrix $A^{T}$ | Vector $A_{1,n}/A_{n,1}$ |
Hermitian matrix ($\mathbb{C}$) Symmetric matrix ($\mathbb{R}$) |
Unitary matrix ($\mathbb{C}$) Orthogonal matrix ($\mathbb{R}$) |
|---|---|---|---|
|
Rows become columns.
\[
A=\ \begin{pmatrix}
1&2
\end{pmatrix}
\quad
A^T=\ \begin{pmatrix}
1\\2
\end{pmatrix}
\]
|
A matrix with either one line or one column.
\[
A_{1,2}=\ \begin{pmatrix}
1&2
\end{pmatrix}
\]
|
This is a matrix equal to its transpose. |
This is a matrix whose inverse is equal to its transpose. |
| Lower triangular $L$ | Strictly lower triangular $L$ | Upper triangular $U$ | Strictly upper triangular $U$ |
|---|---|---|---|
|
All values above the diagonal are 0. |
All values above the diagonal are 0, including the diagonal. |
All values below the diagonal are 0. |
All values below the diagonal are 0, including the diagonal. |
Gaussian elimination
Gauss can be used to solve a system represented as a matrix. The goal is to get a matrix with a strictly increasing number of zeros before the leading coefficients/pivots $p_i$; which are the first non-null values.
The result of Gauss is called the row echelon form. If all leading coefficients are $1$ and there is only $0$ in their columns, the result is called a reduced row echelon form. Both are upper triangular matrices.
The operations you can use to transform the matrix are:
- Swapping two rows: $L_i \iff L_j$
- Adding a row: $L_i = L_i + L_j$
- Adding a row, multiplied by $c$: $L_i \leftarrow L_i + cL_j$
- Subtracting a row: $L_i \leftarrow L_i + -1 * L_j$
- Dividing a row by $c$: $L_i \leftarrow L_i + \frac{1}{c} * L_j$
- Multiply a row by $c$: $L_i \leftarrow c * L_i$
📚 All other operations are examples from $L_i \leftarrow a * L_i + b * L_j$.
➡️ See also: Gauss Example.
Inverse of a matrix
The inverse of a matrix when multiplied by the original matrix, results in the identity matrix. In mathematical notation, $A^{-1}$ represents the inverse of $A$. Only square matrices may be invertible.
The determinant $det(A)$ or $|A|$ is a scalar that we can calculate. When $det(A) \neq 0$, the matrix is invertible, e.g., $A \in Gl_n(\mathbb{R})$.
- $A_{1}=\begin{pmatrix} a \end{pmatrix} \quad \to \quad det(A) = a$
- $D_{n}=\ldots \quad \to \quad det(A) = \prod_1^n a_{ii}$
- $det(AB) = det(A) * det(B)$
- $det(A^t) = det(A)$
- For a square matrix $A_2$:
➡️ The formulas above are also working for upper/lower triangular matrices. Simply consider both as the same as $D_n$.
➡️ You can use Gauss to get a triangular matrix. ⚠️ If you swapped lines $n$ times, you need to multiply the determinant by $(-1)^n$.
➡️ You can also use cofactors to compute the determinant.
➡️ See also: Determinant Example.
To calculate the inverse, we can use:
-
GAUSS 🏔️: we work on two matrices $A_n$ and $I_n$. The goal is to transform $A_n$ to an identity matrix using Gauss. By repeating all operations we did on $A_n$ to $I_n$, we get $A_n^{-1}$.
-
Cofactor Matrix 🐔: we create a matrix of cofactors. We have more calculations, but each calculation is relatively simple.
➡️ See also: Inverse Example.
Cofactor Matrix
We are calling minor of a matrix $A$, the matrix $M_{i,p}$ created by removing the row $i$ and the column $p$ of $A$.
We are calling cofactor $C_{i,p} = (-1)^{i+p} * det(M_{i,p})$.
The cofactor matrix of $A$ is the matrix of the cofactors.
What is convenient is that the minor is often a $M_2$ matrix so we have the formula $ad-bc$ to calculate the determinant.
📚 We usually pick the column $p$ with the highest coefficients.
Definiteness of a matrix
Many calculations will require a specific "definiteness" for the matrix. We can get this property using either eigenvalues or leading minors.
| Name | eigenvalues | minors |
|---|---|---|
| positive definite | $\forall k,\quad \lambda_k \gt 0$ | $\forall k,\quad \Delta_k \gt 0$ |
| positive semi-definite | $\forall \text{k aside $k=e$},\quad \lambda_k \gt 0,\quad \lambda_{e} = 0$ | $\forall k < n,\quad \Delta_{k}>0 \text{. and } \Delta_{n}=0$ |
| negative definite | $\forall k,\quad \lambda_k \lt 0$ | $\forall k,\quad (-1)^k \Delta_k \gt 0$ |
| negative semi-definite | $\forall \text{k aside $k=e$},\quad \lambda_k \lt 0,\quad \lambda_{e} = 0$ | $\forall k < n,\quad (-1)^k\Delta_{k}>0 \text{. and } \Delta_{n}=0$ |
| indefinite | $\exists k\exists n,\quad \lambda_k \gt 0,\quad \lambda_n \lt 0$ | $\Delta_{n} < 0$, and the dimension n is pairwise |
➡️ See also: Definiteness Examples
Leading Minors
From a square matrix $A_n$, we can create a smaller matrix $A_i$ with $i$ the number of rows==columns we kept.
The leading minors or principal minors formula is: $\Delta_{i}=det(A_{i})$.
The leading minors would be:
- \(\Delta_{1}=a_{11}\)
- \(\Delta_{2}=det (A_{2})= \begin{pmatrix}a_{11} & a_{12} \\ a_{21} & a_{22}\end{pmatrix} = a_{11} * a_{22} - a_{12} * a_{21}\)
- ...
- \(\Delta_{n}=det(A)\)
Eigendecomposition of a matrix
The eigendecomposition is a process to generate an expression of matrices allowing us to calculate $A^n$ or $exp(A)$.
- $D$ is a diagonal matrix, with the eigenvalues of $A$ on the diagonal
- $P$ is an invertible matrix that we must find
Eigendecomposition steps
- Calculate $\lambda{I_n}-A_n$
- Solve $det(\lambda{I_n}-A_n) = 0$ to get the values for all $\lambda$ (no order).
- You now have $D$.
For each eigenvalue $\lambda_i$, compute an eigenvector. It's the result of solving $A - \lambda_n * Id_n = 0$ or the following system:
By concatenating all eigenvectors, we got $P$. Compute $P^{-1}$, and check that $P*D^1*P^{-1}=A$.
➡️ $\sigma(A)$ is called spectrum. It's a set of all eigenvalues.
➡️ $\rho$ is called Rho. It's the highest eigenvalue.
➡️ See also: Eigendecomposition Example