Inner Products¶
In this chapter we introduce the idea of an inner product, examine the additional structure that it imposes on a vector space, and look at some common applications. An inner product is an operation on two vectors in a vector space that is defined in such a way as to satisfy certain algebraic requirements. To begin, we will focus only on one specific inner product defined for vectors in \(\mathbb{R}^n\). Later in the chapter we will consider other examples of inner products in \(\mathbb{R}^n\).
The dot product is the most common example of an inner product. For vectors \(U\) and \(V\) in \(\mathbb{R}^n\), the dot product is written as \(U\cdot V\), and is defined in the following way.
For example, we show the computation for \(U\) and \(V\) in \(\mathbb{R}^4\).
This is exactly the sort of calculation that is carried out in matrix multiplications. In fact it is possible to view matrix products as collections of dot products between the rows and columns of the two matrices. Suppose for example that we have the following matrix product with \(U\) as the second row of \(A\) and \(V\) as the third column of \(B\).
We see that \((AB)_{23} = U\cdot V\), the dot product of the second row of \(A\) with the third column of \(B\). In general the entry \((AB)_{ij}\) will be the dot product of the \(i\)th row of \(A\) with the \(j\)th column of \(B\).
Taking advantage of this connection, we could also write the definition of the dot product by viewing the vectors as \(n\times 1\) matrices, and making use of matrix transposes.
Since \(U\) is an \(n\times 1\) matrix, \(U^T\) will be a \(1\times n\) matrix, and \(U^TV\) will be a \(1\times 1\) matrix. In other words \(U^TV\) will be a scalar. If we take this view, we will be able to compute inner products just the same as we would matrix products using NumPy.
import numpy as np
U = np.array([[2],[-1],[2],[3]])
V = np.array([[1],[4],[5],[0]])
dot_product = U.transpose()@V
print(dot_product)
[[8]]
Computing a dot product with this formula is correct and we would not notice any difference in doing the calculation on paper. There is however a slight technical inconvenience in using NumPy operations to apply this formula. The output produced by the code (\(\texttt{dot_product}\)) is not a true scalar, but in fact is another NumPy array object of size \(1 \times 1\). This may have unintended consequences as we include this code in larger calculations. We can keep this in mind and simply reference the value in the \(1\times 1\) array with indices if need be.
print(dot_product[0,0])
8
Another possibility is that we write our own function, called \(\texttt{DotProduct}\) to provide us the dot product in a simple format. The following code is included in the \(\texttt{laguide}\) module for future use.
def DotProduct(U,V):
# =============================================================================
# U and V are NumPy arrays that represents vectors of dimension n x 1.
# DotProduct returns the dot product of U and V
# =============================================================================
# Check shapes of U and V
if (U.shape[1] != 1 or V.shape[1] != 1):
print("Dot product only accepts column vectors.")
return
# Check shape of V
if (U.shape[0] != V.shape[0]):
print("Dot product only accepts column vectors of equal length.")
return
n = U.shape[0]
product = 0
for i in range(n):
product += U[i,0]*V[i,0]
return product
We can now call the function and receive a scalar as output.
print(DotProduct(U,V))
8
Geometric interpretations¶
The dot product provides a way to measure distances and angles in \(\mathbb{R}^n\). Again it is easiest to visualize these measures in \(\mathbb{R}^2\).
The magnitude of a vector \(U\), written as \(||U||\), is defined as \(\left(U\cdot U\right)^{1/2}\). In \(\mathbb{R}^2\) it is plain to see that this definition corresponds to the Pythagorean Theorem.
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}
ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.text(1,2,'$U$')
ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
Calculating the magnitude of a vector is common enough that it will be useful to write a function to carry it out. We will leave it as an exercise in this section, and will include the function \(\texttt{Magnitude}\) in the \(\texttt{laguide}\) module for use in future sections.
from laguide import Magnitude
U = np.array([[2],[3]])
print(Magnitude(U))
3.605551275463989
It is often useful to work with vectors that have magnitude equal to one. Such vectors are called unit vectors. We can construct a unit vector in the same direction as a vector \(U\) by multiplying \(U\) by the reciprocal of its magnitude. We will label this unit vector \(W\).
fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}
ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,0.555,0.832,fc='b',ec='r',**options)
ax.text(1,2,'$U$')
ax.text(0.5,0.3,'$W$')
ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
We can also use the definition of magnitude to define the distance between two vectors \(U\) and \(V\) as \(||U-V||\), the magnitude of the difference. If we picture the vectors as arrows, \(U-V\) goes from the tip of \(V\) to the tip of \(U\). This is consistent with our arrow picture of vector addition since \(V + (U-V) = U\).
fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}
ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,4,2,fc='b',ec='b',**options)
ax.arrow(4,2,-2,1,fc='b',ec='r',**options)
ax.text(1,2,'$U$')
ax.text(3,1,'$V$')
ax.text(3,2.8,'$U-V$')
ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
With common identities from trigonometry, it is also possible to work out the angle between two vectors in terms of the dot product.
fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}
ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,4,2,fc='b',ec='b',**options)
ax.text(1,2,'$U$')
ax.text(3,1,'$V$')
ax.text(1,1,'$\\theta$')
ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
It is worth noting the special case that arises when the angle between \(U\) and \(V\) is a right angle. In this case, since \(\cos{\theta}\) is zero, the dot product \(U\cdot V\) is also zero. When \(U\cdot V = 0\), the vectors \(U\) and \(V\) are said to be orthogonal. In upcoming sections we will learn the importance of orthogonality, and how it can be used to simplify certain calculations
Algebraic properties¶
If \(U\), \(V\), and \(W\) are any vectors in \(\mathbb{R}^n\), and \(k\) is a scalar, then the following statements involving the dot product are true.
\(U\cdot V = V\cdot U\)
\(U\cdot(V + W) = U\cdot V + U\cdot W\)
\((kU)\cdot V = k(U\cdot V)\)
\(U\cdot U \ge 0\)
These algebraic properties are not difficult to prove using the definition of the dot product, but they are important for carrying out calculations with unspecified vectors. More importantly, it is these four properties that make the dot product an inner product.
Exercises¶
Exercise 1: Create a Python function named \(\texttt{Magnitude}\) that accepts as an argument vector in the form of a NumPy array object with shape \(n\times 1\), and returns the magnitude of that vector.
## Code solution here.
Exercise 2: Use the \(\texttt{random}\) module of NumPy to generate random \(4\times 1\) vectors \(U\), \(V\), and \(W\), and a random scalar \(k\). Verify the algebraic properties of the dot product listed above. Try to make use of a conditional statement to check each statement.
## Code solution here.
Exercise 3: Let \(X\) and \(Y\) be the following vectors.
\((a)\) Determine the angle between \(X\) and \(Y\). (You will need the \(\texttt{acos}\) function in Python’s \(\texttt{math}\) module.)
## Code solution here.
\((b)\) Find a vector in \(\mathbb{R}^3\) that is orthogonal to the vector \(X\). Verify your answer with a computation.
## Code solution here.
\((c)\) Construct a unit vector \(Z\) such that \(Z\cdot Y = -||Y||\). Verify your answer with a computation.
## Code solution here.
Exercise 4: Create a Python function named \(\texttt{DifferenceMagnitude}\) which takes two \(n \times 1\) vectors as arguments and returns the magnitude of their difference. Use this function to determine \(|| X - Y||\) where vectors \(X\) and \(Y\) are given below.
## Code solution here
Exercise 5: Create a Python function which takes an \(n \times 1\) vector as its argument and returns the unit vector of the argument vector. Use the NumPy \(\texttt{random}\) module to generate a random \(3 \times 1 \) vector, and test this function on that vector.
## Code solution here
Exercise 6: Find a vector \(Y\) in \(\mathbb{R}^2\) such that \(X\cdot Y = ||Y - X||\) where the vector \(X\) is given below. Is the vector \(Y\) unique ? Verify your answer through a computation.
## Code solution here
Exercise 7: Create a Python function named \(\texttt{Angle}\) which takes two vectors as arguments and returns \(\cos{\theta}\) where \({\theta}\) is the angle between the two vectors. Use this function to show that vectors \(U\) and \(W\) are orthogonal.
## Code solution here
Exercise 8: Given that \(||X+Y|| = 3\), \(X\cdot Y = 2\), find \(||X-Y||\).
## Code solution here