Inner Products

In this chapter we introduce the idea of an inner product, examine the additional structure that it imposes on a vector space, and look at some common applications. An inner product is an operation on two vectors in a vector space that is defined in such a way as to satisfy certain algebraic requirements. To begin, we will focus only on one specific inner product defined for vectors in Rn. Later in the chapter we will consider other examples of inner products in Rn.

The dot product is the most common example of an inner product. For vectors U and V in Rn, the dot product is written as UV, and is defined in the following way.

UV=nuivi=u1v1+u2v2+...+unvn

For example, we show the computation for U and V in R4.

U=[2123]V=[1450]
UV=(2)(1)+(1)(4)+(2)(5)+(3)(0)=8

This is exactly the sort of calculation that is carried out in matrix multiplications. In fact it is possible to view matrix products as collections of dot products between the rows and columns of the two matrices. Suppose for example that we have the following matrix product with U as the second row of A and V as the third column of B.

A[1102212331212001]B=[111414015110]AB=[745168059112]

We see that (AB)23=UV, the dot product of the second row of A with the third column of B. In general the entry (AB)ij will be the dot product of the ith row of A with the jth column of B.

Taking advantage of this connection, we could also write the definition of the dot product by viewing the vectors as n×1 matrices, and making use of matrix transposes.

UV=UTV

Since U is an n×1 matrix, UT will be a 1×n matrix, and UTV will be a 1×1 matrix. In other words UTV will be a scalar. If we take this view, we will be able to compute inner products just the same as we would matrix products using NumPy.

import numpy as np

U = np.array([[2],[-1],[2],[3]])
V = np.array([[1],[4],[5],[0]])

dot_product = U.transpose()@V
print(dot_product)
[[8]]

Computing a dot product with this formula is correct and we would not notice any difference in doing the calculation on paper. There is however a slight technical inconvenience in using NumPy operations to apply this formula. The output produced by the code (dot_product) is not a true scalar, but in fact is another NumPy array object of size 1×1. This may have unintended consequences as we include this code in larger calculations. We can keep this in mind and simply reference the value in the 1×1 array with indices if need be.

print(dot_product[0,0])
8

Another possibility is that we write our own function, called DotProduct to provide us the dot product in a simple format. The following code is included in the laguide module for future use.

def DotProduct(U,V):
    # =============================================================================
    # U and V are NumPy arrays that represents vectors of dimension n x 1.
    # DotProduct returns the dot product of U and V
    # =============================================================================

    # Check shapes of U and V
    if (U.shape[1] != 1 or V.shape[1] != 1):
        print("Dot product only accepts column vectors.")
        return
    # Check shape of V
    if (U.shape[0] != V.shape[0]):
        print("Dot product only accepts column vectors of equal length.")
        return

    n = U.shape[0]
    product = 0
    
    for i in range(n):
        product += U[i,0]*V[i,0]

    return product

We can now call the function and receive a scalar as output.

print(DotProduct(U,V))
8

Geometric interpretations

The dot product provides a way to measure distances and angles in Rn. Again it is easiest to visualize these measures in R2.

The magnitude of a vector U, written as ||U||, is defined as (UU)1/2. In R2 it is plain to see that this definition corresponds to the Pythagorean Theorem.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}

ax.arrow(0,0,2,3,fc='b',ec='b',**options)

ax.text(1,2,'$U$')

ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
_images/Inner_Products_9_0.png
U=[23]||U||=(2)(2)+(3)(3)=13

Calculating the magnitude of a vector is common enough that it will be useful to write a function to carry it out. We will leave it as an exercise in this section, and will include the function Magnitude in the laguide module for use in future sections.

from laguide import Magnitude

U = np.array([[2],[3]])
print(Magnitude(U))
3.605551275463989

It is often useful to work with vectors that have magnitude equal to one. Such vectors are called unit vectors. We can construct a unit vector in the same direction as a vector U by multiplying U by the reciprocal of its magnitude. We will label this unit vector W.

fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}

ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,0.555,0.832,fc='b',ec='r',**options)

ax.text(1,2,'$U$')
ax.text(0.5,0.3,'$W$')


ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
_images/Inner_Products_13_0.png
W=113[23]

We can also use the definition of magnitude to define the distance between two vectors U and V as ||UV||, the magnitude of the difference. If we picture the vectors as arrows, UV goes from the tip of V to the tip of U. This is consistent with our arrow picture of vector addition since V+(UV)=U.

fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}

ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,4,2,fc='b',ec='b',**options)
ax.arrow(4,2,-2,1,fc='b',ec='r',**options)

ax.text(1,2,'$U$')
ax.text(3,1,'$V$')
ax.text(3,2.8,'$U-V$')

ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
_images/Inner_Products_16_0.png

With common identities from trigonometry, it is also possible to work out the angle between two vectors in terms of the dot product.

fig, ax = plt.subplots()
options = {"head_width":0.1, "head_length":0.2, "length_includes_head":True}

ax.arrow(0,0,2,3,fc='b',ec='b',**options)
ax.arrow(0,0,4,2,fc='b',ec='b',**options)

ax.text(1,2,'$U$')
ax.text(3,1,'$V$')
ax.text(1,1,'$\\theta$')

ax.set_xlim(0,5)
ax.set_xlabel('$x_1$')
ax.set_ylim(0,5)
ax.set_ylabel('$x_2$')
ax.set_aspect('equal')
ax.grid(True,ls=':')
_images/Inner_Products_18_0.png
cosθ=UV||U||||V||

It is worth noting the special case that arises when the angle between U and V is a right angle. In this case, since cosθ is zero, the dot product UV is also zero. When UV=0, the vectors U and V are said to be orthogonal. In upcoming sections we will learn the importance of orthogonality, and how it can be used to simplify certain calculations

Algebraic properties

If U, V, and W are any vectors in Rn, and k is a scalar, then the following statements involving the dot product are true.

  1. UV=VU

  2. U(V+W)=UV+UW

  3. (kU)V=k(UV)

  4. UU0

These algebraic properties are not difficult to prove using the definition of the dot product, but they are important for carrying out calculations with unspecified vectors. More importantly, it is these four properties that make the dot product an inner product.

Exercises

Exercise 1: Create a Python function named Magnitude that accepts as an argument vector in the form of a NumPy array object with shape n×1, and returns the magnitude of that vector.

## Code solution here.

Exercise 2: Use the random module of NumPy to generate random 4×1 vectors U, V, and W, and a random scalar k. Verify the algebraic properties of the dot product listed above. Try to make use of a conditional statement to check each statement.

## Code solution here.

Exercise 3: Let X and Y be the following vectors.

X=[312]Y=[420]

(a) Determine the angle between X and Y. (You will need the acos function in Python’s math module.)

## Code solution here.

(b) Find a vector in R3 that is orthogonal to the vector X. Verify your answer with a computation.

## Code solution here.

(c) Construct a unit vector Z such that ZY=||Y||. Verify your answer with a computation.

## Code solution here.

Exercise 4: Create a Python function named DifferenceMagnitude which takes two n×1 vectors as arguments and returns the magnitude of their difference. Use this function to determine ||XY|| where vectors X and Y are given below.

X=[312]Y=[420]
## Code solution here

Exercise 5: Create a Python function which takes an n×1 vector as its argument and returns the unit vector of the argument vector. Use the NumPy random module to generate a random 3×1 vector, and test this function on that vector.

## Code solution here

Exercise 6: Find a vector Y in R2 such that XY=||YX|| where the vector X is given below. Is the vector Y unique ? Verify your answer through a computation.

X=[11]
## Code solution here

Exercise 7: Create a Python function named Angle which takes two vectors as arguments and returns cosθ where θ is the angle between the two vectors. Use this function to show that vectors U and W are orthogonal.

U=[112]W=[201]
## Code solution here

Exercise 8: Given that ||X+Y||=3, XY=2, find ||XY||.

## Code solution here