12
Differentiation
I turn with terror and horror from this lamentable scourge of continuous functions with no derivatives.
— Charles Hermite
In the history of science, a few milestones are as significant as inventing the wheel. Even among these, differentiation is a highlight. With the invention of calculus, Newton created mechanics as we know it. Differentiation is all over science and engineering, and as it turns out, it’s a key component of machine learning as well.
Why? Because of optimization! As it turns out, the extremal points of a function can be characterized in terms of their derivative, and these extremal points can be iteratively found via gradient descent.
In this chapter, we’ll learn what differentiation is, what its origins are, and most importantly, how to use it in practice. Let’s go!
