Scalar

A scalar is a simple numerical value. Variables or constants that take scalar values are denoted by an italic letter, like $x$ or $a$.

Vector

A vector $x$ is an ordered sequences of scalars $(x_1, x_2, \dots, x_N)$, called attributes or elements. Vectors can be visualized as arrows that point to some directions as well as points in a multi-dimensional space. The dimension in a vector is the j-th position in the vector.

A vector is denoted with a bold letter: $\textbf{x}$

Matrix

A matrix is a rectangular array of numbers arranged in rows and columns: e.g.:

Matrices are denoted with capital letters: $X, A$

Set

A set is an unordered collection of unique elements.

A set of numbers can be finite (include a fixed amount of values). Or it can be infinite and include all values in some interval.

Functions

A function is a relation that associates each element $x \in X$, the domain of the function, to a single element $y \in Y$, the codomain of the function.

A function usually is represented like $y = f(x)$, where $x$ is the input, or variable of the function, and $y$ is the output of the function.

A function $f$ has a local minimum in $c$ if for every $x$ in a neighborhood of $c$, we have $f(x) \ge f(c)$. The minimum of all local minima of a function is its global minimum.

A vector function is a function that returns a vector. It can have a scalar or vector argument.

A derivative $f'$ of a function $f$ is a function or a value that describes how fast $f$ grows (or decreases). If the derivative is a constant value, like 5 or −3 , then the function grows (or decreases) constantly at any point $x$ of its domain. if $f'$ is a function, then $f$ can grow at different paces in different regions of its domain. If $f'$ is positive, then $f$ will grow in that point, while if $f'$ is negative $f$ will decrease. Derivative of zero means the function is flat.

The process of finding the derivative of a function is called differentiation.

Gradient is the generalization of derivative for functions that take several inputs (or one input in the form of a vector or some other complex structure).

A gradient of a function is a vector of partial derivatives. You can look at finding a partial derivative of a function as the process of finding the derivative by focusing on one of the function’s inputs and by considering all other inputs as constant values.