Log Loss
Log loss, or logarithmic loss, is a generic term that refers to the negative log-likelihood of the true labels given a set of predicted probabilities.
- Binary Cross Entropy is also known as Binary Log Loss
- Categorical Cross Entropy is also known as Softmax Cross Entropy
Cross Entropy Loss
Cross-entropy loss, in a broad sense, measures the difference between two probability distributions.
while the terms “Cross-Entropy Loss” and “Log Loss” are used interchangeably, the specific context, such as binary or multi-class classification, may lead to the use of more specific names like “Binary Cross-Entropy Loss” or “Categorical Cross-Entropy Loss.”
Binary Cross Entropy Loss
In the context of binary classification, the Cross-Entropy loss is often referred to as binary cross-entropy loss. It measures the difference between the predicted probability distribution and the true distribution for a binary classification problem.
In the context of binary cross-entropy loss, y typically represents the actual label, which is a binary value indicating the correct class membership. Here’s a breakdown:
- y=1 if the true class is the positive class (e.g., the presence of an object, the occurrence of an event).
- y=0 if the true class is the negative class (e.g., the absence of an object, the non-occurrence of an event).
The Log in Log Loss
The choice of using the logarithm in log loss (or binary cross-entropy) is primarily motivated by mathematical and computational reasons. The log loss function has several desirable properties that make it suitable for training and optimizing models in the context of binary classification
- The logarithmic function is smooth and differentiable everywhere, allowing for gradient-based optimization methods. This makes it well-suited for training machine learning models using techniques like gradient descent.
- When dealing with probabilities, especially small probabilities close to zero or large probabilities close to one, using the logarithm helps avoid numerical instability. The log transformation compresses the range of values, making computations more stable.
- The log loss function is convex, which means that there is a unique global minimum. This property facilitates optimization, ensuring that optimization algorithms converge to a single, optimal solution.
- Taking the logarithm of the likelihood provides a natural way to penalize the model more when it makes confident incorrect predictions. This aligns with the intuitive notion that being confidently wrong should be more penalized than being uncertain.
Categorical Cross-Entropy Loss
In the context of multi-class classification, the Cross-Entropy Loss may refer to categorical cross-entropy loss.
The Other Variants of Log Loss
Sparse Categorical Cross-Entropy Loss: Similar to categorical cross-entropy but more memory-efficient when dealing with a large number of classes.
Kullback-Leibler Divergence (KL Divergence) Loss: Measures how one probability distribution diverges from a second, expected probability distribution.
Negative Log-Likelihood Loss (NLL Loss): Used in maximum likelihood estimation problems, often associated with softmax activation.
Weighted Cross-Entropy Loss: An extension of cross-entropy that introduces class weights to address the class imbalance.