why do we use LOG-LOSS as an evaluation metric instead of MSE?

Logarithmic Loss (LogLoss), also known as Cross-Entropy Loss is commonly used as an evaluation metric for classification problems, especially in machine learning and deep learning, because it is well-suited for assessing the performance of models that output probabilities. In contrast, Mean Squared Error (MSE) is typically used for regression problems and Log Loss is used for classification problems. Here’s why LogLoss is preferred for classification tasks:

1. ​Probability Interpretability: LogLoss is based on the probabilities predicted by a classification model. In classification, you’re often interested in not just the class label buy also the model’s confidence in the labels. LogLoss takes into account the probability distribution of predicted classes, providing a more nuanced measure of performance.
2. Sensitivity to Probabilities: LogLoss is more sensitive to the quality of probabilities predicted by a classifier. It heavily penalizes confident but incorrect predictions. This sensitivity is crucial for tasks like binary and multiclass classification, where misclassifying certain samples can have significant consequences.
3. Gradient Descent Optimization: When training a classification model using gradient descent or similar optimization algorithms, LogLoss provides smooth gradients, making it easier to optimize. This is in contrast to MSE, which can have discontinuous gradients for classification problems, making optimization more challenging.
4. Information Theory: Logloss is rooted in information theory and has a clear interpretation as the cross-entropy between the true and predicted probability distributions. It measures how well the predicted probabilities match the true class probabilities, which is a fundamental aspect of classification tasks.
5. Common Standard: LogLoss has become a standard metric for evaluating classification models, making it easier to compare and benchmark different models and algorithms in the field. This consistency in evaluation metrics is valuable for researchers and practitioners.

however, it’s worth noting that the choice of evaluation metric should align with the specific goals of your problem. While LogLoss is widely used for classification, there may be cases where other metrics, such as MSE, are more appropriate. For example, if you are working on a regression-like task within a classification problem (e.g. predicting a continuous value associated with each class), MSE or other regression metrics might be more suitable.

To understand mathematically why Logarithmic Loss (LogLoss) is preferred over Mean Squared Error (MSE) in logistic regression, we can examine the nature of there two loss functions and their suitability for the task.

LogLoss (Cross-Entropy Loss):

In logistic regression, the goal is to model the probability that an input sample belongs to a particular class. Mathematically, this is represented as:

where,

p(y=1|x) is the probability of the positive class,

x represents the input features,

z is a linear combination of the input features and model parameters:

The LogLoss for a single sample can be defined as:

LogLoss = -(y log(p(y=1|x))+(1-y)log(1-p(y=1|X)))

where,

y is the true class label (0 for the negative class, 1 for the positive class).

p(y=1|x) is the predicted probability that the sample belongs to the positive class.

To derive the LogLoss, you can consider it as the negative log-likelihood of the true class labels given the predicted probabilities. The key mathematical property here is that LogLoss heavily penalizes confident but incorrect predictions, as the log of a value close to 0 or 1 approaches negative infinity.

MSE (Mean Squared Error):

In contrast, MSE is primarily used for regression tasks where the goal is to predict continuous values. For logistic regression, you would typically predict probabilities, not continuous values. Mathematically, MSE for a single sample is defined as:

MSE = (y- P(y=1|x))²

The key difference is that MSE measures the squared error between the true and predicted values, but it doesn’t take into account the probabilistic nature of logistic regression. Furthermore, it doesn’t penalize confident but incorrect predictions as heavily as LogLoss does.

Why LogLoss is Preferred:

LogLoss is preferred in logistic regression because it aligns with the probabilistic interpretation of logistic regression. It measures how well the predicted probabilities match the true class labels. It encourages the model to produce well-calibrated probabilities, which is essential in classification tasks where the relative importance of different types of errors (false positives vs. false negatives) can vary.