Calculate Weighted Accuracy Python

An essential tool for evaluating machine learning model performance, especially with imbalanced datasets.

Calculate Weighted Accuracy

Confusion Matrix and Derived Metrics

Metric	Value	Formula
True Positives (TP)	—	–
True Negatives (TN)	—	–
False Positives (FP)	—	–
False Negatives (FN)	—	–
Sensitivity (Recall)	—	TP / (TP + FN)
Specificity	—	TN / (TN + FP)
Overall Accuracy	—	(TP + TN) / Total
Weighted Accuracy	—	(Sensitivity + Specificity) / 2

What is Weighted Accuracy (Python)?

Weighted accuracy, in the context of machine learning and Python, is a metric used to evaluate the performance of classification models, particularly when dealing with imbalanced datasets. Unlike standard accuracy, which treats all misclassifications equally, weighted accuracy considers the relative importance or cost associated with different types of errors. In its simplest form, often referred to as balanced accuracy or simply mean of sensitivity and specificity, it provides a more nuanced view than raw accuracy when class distributions are uneven.

The need for weighted accuracy arises frequently in real-world applications. For example, in medical diagnoses, a false negative (failing to detect a disease) might have far more severe consequences than a false positive (incorrectly diagnosing a healthy patient). In fraud detection, a false negative (missing a fraudulent transaction) is significantly more costly than a false positive (flagging a legitimate transaction). Python libraries like Scikit-learn provide tools to calculate various performance metrics, but understanding the underlying concepts, like weighted accuracy, is crucial for effective model evaluation and selection.

Who Should Use It?

• Data scientists and machine learning engineers working with classification problems.
• Anyone evaluating models on datasets where class imbalance is present.
• Practitioners needing a performance metric that accounts for the differential cost of errors.

Common Misconceptions about Weighted Accuracy:

• Misconception: Weighted accuracy is always the best metric.
  Reality: The choice of metric depends heavily on the specific problem and the costs associated with different errors. Other metrics like Precision, Recall, F1-score, or AUC might be more appropriate in certain scenarios.
• Misconception: Weighted accuracy is complex to calculate.
  Reality: The core concept (average of sensitivity and specificity) is straightforward, and Python tools make implementation easy.
• Misconception: Weighted accuracy is synonymous with precision or recall.
  Reality: While related, weighted accuracy specifically averages sensitivity (recall for the positive class) and specificity (recall for the negative class).

Weighted Accuracy (Python) Formula and Mathematical Explanation

The most common interpretation of weighted accuracy in a binary classification context is the average of the model's sensitivity and specificity. This metric is also often referred to as balanced accuracy.

Let's break down the components:

First, we need to understand the components of a confusion matrix:

• True Positives (TP): The number of instances correctly predicted as positive.
• True Negatives (TN): The number of instances correctly predicted as negative.
• False Positives (FP): The number of instances incorrectly predicted as positive (predicted positive, but actually negative – Type I error).
• False Negatives (FN): The number of instances incorrectly predicted as negative (predicted negative, but actually positive – Type II error).

From these, we derive:

• Sensitivity (Recall or True Positive Rate): The proportion of actual positives that were correctly identified.
  Formula: Sensitivity = TP / (TP + FN)
• Specificity (True Negative Rate): The proportion of actual negatives that were correctly identified.
  Formula: Specificity = TN / (TN + FP)

The Overall Accuracy metric is calculated as:

Overall Accuracy = (TP + TN) / (TP + TN + FP + FN)

Finally, the Weighted Accuracy (Balanced Accuracy) is the average of Sensitivity and Specificity:

Weighted Accuracy = (Sensitivity + Specificity) / 2

This formula gives equal weight to the performance on the positive and negative classes. If the dataset is perfectly balanced and the model performs equally well on both classes, the weighted accuracy will be similar to the overall accuracy. However, with imbalanced classes, it provides a more reliable picture of performance.

Variables Table

Variable	Meaning	Unit	Typical Range
TP	True Positives	Count	≥ 0
TN	True Negatives	Count	≥ 0
FP	False Positives	Count	≥ 0
FN	False Negatives	Count	≥ 0
Sensitivity	True Positive Rate	Proportion	[0, 1]
Specificity	True Negative Rate	Proportion	[0, 1]
Overall Accuracy	Correct Predictions / Total Predictions	Proportion	[0, 1]
Weighted Accuracy	(Sensitivity + Specificity) / 2	Proportion	[0, 1]

Practical Examples (Real-World Use Cases)

Example 1: Medical Diagnosis (Imbalanced Dataset)

Consider a model designed to detect a rare disease. Out of 1000 patients tested:

• 800 are healthy (Negative Class)
• 200 have the disease (Positive Class)

The model correctly identifies 180 patients with the disease and misses 20.

It correctly identifies 750 healthy patients and incorrectly flags 50 healthy patients as having the disease.

Inputs:

• True Positives (TP): 180
• False Negatives (FN): 20 (200 actual positive – 180 predicted positive)
• True Negatives (TN): 750
• False Positives (FP): 50 (800 actual negative – 750 predicted negative)

Calculations:

• Sensitivity = 180 / (180 + 20) = 180 / 200 = 0.90
• Specificity = 750 / (750 + 50) = 750 / 800 = 0.9375
• Overall Accuracy = (180 + 750) / 1000 = 930 / 1000 = 0.93
• Weighted Accuracy = (0.90 + 0.9375) / 2 = 1.8375 / 2 = 0.91875

Interpretation: The overall accuracy is 93%, which sounds high. However, the weighted accuracy is 91.875%. While still good, it highlights that the model's performance on the negative class (Specificity) is slightly better than on the positive class (Sensitivity). This is important because the positive class (having the disease) is rarer, and detecting it correctly (high sensitivity) is crucial. The weighted accuracy gives a more balanced view than overall accuracy in this imbalanced scenario.

Example 2: Fraud Detection System

A financial institution uses a model to detect fraudulent transactions. Out of 5000 transactions:

• 4800 are legitimate (Negative Class)
• 200 are fraudulent (Positive Class)

The model correctly flags 150 fraudulent transactions but misses 50.

It correctly classifies 4700 legitimate transactions and incorrectly flags 100 legitimate transactions as fraudulent.

Inputs:

• True Positives (TP): 150
• False Negatives (FN): 50 (200 actual positive – 150 predicted positive)
• True Negatives (TN): 4700
• False Positives (FP): 100 (4800 actual negative – 4700 predicted negative)

Calculations:

• Sensitivity = 150 / (150 + 50) = 150 / 200 = 0.75
• Specificity = 4700 / (4700 + 100) = 4700 / 4800 = 0.9792
• Overall Accuracy = (150 + 4700) / 5000 = 4850 / 5000 = 0.97
• Weighted Accuracy = (0.75 + 0.9792) / 2 = 1.7292 / 2 = 0.8646

Interpretation: The overall accuracy of 97% might suggest excellent performance. However, the weighted accuracy drops significantly to 86.46%. This highlights a critical issue: while the model is very good at identifying legitimate transactions (high specificity), it struggles to detect actual fraud (lower sensitivity). Missing fraudulent transactions (FN) can be extremely costly. In this case, focusing solely on overall accuracy would be misleading, and improving the model's sensitivity would be a priority.

How to Use This Weighted Accuracy Calculator

Our Weighted Accuracy Calculator is designed to be intuitive and provide immediate insights into your classification model's performance, especially for imbalanced datasets. Here's how to use it effectively:

1. Gather Your Confusion Matrix Data: Before using the calculator, you need the four key values from your model's confusion matrix: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).
2. Input the Values: Enter these four numbers into the corresponding input fields: "True Positives (TP)", "True Negatives (TN)", "False Positives (FP)", and "False Negatives (FN)".
3. Calculate: Click the "Calculate" button. The calculator will instantly process the inputs.
4. Review the Results:
   • Primary Result (Weighted Accuracy): The most prominent value displayed is the Weighted Accuracy (Balanced Accuracy), presented in a large, highlighted format. This is your primary performance metric, especially for imbalanced classes.
   • Intermediate Values: You'll also see the calculated Sensitivity, Specificity, and Overall Accuracy. These provide context and allow for a more detailed analysis.
   • Chart: The dynamic chart visually compares the different accuracy metrics, making it easy to spot discrepancies between overall accuracy and balanced performance.
   • Table: The confusion matrix table provides a detailed breakdown of the input values, along with the formulas used to derive each metric.
5. Interpret the Findings: Compare the Weighted Accuracy to the Overall Accuracy. A significant difference suggests class imbalance issues. Analyze the individual Sensitivity and Specificity values to understand where your model excels and where it struggles. For instance, if detecting the positive class is critical (e.g., disease detection), prioritize high Sensitivity, even if it slightly reduces Specificity.
6. Reset or Copy: Use the "Reset" button to clear the fields and start over with new values. Use the "Copy Results" button to copy the key metrics and assumptions to your clipboard for reporting or documentation.

Decision-Making Guidance:

• High Weighted Accuracy, Similar to Overall Accuracy: Your model performs well and consistently across classes, even with potential imbalance.
• High Overall Accuracy, Lower Weighted Accuracy: Your model is performing much better on the majority class than the minority class. You need to focus on improving performance for the minority class (e.g., through resampling techniques, cost-sensitive learning, or feature engineering).
• Low Sensitivity: The model is poor at identifying positive instances. Critical for problems like disease detection or fraud detection.
• Low Specificity: The model is poor at identifying negative instances. Can lead to many false alarms in spam detection or security systems.

Use the insights gained from this calculator to guide your model improvement strategies and make informed decisions about model deployment.

Key Factors That Affect Weighted Accuracy Results

Several factors can influence the weighted accuracy (balanced accuracy) of a classification model. Understanding these is key to interpreting results and improving model performance:

1. Class Imbalance: This is the most significant factor. When one class has far more samples than others, overall accuracy can be misleadingly high. Weighted accuracy is specifically designed to mitigate this by averaging performance across classes. A highly imbalanced dataset will almost always result in a lower weighted accuracy compared to overall accuracy if the model favors the majority class.
2. Model Performance on Minority Class: Weighted accuracy directly reflects how well the model identifies the positive class (Sensitivity) and the negative class (Specificity). If the model struggles to correctly classify instances of the minority class (leading to low Sensitivity for positive class or low Specificity for negative class if that's the minority), the weighted accuracy will suffer disproportionately compared to overall accuracy.
3. Choice of Evaluation Metric: While weighted accuracy is valuable, it might not be the ultimate goal. Depending on the business objective, maximizing precision, recall (sensitivity), F1-score, or AUC might be more important. For instance, in a scenario where false positives are extremely detrimental (e.g., incorrectly diagnosing a severe disease leading to unnecessary treatment), a high specificity might be prioritized over a balanced score.
4. Feature Engineering and Selection: The quality and relevance of the input features significantly impact a model's ability to distinguish between classes. Poor features lead to poor classification, affecting TP, TN, FP, and FN, and thus all derived metrics including weighted accuracy. Effective feature engineering can dramatically improve both sensitivity and specificity.
5. Hyperparameter Tuning: Model hyperparameters (e.g., regularization strength in logistic regression, depth of trees in random forests) control the model's complexity and learning process. Improper tuning can lead to underfitting or overfitting, negatively impacting performance on both majority and minority classes, thereby reducing weighted accuracy.
6. Data Quality and Noise: Errors, outliers, or noise in the training data can confuse the model, leading to misclassifications. This affects the accuracy of the confusion matrix components. If noise disproportionately affects the minority class, it can heavily skew the weighted accuracy downward.
7. Threshold Selection (for probabilistic models): Many classification models output probabilities. The decision threshold (often defaulted to 0.5) determines the final class prediction. Adjusting this threshold can trade off sensitivity and specificity. While weighted accuracy averages these, understanding the underlying trade-off driven by the threshold is crucial for optimization based on specific error costs.

Frequently Asked Questions (FAQ)

Q1: What is the difference between Overall Accuracy and Weighted Accuracy?
A1: Overall Accuracy calculates the total correct predictions (TP + TN) divided by all predictions. Weighted Accuracy, often called Balanced Accuracy, is the average of Sensitivity (Recall) and Specificity. It's more reliable for imbalanced datasets because it equally considers performance on both positive and negative classes.

Q2: When should I use Weighted Accuracy over Overall Accuracy?
A2: You should strongly consider Weighted Accuracy when your dataset has a significant class imbalance (e.g., one class has 10x more samples than the other). In such cases, Overall Accuracy can be artificially inflated by correctly predicting the majority class, masking poor performance on the minority class.

Q3: Is Weighted Accuracy the same as the F1-Score?
A3: No. Weighted Accuracy is the average of Sensitivity and Specificity. The F1-Score is the harmonic mean of Precision and Sensitivity (Recall). While both are useful for imbalanced data, they measure different aspects of performance. F1-Score heavily emphasizes correct positive predictions (Precision), while Weighted Accuracy balances performance on positive and negative classes.

Q4: Can Weighted Accuracy be 100%?
A4: Yes. Weighted Accuracy can be 100% (or 1.0) if both Sensitivity and Specificity are 100%. This means the model perfectly identifies all positive instances and all negative instances without any errors.

Q5: My weighted accuracy is much lower than my overall accuracy. What does this mean?
A5: This typically indicates that your model performs significantly better on the majority class than on the minority class. The overall accuracy is being boosted by high performance on the abundant class, while the weighted accuracy reveals poor performance on the scarce class. You should investigate why the minority class is being misclassified.

Q6: How does Python's Scikit-learn calculate balanced_accuracy_score?
A6: Scikit-learn's `balanced_accuracy_score` function computes exactly what we've described: the average of recall obtained on each class. For binary classification, this is equivalent to `(Sensitivity + Specificity) / 2`.

Q7: What if I have more than two classes (multi-class classification)?
A7: For multi-class problems, "weighted accuracy" can have different interpretations. A common approach is to calculate sensitivity and specificity for each class individually and then average them, possibly weighted by class support (number of true instances for each label). Scikit-learn's `balanced_accuracy_score` handles this by averaging the recall obtained on each class.

Q8: Should I always aim for the highest possible Weighted Accuracy?
A8: Not necessarily. The "best" metric depends on the specific application's goals and the costs of different types of errors. If, for example, false negatives are far more costly than false positives, you might prioritize sensitivity even if it lowers the overall weighted accuracy slightly. Always align your metric choice with your business or project objectives.