Evaluate a binary classifier with a confusion matrix, precision, recall and a decision-cost comparison.
A fictional system flags help requests for manual review. On a labeled sample of 100 requests it produces 18 true positives, 12 false positives, 6 false negatives and 64 true negatives. A second candidate produces 22, 20, 2 and 56 respectively. No classifier is deployed by this exercise.
| Candidate | TP | FP | FN | TN |
|---|---|---|---|---|
| A | 18 | 12 | 6 | 64 |
| B | 22 | 20 | 2 | 56 |
Positive means the condition your evaluation counts as the target, not a favorable outcome. Define the target and label source before counting errors. Keep a held-out evaluation sample separate from training decisions. This exercise starts with aggregate counts, so it cannot inspect individual label quality or data leakage.
Write the positive-class definition and reconstruct each 2 × 2 matrix.
Evaluation contract
Record your result and evidence here or in a separate file.
Both matrices contain 100 cases and 24 actual positives.
Sum by actual class before interpreting predictions.
Precision asks how many flagged cases are true positives; recall asks how many actual positives are found. Their denominators differ. A zero denominator needs an explicit policy rather than a silent division failure. Report counts alongside rates to keep the sample size visible.
Calculate accuracy, precision, recall and F1 for candidate A.
Metric table
Record your result and evidence here or in a separate file.
A has accuracy 82%, precision 60%, recall 75% and F1 about 66.67%.
Write each fraction using the four cell counts.
A cost model translates the trade-off into the case objective. Here assign 1 review unit to each false positive and 5 missed-case units to each false negative. These are hypothetical teaching costs, not universal weights. Do not add costs for correct predictions unless the model defines them.
Calculate error cost for both candidates and compare with accuracy.
Cost comparison
Record your result and evidence here or in a separate file.
A costs 42 units; B costs 30 even though B accuracy is 78%.
Keep the cost assumptions next to the recommendation.
Aggregate metrics can hide weak performance on a subgroup or changed data. This tiny sample does not establish long-run reliability. Specify useful slices and minimum evidence before comparison, while avoiding conclusions about groups for which no counts exist. A repeated evaluation needs consistent definitions and versioned labels.
Draft a slice-evaluation table with no invented slice results.
Evaluation extension plan
Record your result and evidence here or in a separate file.
Every unmeasured slice is marked not evaluated.
Replace guessed subgroup values with a data request.
A model choice is conditional on the objective, label quality and evaluation sample. State which candidate is preferable under the supplied cost rule and what would trigger reconsideration. Distinguish the evaluated candidate from any production decision; the exercise provides no deployment, human review capacity or monitoring evidence.
Write a one-page evaluation report with a cost-sensitivity question.
Decision report
Record your result and evidence here or in a separate file.
The report recommends B only under the stated case costs.
Add the assumptions that would make the recommendation change.
Deliver the matrices, metric calculator, cost comparison, zero-denominator policy and bounded report. Test a no-positive-prediction case and explain the chosen output for undefined precision.
Metric denominators and the positive class are explicit; no unmeasured slice or deployment outcome is claimed.
| Criterion | Points | Evidence |
|---|---|---|
| Evidence and correctness | 30 | Calculations, source use and results match the supplied case; limitations are honest. |
| Required artifacts | 30 | All five stage outputs and the capstone handoff are present and inspectable. |
| Reasoning and decisions | 20 | Choices, assumptions, alternatives and the case-specific critical condition are explained. |
| Reproducibility | 10 | A reviewer can follow the stated steps and distinguish expected from observed results. |
| Clear communication | 10 | Labels, units, structure and evidence references make the work easy to review. |
Score at least 80/100, supply every required artifact, and satisfy the path-specific critical condition. Rework missing evidence before marking complete.
Candidate A: precision 18/30 = 60%, recall 18/24 = 75%, F1 = 36/54 ≈ 66.67%. Its error cost is 12 + 5×6 = 42. Candidate B costs 20 + 5×2 = 30. Lower accuracy does not automatically mean worse utility under this cost rule.
This evaluates supplied counts, not a trained or deployed model, and does not establish causal fairness.