Classification Evaluation
Project workbook · Path 71

Evaluate a binary classifier with a confusion matrix, precision, recall and a decision-cost comparison.

The case

A fictional system flags help requests for manual review. On a labeled sample of 100 requests it produces 18 true positives, 12 false positives, 6 false negatives and 64 true negatives. A second candidate produces 22, 20, 2 and 56 respectively. No classifier is deployed by this exercise.

CandidateTPFPFNTN
A1812664
B2220256
STAGE 1

Define the positive class

Positive means the condition your evaluation counts as the target, not a favorable outcome. Define the target and label source before counting errors. Keep a held-out evaluation sample separate from training decisions. This exercise starts with aggregate counts, so it cannot inspect individual label quality or data leakage.

Your task

Write the positive-class definition and reconstruct each 2 × 2 matrix.

Required output

Evaluation contract

Record your result and evidence here or in a separate file.

Review gate

Both matrices contain 100 cases and 24 actual positives.

If the gate is not met

Sum by actual class before interpreting predictions.

STAGE 2

Calculate complementary metrics

Precision asks how many flagged cases are true positives; recall asks how many actual positives are found. Their denominators differ. A zero denominator needs an explicit policy rather than a silent division failure. Report counts alongside rates to keep the sample size visible.

Your task

Calculate accuracy, precision, recall and F1 for candidate A.

Required output

Metric table

Record your result and evidence here or in a separate file.

Review gate

A has accuracy 82%, precision 60%, recall 75% and F1 about 66.67%.

If the gate is not met

Write each fraction using the four cell counts.

STAGE 3

Compare decision costs

A cost model translates the trade-off into the case objective. Here assign 1 review unit to each false positive and 5 missed-case units to each false negative. These are hypothetical teaching costs, not universal weights. Do not add costs for correct predictions unless the model defines them.

Your task

Calculate error cost for both candidates and compare with accuracy.

Required output

Cost comparison

Record your result and evidence here or in a separate file.

Review gate

A costs 42 units; B costs 30 even though B accuracy is 78%.

If the gate is not met

Keep the cost assumptions next to the recommendation.

STAGE 4

Inspect uncertainty and slices

Aggregate metrics can hide weak performance on a subgroup or changed data. This tiny sample does not establish long-run reliability. Specify useful slices and minimum evidence before comparison, while avoiding conclusions about groups for which no counts exist. A repeated evaluation needs consistent definitions and versioned labels.

Your task

Draft a slice-evaluation table with no invented slice results.

Required output

Evaluation extension plan

Record your result and evidence here or in a separate file.

Review gate

Every unmeasured slice is marked not evaluated.

If the gate is not met

Replace guessed subgroup values with a data request.

STAGE 5

Write a bounded recommendation

A model choice is conditional on the objective, label quality and evaluation sample. State which candidate is preferable under the supplied cost rule and what would trigger reconsideration. Distinguish the evaluated candidate from any production decision; the exercise provides no deployment, human review capacity or monitoring evidence.

Your task

Write a one-page evaluation report with a cost-sensitivity question.

Required output

Decision report

Record your result and evidence here or in a separate file.

Review gate

The report recommends B only under the stated case costs.

If the gate is not met

Add the assumptions that would make the recommendation change.

Final project

Deliver the matrices, metric calculator, cost comparison, zero-denominator policy and bounded report. Test a no-positive-prediction case and explain the chosen output for undefined precision.

Critical condition

Metric denominators and the positive class are explicit; no unmeasured slice or deployment outcome is claimed.

CriterionPointsEvidence
Evidence and correctness30Calculations, source use and results match the supplied case; limitations are honest.
Required artifacts30All five stage outputs and the capstone handoff are present and inspectable.
Reasoning and decisions20Choices, assumptions, alternatives and the case-specific critical condition are explained.
Reproducibility10A reviewer can follow the stated steps and distinguish expected from observed results.
Clear communication10Labels, units, structure and evidence references make the work easy to review.

Score at least 80/100, supply every required artifact, and satisfy the path-specific critical condition. Rework missing evidence before marking complete.

Worked reference

Candidate A: precision 18/30 = 60%, recall 18/24 = 75%, F1 = 36/54 ≈ 66.67%. Its error cost is 12 + 5×6 = 42. Candidate B costs 20 + 5×2 = 30. Lower accuracy does not automatically mean worse utility under this cost rule.

Boundaries

This evaluates supplied counts, not a trained or deployed model, and does not establish causal fairness.