FrameworkDecision principleArtificial intelligenceDecision-making
Precision against recall: which error you can afford is a business decision
Kimberly Fessel · Precision vs Recall, With a Clear Example
The idea
A classifier can be wrong in two directions: flagging something that is not there, or missing something that is. Precision measures how many of its flags were right; recall measures how many of the real cases it caught. Raising one usually lowers the other. Kimberly Fessel fixes the two definitions in place with a clear example and then works the trade-off. The point for anyone who uses a model rather than builds one is that the trade-off is not technical. A fraud system that misses cases and one that blocks good customers fail differently, and choosing between them is a decision about costs that the business has to own.
Decide which mistake is more expensive before tuning a model; the threshold is a business choice the statistics only express.
When it works
Any model used for screening, fraud, credit or hiring, and the interview question of how you would evaluate a classifier.
When it may fail
Costs are often unknown or contested, and some errors have consequences money does not capture. The frame forces the question; it does not settle it.
How to use it
For a classifier in use, write the cost of a false positive and a false negative in money. Then say where the threshold should sit and why, before anyone shows you an accuracy figure.
Where it came from
Video Precision vs Recall, With a Clear Example · Kimberly Fessel
What the source says. Kimberly Fessel fixes precision and recall in place with a clear example, then works the trade-off between them.
Connected ideas
Says something similar
A model trained on a biased outcome will reproduce it at scaleA model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.
Says something similar
Without evaluation you have a demo, not a systemBuild the test set before the feature, measure every change against it, and keep a person in the loop where the metric cannot be trusted.
Concepts
Put it to work on L3VLUP
More like this
- Post-mortemFounders and companiesDocumentary
A board that accepts a narrative has stopped doing diligence
A compelling story is a claim to test, and the test for a technology is watching it work.
From The Inventor: Out for Blood in Silicon Valley · 2019
- FrameworkInvestingFilm
A cheap measurable signal beats an expensive intuitive one at scale
The market misprices what is hard to measure, so a consistent cheap signal can beat an expensive expert.
From Moneyball · 2011
- Decision principleDecision-makingBook
A good decision and a good outcome are separate things
Judge a decision by what was knowable when it was made, and keep the outcome in a separate column.
From Thinking in Bets · Annie Duke
- TechniqueProductVideo located
A metrics question is a structured hunt: north star, guardrails, then the segment
Clarify the metric, protect the guardrails, then decompose the drop by segment until it is isolated; show the process, not a guess.
From Google Product Manager Metrics Interview: A Worked Mock · Aakash Gupta