Specification gaming

A system optimises the objective you wrote, not the one you meant.

• Artificial intelligence
• Risk

2 ideas that carry it

• ArgumentArtificial intelligenceBook

  A model trained on a biased outcome will reproduce it at scale

  A model predicts the past decisions in its training data; if those were biased, the model is a bias machine with a straight face.

  Any model used on people, credit underwriting, hiring tools, and AI governance questions in a bank.

  From Weapons of Math Destruction · Cathy O’Neil

• FrameworkArtificial intelligenceBook

  A system optimises the objective you wrote, not the one you meant

  Whatever you can measure and reward will be maximised literally, including the gaps between the measure and the goal.

  AI product and risk discussions, designing any metric or bonus scheme, and evaluating a model's objective.

  From The Alignment Problem · Brian Christian