Confidence

Confidence, in a machine learning context, refers to how certain a model reports itself to be about a specific prediction. A model's confidence is only meaningful if it's well-calibrated: a model that says it is 90% confident should be right about 90% of the time for predictions at that confidence level, not just report a high number regardless of actual accuracy.¹

Why calibration determines whether a confidence score can be trusted

A poorly calibrated model can report high confidence on predictions that are frequently wrong, which is why calibration, not just the raw confidence number, determines whether that confidence score is actually trustworthy enough to act on.

Where confidence applies

Not on the drawing; confidence is a property of a model's output for a specific part or situation, not something the part's requirements specify.

Common mistakes

Trusting a model's reported confidence number at face value without checking whether the model is actually well-calibrated; an uncalibrated model's confidence score can be misleading.

Is a model's confidence score always trustworthy?

Only if the model is well-calibrated; an uncalibrated model can report high confidence on predictions that are frequently wrong.

What does 'calibrated' mean here?

That a model's stated confidence level matches its actual accuracy rate at that level, such as being right 90% of the time when it reports 90% confidence.

Why does this matter for manufacturing AI?

Because a confidently wrong prediction that isn't flagged for review can lead to a costly error, which is why calibrated confidence, not just any confidence number, matters for trustworthy automation.