Julien Florkin

AI & Automation

AI in Machine Vision: What It Must Prove Before You Trust It

Imagine a packing line just after a shift change. A new vision system begins flagging far more boxes than it did during the morning demonstration.

Julien Florkin 9 minute read
AI in Machine Vision

Imagine a packing line just after a shift change. A new vision system begins flagging far more boxes than it did during the morning demonstration. Some may be defective. Others may only look different under the new lighting. The supervisor has seconds to decide whether to stop the line, ignore the alerts, or let the system reject items automatically.

That moment contains the real question about AI in machine vision. It is not whether a model can recognize an image. It is whether the complete system can produce a signal that people can act on safely, consistently, and with a clear way to recover when it is wrong.

The useful starting point is therefore narrow: name one visual task, one action triggered by the output, and the consequences of both kinds of error. Only then can a demo become an operating test.

In this article

A vision model does not make the decision

Machine vision is a chain. Light reaches an object. Optics and a camera turn that view into data. Software prepares the image. A model or rule identifies a pattern. Other software translates the output into an alert or action. A person, machine, or workflow then absorbs the consequence.

AI may improve one part of that chain. A learned model can sometimes handle more visual variation than a rigid rule. If you want the technical background, this explanation of AI and deep learning provides a useful continuation. But model sophistication does not repair poor lighting, a moving camera, damaged labels, delayed data, a badly chosen threshold, or an action nobody owns.

This is why an impressive image label is not yet an operating decision. "Possible seal defect: 82%" leaves several questions unanswered:

  • Is 82% high enough to stop, reject, or inspect?
  • What happens to a sound item that is rejected?
  • What happens if a real defect is missed?
  • Who can override the result, and what evidence do they see?
  • How is the outcome fed back into monitoring or retraining?

The NIST AI Risk Management Framework Core makes the same system-level distinction in formal terms: intended purpose, task, context, costs of errors, human oversight, testing, and monitoring all need to be defined. The model sits inside that arrangement; it is not the arrangement.

Start with the harm of being wrong

Teams often ask for an accuracy figure before agreeing on which error matters. That reverses the decision.

Suppose the system is looking for a damaged seal. A false positive rejects a sound package. The cost may be wasted material, extra inspection, or a production delay. A false negative lets a damaged package continue. The consequence may be a customer complaint, spoilage, injury, or another harm depending on what the package contains. Those two errors are not interchangeable, and their relative importance changes with the task.

Even the same camera output can support different decisions. A low-confidence alert may be acceptable if it sends an item to a trained inspector. It may be unacceptable if it triggers automatic disposal or stops a safety-critical process. "Human in the loop" is not a magic phrase either: an exhausted operator facing hundreds of alerts has a very different role from a specialist reviewing a small, prioritized exception queue.

Before discussing a model threshold, write down four things:

  1. the object or event the system must detect;
  2. the action that follows a positive and a negative result;
  3. who bears the cost of each wrong result; and
  4. the maximum delay before a human or system must respond.

Now a metric can serve a decision. Without that map, a single accuracy number hides the errors that matter most. NIST's Core specifically calls for the costs of errors to be examined against the organization's risk tolerance, and for performance to be demonstrated under conditions similar to deployment rather than assumed from a general benchmark.

Test the whole scene, not just the model

Return to the shift change. The model may be unchanged while the scene is not. The lighting angle changed. A supplier's cardboard is slightly darker. A lens has dust on it. The belt is running faster. Staff have started placing boxes closer together. Any of these can alter what the camera sees or how the output reaches the next step.

A representative test therefore samples the operating world, not only a convenient folder of images. It should cover the relevant objects, normal variation, rare but important cases, different shifts, expected environmental changes, borderline examples, and known failure conditions. The results should be separated by conditions that can change the decision. One aggregate score can conceal a serious weakness on a small but consequential subset.

The NIST AI RMF Measure guidance recommends assessing whether measurements generalize beyond their original context, comparing production results with pre-deployment tests, tracking drift and anomalies, testing likely stress conditions, and documenting the limits under which a system can fail safely. Those are useful principles for vision systems because the camera, scene, model, integration, and action can all drift in different ways.

Your test plan should consequently include more than model performance:

  • image quality and camera availability;
  • performance by meaningful operating condition;
  • end-to-end delay from image to action;
  • the volume and quality of alerts reaching people;
  • override, reinspection, and exception outcomes;
  • integration failures and safe fallback behavior; and
  • changes in objects, processes, suppliers, environments, or user behavior.

This broader view is consistent with the general artificial-intelligence system perspective: value and risk emerge from how a capability is used, not from the label attached to it.

Design human recovery before automation

Human oversight works only when a person has enough time, authority, information, and skill to act differently from the model. If every override is discouraged, if the evidence is invisible, or if pausing the line requires executive approval, the supposed safeguard is decorative.

Define recovery while the pilot is still easy to change:

  • Which results go straight through, which require review, and which stop the process?
  • What does the reviewer see besides the model's conclusion?
  • Who owns an unresolved exception?
  • Can the responsible person override, reverse, or pause the output?
  • What happens safely if the camera, model, network, or downstream system fails?
  • How are complaints, near misses, overrides, and corrected labels recorded?

For high-risk AI systems within the scope of the EU AI Act, Article 14 describes human oversight as the ability to understand relevant capabilities and limitations, detect anomalies, avoid over-reliance, interpret output, disregard or reverse it, and interrupt a system when appropriate. The exact legal obligations depend on the use, role, timing, and jurisdiction, so this article does not classify a deployment or provide legal advice. The practical lesson is wider: recovery must be operationally possible, not merely promised.

Video can also observe people, not only products. If workers, customers, faces, license plates, or behavior may be captured, make the privacy purpose, field of view, access, retention, security, and necessity explicit before data collection. The European Data Protection Board's final guidance on video devices is a primary reference for personal-data processing in the European context. Biometric, clinical, employment, mobility, security, and public-space uses deserve a narrower scope and qualified legal, safety, privacy, or domain review before any deployment decision.

Use the visual-decision trust card

Bring the proposal back to one page. Complete this card with the people who own the process, the technology, and the consequences.

Task and object. What exactly must be seen, measured, classified, or tracked? What is outside scope?

Action triggered. Does the output inform, route, flag, reject, stop, or authorize something? Is the action reversible?

Error consequences. Who or what is affected by a false positive, a false negative, an uncertain result, and a delayed result?

Representative conditions. Which objects, environments, shifts, devices, suppliers, populations, and rare cases must the evidence cover?

Threshold and evidence. Which measures support the operating decision? How do results vary across important conditions? What baseline is the system compared with?

Human recovery. Who reviews uncertainty, owns exceptions, overrides an output, and pauses the system? What information and training do they need?

Monitoring. Which signals reveal drift, camera problems, integration failures, changing alert volumes, or worsening outcomes? How quickly must someone respond?

Privacy and security boundary. What data is captured, who can access it, how long is it kept, and which attack or misuse paths must be addressed?

Stop rule. Which result, incident, unknown, or operating change pauses the pilot or sends the design back for review?

A supplier may provide part of this evidence. The accountable organization still needs to connect it to its own scene, workflow, people, risk tolerance, and legal obligations. A result from another site or a curated dataset is a reason to investigate, not proof that the system is ready here.

Choose a bounded pilot, redesign, or stop

A useful pilot is not a small launch. It is a controlled attempt to reduce a named uncertainty.

Start in observation or shadow mode when the consequence permits it: let the system make predictions without controlling the workflow, then compare its output with reliable ground truth and the current process. Predefine the conditions, samples, error measures, review effort, duration, ownership, and stop rule. Include routine variation and deliberately test known weak conditions. Keep a safe fallback available.

At the end, the decision is not simply "Did the model work?" Ask instead:

  • Did the complete chain improve the intended decision under representative conditions?
  • Were the important errors within the agreed tolerance?
  • Could people understand, challenge, and recover from the output?
  • Did the alert and exception workload remain usable?
  • Can monitoring detect meaningful change before harm accumulates?
  • Are the privacy, security, safety, and accountability boundaries supportable?

Proceed only as far as the evidence supports. A narrow assistive use may pass while automatic rejection does not. The camera position may need redesign. The threshold may need to vary by product. The process may need better labels or maintenance before any AI is useful. In some cases, a deterministic rule, a physical fixture, improved lighting, or a clearer manual check will solve the problem more reliably.

That is not a failure of ambition. It is what disciplined automation looks like. The decisive question is the one the supervisor needed at shift change: What must this system prove in our conditions before a person is asked to trust its signal?

Sources

Related reading