---
title: How to design an AI proof of concept and decide whether to move forward
description: How to design an AI proof of concept that produces useful learning,
  supports fair comparisons and informs the decision to move forward.
author:
  name: Walter Jerusalinsky
  title: AI strategy and implementation
  url: https://www.linkedin.com/in/walter-jerusalinsky/
canonical: https://www.idoneos.com/en/applied-ai/proof-of-concept/
image: ../resources/testing.png
source: applied-ai/proof-of-concept/index.md
---

<p class="series-label"><a href="../">AI for real work</a> · 4 of 5 · Test</p>

# How to design an AI proof of concept and decide whether to move forward

<p class="article-byline">By <a href="https://www.linkedin.com/in/walter-jerusalinsky/">Walter Jerusalinsky</a> · AI strategy and implementation</p>

A simple demonstration lets us observe that a tool produces a result. But to decide whether an investment is worthwhile, we need to understand more: the quality it achieves, how much work it requires and under which conditions we can expect similar behaviour.

A *proof of concept* studies a defined uncertainty. It can help us find out whether we can locate the necessary information, achieve acceptable quality or obtain an improvement after accounting for review effort.

Designing the proof of concept starts by defining the main decision we need to make.

## Formulate a question that can be answered

“See whether it works” leaves what we will observe undefined. We can make the question more precise, for example: does this AI assistance let us complete a task in less total time while preserving the required quality?

The hypothesis must define an acceptable result. If we are evaluating a synthesis, we need to establish what information it must retain, which errors require its rejection and how it will be checked. If we are evaluating a recommendation, we must make explicit which conditions make it relevant.

Before running the test, it helps to agree on which results would justify moving forward, which would require changes and which would make stopping this line of work reasonable. This gives us a reference for interpreting what we observe.

## Build a useful comparison

The *baseline* describes the metrics for how the task is performed today, before any changes: time, quality, cost and difficulties. It must cover the process needed to obtain a usable result—in other words, a way of measuring that applies equally before and after the change.

We then select cases that represent the relevant variety within the scope. Both typical cases and those that test the limits matter: incomplete information, ambiguities or situations that need to be referred elsewhere.

Some cases can be used to develop and adjust the solution. It helps to reserve others for evaluation once the version has been fixed. If we keep making adjustments against the same examples, it becomes difficult to know how the solution will respond to new cases.

The way the comparison is organised also matters. A person solving the same case twice may benefit from what they learned the first time. We can distribute equivalent cases or alternate the order between participants, recording the procedure used.

## Measure the whole task

Suppose, as a hypothetical example, that a task takes 18 minutes. AI assistance produces a draft in one minute, but four minutes of review and three of correction are then needed. The assisted process takes eight minutes.

The relevant difference would be ten minutes to reach an acceptable result. We must also record costs, errors and variation between cases. A favourable average can hide situations that require substantial intervention.

The diagram retains the stages in the calculation to show which time we are comparing.

<figure class="editorial-figure chalkboard-figure">
  <img src="../resources/testing.svg" width="800" height="800" alt="Illustrative comparison awaiting measurement: an 18-minute current process and an 8-minute assisted process, made up of 1 minute of preparation, 4 of review and 3 of correction. Compare quality and cost too." loading="lazy">
  <figcaption>Hypothetical example: the assisted time includes review and correction. The test must measure the whole task and check its quality. <a class="diagram-expand" href="../resources/testing.svg">View larger diagram</a>.</figcaption>
</figure>

Quality evaluation should help locate the causes of any problem. When a solution consults documents, it helps to distinguish whether it found the relevant information and whether it correctly developed the response from that information. The [Amazon Bedrock evaluation documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html) addresses both capabilities; separating them helps direct the corrections needed.

## Interpret what the evidence allows us to claim

The result must retain its context: the cases used, sources, solution version and evaluation procedure.

A small test can show feasibility and reveal difficulties. The confidence needed to expand use depends on the variety of the work and the potential consequences of errors. The absence of failures in observed cases leaves questions open about situations that have not yet been evaluated.

If the results obtained justify moving forward, a pilot will allow us to study use within the activity: interruptions, people's participation, connections and maintenance effort.

The test will have served its purpose when it allows us to explain what worked, under which conditions, what remains to be resolved and why the evidence supports the next decision.

<div class="next-step">
  <p class="eyebrow">Next step</p>
  <h2>Build on what has shown value</h2>
  <p>The test provides a firmer basis for deciding. We can now examine what it takes to turn a validated capability into something people can use every day.</p>
  <div class="action-group"><a href="../integrate-into-daily-work/" role="button">Plan the integration</a></div>
</div>
