AI for real work · 4 of 5 · Test
How to design an AI proof of concept and decide whether to move forward
An idea becomes more convincing when we can see it work. What was once a possibility takes shape and produces results we can observe. That moment brings enthusiasm and an opportunity to learn.
To make use of it, we need to understand what we are seeing. A favourable result becomes valuable when we know how it was obtained, what it represents and under which conditions it could be repeated.
Designing a proof of concept means preparing that observation. We want the effort of building something to leave us with a useful answer for the next decision.
Start with the question we need to resolve
After assessing readiness, we have a better understanding of the project’s purpose, information and conditions. Uncertainties remain. The test should focus on one of those that determine whether we can move forward.
We can state the improvement we expect and describe what would need to happen for us to consider it feasible. That relationship between an expectation and an observation forms the working hypothesis.
It should be precise enough for us to recognise the result. We need to understand which capability we are testing, what part of the work it covers and the quality required. We must also define what falls outside the scope.
That boundary matters because it makes the experience interpretable. Trying to resolve too many questions at once makes it difficult to understand afterwards what worked, why it worked and what we should change.
Build a meaningful comparison
Understanding an improvement requires knowing our starting point. We need to observe how the work is currently done and the effort required to produce an acceptable result.
That reference makes comparison possible. If we measure time, we must define where the measurement begins and ends. If we assess quality, we need agreement on what a correct result means and who has the expertise to judge it. If we estimate cost, we must define which resources are included.
We then need a collection of work that is sufficiently representative of the chosen scope. It should cover the variety that matters: usual situations, foreseeable difficulties and the conditions that call for human involvement.
During development, some of this material will help us adjust the solution. Another part should be reserved for evaluation afterwards. This lets us observe its response to work that was not used to guide those adjustments.
The comparison should also consider people’s participation. Their experience, prior knowledge of a task and time to become familiar with the tool can influence what we observe. Recording those conditions helps us interpret the result more carefully.
Measure what the business needs
Measurement should follow the improvement we want. We may care about the time required, quality, the capacity to complete work or a combination of these dimensions.
In every case, we should observe the sequence through to a usable result. That includes the preparation, review and corrections a person still needs to make. We can then understand how much effort moves elsewhere and how much is genuinely reduced.
Quality calls for a similar analysis. We need to know which errors occur, how often they occur and what their consequences would be. An average can hide an important difficulty, so we must also examine the variety of results and the conditions that explain it.
Cost relates to that complete sequence of work. Alongside technical services, it includes people’s participation and the maintenance needed to repeat the improvement. The test can help us estimate these resources on a firmer basis.
When a solution uses information to produce answers, it is useful to examine both what it finds and what it builds from that material. The Amazon Bedrock evaluation documentation distinguishes these two capabilities. Analysing them separately helps locate the source of a problem, even when using another technology.
Agree in advance what will justify moving forward
Before running the test, we should decide how we will use its results. We need to agree on conditions of quality, effort and cost that would justify the next step.
Those conditions depend on the work and its consequences. They should be clear enough to keep enthusiasm from changing the basis of judgement. It also helps to set the scope of the investment and the participation of those who will evaluate it.
At the end, we may find favourable evidence, a difficulty worth adjusting or a relationship between value and effort that suggests stopping. All three conclusions can be useful when they are well supported.
The report should retain the version tested, the conditions, the work used and the observations made. Its limits are part of the result. Knowing where evidence is still missing guides the next investment and keeps a conclusion within what the test can support.
Bring technical capability closer to real work
A proof of concept studies the technical feasibility of a bounded capability. Its results help us decide whether to take it into a pilot with people and controlled conditions of use.
The pilot expands what we need to observe. We want to understand how people use the solution, what effort adoption requires and whether the improvement holds within the complete process. That experience informs development of the version intended for everyday use.
Each stage adds a question and evidence. Technical capability opens a possibility; people’s participation and measurement of the work help us understand its value. When that evidence supports moving forward, it is time to prepare the integration.