OpenAI Evals: keep your app’s quality checks working

Keep testing part of the operating plan

If your app team uses OpenAI’s hosted Evals service to check AI answers, ask where those checks will run next. The service has entered a new stage of its retirement, and the team needs a working replacement before the final closure.

OpenAI’s timeline says existing Evals became read-only on October 31. The Evals dashboard and its software connection are scheduled to shut down on November 30. The original announcement was June 3; this week’s change is the read-only stage.

Evals are structured checks that compare an AI result with the behaviour your business expects. This notice concerns OpenAI’s named hosted product, not the idea of testing AI or every testing tool your team might use.

Read the announcement: OpenAI: retirement timeline and October 31 read-only stage.

Preserve the work behind the score

Ask the technical owner which app checks depend on this service. Keep the relevant test examples, instructions and judging rules, along with any previous results your business needs and is authorized to retain.

A headline score is not enough to recreate a check. The replacement needs the same business question and a clear account of what a good answer includes. If a case contains customer information, keep it within your approved handling process.

OpenAI’s migration guide describes manually recreating tests. It does not promise an automatic export that transfers the full dashboard and its history into another product. Confirm what your account can retrieve while the service remains available.

Test the replacement before trusting it

OpenAI recommends Promptfoo, a tool that lets a technical team keep test definitions alongside application files and run them on its own systems. Another suitable testing process may also fit your requirements.

The official guide maps examples, prompts, model settings and judging criteria into a new configuration. A new run is separate from earlier OpenAI Evals runs. Similarity scores and AI-based judging may differ between systems.

Have the technical owner recreate one important check first and compare the actual answers and pass-or-fail decisions. Do not accept a matching total score as the only evidence that the test still catches the mistakes that matter.

Assign time for the transition

Our perspective: treat the move as part of maintaining the app, with an owner and an agreed completion date. Avoid a gap where updates ship without the checks the team normally relies on.

OpenAI also schedules Agent Builder to close on November 30, while ChatKit remains available. That is a separate dependency to check if your app uses it. This notice does not mean all OpenAI app services are closing.

We have not tested your setup or performed a migration. Ask for a plan covering the replacement, ongoing model usage costs and responsibility for keeping the tests useful.

1. This week, identify the app checks that use hosted OpenAI Evals and assign a transition owner.
2. Preserve the available examples, instructions, criteria and results your business needs.
3. Recreate one representative check and have its business owner review whether it judges the result correctly.
4. Move the remaining checks and confirm regular runs before November 30, leaving time to resolve differences.