A demo built on a language model can come together in an afternoon. A feature that thousands of users rely on is a different job. Models are non-deterministic, they occasionally return the wrong format, and costs can climb quietly. Here's the checklist we work through before any LLM feature goes live.
1. Define What Good Looks Like
Before writing prompts, collect 20 to 50 real examples of inputs and the outputs you'd be happy with. These become your evaluation set. Without them, "it seems to work" is the only test you have.
2. Build Evals, Not Just Prompts
Run every prompt or model change against the evaluation set and score the results. Some checks are simple, like valid JSON or length limits. Others need a rubric or a second model to grade answers. The point is to catch regressions before users do.
If you can't measure a change, you can't tell whether it made things better.
3. Validate Every Output
- →Check the structure: parse the response and reject anything that doesn't match the expected schema.
- →Check the content: flag banned words, made-up product names or numbers outside a sensible range.
- →Retry with context: if validation fails, retry once with the error message included in the prompt.
4. Plan for Failure
Providers have outages and slow days. Set timeouts, add a fallback model from a second provider, and decide what the user sees if both fail. A clear "we couldn't generate this right now" message beats a spinner that never ends.
5. Put Limits on Cost
- →Cache repeated requests so identical questions don't cost twice.
- →Route by difficulty: send simple tasks to smaller, cheaper models.
- →Set per-user and per-day limits with alerts before the bill surprises you.
- →Track cost per request next to quality, so you can see the trade-off.
6. Watch It in Production
Log inputs, outputs, latency and cost for every call, with personal data removed. Review a sample each week, add interesting failures to your evaluation set, and keep tuning. LLM features aren't finished at launch. They improve with usage.
- ✓Evaluation set of real examples, scored automatically
- ✓Output validation with retries
- ✓Timeouts and a fallback model
- ✓Cost limits, caching and alerts
- ✓Logging and a weekly review
These steps add a little time up front and save a lot of firefighting later. They're part of every AI feature Devex Hub ships.
Want help putting this into practice?
We scope the use case with you, build it on your real data and stay on after launch.
Book a call
