---
title: "Operating notes — Context Field Notes"
description: "What on-call for an AI feature actually looks like."
canonical: "https://press.getalchemystai.com/books/context-field-notes/operating-notes"
last_updated: "2018-10-20"
book: "Context Field Notes"
---

# Operating notes

Running an AI feature in production feels familiar for the first week and strange after that. The dashboards look like any other service, but the incidents do not.

## Incidents without errors

The most common incident is not a crash. It is a slow change in answer quality after a dependency changed: a new model version, a re-indexed corpus, a prompt edit that seemed harmless. Nothing returns a 500. Users simply start trusting the feature less.

The defence is boring and effective. Keep a frozen evaluation set, run it on every change to prompts, models or retrieval, and alert on regressions the same way you would alert on latency.

## What to log

- The exact prompt and retrieved context for a sample of requests.
- Model, version and parameters for every call.
- User corrections and thumbs-down events, joined to the request that caused them.

Logging the context, and not only the answer, is what turns "the bot said something odd" into a reproducible bug report.

## Saying no

Finally, teach the system to decline. An assistant that says _I don't have enough information to answer that_ when retrieval comes back empty will be trusted far longer than one that always has an answer.

This paragraph had an inline handler that the sanitizer removes.

---

- Previous: [Retrieval in practice](https://press.getalchemystai.com/books/context-field-notes/retrieval-in-practice.md)
- Contents: [Context Field Notes](https://press.getalchemystai.com/books/context-field-notes.md)
