Establish and Document the Application's AI and Model Configuration

This chapter explains how to establish and document the configuration of any Artificial Intelligence (AI) capabilities embedded in an Application.

Why AI and Model Configuration Deserves Its Own Treatment

Applications increasingly embed AI, for example to classify documents, answer questions, generate content, or act as agents. AI behavior is determined by configuration that does not resemble traditional settings: the model chosen, its version, the prompts and instructions it receives, the data it can retrieve, and the guardrails that constrain it. Small changes to any of these can change the Application’s behavior significantly, and model providers frequently change or retire models.

Establish the AI and Model Configuration

For each AI capability in the Application, establish:

• The model and provider used, and the model version.

• Prompts and system instructions.

• Retrieval sources the AI may draw on, and their access controls.

• Parameters that affect behavior, such as settings that control randomness or response length.

• Guardrails, such as content filters, permitted actions, and human approval requirements.

• Evaluation thresholds and the tests used to confirm acceptable behavior before changes are released.

• What happens if the model is unavailable or is retired by its provider.

Document and Share the AI and Model Configuration

Record the AI and model configuration in the Application Documentation, version prompts and instructions with the Application’s Releases, and record Instance differences explicitly. Track model deprecation and retirement dates as expiring dependencies. Update the configuration documentation with every change, including changes made by the model provider.

Best Practice: Establish and Document the Application’s AI and Model Configuration

It is a best practice to both establish and document the solution for configuring an Application’s AI: record its models, versions, prompts, retrieval sources, parameters, guardrails, evaluation thresholds, and fallback behavior, and version them with the Application’s Releases.

Benefits: Documented and versioned AI configuration makes AI behavior reproducible and reviewable, so that changes in behavior can be traced to their cause and governed like any other change. It also prepares the enterprise for model changes and retirements imposed by providers, so that they do not come as a surprise.

Connections to Related IF4IT Practices and Inventories

The Enterprise AI Governance Best Practices addresses the governance of AI across the enterprise. The AI Agents Inventory and Attributes describes how AI agents are recorded.