Non-Functional Requirements (NFRs) Framework for Software Systems

Overview

AI-Enabled Systems and Responsible AI Non-Functional Requirements (NFRs) define how AI-enabled systems, AI agents, machine learning models, generative AI capabilities, prompts, outputs, training data, retrieval sources, embeddings, tools, automated decisions, and human-in-the-loop workflows must be governed, validated, monitored, explained, secured, and controlled.

These requirements are different from using AI to generate NFRs. This PART focuses on NFRs for software systems that contain AI capabilities. Such systems may require additional attention to validity, reliability, safety, misuse prevention, prompt injection resistance, privacy, explainability, fairness, human review, model drift, output quality, retrieval governance, tool-use permissions, and AI-specific evidence.

Best Practice: Define AI validity and reliability non-functional requirements

Description

AI validity and reliability NFRs define how well an AI-enabled system must perform for its intended use, target users, data context, decision context, and operating environment. They should address evaluation datasets, success metrics, failure thresholds, repeatability expectations, known limitations, confidence handling, and revalidation triggers.

Benefits

Validity and reliability requirements reduce the risk that AI outputs are treated as trustworthy when they are inaccurate, inconsistent, unsupported, outside intended use, or insufficiently evaluated.

Example non-functional requirements

• The AI-enabled capability shall meet approved accuracy, quality, and failure-rate thresholds on an evaluation dataset representative of the intended use case before production release.

• Validation method: Validate through controlled model or system evaluation using an approved evaluation dataset, defined scoring rubric, and stakeholder-reviewed acceptance thresholds.

• Example validation evidence: Evaluation dataset description, scoring rubric, evaluation report, threshold approval, reviewer signoff, and release readiness evidence.

• The system shall identify when an AI output is below the approved confidence or quality threshold and shall route the case to an approved fallback or human review process.

• Validation method: Validate through scenario testing using low-confidence, ambiguous, and out-of-scope inputs and verify fallback or human review routing.

• Example validation evidence: Scenario test results, output quality scores, fallback workflow evidence, human review records, and defect/remediation records.

Related stakeholders

Typical stakeholders include product owners, AI/ML engineers, data scientists, domain experts, architects, QA teams, risk stakeholders, and business owners.

Related lifecycle phases

AI validity NFRs are defined during use-case discovery, model/system design, data preparation, and risk assessment; validated during evaluation, testing, user acceptance, release readiness, production monitoring, and periodic revalidation.

Best Practice: Define AI safety and misuse-prevention non-functional requirements

Description

AI safety and misuse-prevention NFRs define how an AI-enabled system prevents or reduces harmful, inappropriate, unauthorized, unsafe, or unintended use. They should address prohibited uses, misuse scenarios, content safety, unsafe recommendations, abuse prevention, escalation rules, and fail-safe behavior.

Benefits

Safety and misuse-prevention requirements help reduce harm from unsafe outputs, excessive automation, inappropriate recommendations, abuse, malicious prompts, and use outside the approved operating context.

Example non-functional requirements

• The AI-enabled system shall block or safely respond to requests that are outside approved use, violate defined safety policy, or seek prohibited actions.

• Validation method: Validate through misuse testing, safety policy test suites, red-team prompts, and review of blocked or redirected outputs.

• Example validation evidence: Misuse test report, safety policy test results, red-team findings, blocked-output logs, and safety review approval.

• AI-generated recommendations that could materially affect customer, patient, employee, financial, legal, or operational outcomes shall require approved human review before execution or communication as a final decision.

• Validation method: Validate through workflow testing that verifies human review, approval, rejection, override, and audit logging before final action.

• Example validation evidence: Workflow test report, review queue records, approval/rejection logs, override evidence, and risk stakeholder signoff.

Related stakeholders

Typical stakeholders include product owners, risk stakeholders, legal teams, compliance teams, safety stakeholders, domain experts, AI teams, UX teams, and operations teams.

Related lifecycle phases

AI safety NFRs are defined during use-case approval, risk assessment, design, and governance; validated during safety testing, red-team testing, user acceptance, release readiness, monitoring, and incident review.

Best Practice: Define AI security and resilience non-functional requirements

Description

AI security and resilience NFRs define how AI capabilities, model endpoints, orchestration layers, prompt flows, retrieval pipelines, vector stores, tools, agents, and supporting infrastructure are protected from attack, misuse, disruption, data exfiltration, service exhaustion, and dependency failure.

Benefits

These requirements help protect AI-enabled systems from AI-specific and traditional attack paths while preserving availability, safe fallback, and controlled behavior under degraded conditions.

Example non-functional requirements

• AI model endpoints and orchestration APIs shall require approved authentication, authorization, rate limiting, logging, and monitoring before production use.

• Validation method: Validate through API security testing, access-control testing, rate-limit testing, and monitoring review for the AI endpoint and orchestration layer.

• Example validation evidence: API security test report, access-control matrix, rate-limit test results, logs, dashboard evidence, and security approval.

• The AI-enabled system shall degrade safely when the model provider, retrieval source, vector store, or tool dependency is unavailable or exceeds approved latency thresholds.

• Validation method: Validate through dependency failure simulation and latency injection testing to verify fallback, user communication, logging, alerting, and recovery behavior.

• Example validation evidence: Dependency failure test report, latency test results, fallback screenshots, alert records, logs, and resilience review evidence.

Related stakeholders

Typical stakeholders include AI architects, security architects, platform engineers, SRE teams, AI/ML engineers, developers, QA teams, and operations teams.

Related lifecycle phases

AI security and resilience NFRs are defined during architecture and risk assessment; validated during security testing, integration testing, resilience testing, release readiness, production monitoring, and incident review.

Best Practice: Define prompt injection and insecure output handling non-functional requirements

Description

Prompt injection and insecure output handling NFRs define how an AI-enabled system resists malicious, indirect, or accidental instructions that attempt to alter intended behavior, bypass policy, expose data, manipulate tools, or create unsafe downstream actions. They also define how AI outputs are sanitized, constrained, reviewed, and handled by downstream systems.

Benefits

These requirements reduce the risk that retrieved content, user input, embedded instructions, or generated output manipulates the AI system or compromises downstream applications, data, users, or workflows.

Example non-functional requirements

• The AI-enabled system shall detect, block, or neutralize direct and indirect prompt injection attempts that request policy bypass, credential disclosure, unauthorized tool use, or hidden instruction execution.

• Validation method: Validate through prompt injection test suites, indirect prompt injection tests using retrieved content, and red-team review of bypass attempts.

• Example validation evidence: Prompt injection test results, red-team findings, retrieval test records, blocked attempt logs, and remediation evidence.

• AI-generated output used by downstream systems shall be validated, sanitized, or approved before being executed as code, queries, commands, configuration, or workflow actions.

• Validation method: Validate through downstream integration tests, output sanitization tests, code/query review tests, and policy enforcement checks.

• Example validation evidence: Output handling test report, downstream integration logs, sanitization evidence, rejected unsafe output samples, and security approval.

Related stakeholders

Typical stakeholders include security teams, AI engineers, developers, architects, QA teams, UX teams, operations teams, and risk stakeholders.

Related lifecycle phases

Prompt injection and output handling NFRs are defined during AI architecture, prompt design, retrieval design, and integration design; validated during security testing, red-team testing, integration testing, release readiness, and production monitoring.

Best Practice: Define AI privacy and sensitive-data protection non-functional requirements

Description

AI privacy and sensitive-data protection NFRs define how prompts, outputs, embeddings, retrieval content, model inputs, evaluation data, logs, feedback, human-review records, and tool interactions handle personal, sensitive, confidential, regulated, or protected data. They should address minimization, masking, retention, logging, access control, third-party model use, and cross-border processing.

Benefits

AI privacy requirements reduce the risk of exposing sensitive data through prompts, outputs, logs, vector stores, retrieval sources, model providers, human-review workflows, or unintended training/evaluation reuse.

Example non-functional requirements

• The AI-enabled system shall not send personal or sensitive data to external model providers unless the data transfer is approved, protected, logged, and aligned with privacy and contractual requirements.

• Validation method: Validate through data-flow review, model provider configuration review, privacy review, logging review, and contract/control review.

• Example validation evidence: Data-flow diagram, provider configuration, privacy approval, contract/control mapping, log samples, and DPIA or privacy review evidence.

• Prompts, outputs, embeddings, and AI interaction logs containing sensitive data shall follow approved retention, access-control, masking, and purge requirements.

• Validation method: Validate through storage inspection, access-control testing, log review, retention configuration review, and purge test execution.

• Example validation evidence: Retention configuration, access test results, log samples, masking evidence, purge test report, and privacy approval.

Related stakeholders

Typical stakeholders include privacy teams, security teams, AI teams, data owners, legal stakeholders, compliance teams, architects, developers, and operations teams.

Related lifecycle phases

AI privacy NFRs are defined during data-flow analysis, privacy review, architecture, and vendor review; validated during testing, privacy assessment, release readiness, production monitoring, and recurring privacy control review.

Best Practice: Define AI transparency, explainability, and interpretability non-functional requirements

Description

AI transparency, explainability, and interpretability NFRs define what users, operators, reviewers, auditors, and affected stakeholders must be able to understand about AI outputs, recommendations, limitations, sources, confidence, decision factors, and human review. They should align to the risk and impact of the AI use case.

Benefits

These requirements improve trust, accountability, reviewability, user understanding, auditability, and safe adoption of AI-enabled capabilities.

Example non-functional requirements

• The AI-enabled system shall identify when content, recommendations, or summaries are AI-generated and shall communicate relevant limitations to intended users.

• Validation method: Validate through UX review, content review, user acceptance testing, and review of generated-output labeling behavior.

• Example validation evidence: UX review record, generated-output screenshots, UAT results, content review notes, and product approval.

• For AI-assisted decisions requiring human review, the system shall provide reviewers with relevant input data, retrieval sources, confidence indicators, and rationale or explanation sufficient for review.

• Validation method: Validate through reviewer workflow testing and expert review of whether the provided explanation and supporting context are sufficient for the decision type.

• Example validation evidence: Reviewer workflow test report, explanation samples, source citation records, confidence display evidence, and domain expert signoff.

Related stakeholders

Typical stakeholders include product owners, domain experts, UX teams, legal/compliance stakeholders, risk stakeholders, AI teams, auditors, and affected business owners.

Related lifecycle phases

Transparency and explainability NFRs are defined during use-case design, UX design, governance review, and risk assessment; validated during UAT, expert review, compliance review, release readiness, and periodic monitoring.

Best Practice: Define AI fairness and harmful-bias management non-functional requirements

Description

AI fairness and harmful-bias management NFRs define how the system identifies, evaluates, mitigates, monitors, and governs unacceptable bias or materially different performance across relevant groups, contexts, regions, languages, use cases, or operational conditions. Requirements should be appropriate to the system impact and available data.

Benefits

Fairness requirements reduce the risk of discriminatory, inequitable, or unjustified outcomes and help create accountable governance for high-impact AI-enabled systems.

Example non-functional requirements

• AI-enabled decision-support outputs for high-impact workflows shall be evaluated for harmful bias across approved relevant population or context segments before production release.

• Validation method: Validate through subgroup performance analysis, fairness review, issue triage, remediation, and stakeholder signoff.

• Example validation evidence: Fairness evaluation report, subgroup analysis, bias issue register, remediation plan, and governance approval.

• Material changes to model, retrieval data, decision policy, or target population shall trigger review of fairness and harmful-bias management requirements.

• Validation method: Validate through change impact review and confirmation that fairness revalidation was performed or formally deemed not applicable.

• Example validation evidence: Change impact assessment, fairness revalidation record, approval notes, exception record, and release decision evidence.

Related stakeholders

Typical stakeholders include risk teams, legal teams, compliance stakeholders, domain experts, product owners, AI/ML teams, data owners, and governance bodies.

Related lifecycle phases

Fairness NFRs are defined during use-case approval, data analysis, model evaluation, and risk assessment; validated during testing, governance review, production monitoring, change review, and periodic model/system review.

Best Practice: Define AI human review, escalation, and override non-functional requirements

Description

AI human review, escalation, and override NFRs define when humans must review, approve, reject, modify, escalate, or override AI-generated outputs, recommendations, decisions, or actions. They should identify review roles, approval thresholds, escalation triggers, override authority, audit logging, and response timelines.

Benefits

Human review requirements help keep accountability with qualified stakeholders and reduce the risk of over-automation, excessive agency, unsafe action, or unreviewed high-impact decisions.

Example non-functional requirements

• AI-generated recommendations that exceed approved risk or impact thresholds shall require qualified human approval before execution, communication, or downstream action.

• Validation method: Validate through workflow testing using low-risk and high-risk cases and verifying approval routing, rejection, override, and audit logging behavior.

• Example validation evidence: Workflow test report, approval logs, rejection/override records, role mapping, and risk approval evidence.

• Authorized reviewers shall be able to override AI recommendations and record a reason for the override.

• Validation method: Validate through reviewer UI testing, API testing where applicable, and audit log review of override reason capture.

• Example validation evidence: Reviewer test results, override screenshots, audit log samples, role authorization evidence, and UAT approval.

Related stakeholders

Typical stakeholders include business owners, domain experts, operations teams, risk stakeholders, compliance teams, UX teams, product owners, AI teams, and audit stakeholders.

Related lifecycle phases

Human review NFRs are defined during workflow design, risk assessment, UX design, and governance review; validated during UAT, operational readiness testing, audit review, and production monitoring.

Best Practice: Define AI model, prompt, retrieval-source, and tool governance non-functional requirements

Description

AI model, prompt, retrieval-source, and tool governance NFRs define how AI assets and runtime components are inventoried, versioned, approved, tested, changed, monitored, retired, and traced. They should address model version, prompt version, system instructions, retrieval-source ownership, content freshness, tool permissions, tool-use boundaries, and change control.

Benefits

AI governance requirements reduce uncontrolled change, unknown model behavior, stale retrieval content, unauthorized tool use, and untraceable AI outputs.

Example non-functional requirements

• Production AI prompts, model configurations, retrieval sources, and tool permissions shall be versioned, approved, and traceable to release records.

• Validation method: Validate through configuration repository review, release traceability review, and comparison of deployed AI configuration against approved release records.

• Example validation evidence: Prompt/configuration version history, release record, retrieval-source inventory, tool permission review, and approval evidence.

• AI tools capable of changing data, initiating transactions, sending communications, or calling external systems shall enforce least-privilege permissions and approved action boundaries.

• Validation method: Validate through tool permission review, action-boundary testing, negative testing for unauthorized actions, and audit log review.

• Example validation evidence: Tool permission matrix, negative test results, action logs, least-privilege review, and security/governance approval.

Related stakeholders

Typical stakeholders include AI architects, product owners, data owners, knowledge owners, security teams, platform teams, operations teams, risk stakeholders, and governance bodies.

Related lifecycle phases

AI governance NFRs are defined during AI architecture, solution design, data/retrieval design, and governance planning; validated during configuration review, integration testing, release readiness, production monitoring, and change review.

Best Practice: Define AI output monitoring, drift detection, and quality feedback non-functional requirements

Description

AI output monitoring, drift detection, and quality feedback NFRs define how the system monitors AI behavior after release, detects changes in input patterns or output quality, captures feedback, identifies degradation, triggers review, and supports retraining, reconfiguration, or rollback decisions where applicable.

Benefits

Monitoring and drift requirements help teams detect when AI-enabled systems degrade over time or operate outside expected context. They also support continuous improvement and governance.

Example non-functional requirements

• The AI-enabled system shall monitor approved output quality indicators and alert accountable owners when quality drops below approved thresholds.

• Validation method: Validate through monitoring configuration review, threshold simulation, alert testing, and review of ownership routing.

• Example validation evidence: Monitoring dashboard, threshold configuration, alert test results, owner routing evidence, and operations approval.

• Material model drift, retrieval quality degradation, or repeated human-review rejection patterns shall trigger documented investigation and revalidation.

• Validation method: Validate by simulating or reviewing qualifying drift/rejection scenarios and verifying that investigation, triage, and revalidation workflows are triggered.

• Example validation evidence: Drift report, rejection trend report, investigation ticket, revalidation record, and governance review notes.

Related stakeholders

Typical stakeholders include AI/ML teams, product owners, data scientists, domain reviewers, SRE teams, operations teams, risk stakeholders, and governance bodies.

Related lifecycle phases

AI monitoring NFRs are defined during monitoring design and governance planning; validated during release readiness, production monitoring, model/system review, incident review, and continuous improvement cycles.

Best Practice: Define AI validation and evidence non-functional requirements

Description

AI validation and evidence NFRs define the proof required to show that AI-enabled system requirements have been satisfied. Evidence may include evaluation reports, red-team results, safety tests, bias reviews, privacy reviews, prompt injection tests, output quality reports, human-review records, monitoring dashboards, drift reports, and approvals.

Benefits

AI validation evidence makes AI-enabled behavior reviewable, repeatable, governable, and auditable. It helps teams avoid relying on demonstrations, vendor claims, model reputation, or informal user impressions as proof of readiness.

Example non-functional requirements

• Each AI-enabled production capability shall have documented validation evidence for intended use, limitations, privacy, security, safety, human review, and monitoring before release.

• Validation method: Validate through AI release readiness review and verify that each required evidence category is complete, reviewed, and approved.

• Example validation evidence: AI validation package, release readiness checklist, evaluation report, privacy/security review, monitoring evidence, and approval record.

• AI validation evidence shall be retained with release records and shall be available for incident, audit, risk, compliance, and governance review.

• Validation method: Validate through evidence retrieval sampling for released AI capabilities and confirm traceability to release, model, prompt, retrieval source, and owner.

• Example validation evidence: Evidence repository links, release record, model/prompt version record, retrieval-source inventory, audit sample evidence, and governance approval.

Related stakeholders

Typical stakeholders include AI governance teams, AI/ML teams, product owners, security teams, privacy teams, compliance teams, risk stakeholders, audit stakeholders, and domain experts.

Related lifecycle phases

AI validation and evidence NFRs are defined during AI governance planning and release planning; validated during evaluation, specialized testing, release readiness, production monitoring, recurring review, incident review, and audit review.