AI Security Evaluation Framework
SCS uses an internal evidence-first assessment framework to test AI systems across chatbots, RAG systems, agents, tool workflows, internal copilots, and Microsoft 365 Copilot exposure scenarios.
How the framework works
- Target configuration and architecture selection
- Recommended suites and categories by architecture
- Identity matrix testing across users and roles
- Architecture-specific runners
- Category-specific evaluators
- Findings with severity, evidence, remediation, and report sections
Architectures covered
- Chatbot
- RAG
- Agent
- Tool workflow
- Internal copilot
- Microsoft 365 Copilot
Evidence model
- Assistant output
- Citations and retrieved documents
- Source paths and connector IDs
- Sensitivity labels
- Tool calls
- Canary matches
- Identity, role, and persona context
- Optional inventory and access-path context
How the SCS evaluation framework is applied.
The framework is used to support operator-led assessment work. It helps structure evidence and reporting, but findings still require practitioner review and scoped interpretation.
Why SCS maintains its own evaluation framework.
AI security testing has no settled equivalent of a network scan. Architectures differ enough that a fixed checklist either misses the relevant attack surface or reports findings that do not apply. SCS therefore maintains an internal framework that selects test coverage from the deployed architecture.
Framework structure
| Architecture intake | What the system is: model access, retrieval, tools, identities, tenancy, and downstream actions. |
| Risk selection | Which categories apply to that architecture, so an agent is not tested as though it were a chatbot. |
| Declarative test suites | Reusable cases per risk category, configured to the specific target rather than rewritten each engagement. |
| Evidence capture | Raw output, context indicators, latency, and observable system effect recorded for every case. |
| Evaluator review | Automated evaluation narrows the set; an operator confirms each candidate finding before it is reported. |
| Retest definition | Every confirmed finding produces a named case that can be replayed after remediation. |
What the framework is, and what it is not
This is an internal methodology, not a certification, standard, or compliance regime. SCS does not issue a framework certificate, and an assessment conducted under it should not be presented as accreditation against a published standard.
Its purpose is repeatability. Because coverage is selected from architecture rather than from a fixed list, two engagements against similar systems receive comparable treatment, and a system reassessed a year later can be compared against its own prior results.
The framework is also what makes retest meaningful. Since each finding is expressed as a declarative case rather than as prose, the same conditions can be replayed exactly after a fix, and the result is a direct comparison rather than a fresh judgment call.
AI Security Evaluation Framework questions.
What is an AI security evaluation framework?
It is a structured method for deciding what to test in an AI system, how to capture evidence, and what qualifies as a finding. SCS uses an internal, evidence-first framework that selects coverage from the deployed architecture rather than from a fixed checklist.
Is this framework a certification or a published standard?
No. It is an internal SCS methodology. It produces assessment evidence and remediation guidance, not accreditation, and an engagement under it should not be represented as certification against a published standard.
Which AI architectures does the framework cover?
Chatbots and assistants, retrieval-augmented systems, agents with tool and API access, internal copilots, and Microsoft 365 Copilot exposure scenarios. Coverage is selected per engagement based on which of those patterns the system actually uses.
How does the framework decide what counts as a finding?
A candidate becomes a finding only when an operator reproduced the behavior against the deployed system and captured evidence of a real consequence, such as data disclosure, policy bypass, unsafe action, or a broken workflow assumption. Unusual output alone does not qualify.
Does the framework support retesting after remediation?
Yes. Each confirmed finding is expressed as a named, replayable case, so the same conditions can be run again after a fix and compared directly against the original result.