AI-security research progression
The educational foundation uses familiar cybersecurity exercises: object authorization, web security, Linux and forensic investigation. The website includes a two-record authorization fixture and links to independent training providers. The AI research fixture executes a deterministic export-control rule; it does not pretend that a language model is connected. A future prompt-injection lab must explain that a production secret should not be protected solely by a system instruction.
Generation 2 introduces a retrieval pipeline. Record corpus version, document identity, retrieval ranking, trust label and retrieved context. Distinguish poison ingestion, poison retrieval, model influence and actual disclosure. Indirect prompt injection is an established attack class in which adversarial instructions arrive through material an application consumes.
Generation 3 connects an agent to tools through explicit identities and authorization. Observe a proposed tool call, gateway decision, downstream request and actual resource effect separately. MCP tests should pin the relevant protocol version and distinguish protocol conformance from application authorization. MCP security guidance discusses confused deputies, token passthrough, session attacks and other integration hazards.
Generation 4 composes applications, agents, models, retrieval, tools and cloud resources. Investigate delegation, asynchronous actions and control interactions only after the simpler contracts work. Multi-agent behavior multiplies possible causal explanations and is a later extension, not an initial product dependency.
State-based evaluation
Make state-based evaluation a core capability, while explicitly recognizing existing research such as AgentDojo. The proposed differentiator is validated correspondence between the scenario property, observation points and evaluator across controlled environment changes.
For disclosure, compare a synthetic protected value or a defined transformation of it with evidence at an unauthorized sink. For a forbidden write, check the destination service's durable state and authenticated audit record. For a privilege change, compare effective authorization before and after the run. For remediation, verify restored state and rerun the attack plus the legitimate task.
Exact matching misses paraphrase and encoding; semantic judges can misclassify or be influenced by adversarial text. Use deterministic checks where ground truth is available. Treat model-based assessments as fallible supporting signals with separately measured agreement, and never give an evaluation model authority to change the environment or its own rubric.
Measure security and legitimate task completion separately. A system that refuses every request may block an attack while becoming useless. Report attempted action, permitted action and completed effect as distinct events, and document whether a test evaluates confidentiality, integrity, availability or functional utility.
Research questions and planned experiments
Experiment A — semantic validity. Create a proposed corpus of 20 reviewed scenarios from the support-agent family, each with benign and vulnerable witnesses. Produce five controlled variants per scenario. Compare independently maintained deployment/grader scripts with the proposed contract compiler. Measure valid-run fraction, detected specification defects, manual repair time and undetected semantic drift. Have reviewers assess variants without knowing which authoring method produced them.
Experiment B — evaluation validity. Build a proposed set of 200 synthetic runs containing legitimate actions, blocked attacks, successful side effects, misleading model statements and missing evidence. Compare output-only scoring, an existing state-aware harness, and the proposed evaluator. Measure precision, recall, inconclusive rate and inter-reviewer agreement against independently reviewed resource evidence. Balance classes and publish the confusion matrix so prevalence cannot hide poor discrimination.
Experiment C — reproducibility. Repeat ten selected scenarios over at least 30 runs per condition, using pinned fixtures first and a live model second. Vary one dependency at a time: model identifier, retrieval index, policy or tool schema. Measure initial-state equivalence, verdict agreement, outcome distributions and time to reproduce a finding. Use this pilot to estimate variance and plan an adequately powered follow-up.
Experiment D — observability. Inject duplicate events, missing spans, delayed records, clock skew and forged guest logs into controlled traces. Compare timestamp joins with explicit provenance and source-sequence joins. Measure incorrect causal edges, unrecognized missing evidence, detection delay and evaluator changes. Remove each sensor in an ablation to identify the minimum defensible evidence set.
Experiment E — target classroom pilot. Progress through 1, 5, 10, 25 and 50 simultaneous synthetic sessions before involving a workshop. Measure readiness success, p50/p95 startup latency, tail task latency, session interference, teardown completion, abandoned resources, cost per session-hour and evidence loss. Fifty sessions is a target test condition, not a present capacity claim.
Experiment F — authoring and transfer. Compare expert-template authoring with constrained generation using counterbalanced task order and equivalent scenario difficulty. Track total time including rejected proposals and repair. After classroom validation, ask enterprise design partners to reproduce one representative agent defect using the same contract. Measure adaptation effort; do not infer enterprise utility from student satisfaction alone.