I build LLM agents that find, exploit, and patch vulnerabilities. Then I try to prove they don’t work.

Takes raw CodeQL alerts and runs them through contextual reasoning + structured evidence validation to filter false positives. Evaluated 10 frontier LLMs across 6 model families (Gemini, GPT, Grok, Mistral, DeepSeek, Qwen) on two benchmarks: OWASP Java Benchmark (1,974 cases / 10 CWE categories) and CWE-bench, a real-world dataset of 755 CodeQL alerts across 56 project–CVE pairs from 37 open-source Java repositories. CWE-specialized prompting improved F1 by up to +0.26 on real-world code.

LLMsCodeQLPythonStatic AnalysisMulti-Stage Prompting

Read the paper Source

Research thread at the University of Edinburgh (Mitacs Globalink Research Award). The pipeline runs worktree → docker build → exploiter → patcher → post-patch exploit re-run → regression → verifier, with the exploiter required to write and validate a proof-of-vulnerability in Docker before any patch is attempted. The comparison is an ablation against itself: both arms use the same agent, the same model, and the same system prompt, and the only thing that varies is whether the exploit evidence reaches the patcher — holding prompt wording fixed deliberately, to avoid confounding "more evidence" with "differently worded prompt". Upstream alerts come from vendored third-party IRIS (ICLR 2025) over CWE-Bench-Java. The evaluation harness is a reference-anchored LLM-as-judge scoring generated patches against the real upstream fix across 8 dimensions, with K=3 ensemble medoid selection, aggregates computed in code rather than asked of the model, hard gates on vulnerability-elimination and regressions, and a noise floor measured before the harness was used to compare anything. The mechanism diagnosis: a single proof-of-vulnerability is a narrowing signal, so an evaluation gate defined on that one path rewards patches that close it rather than the bug class behind it.

LLM AgentsAblation DesignLLM-as-JudgeDockerPythonCWE-Bench-Java

Production agentic IDE in the Cursor product space. Shipped: a six-tool subagent surface (DelegateTask / SendMessage / WaitForTask / CheckTaskOutput / StopTask / ListTasks) with headless execution, foreground/background promotion, and notification-queue injection back into LLM conversation history; a TDD-style RED→GREEN multi-agent DAG orchestrator with Mission Control dashboard; chain-of-density + KV-cache-aware summarization with unified context-budget tracking; the prepare→permission→execute tool lifecycle with path-scoped Bash/Read/Write/Edit/Glob; SWE-Bench and Aider-Polyglot evaluation infrastructure; and an MCP server exposing the test-and-break loop to AI agents. Also designed and ran a SWE-bench-with-vs-without-GraphRAG experiment over an 18,000-LoC code-knowledge-graph subsystem; the negative result (no measurable improvement) informed the team’s no-ship recommendation.

TypeScriptElectronReactLLM AgentsMCPSWE-BenchDocker

Visit

SFU lab project. Eight-phase LangGraph state machine drives the full exploit-and-patch lifecycle per CVE: parallel PoC analysis across 7 sources (GitHub, GitLab, Exploit-DB, PacketStorm, Nuclei, Metasploit, vendor advisories) + advisory enrichment, 0–10 composite PoC scoring with a synthesis fallback below threshold, parallel dockerized vulnerable + patched builds, automated exploit validation that verifies EXPLOIT_SUCCESS on vuln and EXPLOIT_FAILED on patched, and a three-layer hallucination defense at validation (filesystem-grounded verdict, fresh-context re-read, persistent audit trail). Planned bolt-on: hybrid retrieval-augmented context layer (pgvector + BGE-large dense + BM25 sparse → RRF fusion → bge-reranker-v2-m3 cross-encoder) over the 89-CVE artifact corpus plus ingested public-source documents, with a Recall@10 / MRR / nDCG / p95-latency eval harness and an ablation measuring whether retrieval-augmented context improves the synthesis-fallback path.

LangGraphLangChainDockerPythonClaude Code SDK

Farpoint client engagement for BDC (Canada’s federal Crown corporation for entrepreneurs, serving the Senior Management Committee, Board, and Shareholder). End-to-end daily-cycle pipeline: multi-source poll (RSS / Atom / sitemap / JSON-feed) with a Copilot Studio Bing-search fallback agent for un-crawlable sources → deterministic noise / URL / recency filter + SHA-256 / SimHash / Jaro-Winkler dedup → in-process Xenova/all-mpnet-base-v2 ONNX embeddings (768-dim, ~15–30 ms/article CPU, cached per row) → cosine clustering against open same-category clusters at 0.65 threshold → top-K cluster selection (N=12 per category × ~7–10 BDC strategic pillars) → Microsoft Copilot Studio judge over Direct Line with strict JSON-schema validation → decision-ready Signal cards (title / summary / BDC angle / composite score). Idempotent at every stage with crash-recovery via durable per-run state in PostgreSQL.

Next.js 16TypeScriptPostgreSQLPrismaMicrosoft Copilot StudioDirect LineONNX RuntimeSentence Embeddings