The AI security landscape is shifting rapidly, and the biggest players are making their moves. Case in point: OpenAI recently announced the acquisition of Promptfoo, an open-source AI security platform.
The Promptfoo acquisition isn’t just another tech acquisition. The acquisition signals a critical pivot in how enterprises approach generative AI. As OpenAI’s CTO of B2B Applications, Srinivas Narayanan, noted, the goal is to integrate “deep engineering expertise in evaluating, securing, and testing AI systems at enterprise scale” directly into OpenAI Frontier.
The acquisition underscores a reality developers have been grappling with: deploying AI agents into production without systematic testing is a recipe for disaster. Risks like prompt injections, data leaks, and unpredictable agent behavior demand robust, scalable solutions. For a comprehensive comparison of Promptfoo with other AI security tools, see The New Triad of AI Security: Promptfoo, Strix, and CAI.
What is Promptfoo?
Figure: The Promptfoo services ecosystem includes automated red teaming, AI-specific SAST, complete lifecycle model security, enterprise MCP proxy, self-improving guardrails, and core evaluations for prompts, RAGs, and agents.
At its core, Promptfoo is an open-source CLI tool and library designed for evaluating and red-teaming Large Language Model (LLM) applications.
Think of Promptfoo as test-driven development (TDD), but specifically tailored for the chaotic, probabilistic nature of AI. Instead of relying on manual trial-and-error—tweaking a prompt, testing the prompt with a few queries, and hoping for success—Promptfoo provides a structured, automated framework to ensure your LLMs behave as expected.
Promptfoo’s Complete AI Security Ecosystem
While Promptfoo began as a tool for evaluating prompts, Promptfoo has rapidly evolved into a comprehensive suite of security and evaluation services designed for the entire AI lifecycle. Today, Promptfoo offers a robust ecosystem of tools to secure everything from raw model files to complex, agentic workflows.
Here is a breakdown of the core services currently offered by Promptfoo:
Automated Red Teaming for Agents & RAGs
Instead of waiting for a manual security audit, Promptfoo acts as an automated adversary. By simply running npx promptfoo@latest redteam setup, the platform simulates real users to generate custom attacks tailored to your specific application. This uncovers a wide range of vulnerabilities, including direct and indirect prompt injections, customized jailbreaks, PII leaks, toxic content generation, and insecure tool usage by autonomous agents.
AI-Specific SAST (Static Application Security Testing)
Promptfoo shifts AI security to the “left,” allowing you to find vulnerabilities before the code is ever merged. It performs deep data flow analysis to catch LLM-specific risks that traditional generic security scanners miss. This is integrated directly into the developer workflow:
- IDE: Real-time scanning with inline fix suggestions.
- PR Review: Automated security findings directly in your pull requests.
- CI/CD: Native CLI integration for any deployment pipeline.
Complete Lifecycle Model Security
Whether you are leveraging proprietary foundation models (like OpenAI or Anthropic) or hosting open-weight models (PyTorch, TensorFlow, ONNX), Promptfoo secures the models from development through deployment. Promptfoo scans raw model files for malicious code and backdoors, tests model behavior against real attacks, and generates ready-to-use compliance reports aligned with major frameworks like OWASP, NIST, and the EU AI Act.
Enterprise MCP Proxy
As AI agents increasingly rely on the Model Context Protocol (MCP) to interact with external tools and databases, securing these connections is critical. Promptfoo provides an Enterprise MCP Proxy that gives organizations complete oversight of MCP activity. Security teams can whitelist approved MCP servers, grant granular access to specific applications, and monitor traffic in real-time to prevent sensitive data and PII from leaking to untrusted servers.
Self-Improving Guardrails
Static security rules quickly become obsolete against evolving AI threats. Promptfoo offers dynamic runtime guardrails that continuously learn from red team feedback and real-world attack attempts. These guardrails automatically adapt to new attack patterns and enforce custom company policies. They can be deployed in minutes, either in the cloud or on-premises, integrating seamlessly into any existing AI workflow.
Core Evaluations for Prompts, RAGs, and Agents
The foundational capability that put Promptfoo on the map remains stronger than ever. Developers can start testing the performance, accuracy, and reliability of their models, prompts, and tools in minutes using npx promptfoo@latest init. Promptfoo runs completely locally and integrates directly with your app—requiring no heavy SDKs, forced cloud dependencies, or mandatory logins.
How Promptfoo Works: A Declarative Approach
One of Promptfoo’s biggest strengths is simplicity. Promptfoo relies on declarative YAML configurations, allowing you to define test cases without wrestling with heavy Python notebooks or writing extensive boilerplate code.
A typical Promptfoo configuration involves three main components:
- Prompts: The templates you want to test (e.g.,
Convert this English text to {{language}}: {{input}}). - Providers: The AI models you are targeting. Promptfoo supports over 60 providers, including OpenAI (GPT-4o, GPT-o1), Anthropic (Claude), Google (Gemini), and local models via tools like Ollama.
- Tests (Inputs & Assertions): The variables you want to inject into your prompts and the rules (assertions) the output must follow.
Example: Evaluating a Translation Prompt
Running promptfoo eval executes a matrix: testing each prompt, against each provider, with each set of variables.
prompts:
- 'Convert the following English text to {{language}}: {{input}}'
providers:
- openai:gpt-4o
- vertex:gemini-3-flash-preview
tests:
- vars:
language: French
input: Hello world
assert:
- type: contains
value: 'Bonjour le monde'
- vars:
language: German
input: How's it going?
assert:
- type: contains
value: 'wie gehts'The Power of Assertions
While you can manually review the outputs in Promptfoo’s web UI, the real power lies in assertions. These automatically grade the LLM’s responses based on criteria you define.
Promptfoo offers various assertion types:
- String Matching: Checking if the output contains a specific word or JSON format (
type: contains,type: contains-json). - Semantic Similarity: Using embeddings to check if the meaning of the output matches an expected result, rather than relying on exact keyword matches (
type: similar). - Custom Logic: Running JavaScript or Python scripts against the output to validate complex conditions.
- LLM-as-a-Judge: Using another LLM (a “rubric”) to evaluate the output (e.g.,
type: llm-rubric,value: does not describe self as an AI, model, or chatbot).
Why Developers Choose Promptfoo
The acquisition by OpenAI isn’t surprising when you look at why developers have flocked to Promptfoo:
- Developer-Friendly & Fast: Promptfoo features live reloads, caching, and concurrency, making iterative prompt engineering significantly faster.
- Agnostic & Flexible: Promptfoo isn’t locked into one ecosystem. You can test OpenAI against Anthropic, use Python or Javascript for custom variables, and load test cases from YAML, CSVs, or even Google Sheets.
- Local & Private: By default, Promptfoo evaluations run locally on your machine, communicating directly with the LLM APIs. Your proprietary data and test cases don’t need to be uploaded to a third-party evaluation platform.
- CI/CD Integration: Promptfoo is designed to run in continuous integration pipelines, ensuring that every code (or prompt) change is automatically evaluated for regressions or security flaws before hitting production.
Conclusion: The Shift to Test-Driven AI
OpenAI’s acquisition of Promptfoo highlights a maturing AI industry. The novelty of chatbots has worn off; the focus is now on building reliable, secure “AI coworkers” and agents that interact with sensitive enterprise data.
Promptfoo provides the systematic approach required for this next phase. Promptfoo moves AI development away from “vibes” and manual testing, toward rigorous, automated evaluation and red-teaming. As AI systems become more complex and autonomous, tools like Promptfoo will be the essential guardrails keeping them on track.
To further enhance your AI security strategy and implement Promptfoo, contact me on LinkedIn or [email protected].
Relevant Resource List
- Promptfoo GitHub Repository: https://github.com/promptfoo/promptfoo
- Promptfoo Documentation: https://promptfoo.dev/docs
- OpenAI Announcement: Integrating Evaluation into B2B Applications
- OWASP Top 10 for LLMs: https://owasp.org/www-project-top-10-for-large-language-model-applications/
- Related: The New Triad of AI Security: Promptfoo, Strix, and CAI
Frequently Asked Questions (FAQ)
What is Promptfoo?
Promptfoo is an open-source CLI tool and library for evaluating and red-teaming LLM applications. It allows developers to test prompts, models, and RAG pipelines systematically using YAML-based configurations.
How does Promptfoo support red teaming?
It includes built-in capabilities to simulate common attacks like jailbreaks, PII leaks, and prompt injections, allowing you to automatically test if your LLM resists malicious inputs.
Can I use Promptfoo with models other than OpenAI?
Yes, Promptfoo is model-agnostic. It supports over 60 providers, including Anthropic (Claude), Google (Gemini), and local models via Ollama or other local runners.
What is 'LLM-as-a-Judge' in Promptfoo?
This is an assertion type where Promptfoo uses a separate, often more capable LLM to evaluate the output of your target model based on custom rubrics (e.g., 'Is the response helpful and professional?').
Why is Promptfoo useful for CI/CD pipelines?
It allows teams to treat prompts like code, running automated evaluation suites on every commit. This ensures that changes to a prompt don't accidentally introduce regressions or new security vulnerabilities.
