The NIST, the U.S. Commerce Department agency has re-released a testbed designed to measure how malicious attacks particularly attack that “poison” AI model training data might degrade the performance of an AI system.
The tool dubbed as Dioptra, the modular, open source web-based tool, initially released in 2022, seeks to help companies training AI models and the people using these models assess, analyze, and track AI risks. Dioptra can be used to benchmark and research models, as well as to provide a common platform for exposing models to simulated threats in a “red-teaming” environment.
Dioptra debuted alongside documents from NIST and NIST’s recently created AI Safety Institute that lay out ways to mitigate some of the dangers of AI, like how it can be abused to generate nonconsensual pornography. It follows the launch of the U.K. AI Safety Institute’s Inspect, a toolset similarly aimed at assessing the capabilities of models and overall model safety.
Dioptra is also the product of President Joe Biden’s executive order (EO) on AI, which mandates that NIST help with AI system testing. The EO establishes standards for AI safety and security, including requirements for companies developing models to notify the federal government and share results of all safety tests before they’re deployed to the public.
It delivers five products collectively as follows
- Preventing Misuse of Dual-Use Foundation Models – NIST AI 800-1
- Testing How AI System Models Respond to Attacks
- Mitigating the Risks of Generative AI – NIST AI 600-1
- Reducing Threats to the Data Used to Train AI Systems -NIST SP 800-218A
- Global Engagement on AI Standards – NIST AI 100-5
NIST doesn’t assert that Dioptra can completely de-risk models. It does propose that Dioptra can shed light on which sorts of attacks might make an AI system perform less effectively and quantify this impact to performance.
However, Dioptra only works out-of-the-box on models that can be downloaded and used locally, like Meta’s expanding Llama family. Models gated behind an API, such as OpenAI’s GPT-4o, can’t be tested with Dioptra for now.
