At the 2023 Defcon hacker conference in Las Vegas, prominent AI technology companies partnered with groups that pursue algorithmic integrity and transparency to drive thousands of participants into generative AI platforms to uncover weaknesses in these critical systems. The “red team” exercise, which was also backed by the U.S. government, was a step toward scrutinizing these increasingly influential but opaque systems. Now, Humane Intelligence, a nonprofit that conducts ethical AI and algorithm evaluation, is taking this model a step further. The group issued a call for participation on Wednesday at the National Institute of Standards and Technology, inviting any U.S. resident to participate in a qualifying round of a nationwide red team exercise to evaluate AI office productivity software.
The preliminary rounds will be held online and open to developers and the public as part of NIST's AI challenge, “Assessing the Risks and Impacts of AI (ARIA).” Participants who make it through the preliminary rounds will participate in an in-person red team event at the Conference on Applied Machine Learning in Information Security (CAMLIS) in Virginia at the end of October. The goal is to expand the capacity to conduct rigorous testing of generative AI technologies for security, resilience, and ethics.
“The public who use these models don't really have the ability to judge whether they're fit for purpose,” said Theo Skedas, chief of staff at Humane Intelligence, “so we want to democratize the ability to do the evaluation so that anyone who uses these models can evaluate for themselves whether they're meeting their needs.”
At the final CAMLIS event, participants will split into red teams that will attempt to attack the AI system and blue teams that will work on defense. They will use the AI 600-1 profile, part of NIST's AI Risk Management Framework, as a benchmark to measure whether red teams can produce results that violate the system's expected behavior.
“NIST's ARIA uses structured user feedback to understand real-world applications of AI models,” says Rumman Chowdhury, founder of Humane Intelligence, a contractor for NIST's Emerging Technologies Office and a member of the U.S. Department of Homeland Security's AI Safety and Security Committee. “The ARIA team is primarily experts in socio-technical testing and evaluation, and we are leveraging that background to advance the field toward rigorous scientific evaluation of generative AI.”
Chowdhury and Skeerdas said the partnership with NIST is just one in a series of AI red team collaborations Human Intelligence will announce with U.S. government agencies, international governments and NGOs in the coming weeks. The effort aims to make it more common for companies and organizations that develop algorithms that are currently black boxes to provide transparency and accountability through mechanisms such as the “Bias Bounty Challenge” to reward individuals who find problems or unfairness in AI models.
“The community should be broader than programmers,” Skærdas says. “Policymakers, journalists, civil society, and non-technical people should all be involved in the process of testing and evaluating these systems, and we need to ensure that under-represented groups, like speakers of minority languages and people with non-majority cultures and perspectives, can participate in this process.”
