‘Over 800 Biases Uncovered’ As Pentagon Ends AI Chatbot Pilot Program For Military Medicine

The US Department of Defense’s Chief Digital and Artificial Intelligence Office (CDAO) has concluded a pilot program focused on using AI chatbots in military medical services.  

In a Jan. 2 announcement, the DoD said the Crowdsourced AI Red-Teaming (CAIRT) Assurance Program pilot focused on using large-language models (LLM) for clinical note summarization and as medical advisers in the military.

It comes as more AI firms have begun offering their products to the US military and defense contractors to investigate their usefulness in military applications.

CoinTelegraph’s Stephen Katte reports that, according to the DoD, the pilot was a red-teaming effort conducted by technology nonprofit Humane Intelligence.

It attracted over 200 independent external participants, including clinical providers and healthcare analysts, who compared three prominent chatbot models.

Analysts from the Defense Health Agency and the Uniformed Services University of the Health Sciences also collaborated with the other participants, testing for potential system weaknesses and flaws while the chatbots were used.

According to the DoD, the pilot discovered a few hundred possible issues when using chatbots in military medical applications.

“The exercise uncovered over 800 findings of potential vulnerabilities and biases related to employing these capabilities in these prospective use cases.”

“This exercise will result in repeatable and scalable output via the development of benchmark data sets, which can be used to evaluate future vendors and tools for alignment with performance expectations,” the DoD said.

The Chief Digital and Artificial Intelligence Office’s lead for the initiative, Matthew Johnson, said the results will also be used to shape future DoD research and development of Generative AI (GenAI) systems that may be deployed in the future.

The CDAO was established in June 2022 to oversee and advance the integration of digital and artificial intelligence technologies within the US military and defense operations.

Last November, a bipartisan US congressional commission said the country should focus on developing an initiative similar to the Manhattan Project to advance artificial intelligence development amid growing competition with China.

Among a list of specific recommendations, the commission said the US secretary of defense should mark AI projects with the highest national priority designation.

Meanwhile, social media and tech firm Meta has started offering its artificial intelligence model Llama to the US military and defense contractors for national security purposes.