VHA lacks ‘formal mechanism’ for mitigating clinical AI chatbot risks, watchdog says
The analysis, released on Thursday by VA’s Office of the Inspector General, found that uses of generative AI chatbots across the Veterans Health Administration — VA’s healthcare arm — lacked the necessary oversight to mitigate potential risks resulting from these tools’ output. AI-powered chatbots have been documented to produce inaccuracies, such as omitting relevant data or generating false information.
OIG said it examined two chatbots used by VHA: VA GPT, “a general use AI chat tool created by VA,” and Microsoft 365 Copilot chat, “a general use AI chat tool that the VA provides to all VA staff.” The watchdog said both of these tools are dependent on clinical prompts to work and do not have access to web searches, meaning that “the chat tools’ knowledge base is not current.”
In its review, OIG said it found that VHA’s healthcare-related uses of AI “are driven by an informal collaboration” between the leadership of VA’s National Artificial Intelligence Institute and the department’s chief AI officer. Following interviews with VHA staff, however, the watchdog said it found these units did not properly coordinate with VA’s National Center for Patient Safety when it came to clinical uses of these generative AI tools.
OIG also said that VHA “does not have a formal mechanism to identify, track, or resolve risks associated with generative AI,” and expressed concern “about VHA’s ability to promote and safeguard patient safety without a standardized process for managing AI-related risks.” The report added that “not having a process precludes a feedback loop and a means to detect patterns that could improve the safety and quality of AI chat tools used in clinical settings.”
VA listed 227 AI use cases in its reported 2024 public inventory, with applications ranging in use from advanced medical devices to high-powered predictive algorithms designed to identify veterans at high risk of suicide. Of the totals reported in VA’s most recent AI inventory, 145 of these uses were listed as safety- or rights-impacting — meaning that these technologies could have a “significant effect” on an individuals’ wellbeing or legally-protected liberties.
In prior Nextgov/FCW reporting on VA’s use of AI to prevent veteran suicide, department officials stressed that such uses of AI are only meant to augment clinician outreach or bolster crisis line training efforts — an approach that researchers and veterans advocates have said is the only way these tools should be used when it comes to mental health interventions.
]]>