DISA’s updated tech watchlist gets specific on AI capabilities

As the entire Pentagon works to better understand artificial intelligence and machine learning, the Defense Information Systems Agency is taking a more comprehensive approach to analyzing AI capabilities and their potential use cases.

In its so-called “Tech Watchlist” for fiscal 2024, DISA now has multiple items that cover specific capabilities and issues related to artificial intelligence — from leveraging large language models to what guardrails are needed when deploying AI. The new categories have been divided up from a single item that was dedicated to tracking overarching AI/ML technologies, according to Steve Wallace, DISA’s chief technology officer and director of emerging technologies.

The update comes as the Pentagon pushes for improved and responsible deployment of AI. Last week, the Department of Defense released its new Data, Analytics, and AI Adoption Strategy. For DISA, the specific areas will help inform how it leverages the technology in the future, Wallace said Monday while unveiling the new watchlist at DISA’s annual forecast to industry event.

“There is nothing more important than the ability to operate a system that you deliver. You could deliver the most spectacular, exquisite capability in the world, but if it is too hard to operate then you’ve really not done a whole lot because it’s likely down more than it is up,” he said. “So, how are we going to insert artificial intelligence into that chain?”

DISA’s watchlist — which is routinely updated at least every fiscal year, but oftentimes more frequently — is a broad inventory of over two dozen new technologies the agency has identified as ones it’s interested in pursuing. The items are divided by maturity levels, from technologies just being surveyed to actual prototypes that the agency is gearing up to deploy, such as the recently awarded Thunderdome zero-trust cybersecurity effort.

One new category in the “monitor” subdivision tackles AI operations and was born from conversations the agency is having with the Defense Innovation Unit (DIU) about a prototype it has in the works, Wallace said. The AI-enabled prototype is “taking log sets from given infrastructure components and given systems and then using a model to train itself over the course of weeks or months to do predictive analysis or predictive reliability analysis,” Wallace said.

DISA is hoping to learn some lessons from the prototype work in order to improve how it addresses vulnerabilities in the Defense Department’s networks by using artificial intelligence to predict them far in advance.

The agency has also added an item in the “monitor” subdivision of the watchlist that is dedicated to trust, risk and security management of artificial intelligence. Wallace noted that this category will explore how the Pentagon can keep pace with its adversaries’ use of AI, while also identifying technology guardrails DISA will need to implement.

Wallace pointed to some capabilities that act as a proxy for publicly available large language models as an area of concern. These tools will take in queries from users, run them against a contained dataset, and eventually pass them to public large language models such as OpenAI’s ChatGPT or Google’s Bard, he said.

Broadly, large language models are deep learning algorithms that are trained with massive datasets to recognize, summarize, translate, predict and generate convincing, conversational text and other forms of media in a manner that mimics a real human being.

“The department has a number of concerns against querying those commercial, large language models,” Wallace said. “How are we going to protect ourselves? How are we going to deal with that until such time that these models exist in a [government] cloud or higher environment?”

Despite the concerns, DISA hasn’t sworn off large language models as the fiscal 2024 technology watchlist renamed its generative AI category, added earlier in 2023, to “concierge AI.” The technology currently sits on the “planning” subdivision of the watchlist — meaning the agency has an understanding of how it could impact and integrate into missions.

“What we’re looking for out of a concierge AI … is the idea of having for the business side of the house that ChatGPT-like experience of being able to interact and ask questions of data sets,” Wallace said. “The other side of that coin that we sort of bundle under this, although it’s a slightly separate effort, is how are we helping the analysts. On the security platforms, how are you helping the analyst more quickly diagnose threats [and] put the pieces together far more quickly?”

It’s possible DISA will award an other transaction agreement for a prototype of the capability in fiscal 2024, he added.

Although the watchlist offers a snapshot of DISA’s technological endeavors, the agency does have a technical strategy in the works that will serve as a more specialized counterpart to DISA’s overall strategy, Wallace said. Although the strategy is still in its first draft, Wallace teased three main points it will cover: simplification of systems, integration across the agency and iterative delivery of new capabilities.