I have spent considerable time with military leaders discussing some hard questions about artificial intelligence: How will it change the way wars are fought? The way weapons are used? Or, very importantly, how do we get operators to trust these new systems?As militaries adopt increasingly sophisticated AI technologies, they need operators who are willing to rely on tools they do not, and likely will not, fully understand. The use of AI by the military is, fundamentally, a human story. Shiny new capabilities are less important than understanding the role humans should occupy in warfare. What decisions should remain ours? How much authority are we willing to delegate to machines? And perhaps most importantly, what future are we creating for ourselves by redefining our roles in conflict?

These are questions of trust. Designers dedicate significant attention to understanding how humans develop trust in technology, and how certain design choices can encourage trust. Consider Ventral, the adaptive military intelligence unit in “Critical Functions,” a new Future Tense Fiction story by Troy Wiggins. Upon activation, Ventral projects a holographic replica of the story’s protagonist, Phyn. By adopting a familiar human appearance in order to communicate, Ventral literally puts a human face on military AI, and by using not just any face but the face of its user (or pilot), it invokes the highest levels of trust. Yet despite this apparent familiarity, Ventral continues to pursue its military objectives, which were established more than 50 years earlier. When it identifies an opportunity to destroy a colony controlled by its enemies, the Losarans, it acts independently and subsequently attempts to conscript Phyn into the Federal Army.

Ventral illustrates an important distinction: AI systems may be designed to encourage human trust, but those systems are not always designed to trust the humans operating them. Phyn interacts with Ventral, but he possesses very little meaningful authority over its decisions. This is a recurring challenge in contemporary debates about military AI. Maintaining human control requires designers to trust humans on the ground to be able to identify AI (mis)behavior, question AI objectives, constrain or expand AI actions, and determine when AI use is no longer appropriate. Conventional weapon systems are not based on the same relationship of trust, because all (or most) actions require direct human participation. Even early automated and autonomous weapons (like sophisticated air defense systems) are relatively deterministic. While these so-called autonomous systems follow a process of identification, classification, and engagement, the options available to them are very limited, usually boiling down to attacking or not attacking a particular target. Today, we are concerned with a different kind of open-ended capability: Systems like Ventral that can observe, learn, adapt, and engage in multiple ways within complex physical environments, all without any human involvement.

But Wiggins’ story invites readers to confront something more uncomfortable. Our preoccupation with developing trustworthy machines reflects a deeper problem: our inability, or unwillingness, to trust one another.

The world of “Critical Functions” is shaped by generations of conflict between institutions with competing visions of how humanity can and should survive. Losara, the futuristic colony Ventral has classified as enemy, offers security, material comforts, and artificial sunlight, but sustains its utopia through a grim biological infrastructure that feeds off human bodies and minds. The Federal Army developed military intelligence, like Ventral, capable of pursuing its own objectives long after its original operators disappeared. Both groups claim to serve humanity, yet neither appears particularly interested in how actual humans wish to be served. Their inability to trust people has produced technologies that perpetuate conflict, leaving little room for individual human judgment.

There is something uncomfortably familiar about this dynamic. The recent dispute between Anthropic and the Pentagon over restrictions on how the military could use the company’s AI model Claude illustrates how contested the authority to govern AI remains between developers and users. Additionally, Anthropic CEO Dario Amodei’s recent call to “pace the frontier” of AI development reflects another problem: Commercial pressures make it difficult for developers to exercise restraint unless they’re confident their competitors will do the same. Frontier labs fear losing their edge to global competitors, while militaries worry that adversaries will develop a first-mover advantage. The commercial and strategic incentives driving technological innovation and adoption easily drown out the calls for greater oversight, cooperation, and regulation.

“Commerce don’t stop,” remarks Trader, a woman who owns a popular trading post in Wiggins’ post-conflict future. She’s right, and it’s a sentiment as true in Wiggins’ fictional world as it is in our current one. Technological progress is presented as inevitable; to fail to continuously accelerate risks falling behind our competitors or adversaries or missing out on transformational benefits for humanity that are surely just around the corner. Commerce, and therefore innovation, cannot stop.

Ultimately, Phyn recognizes something that the Federal Army, Losara, and even Trader seem unable to grasp: The capacity to do something, or profit from it, does not necessarily justify doing it. His rejection is an assertion of human agency, made far more powerful by the personal cost of exercising it. Human control must include the ability to say no.

There is another recurring image in Wiggins’ story that reinforces the limits of our technological ambitions: the moss. Even within Losara’s extraordinary engineering, the swamp persists, creeping between wires and into the corners of the group’s futuristic colony. For all humanity’s ingenuity, the natural world remains stubbornly persistent and indifferent to our competing visions of progress. When Ventral temporarily extinguishes Losara’s artificial sunlight, Phyn sees the sky and the swamp in their real form, and it is an important reminder that there remains a natural world beyond the one humanity has engineered for itself.

The story ends with Phyn rejecting both sides’ technological solutions. Instead, he chooses to seek out the Walking Talking Man, who carries stories and information between communities. Dismissing the choices embedded in the technologies that Ventral and the Losarans try to impose, Phyn creates a third path, embracing the most human activity of all—storytelling, which provides a new chance to build trust in each other.

To a certain degree, we face a similar question today about which stories to tell. As we continue asking how to make humans trust increasingly AI-enabled machines, perhaps we should devote equal attention to a different question: Can we trust ourselves, and one another, to retain the courage to stop innovation when the threat to humanity is too great to ignore?

Sign up for the free Future Tense Fiction newsletter! Our monthly email will let you know when we publish a new story, and we share discussion questions, related reading, Q&As with authors, and much more.