The AI Security Question Leaders Should Be Asking Instead
Security leaders are under pressure to answer a familiar question: What data was this model trained on? It is a reasonable question. It speaks to confidentiality, intellectual property, and the fear that proprietary information could become part of a model's general knowledge. But according to one security officer, stopping there leaves dangerous gaps in a layered security program. The better question is: How do you maintain authority and integrity of the data in question?
That shift in framing matters because AI systems are no longer just passive tools. They are increasingly embedded in workflows, making decisions, calling APIs, and communicating with other agents. The risk is not only what a model knows. The risk is what a model can do, with whose permissions, and under what controls. Without clear answers, organizations can deploy capable AI agents that exceed their intended authority or act on data they should never touch.
The Wrong Question and the Better One
When security leaders gather, the training-data question dominates. It is often driven by confidentiality concerns. Companies want assurances that their source code, customer records, internal research, or strategic plans will not leak into a model that later serves competitors or the public. That concern is valid. Data leakage, model inversion, membership inference, and prompt-based extraction are real risks. But the training-data question is backward-looking. It asks what already happened. It does not answer who can access the model, what the model can access, or how those permissions are enforced at runtime.
The better question is about authority and integrity. Authority asks who or what has permission to read, modify, or act on data. Integrity asks whether the data and the actions taken on it can be trusted. For AI agents, these questions become operational. An agent may be authorized to summarize documents but not to delete them. It may be allowed to query a customer database but not to export the results. It may be permitted to draft an email but not to send it without human approval. These distinctions are not academic. They are the difference between useful automation and a new class of insider threat.
Least Privilege Still Matters
For decades, least privilege has been a cornerstone of enterprise security. The principle is simple: give users, systems, and processes only the access they need to perform their function, and no more. Authentication and access controls enforce that principle. AI does not change the need for least privilege. It makes it more urgent. An AI agent can operate at machine speed, across many systems, and without fatigue or hesitation. If it is overprivileged, it can cause damage faster than any human user.
Modern identity controls help. Session-based role-based access control, often implemented through protocols such as OIDC and on-behalf-of token flows, allows organizations to scope permissions to a specific context. Instead of giving an agent broad, long-lived credentials, security teams can grant narrow, short-lived access that reflects the task at hand. Appropriate permission scoping ensures that an AI agent never exceeds its technical capability. Even if the model is manipulated through prompt injection or other attacks, the blast radius is limited by what the agent is actually allowed to do.
Human-in-the-Loop and Agent-to-Agent Communication
As AI agents begin to communicate with other agents, the control challenge grows. One agent may request data from another. A chain of agents may execute a multi-step workflow. Each hop introduces new opportunities for misunderstanding, manipulation, or privilege escalation. Strict human-in-the-loop architectures become more important, not less. The goal is not to block automation but to position human operators where they can recognize and correct issues before damage occurs.
Human-in-the-loop can take many forms. High-risk actions such as payments, production changes, or data deletion can require explicit approval. Anomalous behavior can trigger review. Agents can be required to explain their reasoning and cite sources. The exact design depends on the business context, but the principle is consistent: authority should be delegated carefully, and accountability should remain clear. If an AI agent acts, a human or a governed process should be able to answer why.
The Attacker's Toolkit Three Years Later
The offensive security landscape has changed dramatically. Three years ago, finding an exploitable condition, designing an exploit, and deploying it required significant expertise. Today, AI has eroded that asymmetry. Attackers can use AI to scan for vulnerabilities, generate proof-of-concept code, craft phishing messages, and adapt to defenses more quickly. The result is not simply that attacks are faster, though speed is a major factor. The deeper change is that the skill barrier has dropped.
Attackers who previously lacked the ability to exploit a flaw can now leverage AI to bridge the gap. In some cases, they can move faster than skilled defenders. This changes the economics of attack. When the cost of finding and exploiting vulnerabilities falls, the volume of attacks rises. The security industry has already seen a sharp increase in vulnerability reporting. National vulnerability databases have struggled to classify reports quickly enough, leaving many vulnerabilities unclassified. The bottleneck is no longer discovery alone. It is triage, prioritization, and remediation.
Doing More With the Same Team
Security teams are famously understaffed. AI has not magically solved that problem. In one organization, the security team processed just over nine times the number of vulnerabilities in a year compared with the previous year, while maintaining the same staffing level. That increase would have overwhelmed a traditional process. By using large language models and supporting automation harnesses, the team handled the surge without adding headcount and simultaneously reduced its mean time to remediate by just over five percent.
That outcome was not trivial. It required experienced human engineers to design the system, integrate the tools, and keep the workflow current as models evolved. The lesson is not that AI replaces security staff. It is that AI can amplify a skilled team, but only if the team knows how to use it. Investing in infrastructure around AI tools is as important as the tools themselves. Without that investment, AI can create new noise, false positives, and unverified conclusions.
Hiring for an AI-Dependent Security Program
AI is changing what organizations look for when hiring security talent. Current AI systems can produce work on par with many entry-level analysts and engineers. They can write scripts, summarize alerts, draft reports, and suggest remediations. But they also produce confidently wrong outputs. The most valuable skill is often the experience to recognize when an AI is wrong and to correct it. That requires a broad range of knowledge across diverse technical systems, as well as familiarity with the compliance and governance requirements that drive many security programs.
This raises the bar for all talent. It also creates a paradox: entry-level roles that once provided training grounds for new professionals are increasingly automated. Without those roles, how will the next generation gain the experience needed to operate AI systems effectively? Organizations need to think deliberately about apprenticeship, mentorship, and rotational programs. Otherwise, they may save on junior headcount today and struggle to fill senior roles tomorrow.
Model Collapse and the Need for Input Diversity
There is another concern that is less often discussed: generational processing and model collapse. If the majority or all inputs into AI systems are themselves generated by AI systems, the quality and diversity of training data can degrade. This is similar to the need for genetic diversity in a population. Large language models need input diversity that today still requires human curation. If organizations minimize human labor at the same time they need it most, they may weaken the very systems they depend on.
Labor and data are often treated as separate problems. Historically, human labor was how fresh, diverse inputs were provided to systems. Reducing that labor may reduce the diversity of future data. For security teams, this is not an abstract concern. Detection, response, and governance all depend on high-quality, varied data. If that data becomes homogeneous or self-referential, models may become less reliable just as attackers are becoming more capable.
Application Security Moves Earlier
One of the most promising developments is in application security. AI can help move security concerns earlier in the software development life cycle. Instead of waiting for a bottlenecked review period, AI can evaluate code and designs against a curated list of criteria and policies. It can surface concerns and offer guidance in real time. Developers can propose and implement solutions with specialist review, often avoiding the need for additional architecture and design requirements to filter back down the pipeline.
This shift does not eliminate the need for human expertise. It changes where that expertise is applied. Security specialists can focus on policy, threat modeling, and complex decisions while AI handles routine checks. Teams report that they no longer feel like they are playing catch-up to stay current with changes. Basic tasks that once depended on external development teams can become internal side projects, because with explicit review, the team knows the LLM can handle them. That can make a significant difference in cadence and responsiveness.
Source:Help Net Security News