Chapter 5: Manipulating Delegated Judgment

Manipulation of delegated judgment is not a new problem with a new name. Influence operations, disinformation campaigns, and reputation attacks have existed as long as people have relied on information from others to make decisions. What has changed is the surface area available for intervention. When most of the information reaching a decision-maker arrived through personal relationships, physical media, and direct observation, the number of points at which that information could be shaped was relatively limited. When it arrives primarily through algorithmic systems, those systems become the intervention point, and they are considerably more accessible and more scalable than anything that came before.

The practical implication is that an adversary who wants to shape a specific person's decisions no longer needs to get close to that person. They need to get close to the systems the person trusts to make recommendations on their behalf. In many cases, those systems are designed to be open to exactly the kind of input that adversarial manipulation exploits.

Reputation Engineering

The most direct form of judgment manipulation involves shaping how a person or organization appears within the systems that assess credibility, authority, and trustworthiness. Search rankings, review platforms, professional network signals, and the data broker profiles that feed background check and due diligence tools are all systems designed to surface reputation-relevant information. They are also systems that can be engineered.

Positive reputation engineering, building synthetic credibility for a person or entity that has not actually earned it, is a well-documented phenomenon in contexts ranging from fraudulent reviews to manufactured professional histories to coordinated campaigns that inflate the apparent authority of a source or individual. The practical consequence for anyone relying on these signals to make decisions about who to trust, who to hire, who to engage in a transaction, or which information to believe is that the signal they receive may reflect engineering rather than reality. The more important the decision and the higher the potential gain from shaping its outcome, the greater the incentive for someone to invest in shaping the signals that inform it.

Negative reputation engineering, deploying information designed to damage how a person appears to search algorithms, review systems, or data aggregators, operates on the same infrastructure with the opposite objective. A coordinated effort to seed negative information about a person across sources that algorithmic systems treat as authoritative can produce results that appear in due diligence processes, background checks, and news searches without any of the individual sources being obviously fabricated. The aggregate picture that emerges can be genuinely misleading even when no single data point is technically false.

Synthetic Authority

A related but distinct mechanism involves the construction of artificial credibility for sources, advisors, and intermediaries rather than for the subjects of decisions. If a person relies on a set of sources, analysts, or advisors to inform their judgment, and those sources can themselves be made to appear more authoritative than they actually are, the manipulation operates at one remove from the decision-maker while producing the same effect on the decisions they make.

This matters particularly in contexts where high-visibility individuals operate through intermediaries whose judgment they have come to trust. A research analyst, a procurement advisor, a communications consultant, or a financial intermediary whose apparent credentials have been inflated, whose track record has been manufactured, or whose conflicts of interest have been obscured represents a point at which the information environment reaching a principal can be systematically shaped without the principal ever being directly targeted. The manipulation goes through the trusted intermediary rather than at the principal, which makes it harder to detect and easier to sustain.

The rise of AI-generated content has extended this problem in a direction worth understanding directly. A synthetic analyst, a fabricated expert, an AI-generated advisor persona with manufactured history and apparent credentials can now be constructed at a cost and speed that makes it viable for a much wider range of actors than could previously have sustained such an operation. The question of whether the sources informing a decision are what they appear to be has become considerably more complex to answer than it was five years ago.

Information Poisoning

Information poisoning operates at the level of the data that algorithmic systems consume rather than at the level of the systems themselves. If the data flowing into a recommendation system, a market intelligence platform, or a research aggregator can be shaped, the outputs of those systems will reflect that shaping without any direct intervention in the systems themselves. The pipeline gets contaminated at a point upstream from where the decision-maker encounters it.

This is particularly relevant for decisions that depend on aggregated signals rather than primary sources. An executive whose investment thesis is informed by market intelligence tools that aggregate analyst sentiment, news coverage, and social media signals is operating on a composite picture that can be shifted by coordinated activity across any of those input channels. The composite looks authoritative because it represents many sources, but if those sources have been systematically influenced, the composite inherits that influence without making it visible.

At the individual level, information poisoning can also operate through the specific algorithmic profile that platforms maintain for a person. The content a person sees, the connections a platform surfaces for them, and the recommendations they receive are all downstream of the behavioral model those platforms have built from their past activity. That model can be influenced by activity that is not their own, specifically by activity conducted through accounts associated with them, by information about them injected into the data layer, or by coordinated behavior that shapes how they appear to the platforms that serve them.

Workflow and Process Manipulation

As more consequential decisions move into AI-assisted workflows, a class of manipulation that targets those workflows rather than the underlying data becomes increasingly relevant. Prompt injection, the technique of embedding instructions within content that an AI system will process, represents the most technically specific example, but the broader category is any intervention that shapes how an AI assistant, research tool, or decision-support system processes and presents information to the person relying on it.

From a risk standpoint, the significance of this category is not primarily technical. It is the shift in attack surface that it represents. When a principal relies on an AI system to summarize research, filter communications, evaluate options, or prioritize considerations, the principal is no longer just trusting the underlying information. They are trusting the processing layer between the information and themselves. Anyone who can shape what that layer surfaces, suppresses, or frames has a pathway to the principal's judgment that does not require access to the principal directly.

This is a meaningful extension of the layered information environment problem identified in Chapter 4. Each layer of AI assistance that sits between a person and the raw information they are acting on represents a potential point of intervention, and the number of those layers is growing considerably faster than awareness of what they represent as an exposure surface.

Why Detection Is the Hard Problem

The mechanisms described in this chapter share a property that distinguishes them from most other forms of adversarial action: the person being targeted experiences nothing that signals the targeting. A successful recommendation manipulation produces a recommendation that feels normal, a synthetic authority figure presents exactly the appearance of a genuine one, and poisoned information looks like information. The intervention is invisible by design because visibility would defeat its purpose.

This creates a detection challenge that is qualitatively different from what most security disciplines are designed to address. Network intrusion leaves artifacts, physical breach leaves evidence, and financial fraud leaves anomalies that trained investigators know to look for. Successful manipulation of delegated judgment, by contrast, leaves a person who made a decision they believe was their own, based on information they believe was accurate, through a process they believe was sound. The absence of distress is not evidence of the absence of manipulation. It may simply be evidence that the manipulation worked.

Detection in this context requires something that most individuals and organizations have not built: an independent assessment of the information environment itself, conducted by someone who is not inside it. Chapter 6 begins to address what maintaining genuine judgment looks like in an environment where the manipulation described here is possible, and Chapter 7 offers a framework for the specific questions worth building into how consequential decisions get made.