How many times have you been under pressure — the client or the boss needs answers by yesterday — and in front of you sits the inviting interface of Perplexity or Claude, promising to do in minutes the work that would take you hours, if not days?
And here comes the deafening song of yet another digital siren: hit the “Analyse” button and trust the output blindly.
I was chewing on this dilemma when, a while back, I read Nico Dekens‘ post, “The Slow Collapse of Critical Thinking in OSINT Due to AI”, and I decided it wasn’t the umpteenth piece on “the AI revolution” — it was something more, something that concerned me directly.
It’s a warning sign the OSINT community can’t afford to ignore.
A NECESSARY DISCLAIMER: this is not a post against Artificial Intelligence (LLMs and the like).
We use it here, extensively and in all sorts of ways, with great satisfaction.
This post is meant, rather, as a reflection on how we are using it. I want to start from Dekens’ observations to explore an idea that fascinates me more and more: AI, used without a robust critical framework, can become a spectacular amplifier of cognitive biases and a cause of atrophy in our fundamental analytical skills.
The greatest danger, in my view, isn’t that these tools are harmful or inaccurate. The risk is that they’re too good at creating a pleasant, frictionless, almost idyllic user experience. And it’s that very perfection that lulls our critical faculties to sleep.
The cleaner the interface and the faster the answer, the higher the risk that the analyst delegates to it not just the heavy lifting, but the reasoning itself.
The Comfortable Illusion of Automation
Dekens points the finger at our growing, uncritical dependence on automated tools. This cognitive bias has a precise name: automation bias, the human tendency to trust decisions made by an automated system, downgrading one’s own judgement even in the face of contrary evidence.
Picture this scenario: a junior OSINT analyst is using TurboWhatsit 3000, the latest-generation AI-based OSINT tool, capable of analysing thousands of social media posts and handing you a thousand oh-so-useful analyses. Mid-analysis, the account under examination is flagged as a “bot” with a 95% confidence score, no explanation given. The junior analyst — maybe under pressure (as if that ever happens…) — sees that reassuring number and closes the analysis on the spot, reporting to the client that the account is a bot. The label is already stuck on, the system flagged it, and you wouldn’t want to contradict the system, would you?
And so?
And so the manual verification an experienced analyst would have done by instinct is gone: checking posting-frequency data, the originality of the profile pictures (a quick reverse search?), linguistic inconsistencies, the nature of the interactions with other users, and so on.
In this scenario, the tool didn’t just provide a lead — it short-circuited the investigative process. It replaced what should be an investigation with a mere classification.
The real drama, though, doesn’t lurk in the blatant errors. An LLM claiming the Eiffel Tower is in Rome is easy to debunk (for most people). The deadly risk for an intelligence analysis comes from outputs that aren’t obviously wrong, but plausibly wrong. AI is a master at producing content that looks correct, uses the right terminology and cites believable sources. It creates that veneer of authority that turns manual verification into an almost heroic act of scepticism, a pointless waste of time (but don’t you want to trust this very latest release of your favourite LLM?).
This problem, born at the individual level, can become a systemic vulnerability when it’s baked into an organisation’s standard operating procedures. For the sake of efficiency, scalability and standardisation, larger and more mature companies integrate these tools into their production pipelines through procedures along the lines of: “Step 1: enter the target into Tool X; Step 2: analyse the results with a confidence level above 90%, etc.”.
This way, the company risks institutionalising automation bias. The analyst who questions the tool’s result is no longer seen as diligent, but as inefficient — the one who “doesn’t follow the procedure” and slows the workflow down. A perverse feedback loop sets in: the company trusts the tool, the procedures reinforce that trust, and analysts are trained to trust the tool more than their own instinct. The whole intelligence function risks becoming fragile, hostage to the intrinsic flaws of an algorithm that no one on the team designed, knows how it works, or is able to fully understand.
And this brings us to the gravest strategic consequence. A systemic dependence on a handful of commercial tools becomes a single point of failure for adversarial manipulation. A hostile actor, well-resourced and studying how a popular OSINT tool works, can craft and spread data designed specifically to be misread by that algorithm. They can poison the information well of every single organisation that “drinks from that source”, leading entire intelligence communities to the wrong conclusions.
Augmented Bias: When AI Confirms Our Worst Mistakes
If the machine’s error is a problem, there’s something far more insidious. It’s not AI getting it wrong on its own. It’s AI helping us get it wrong more efficiently, amplifying our own cognitive biases.
The most powerful and dangerous of them all is confirmation bias: our natural tendency to seek, interpret and remember information in a way that confirms our pre-existing hypotheses, ignoring everything that contradicts them. It’s an intelligence analyst’s worst enemy.
And here, modern large language models (LLMs) become a double-edged sword. An LLM isn’t a search engine that finds facts; it’s a text-generation engine that completes word sequences in a statistically probable way. That means it can be led, knowingly or not, to generate “evidence” in support of a premature theory.
Picture a CTI (Cyber Threat Intelligence) analyst who suspects, on a hunch, that a certain APT (Advanced Persistent Threat) group is behind an attack. Instead of asking an LLM “analyse this attack’s data”, they might ask: “find evidence linking APT-X to this incident”. The obedient AI will hunt for all the evidence that lets it produce a plausible text, rich in context and technical jargon, that builds connections, highlights similarities in TTPs (Tactics, Techniques, and Procedures) and constructs a narrative confirming the analyst’s initial suspicion.
The AI isn’t lying. It’s simply doing the job it was assigned. It has become our own personal, extremely powerful echo chamber, a tireless generator of “evidence” for our worst hunches.
How do you defend against this? The antidote isn’t technological, it’s methodological. This is where structured analytic techniques stop being an option and become a vital necessity. To start, I’m thinking in particular of the Analysis of Competing Hypotheses (ACH), a methodology developed by Richards Heuer Jr. for the CIA (Psychology of Intelligence Analysis is freely available). ACH forces the analyst to do the opposite of what confirmation bias would suggest: instead of looking for evidence in favour of their preferred hypothesis, they must list all the plausible alternative hypotheses and weigh each piece of evidence against every one of them. It’s a slow, deliberate, cognitively tiring process. It’s the exact opposite of pressing a button.
In an age of AI, ACH and similar techniques could become a kind of cognitive emergency brake against the avalanche of confirming “evidence” an AI can produce in seconds. Or what if we forced the LLM itself to follow a process like the one shown here?
The combination of AI and cognitive bias creates a new and dangerous form of intelligence failure: reports that are “analytically sound” but factually wrong. The report might be well written, internally consistent, and apparently backed by a mass of data. But if that “data” was generated or poorly collected by an AI to satisfy the analyst’s flawed request, the conclusion — however logical — is built on sand. This makes peer review nearly impossible. A reviewer reads the report, the logic holds, the cited “evidence” seems to support the conclusion. How are they to know that the cornerstone of that argument isn’t a verified fact, but text generated by an LLM? The only way would be to redo the entire job from scratch, wiping out the efficiency gains. It’s no longer just a failure of logic; it’s a failure of provenance.
From Analyst to Validator: The New Frontier of OSINT Skill
All of this leads us to an inescapable conclusion. The OSINT analyst’s role is undergoing a radical transformation. If in the past our main value lay in the ability to find information hidden in the noise, in the age of AI — where information is abundant and hard to verify — our value is shifting towards the ability to validate, contextualise and critically assess information.
As a result, the very definition of “senior analyst” is changing too. It’s no longer, or not only, the person who knows Google’s advanced search operators by heart or has access to niche tools. The senior analyst of the future is the one who possesses a deep methodological scepticism. It’s the person who, faced with an AI output, doesn’t ask “what is it telling me?” but rather “why is it telling me this?”. They wonder what training data led the AI to that conclusion, what implicit biases that data might contain, and what the most likely ways are for that specific model to fail or be fooled.
Skill is shifting from search to meta-cognition, from knowing the tools to verifying the algorithmic source.
To find our bearings in this shift, we might think about changing our approach: stop treating AI as software and start treating it as a new, peculiar source of HUMINT (Human Intelligence).
Think of AI as an informant. This informant is incredibly fast, has read almost everything ever published on the internet, and can recall the tiniest details. But, like any informant, it has its flaws. It’s prone to exaggerating to please its handler (you). Sometimes it makes up details — the hallucinations — to fill the gaps in its knowledge (even though Anthropic has greatly mitigated this problem with the release of its latest models, which were then promptly pulled from availability for non-US users). It isn’t always aware of its own prejudices, absorbed from the mountain of text it learned from. And, above all, its “access” to information (the training data) is often… murky.
As you would with any human source, its output must never be taken at face value. It must be independently verified. Its reliability must be constantly reassessed over time. And its motives and limits must always be questioned.
This paradigm shift has enormous implications for how intelligence teams should hire, train and structure themselves. Maybe we should start looking less for certifications on specific tools and more for character traits like intellectual curiosity, cognitive humility (the ability to say “I don’t know” and to question your own certainties) and a natural predisposition to scepticism.
Training programmes should evolve. Instead of teaching only how to use a tool, they should train analysts to break it. They should be built on simulations where analysts are presented with AI-generated outputs — some correct, some subtly wrong, some blatantly manipulated — and their job is to separate the wheat from the chaff. It’s not entirely pie-in-the-sky reasoning: SANS has already run a webcast on detecting AI in OSINT investigations. The goal isn’t to create expert users, but expert validators — people with “cognitive antibodies” against algorithmic disinformation. Perhaps we’ll see new specialised roles emerge: the “AI Output Validator” or the “Algorithmic Red Teamer”, whose sole purpose is to play devil’s advocate against the conclusions of automated systems.
To make this shift more concrete, I tried to map out how our workflow is evolving.
| Process Stage | Traditional OSINT Workflow (Human-Centred) | Augmented OSINT Workflow (AI-Assisted) | New Cognitive Risk Introduced |
|---|---|---|---|
| 1. Data Collection | Manual search, scripting, API use, targeted scraping. The analyst sees and selects the sources. | Input of a generic prompt. The AI performs a massive collection from opaque sources (its training set). | Algorithmic Selection Bias: the analyst loses control of and visibility into the primary sources. |
| 2. Processing | Manual reading, categorisation, translation and synthesis. The analyst is immersed in the raw data. | The AI automatically summarises, translates and categorises thousands of documents in seconds. | Loss of Context: the analyst no longer “feels” the data, missing nuance, sarcasm or weak signals. |
| 3. Analysis | Hypothesis formulation, manual connection of the dots, evidence assessment. A slow, deliberate process. | The AI suggests connections, identifies entities and proposes hypotheses based on statistical patterns. | Automation Bias & Confirmation Bias: the analyst is pushed to accept the most “probable” hypothesis suggested by the AI. |
| 4. Production | Report writing based on manual analysis. Every claim must be backed by a traceable source. | The AI can generate report drafts, complete sections and create explanatory text based on its findings. | Provenance Risk: the report contains claims whose ultimate origin is an opaque statistical model, not a verifiable source. |
So now what?
Why is all this useful to you, today, in your work?
Because ignoring this shift isn’t an option. Our adversaries are already using it.
November 2025: Anthropic documented the first cyber-espionage campaign orchestrated by an AI (the GTG-1002 case, here the full technical report). A state-sponsored group manipulated a commercial AI agent into autonomously carrying out roughly 90% of the operations — reconnaissance, vulnerability discovery, lateral movement, exfiltration — against some thirty targets across technology, finance, government and manufacturing, at a request rate impossible for any human operator. The detail that hits close to home: the tools were the usual ones (nmap, sqlmap and friends), chained together via MCP — the very orchestration infrastructure we analysts celebrate for its efficiency. The difference between an investigation and an attack no longer lies in the toolbox: it lies in who’s holding the wheel. Independent analyses of the case converge: SOCRadar, SOCFortress, ExtraHop, Astrix, LastPass.
And they’ll use it to deceive us, to manufacture disinformation on an industrial scale, to cover their tracks. And we’ll use AI to defend ourselves, to analyse enormous datasets, to find the signal in the noise. The match will be played on this field. Surviving and thriving in this new environment requires an upgrade not of our software, but of our mindset.
We can’t stop the tide, but we can learn to ride it. Here’s a small repertoire of practical strategies you can start applying right away.
- Default Distrust Principle (Zero Trust for AI). Borrow the zero trust concept from cybersecurity and apply it to analysis. Never trust an algorithmic output by default. Reverse the burden of proof. Don’t ask yourself “is this result true?”. Ask: “how could I prove it’s false?”. Actively look for evidence that contradicts it. Treat every AI claim as a hypothesis to verify, not as an established fact.
- Mandatory Triangulation. A golden rule of intelligence that becomes even more vital. Never accept a critical data point, a key conclusion or an identification from a single AI tool without independently verifying it through at least two other sources. These can be: (a) another AI tool, ideally based on a different model; (b) a manual search of primary sources (archives, public records, original sites); or (c) data from another intelligence discipline, if you have access to it.
- “Squeeze” your Tool. Before integrating a new AI tool into your workflow, give it some time. Don’t just use it. Study it. Read the technical documentation, the white papers, other analysts’ reviews. Try to understand what data it was trained on and what its stated limits are. Then put it to the test: feed it queries whose exact answer you already know and watch how it behaves. Test it with ambiguous or deliberately misleading data. Understanding its potential failure points is as important as knowing its capabilities.
- Demand provenance. If you use an AI agent that orchestrates OSINT tools (today the standard is MCP: shodan, maigret, whois and half the OSINT analyst’s arsenal now talk directly to LLMs — soxoj’s curated list gives a sense of the ecosystem), the rule is one: the model must run the tool and read its real output, never “remember” it. A whois that was run is data; a whois remembered by an LLM is hearsay.
We’re living through a fascinating and dangerous moment of transition. AI hands us powers that the generations of analysts before us could only dream of. But every new power brings new responsibilities and new, invisible vulnerabilities.
Our ability to exercise critical thinking, scepticism and methodological rigour has never been so important. These aren’t soft skills; they’re the core competencies that will set us apart from the machines we use, and that will let us stay relevant, effective and — above all — trustworthy.
This leaves me with a question, which I put to myself as much as to you: are we building tools that make us smarter, or tools that let us be more quickly and more convincingly stupid?
And if artificial intelligence is our new informant, who’s writing its reliability report?
Leave a Reply