Weighing smoke: why GEO dashboards are mostly useless

By Iain, 05 Jul 2026

The promise of a tool that gives you similar search visibility to the pre-agent world is understandably seductive. But before you eagerly hand over your cash for a platform, consider whether an afternoon’s work a month might be a more cost-effective alternative.

I recently wrote about the GEO chimera, the cottage industry promising to optimise your brand into AI answers. My argument was that the levers being sold mostly did not exist. Another part of the industry is selling gauges rather than levers. Estimates of the spend on AI visibility tracking now run past $100 million a year, across hundreds of dashboards charting your brand’s presence in ChatGPT the way Moz once charted your rank in Google. The pitch is falling on the fertile ground of brand marketers standing on the precipice of everything they previously relied upon, staring into the coming void.

There is a post-millennial feel to all of this. The Hollywood screenwriter William Goldman is famously quoted as saying, “nobody knows anything.” The lesser-known part of his quote is, “every time…it’s a guess and, if you’re lucky, an educated one.” That is agentic search visibility in a sentence.

Most current tools run prompts daily, average the runs, report a visibility percentage with a trend line and a confidence interval. The averaging is not the problem per se. The problem is everything it is averaging, because the prompts in the measurement basket are generally steered by the analytics vendor, the query volumes weighting them are modelled fiction, the query surface being sampled is not one a human uses, and nobody has shown that the resulting score predicts a single outcome or conversion that is important to a business. Yet these tools are multiplying like vape shops on a dying high street.

The truth is the things we’re desperately reaching for do not really exist. There are minimal mechanisms to improve your visibility in search agent responses. And there is no way to reliably track and measure that visibility. After a quarter century of SEO and SERPs, this is a bitter pill that many are still unwilling to swallow.

The tasty candy currently being offered as an alternative is what I call precision laundering. Run a hollow measurement often enough that the noise cancels, and out comes a seemingly stable, decimal-pointed number that passes for knowledge. But while averaging buys a number, it cannot buy validity.

The dice cup

In January, Rand Fishkin published a detailed study of the search agent tracking industry. With Patrick O’Donnell of Gumshoe.ai, he recruited 600 volunteers to run twelve brand-recommendation prompts through ChatGPT, Claude and Google’s AI, 2,961 times in total, across categories from chef’s knives to cancer hospitals. Fishkin went in expecting to prove that tracking search agent responses was pointless. The results were stranger than that.

Ask a search agent for brand recommendations a hundred times, and the odds of getting the same list twice are under one in a hundred. The same list in the same order runs closer to one in a thousand. The brands change, the ordering changes, even the number of items changes. Fishkin’s conclusion on position tracking was blunt. Any tool that gives a “ranking position in AI”, he wrote, “is full of baloney.”

This is not one maverick standing alone. An arXiv study of repeated GEO measurements found that the set of brands returned for an identical prompt overlapped only 45-59% between runs, with wide variance, and concluded that single observations of search agent visibility are misleading and that visibility must be treated as a probability across repeated runs. A second paper makes the statistical point directly. Citation metrics from generative engines are random variables, not fixed values. Ahrefs, coming at it sideways, found that Google’s AI Mode and AI Overviews cite different sources 87% of the time for the same query.

This churn cannot be controlled for. A language model does not look up an answer. It generates one token at a time, choosing each word from a range of plausible options with a built-in measure of randomness (known as “temperature”). That randomness is a vital setting because it is what makes the output read as fluent rather than canned. Ask the same question twice, and the answers drift apart on their own.

Add web search and the dice get rolled twice more: once when the model breaks your question into its own set of search queries (referred to as the “fan out”), and again when it picks which of the pages it fetched are worth citing. Even the settings meant to hold steady wobble once a system is live, where requests are batched together, silently routed to whichever copy of the model is free, and swapped onto new versions partway through whatever you thought you were measuring.

So the variance Fishkin measured is not immaturity that will calm down as the category develops. It is the product working as designed, and a platform that removed it would be shipping a worse product, one that answers every user with the same canned response. So visibility in a search agent is a distribution, permanently.

Underneath the churn, Fishkin did at least find something stable. The consideration set, the pool of brands the model draws from at all, held steady even as the ordering scrambled. Sony, Bose and Apple appeared in nearly every headphone run, with top brands in tight categories showing up in 90 to 100% of responses, while broad categories like brand design agencies scattered into the 30s and 40s.

When Amanda Natividad had 142 humans write their own prompts for the same intent, the phrasings scored a semantic similarity of 0.081, so about as similar as a haiku and a shopping list. But the consideration sets held anyway. It appears that you are either in the pool or not, and membership can only be estimated slowly over 60-100 runs. That is a narrow but valid signal, but it is the only one.

The wrong window problem

Say the tool runs its prompts a hundred times a week and reports a probability. It is still pointing the instrument through the wrong window. Most trackers query vendor APIs because they are cheap and scriptable. But GPT 4.5, accessed via an API endpoint, is not ChatGPT. The consumer product has memory, custom instructions, conversation history, account context, location inference and a system prompt the API caller never sees. It routes between model variants, and the variants often disagree with each other.

A reverse-engineering study of ChatGPT’s search stack found that GPT 5.2, 5.3 and 5.4 share a knowledge cutoff and a family name yet produce different fan-out queries and retrieve different sources, then cite different pages from the same prompt. More than 90% of ChatGPT’s weekly users are on the free tier, where the default experience runs fewer searches and produces fewer citations than the paid tiers the tool tends to sample.

A tracker is therefore a mystery shopper filing reports averaging hundreds of visits from a showroom that no customer ever visits. What you get is a tight confidence interval around a population that contains few if any of your actual customers. To add insult to injury, the showroom is often completely refitted most nights without notice.

On May 7th this year, OpenAI started rendering clickable brand links inside answers, and referral traffic jumped 157.7% week over week while the share of visits landing on brand homepages went from roughly 30% to 60% and stayed there. On March 4th, the default model switched to GPT 5.3 and the average number of domains cited per answer dropped from 19 to 15. Trying to optimise and measure this ever-shifting landscape appears to be a fool’s errand. Those of us who lived through Google’s Florida, Panda and Penguin algorithm updates understand that the ground can shift beneath you. But when it moves in a consequential and constant way, it quickly becomes unmanageable.

Inventing the denominator

Google’s Search Engine Results Pages (SERPs) were tangible. I know this from painful experience: a marquee product dropping from position two to three, and senior managers rushing around with their hair on fire, demanding to know how we could immediately recover. So SERPs were a double-edged sword, but a sword nevertheless.

Agent prompt volume is modelled, not measured. No lab publishes query data, and the “AI keyword planners” that sell it are either extrapolating from opt-in panels that capture a fraction of a per cent of traffic or resolving your prompt back to a Google keyword and calling the result “AI demand”. Vendors of AI search visibility platforms typically build estimates from clickstream panels and extrapolation models. The clickstream panels skew toward a small, unrepresentative sliver: desktop, mostly Chrome, no apps, no mobile, drawn from people who were paid or unknowingly opted in to have their browsing watched.

When Profound started showing prompt volumes, 26,000 monthly searches for “social listening tool” and the like, the SEOs Steve Toth and William Alvarez compared the figures against keyword volumes and described the numbers, in public posts, as a scam. Conductor, a company that sells adjacent tooling and has every commercial incentive to stay quiet, published a piece calling any AI prompt volume metric fundamentally misleading and warning that resource decisions built on it are at best a gamble.

The product vendors perhaps realise they are on shaky ground, which is why their write-ups read more like incantations than sound methodologies. Writesonic, for example, explains its numbers as an “ensemble methodology with mathematical bias correction”. Translated from corporate-ese, this is guessing twice and averaging. A deeper problem is not the sourcing but the object itself. Keyword volume worked because millions of people typed the same or very similar strings into a box, and the party that owned the box logged it.

Prompt volume has no equivalent. Nobody who owns the box is publishing the log, so what’s sold as prompt volume is a panel vendor’s sliver of traffic, extrapolated, or a prompt resolved back into a Google keyword and re-badged. Conversational prompts also fragment across twenty-word phrasings of similar intent, which is exactly what the 0.081 similarity score mentioned above measured. So the “volume” of any single prompt is as much an artefact of how a vendor chooses to bucket the many paraphrases as anything else.

The composite scores also disagree with each other. Digiday found marketers running the same brand through rival platforms and getting inconsistent results, which is what happens when every vendor averages a different invented basket over a different sampled surface. Two thermometers in two different incorrect rooms will still provide precise readings. An average has to be an average of something that can be used to make a decision, and no independent work I can find ties a visibility score to revenue or any other tangible conversion.

Some tools bolt a tracking pixel and a server-log reader onto their dashboards. They catch AI crawlers fetching your pages and referral clicks from services like ChatGPT. The referral half is real. Someone did land on your site from an AI answer, and it is a number grounded in observation rather than a model. It is also the same server log you could read yourself for free. The catch is that the pixel records who arrived, not what put your page in the answer, and it records that arrival separately from the visibility score generally charted beside it. Putting a true referral figure next to a modelled visibility score does not make the score any more valid. This is just precision laundering in a different guise, using a real number to lend credit to the invented one.

An industry whose entire product is measurement should be drowning in published methods. The opposite seems to be the case. Strip out the laundered averages and the invented denominators and a residue remains. It is narrow and cheap, and mostly it does not require buying an expensive platform subscription. It boils down to five simple steps.

Measure membership, not rank

The consideration set is the one construct that survived the SparkToro data. If you must track something, track the probability of appearing across a basket of intent-varied prompts, sixty to a hundred runs per basket, and treat the number as a coarse presence gauge rather than a steering wheel. It will tell you whether you are in the pool and whether something large has shifted. It will not tell you anything at a more granular level.

Fix what the machines say, at the source

The Tow Center at Columbia found AI search engines failed to retrieve correct citation information in over 60% of 1,600 test queries. That error rate is the strongest argument for monitoring your own brand, because incorrect pricing and dead product features propagate into answers, and the fix is easy. Find the source page the engine cites and correct it.

Do the retrieval plumbing

ChatGPT’s live search runs mainly on Bing’s index. Seer Interactive found that 87% of ChatGPT citations matched Bing’s top results, compared with 56% for Google. So verify your site in Bing Webmaster Tools and wire up IndexNow, then allow OAI-SearchBot plus OpenAI’s published IP ranges, which is the only inclusion requirement OpenAI itself states. Serve plain server-rendered HTML, because OAI-SearchBot does not render JavaScript and parse rates collapse from 94% on static pages to 23% on client-rendered ones.

Then use the one first-party gauge that exists. Microsoft’s AI Performance report in Bing Webmaster Tools, launched in February, shows which pages Copilot and Bing’s AI cited, as well as the grounding queries that fetched them. It is free, which tells you something about what the paid tools are really worth.

Win the machine queries, then the editorial record

When ChatGPT answers, it does not search your prompt. It rewrites it into a handful of machine-generated sub-queries (the “fan-out”), and runs those against the index. ALM Corp’s analysis found 95% of fan-out queries carry next to no human search volume, meaning the queries that decide whether your page gets retrieved are ones no keyword tool tracks, because no human ever types them. A Search Engine Land case study on New York hotels shows the consequence. For the same fan-out queries, Google results slightly favoured the Baccarat, while Bing results clearly favoured the Fifth Avenue Hotel. Since ChatGPT retrieves from Bing, the Fifth Avenue appeared in 20% of its answers across 68 trials, the Baccarat in 1.5%. So leading the human-visible rankings in the wrong index bought almost nothing.

Meanwhile, membership of the consideration pool is built in the editorial record rather than on your own site. SE Ranking found sites with over 32,000 referring domains are 3.5 times more likely to be cited than sites with under 200. Which is to say one proven tactic that meaningfully influences GEO is actually a basic SEO principle, and Digiday’s panel of practitioners has been saying as much for months.

Read your own logs

Server logs show which bots fetch what and at what depth, and whether your robots.txt decisions align with the extraction economics in my chimera article. The monitoring probes appear there too. Everything reaching into your site leaves a trace in your logs, and reading those traces costs nothing.

The size of the prize

AI chatbots barely register as a traffic source. Cloudflare Radar puts the combined share of chatbots at 0.29% of search referrals in May 2026, while Google took 87.6%. Measured against a single site’s traffic rather than the whole web, Conductor benchmarks AI at around 1%, where organic brings 25%. However, the visitors who do arrive are unusually valuable. Similarweb reports that ChatGPT referrals convert at 7.1%, a rate only paid search beats. And whatever your analytics report shows, the true number is higher because an estimated 70% of AI visits have no referrer and are filed as direct traffic. A small channel, then, but one that sends good visitors yet is hard to tease out in the measurements.

The promise of a tool that gives you similar search visibility to the pre-agent world is understandably seductive. But before you eagerly hand over your cash for a platform, consider whether an afternoon’s work a month might be a more cost-effective alternative. Track properly or not at all: run each prompt 60 to 100 times and read the result as a rough probability, because a handful of spot checks is just noise. Fix any incorrect on-site information. Do the Bing plumbing. Earn third-party mentions. Watch the server logs. Use Bing’s free citation report. That really is the long and the short of it.

And no, I didn’t have “more reliance on Bing” on my 2026 bingo card either. We live in truly strange times.