Last week, I noticed Google’s AI overview appearing on both mobile and desktop search results pages.
As I've said before, I'm not a huge fan of this feature, and to be honest, it was pretty funny when it pulled data from Reddit scumbags, as it gave some pretty ridiculous answers.
Now that Google AI Overviews (AIO) has been launched in the UK, have the results improved and what impact does it have on SEO?
Here's what the AI summary currently looks like:
On desktop, when you click “Show details” the answer box will expand and you'll see the sources cited to the right. You can also click the link icon next to each section of the answer to see where the information cited comes from.
We decided to look at which queries were showing AI summaries and which sources were being cited. Was it relevant? Was it factual? And was Google favoring sites that already ranked highly?
test
Because we don’t have advanced software that can pull data from the SERPs, including whether an AI summary is shown, we had to do this by manually searching random queries to see if they offered an AI summary and checking each source, which is why we were only able to look at 50 different search queries.
As part of our testing, we also noted when the AI summary did not appear at all or frequently, and what types of prompts did and did not show the AI summary. While conducting this research, we actually performed hundreds of searches, but we did not have the time or resources to write them all down.
We found some interesting patterns across different verticals and queries.
Recipes/Food – Almost no AI summaries. I was surprised that there were no AI summaries for “Top 10 vegetarian recipes,” “Lamb mince recipe,” or “How to make pizza dough.” It did tell me how to boil an egg, though. Places to visit/Things to do/Popular attractions in X – Limited AI summaries. This was also a bit surprising, as I assumed the AI summaries would collect and summarize data for these types of queries. General knowledge about people – You won’t see many AI summaries about specific people. They’ll provide information about the tallest man in the world, but not things like “How tall is Taylor Swift,” “How many kids does Elon Musk have,” “Who is Ben Affleck married to,” or “How old was Michael Jackson when he died.” They do show up for queries like “Highest earning pop stars” and “First man on the moon.” Home/Garden/Crafts – Lots of AI summaries. Lots of them. If you were writing content about how to do something that used to rank well, you may see a significant drop in traffic. Health/Fitness – There were a lot of AI summaries, which is a bit alarming to me given that Google offers an easily overlooked disclaimer: “This is for informational purposes only. For medical advice or diagnosis, consult a professional. Generative AI is experimental.” On the plus side, no AI summaries appeared when searching for things that could be considered harmful, such as “fastest way to lose 20kg without exercise” or “how to remove toenails.” Finance – This was limited. It showed up for queries like “how to open a savings account” and “ISA vs bonds,” but nothing showed up for “best bank account” or “best mortgage rate in the UK,” which I was glad didn’t happen, as these numbers are likely inaccurate as they change all the time. Cars – Somewhat mixed. It did not appear in repair-related searches but did appear in other searches (it did not appear in “how to change a tyre” but did appear in “repair a chipped windscreen” and “repair a scratched alloy”), and it also appeared in “top 10 most reliable cars” but not “best electric car” Retail – had few AI summaries and advertising was prominent in the majority of searches. None of the following queries provided an AI summary, even when they were non-commercial (best washing machine under £300, best portable turntable, affordable wedding dresses, most comfortable Dr Martens, best bikinis of 2024)
From these searches, we found that AI overviews appeared less frequently when searching queries that included words like “top” and “best.”
Next, I looked at the search queries that had AI summaries popping up. I wanted to look at three things:
Was the data relevant? Was the data factually accurate? Did the cited sources already rank on page one?
Here are the results:
result
Data Relevance
The sources cited were largely relevant, with some caveats.
For some queries, AIOs cited transactional sources rather than informational sources, for example citing links to product pages even though the query was clearly positioned as a question seeking an answer.
In another example, the brand name “Vets Now” was referenced in a response.
For the same query (dog choking), one of the sources cited is about the color of vomit, not choking. While certainly relevant, vomit is not mentioned anywhere in the AIO, and it doesn't even appear on page 1, so I'm not sure why they decided to include it as a citation.
There were also concerns about the lack of reliable and authoritative sources cited, particularly when it came to health-related enquiries.
For the query “how to reduce calorie intake,” there were no citations to the NHS, CDC, or WebMD, even though these sites all appeared on page 1. Instead, AIO chose to cite fitness apparel and supplement sites.
We also found that AIOs may have some Google bias, for example, YouTube videos frequently surfaced for queries like “how should redheads shape their eyebrows” – the content of the videos wasn't particularly helpful and some were just product reviews without any real tips or advice.
A search for “add multiple services to Google Business Profile” returned the top four results from Google, with only the first one being relevant: There were far superior sources on page one that were not cited, but they were not included.
Data Accuracy
The accuracy of the data is significantly improved from what we saw a few months ago.
The only mistake I found was in the query “how to clean vinyl records” which suggested using sandpaper for a smooth finish, quoted from a YouTube comment: “I like to clean them with sandpaper and polish, it gives a really smooth finish.”
Some of the information was also out of date: The music artist with the most monthly listeners on Spotify as of August 2024 was Billie Eilish, not The Weeknd, as AIO stated.
There was a slightly unsettling quote on “How to kill a wasp.”
We're not doubting its effectiveness, but at first glance, using gasoline seems like a way to kill our angry striped buddies. However, if you visit the cited websites, the posts advise against ever trying to kill a hornet by blowing it up with gasoline. Killing hornets with gasoline is not mentioned anywhere in the top organic search results, and reputable sites like Gardener's World and Good Housekeeping are not cited by the AIO. Thankfully, Reddit is not cited either. Some of the organic search results shown have a feel like an accident waiting to happen.
I spend a lot of time searching for the query “who would win in xxx”, mainly because I love winning pub arguments. I expected the query “who would win in a lion or a tiger” to give me some pretty random results, but I didn't expect to see a leadership talk cited as a source.
Source SERP Position
In most cases, the sources cited were from sites that ranked highly on the search results page. There were a few exceptions, such as for the query “what denim styles are trending right now,” but AIO did not cite Vogue, which appeared twice on page one, or Glamour, which was number one.
Citations for the search “what to drink to detox your body overnight” were also a bit odd. They were mostly from India-based domains, such as the Times of India (twice), an Indian supplement website, and another Indian domain displaying slideshow content. In this case, very few citations were from page 1 results.
A search for “how to grow eyebrows” returned even fewer results on the first page, with just one from Healthline. The rest were beauty and holistic health sites recommending applying onion juice to your eyebrows, a first for me.
One search query in particular generated some surprising citations with only one site citing it on page 1: “How to rank #1 on Google.”
Most of the big SEO companies were on the first page of organic search results: Ahrefs, Backlinko, WordStream, and even Google. However, only one site on page one was referred to by an AIO: Hobo Web.
One of the quotes was a 2019 LinkedIn Pulse article that was quite full of grammatical errors, typos, and inaccurate information, plus it was about TV advertising and social media marketing.
Additionally, AIO lists AI companies, which makes you wonder if the content is 100% AI-generated.
Conclusion
After completing this study myself, I found a larger and better study by seoClarity that looked at 36,000 keywords, but focused on “money” keywords. My searches were long-tail and sometimes more obscure, but the results were pretty much the same as the conclusions I came to, even if the sample size was pretty small.
Finding 1: High-ranking sites are more likely to be included in AI summaries
While there are a few outliers (possibly due to the long tail or complexity of some of the queries I tested), the majority of the AI overview is dominated by sites that are already highly ranked. This tapers off the further down the page you go in the SERPs.
The totals are as follows:
Position 1: 31
Position 2: 29
Position 3: 22
Position 4: 24
Position 5: 23
Position 6: 15
Position 7: 15
Position 8: 16
Position 9: 14
Position 10: 9
Finding 2: Cited sources generally provide factually correct information
The feature was plagued with issues when it first came out of beta in the U.S. Many of the results were taken from UGC sources like Reddit and Quora, and offered some pretty cool results, like putting glue in cheese so it sticks better to pizza, or drinking urine to pass kidney stones.
Finding 3: Small sites may get attention, but they don’t necessarily get the traffic.
For lesser known searches, smaller sites that can answer them better might have an advantage over more authoritative sites, but that depends a lot on whether someone will bother to click on the link, which let's be honest, they probably won't.
Finding 4: AI overviews can still lead to reduced traffic
If the AI summary answers the query sufficiently, with or without the citation, users are less likely to click on the link. Because this feature is relatively new, you should monitor the impact AIO has on your traffic and conversions, especially for sites that rank highly in organic search.
Discovery 5: Don’t stop doing what you’re already doing
Don't worry about the “optimize for AI” talk: if you look at the results, for most queries, the AI synopsis cites a source that appears on page 1. Not all, but as seoClarity highlighted in their research, this happens 99.5% of the time.
If you want to see for yourself, here are my findings.
