Top-Level Domain Bias in Search Engine Indexing and Rankings

As a search engine optimization (SEO) and domain name consultant, one of the questions I get asked most often about domain names is whether or not the domain name or TLD (Top-Level Domain) matters. Will the domain name ending have an effect on SEO or search engine rankings. Are certain domain name endings preferred by the search engines over other domain name extensions? I decided to answer this question based on search engine optimization testing and not just on my personal and professional experience.

In order to begin to answer this question, we have to register domain names, set up websites, set up unbiased SEO testing software, and track the results. That’s exactly what I did — and I used the help of two different organizations, Digitaleer and SIA (SEO Intelligence Agency). Clint Butler from Digitaleer help set up and monitor the testing, SIA helped provide the software and testing, and Hartzer Consulting provided the analysis for this research.

SEO TLD Domain Test Methodology

In order to properly test, in an unbiased way, whether or not the domain name ending has any impact on indexing and search engine rankings, we set up SEO testing. We will call them TLDs (Top-Level Domains) going forward in this document. The methodology we used was the following:

Pick a list of TLDs based on popularity and the number of domain name registrations. Pick both old and new TLDs, as well as ccTLDs (country code TLDs). Make up a word and register that word in all those TLDs. Choose one made-up keyword per domain name. Set up one website per domain name. Track the indexing of each website. Track the rankings of each keyword on each website. Track the progress and report the results.

Before I chose the domain names and the TLDs for the testing, I consulted with several domain name industry professionals and domain name investors. I wanted to make sure that the TLDs that were picked were acceptable for the test. Several recommended that .COM, .NET, and .ORG be used in the testing, as well as ccTLDs be used, as well. I chose the most popular new gTLDs based on domain name registrations, as well as popular ccTLDs (country code TLDs). I used data from nTLDSTats to help with these decisions.

Ultimately, the following TLDs were chosen for the test:

I “randomly” chose a made-up word that does not exist. In picking the word for each domain name, I chose the name of a popular domain name conference, NamesCon, and spelled it backward. At the time of domain name registration, there were only 16 search engine results for this made-up word in Google.com’s search engine results, so it was a good prospect to use. It’s important to note that the keywords chosen (one per website) are not related in any way to the made-up word (NamesCon backward). We didn’t try to rank for “nocseman” as a keyword, and that keyword is only present in the domain names. It is not present anywhere on the websites that were built.

Initially, we wanted to test the indexing of each of the websites. We let the websites sit for a certain number of days, and then when they weren’t indexed (no pages were indexed), we verified the websites in Google Search Console and manually submitted one page per site. Then, after a certain number of days, we used SEO indexing software to help index the rest of the pages that were not indexed. There were 25 pages per domain name for a total of 375 web pages.

We also tracked the search engine rankings of each keyword — one keyword was assigned per domain name.

We monitored the progress and reported the results in a few ways:

• The SEO Testing Software we used checked indexing and rankings of the assigned keyword for each domain name. We strongly believe in the method we used to check the indexing and rankings using the software.
• I manually performed a search at designated times using the site: search operator in Google to see how many pages were indexed.
• The other way I used was to review Google Search Console data and the reports that are provided in Google Search Console.

Timeline of the SEO Testing

We took several days to perform the SEO testing of these TLDs. We initially registered the domain names, created the websites, watched the indexing process of all pages, and the looked and analyzed the search engine rankings. Here is the timeline for the SEO testing:

Once the domain names were registered, the websites were set up, and the SEO testing software was in place, I tracked the progress, made some adjustments to the indexing process. For example, it was necessary to verify the websites in Google Search Console and manually submit one page using the URL Inspection Tool in Google Search Console. We previously had also tried to force indexing by submitting one page from each website using the Mobile-Friendly Test tool by Google. That did not work, so we were forced to use the URL Inspection Tool.

Previously, once a domain name was registered, the search engines would start crawling the websites, looking for new pages. This has changed drastically in the past few years, making it now necessary to “force” Google to crawl and index pages. For example, I now recommend using the URL Inspection Tool to ask Google to crawl a new page or re-crawl a web page that has been updated.

During the testing phase, and when the testing was complete, we discovered many different issues that are worth noting. For example, here are some of the highlights of the results:

• Out of 15 TLDs tested, two ccTLDs’ websites were determined to be in the main language of that country. This is despite there being no indication of what language the website is in — no hreflang tags or language declaration. Google just assumed the website was in that language based on the ccTLD. Furthermore, search results of any pages on that site include an option to translate the page into the other language. The .DE was considered to be a website in German. And the .CO was considered to be a website in Spanish.
• One website almost immediately got 40 percent of its pages indexed in Google: the .XYZ website. Google was very aggressive in crawling this one particular website, on the .XYZ TLD. More than any of the other domain names.
• A few websites were very difficult to get their pages indexed. Some failed miserably despite going through the same process as other domain names and websites.
• A few of the websites had their main keyword rank very well very quickly. Some took longer than others. One failed miserably and did not get pages indexed and didn’t have its keyword rank even in the top 100 search results in Google.
• One of the websites ended up getting none of its pages indexed according to our SEO Testing Tool, which was the .CO.UK ccTLD domain. We did find some pages indexed using other methods to check pages indexed.
• Checking for pages indexed reveals different results based on how I checked. Google Search Console has certain data, the site: search operator shows other data, and the SEO Software Testing Tool shows other data.
• Searching for the made-up word that only appears in all the domain names and not in content on the websites, there is, in fact a clear winner. Only one web page ranks first for this keyword. You can search Google for the made-up word “nocseman” and see the current results. Again, this keyword only appears in the first part of the domain name, nowhere else in the content on any of the websites.

When it comes to checking pages indexed, my professional opinion is that one particular method is the most reliable and “most fair” and unbiased method. While the other methods are typically used in the search engine optimization industry by many professionals, I am now questioning the data, as all sources show different data at the end of the day.

Even though Google does say that there are a certain number of pages indexed, if we rely specifically on Google Search Console data, we had clear winners when it comes to pages indexed. The .XYZ, .CLUB, .ONLINE, and .CO.UK TLDs performed the best and had more pages indexed. We have a clear loser, though: The .ORG site has only 17 pages indexed.

• Winner: .XYZ, .CLUB, .ONLINE, .CO.UK for pages indexed according to Google Search Console
• Loser: .ORG for pages indexed according to Google Search Console

When it comes to search engine rankings, at the conclusion of the test, 10 of the sites ended up having pages rank for their keyword in 55 positions in the top 100 search engine results. One could be considered a “loser” since it had its keyword rank in no positions in the top 100. This could potentially be a bias with the co.uk TLD, as the SEO software checked Google.com and not Google.co.uk. But, other ccTLDs tested, including the .DE and the .IN did have 55 positions in the top 100 search results. The others that didn’t do well were .ICU, .TOP, .SITE, and .VIP.

This is only a highlight of the data that was analyzed during the testing. I looked at a lot more data, such as impressions, clicks, average position, and more.

Source: (http://www.circleid.com/)