.Google.com's John Mueller responded to a question concerning why Google indexes webpages that are actually forbidden from crawling by robots.txt as well as why the it is actually safe to overlook the relevant Browse Console files about those crawls.Crawler Web Traffic To Query Criterion URLs.The person talking to the concern recorded that bots were actually making links to non-existent concern specification Links (? q= xyz) to webpages with noindex meta tags that are actually additionally blocked out in robots.txt. What triggered the concern is actually that Google is crawling the hyperlinks to those pages, obtaining obstructed by robots.txt (without envisioning a noindex robotics meta tag) after that receiving shown up in Google.com Explore Console as "Indexed, though shut out through robots.txt.".The person asked the complying with concern:." But listed here's the huge question: why would certainly Google.com index web pages when they can't also see the material? What is actually the advantage during that?".Google.com's John Mueller confirmed that if they can not creep the webpage they can not find the noindex meta tag. He also makes an appealing acknowledgment of the website: hunt driver, encouraging to ignore the outcomes given that the "normal" users will not see those outcomes.He composed:." Yes, you're right: if our company can not crawl the web page, our team can not view the noindex. That claimed, if our experts can not creep the web pages, after that there's certainly not a whole lot for our company to index. Therefore while you might see several of those pages with a targeted site:- query, the normal customer will not view them, so I wouldn't fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it simply means the URLs are going to find yourself being crept (as well as find yourself in the Explore Console report for crawled/not indexed-- neither of these statuses create issues to the remainder of the internet site). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the constraints in using the Website: search advanced hunt operator for diagnostic explanations. Among those factors is because it's not hooked up to the frequent search index, it's a distinct point altogether.Google's John Mueller commented on the site search operator in 2021:." The brief answer is actually that a website: concern is actually not meant to be total, nor used for diagnostics functions.A web site inquiry is a details type of hunt that limits the end results to a certain internet site. It's basically only words site, a bowel, and afterwards the web site's domain name.This query restricts the outcomes to a details web site. It is actually not suggested to be a complete collection of all the pages coming from that internet site.".2. Noindex tag without using a robots.txt is actually great for these type of situations where a bot is connecting to non-existent pages that are getting found out by Googlebot.3. Links along with the noindex tag will definitely generate a "crawled/not recorded" entry in Browse Console and also those will not have a negative impact on the rest of the internet site.Review the inquiry and also respond to on LinkedIn:.Why would Google mark pages when they can not even find the material?Featured Image through Shutterstock/Krakenimages. com.