.Google's John Mueller responded to a question regarding why Google.com marks web pages that are actually prohibited from crawling through robots.txt as well as why the it is actually safe to dismiss the relevant Browse Console reports concerning those creeps.Robot Traffic To Concern Criterion URLs.The person asking the concern documented that crawlers were developing hyperlinks to non-existent concern guideline Links (? q= xyz) to pages with noindex meta tags that are also blocked in robots.txt. What urged the inquiry is actually that Google is actually crawling the links to those web pages, receiving obstructed by robots.txt (without watching a noindex robots meta tag) then obtaining shown up in Google.com Look Console as "Indexed, though blocked by robots.txt.".The individual inquired the complying with concern:." However listed below's the significant question: why would certainly Google index webpages when they can't even view the content? What's the conveniences during that?".Google.com's John Mueller affirmed that if they can't crawl the web page they can not see the noindex meta tag. He likewise makes an appealing acknowledgment of the website: hunt operator, advising to dismiss the results given that the "normal" individuals won't view those results.He wrote:." Yes, you're correct: if our team can not creep the webpage, our team can not observe the noindex. That pointed out, if our team can't crawl the pages, at that point there's certainly not a whole lot for us to index. So while you may see some of those webpages along with a targeted web site:- query, the typical user won't see all of them, so I would not bother it. Noindex is likewise fine (without robots.txt disallow), it simply means the Links will wind up being crept (as well as wind up in the Explore Console document for crawled/not recorded-- neither of these conditions create concerns to the remainder of the internet site). The fundamental part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limitations in using the Internet site: search progressed search operator for diagnostic explanations. One of those factors is actually given that it is actually certainly not attached to the regular search mark, it is actually a distinct factor altogether.Google.com's John Mueller talked about the internet site hunt driver in 2021:." The brief answer is that a website: inquiry is actually certainly not meant to become total, neither used for diagnostics functions.An internet site question is a details sort of hunt that restricts the outcomes to a particular web site. It's essentially merely the word site, a digestive tract, and afterwards the internet site's domain.This query limits the outcomes to a specific internet site. It is actually not meant to become an extensive selection of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is actually alright for these kinds of circumstances where a crawler is actually connecting to non-existent pages that are getting uncovered through Googlebot.3. URLs with the noindex tag will certainly generate a "crawled/not catalogued" entry in Explore Console and also those won't possess an unfavorable result on the remainder of the web site.Check out the inquiry and address on LinkedIn:.Why will Google.com mark web pages when they can not even view the content?Included Picture through Shutterstock/Krakenimages. com.