What Is robots txt and Why It Cannot Deindex Anything

Robots.txt lives at the root of a protocol and domain, and you read it by adding /robots.txt to the host, per Google Search Console Help.^[1] Its job is narrow: it tells automated crawlers which paths they may request. The part nobody repeats often enough is that it has almost nothing to do with whether a page appears in search results, and treating the file as an index switch is how site owners quietly damage their own crawl budget.

The Line of Text That Decides What a Crawler Fetches

Google's own documentation is blunt about the scope: robots.txt manages crawler traffic, and it is not a mechanism for keeping a web page out of Google (per Google Search Central). Read that twice, because it kills the assumption most people bring to this file.

A robots.txt file is a plain text document served from the root of a domain. It contains groups, and each group pairs one or more user-agent lines with rules. Only two rule types matter in practice: Disallow, which tells a crawler not to request a path, and Allow, which carves an exception back out of a broader disallow.

Anything the crawler requests that you did not disallow is fair game by default. There is no implicit whitelist and no implicit blacklist. The file's silence permits, it never forbids.

User-agent: *
Disallow: /cart/
Disallow: /search?
Allow: /search/about

Sitemap: https://example.com/sitemap.xml

The User-agent: * line addresses every crawler that has no more specific group. A group naming a particular bot overrides the wildcard one for that bot, so you can let a shopping bot through while blocking a scraper. The Sitemap: line is not a directive at all; it is an advertisement, and it only counts if you point at the live XML file.

That format is the entirety of it. That is also why the file is such a poor control surface: it expresses permission about fetching, and nothing else. For the fuller picture of what a disallow rule actually buys you, our breakdown of how little a disallow rule protects covers where the limits bite in practice.

Writing Your Own Directives Without Breaking the Site

Fetch the current file before you touch anything. A surprising number of "my traffic dropped" tickets trace to a disallow rule someone pasted a year ago and forgot.

1. Open the file in a browser by adding /robots.txt to your domain.
2. Copy the live content somewhere you can restore it from, in full.
3. Add or edit only the paths you can name and justify.
4. Save, then reload the file in the browser to confirm what is actually being served.
5. Compare your live output against a robots txt example you trust, line by line.
6. Resubmit or check the URL in Search Console, since a listing that has returned 404 for 30 days drops out of the report even though Google keeps checking it in the background.

Step five is where most edits die. A stray capital letter in Disallow, a missing slash, or a group with a user-agent line and no rules underneath it will serve a file that looks correct and behaves like a blank one.

Then there is the factor that makes raw, quiet speed a liability rather than an advantage: RFC 9309, the 2022 standard that defines the protocol, says crawlers should not use a cached copy of the file for more than 24 hours unless it is unreachable. Push a bad rule live and a crawler can be acting on it inside a day. There is no soft-launch window and no staging environment at this layer.

Which is why we push one rule on anyone editing a production file: never publish a disallow you cannot undo from memory. Write the old block down first. If you cannot recite what you deleted, you have not finished the job.

The Edits That Quietly Cost You Crawl Budget

Every path you disallow stops the crawler from requesting it. That sounds tidy until you remember what lives on those paths.

Internal links on a disallowed page still exist in the HTML of every page that points to it. A crawler reading those links finds a target it is not allowed to fetch, and that relationship goes unused. Disallow enough of your own URLs and you strip out the internal link graph your site depends on to distribute authority. The pages that remain look, from the crawler's side, increasingly isolated.

The wildcard Disallow: / is the most destructive single line in the file, and people paste it by accident constantly. A staging rule copied into production takes the whole domain out of the crawl queue in one save. If you have ever seen a site vanish from a report overnight, that is usually the line.

The more interesting failure is the opposite one. Some site owners block a URL, see it disappear from a rank tracker, and conclude the page is deindexed. It is not. A page blocked by robots.txt can still surface in search results, just without the description Google would otherwise pull from the page.^[2] Google can see a link to it, can decide it is relevant, and can show a bare URL with no snippet. You removed your ability to control the appearance, not the listing itself.

Two more small traps earn their own paragraph. Inurl robots txt searches are how people check what competitors expose, and the results frequently show a platform's stock template rather than anything a human wrote. A robots.txt checker tool verifies syntax, not intent, so it will happily confirm a rule that blocks your money pages. Neither tells you whether the file is doing your site any favors.

That is what a scoring engine looks at when it decides whether a published page is worth indexing in the first place, and it is a different question from the one the file answers. If you want the deeper version, the false confidence a disallow rule creates goes further into how a blocked URL keeps earning impressions you cannot see.

When to Leave the File Alone

Here is the honest split. You need a custom robots.txt file in exactly two situations: you have a real, stable pattern of URLs that waste crawl budget, or you run multiple environments under one domain and need to fence a staging area off.

If neither applies, the highest-value edit is deletion. That is true for most WordPress installations, where a plugin or a host ships a working default file and a hand edit only introduces the risk of a typo. We have written about why the stock file is usually right and where the exceptions actually sit, and the short version is that a platform-managed file fails safe more often than a hand-rolled one.

The decision comes down to who owns the URL structure.

• If you control paths and generate them in code, a custom file is worth the maintenance.
• If a CMS, a theme, or a page builder generates your URLs, the platform's default is almost always the better bet.
• If your site is too young to have wasted crawl budget, you have nothing to block yet and a custom file is premature.

The last signal is the one people skip. Crawl budget only becomes scarce once you have more URLs than a crawler will fetch in a reasonable window. A fifty-page site has no budget problem, and adding a fragile file to a tiny site swaps a nonexistent risk for a real one.

The Reach of a Crawl Directive, and Its End

Here is where the layer stops. A robots.txt file governs fetching. It cannot affect a page's ranking, cannot build authority, and cannot notice when a page you published six weeks ago loses position. Backlinks are not built for you. We track authority and surface the gaps, but link building is outbound work, and no configuration file changes that. Monthly article allowances differ by plan, and rank is measured on days 1, 3, 7 and 14, with paid plans refreshing a page that slips against a fresh read of page one.

So the file is a small part of a larger operation, and the part most teams can hand off entirely. GrowGanic reads the pages ranking on page one, writes for the intent they show, cites only statistics from sources it checked, publishes straight to your CMS, and tracks AI Overview visibility next to Google rankings. If you need a look at what a page actually ranks for before you decide anything about crawl rules, the diagnosis comes first and the configuration second.

Free gives you one evidence-checked article. Pro runs up to 30 a month, and Business up to 150. See the current numbers at growganic.io/pricing, or send us your domain and we will show you what the pipeline produces before you pay anything.

Stop writing articles. Start shipping them.

Related reading

• SEO Robots.txt: Why Your Disallow Rules Are Worth Less Than You Think
• Where to Find robots.txt and What the File Reveals About Crawling
• Best SEO Automation Tools: The Ones That Fix What They Publish

Frequently Asked Questions

What is robots.txt used for?

Its single job is managing crawler traffic. You put a robots.txt file at the root of a domain, and crawlers read it before requesting other URLs on that host. In it you list paths a crawler should not fetch, and optionally advertise where your sitemap lives. Blocking a path reduces what a crawler requests, which matters when you have thousands of low-value URLs consuming a limited fetch allowance. It has no bearing on whether those URLs can rank. Anything a crawler may fetch is then governed by the page itself, by internal links, and by whatever signal the page earns.

What does "blocked" by robots.txt mean?

It means a crawler was told not to request that exact path. It does not mean deindexed, hidden, or penalized. Google can still list a blocked URL in search results if other pages link to it, but the result may carry no snippet because Google could not read the page content. You also lose the ability to shape how that listing appears. In Search Console, "blocked by robots.txt" is a status about exclusion from the crawl queue, not a report card on the page's quality or its ability to rank for a query.

How to open a robots.txt file?

Add /robots.txt to the host in any browser, exactly the way Google Search Console Help describes it. So a site at example.com serves its file at https://example.com/robots.txt. There is no admin panel involved and no login required; it is a plain text file served like any other URL, which means you can fetch it from a phone. If the page returns a 404, no file is published and crawlers treat that as unrestricted access. That is worth checking before you assume a directive exists because a tutorial said it should.

Is robots.txt legally enforceable?

The protocol is a convention, not a law, and courts have treated it as evidence of a website owner's stated boundaries rather than a binding contract. RFC 9309 describes what compliant crawlers should do, and reputable ones follow it, including the major search engines. Nothing physically prevents a scraper from ignoring the file and requesting every path anyway. So treat it as a request honored by good actors, not a security control. If you need real protection for private content, put it behind authentication, not behind a text file.

Sources

1. Google Search Console Help
2. Google Search Central