Robots.txt for Multilingual Sites: A Practical Crawl Control Guide

If you run an overseas content site with multiple language versions under subdirectories like /en/, /ja/, or /es/, your robots.txt file is probably not doing what you think it is. A single Disallow line written for your default language root will silently ignore every other language path. This guide walks through the exact problem, the safe fix, and the verification steps you need.

Why Crawl Control Matters for Multilingual Sites

Search engines allocate a crawl budget to each domain. For a site with dozens of language versions, that budget gets split across every path prefix. Low-value template pages — think auto-generated profile listings under /people/, filter pages, or paginated archive shells — consume that budget without adding ranking value. Worse, if those pages are thin or duplicated across languages, they can dilute your site’s overall quality signals.

The goal is simple: tell crawlers which paths to skip, in every language, without accidentally blocking anything useful. But robots.txt syntax is path-based, not language-aware. A rule for /people/ only matches that exact prefix. Your /ja/people/ and /es/people/ directories are completely different URLs from the crawler’s perspective.

The Default-Language-Only Trap

Here is the most common mistake. Your site’s default language is English, served from the root. You add this line:

User-agent: *
Disallow: /people/

You test it. It works. You move on. Three months later, you check Search Console and see /ja/people/ pages indexed with thin content. Why? Because Disallow: /people/ does not match /ja/people/. The crawler treats /ja/ as a distinct top-level directory. Your rule only blocks the English version.

The fix is not clever. You need one Disallow line for every language prefix that contains the blockable path. If you have five languages, you write five lines. It is repetitive, but it is also predictable and debuggable.

The Wildcard Trap

You might be tempted to write something like:

User-agent: *
Disallow: /*/people/

This looks efficient. It appears to match any language prefix followed by people. But wildcards in robots.txt are dangerous for two reasons.

First, the * matches any number of characters, including slashes. /*/people/ will also match /some/other/nested/people/ if such a path exists. If you later add a legitimate section like /tools/people-finder/, this rule might block it. Second, wildcard behavior is not fully consistent across all search engines. Google supports * and $, but other crawlers may interpret them differently or ignore them entirely. Your carefully crafted pattern might silently fail on Bing or Yandex.

The safer approach is explicit per-language rules. Yes, it means more lines. But each line is obvious, testable, and works identically across every crawler.

Per-Language Rules: A Full Example

Assume your site has four languages: English at the root, Japanese at /ja/, Spanish at /es/, and German at /de/. You want to block the /people/ profile listing pages in every language. Your robots.txt should look like this:

User-agent: *
Disallow: /people/
Disallow: /ja/people/
Disallow: /es/people/
Disallow: /de/people/

# Allow everything else
Allow: /

Notice the pattern. The default language root gets its own line. Each language prefix gets an identical line with the prefix prepended. This covers all cases with zero ambiguity.

What if you also have a blockable admin path? Add it for each language the same way:

User-agent: *
Disallow: /people/
Disallow: /ja/people/
Disallow: /es/people/
Disallow: /de/people/
Disallow: /admin/
Disallow: /ja/admin/
Disallow: /es/admin/
Disallow: /de/admin/

Keep the list flat and alphabetical by path. This makes it easy to spot missing languages when you add a new locale later.

Implementation Options: Static, Per-Language, or Dynamic

You have three ways to deploy these rules.

Option 1: One static robots.txt at the domain root. This is the simplest and most common. You maintain a single file with all per-language rules. The downside is that the file grows as you add languages. For five languages and three blocked paths, you have fifteen lines. Still manageable.

Option 2: Per-language robots.txt files. You can place a separate robots.txt inside each language directory, like /ja/robots.txt. This works, but it fragments your configuration. You must remember to update every file when rules change. Most site owners find this error-prone.

Option 3: Dynamic generation. If you use a framework like Next.js, you can generate robots.txt programmatically. A robots.ts file can read your site’s locale list and emit the correct Disallow lines for each language. This is the most maintainable approach for large sites.

This generates the exact same output as the static file but updates automatically when you add a locale. Choose dynamic generation if your site already uses server-side rendering. Otherwise, stick with the static file — it is easier to audit.

How to Verify Your Robots.txt

Writing the rules is only half the job. You must verify they work.

First, fetch your robots.txt in a browser and confirm the content matches your intent. Then use Google Search Console’s robots.txt tester to check specific URLs. Paste https://yoursite.com/ja/people/ and confirm the tester reports “Blocked by robots.txt”. Repeat for each language version of each blocked path.

Second, use the URL Inspection tool to request indexing for a blocked URL. It should show that Google cannot crawl it. Remember: a blocked page will eventually drop out of the index, but only if Google re-crawls it and sees the block. This can take weeks.

Third, check your server logs or Search Console crawl stats after a week. You should see fewer requests to the blocked paths. If you still see crawls, your rules are wrong — usually a missing language prefix.

Robots.txt vs. Noindex: Know the Difference

Robots.txt is a crawl directive. It tells crawlers not to fetch a URL. It does not tell them to remove an already-indexed page from search results. If a page is already in Google’s index and you add a Disallow rule, Google will eventually drop it, but only after re-crawling and seeing the block. This can take a long time.

If you want a page gone from search results immediately, use a noindex meta tag or X-Robots-Tag HTTP header. This is an index directive — it tells Google to exclude the page from results even if it is crawled.

The correct strategy for low-value template pages is often both:

1. Add Disallow rules to robots.txt to stop wasting crawl budget.
2. Add noindex to the pages themselves to remove them from the index quickly.

For multilingual sites, apply noindex consistently across all language versions. A noindex on /en/people/ does not affect /ja/people/. You need the tag on every locale’s template.

Final Checklist

Before you deploy your robots.txt changes, run through this list:

• List all language prefixes on your site, including the root default.
• For each blocked path, write one Disallow line per language prefix.
• Avoid wildcards like /*/people/ — use explicit paths only.
• Test every blocked URL in Search Console’s robots.txt tester.
• Add noindex tags to pages you want removed from search results quickly.
• After deployment, monitor crawl stats for one week to confirm reduced requests.

Robots.txt for multilingual sites is not hard. It just requires discipline. Write explicit rules for every language, verify each one, and combine crawl control with noindex for fast index removal. Your crawl budget — and your thin-content metrics — will thank you.

(Figures and case details in this article are as reported in the source material and have not been independently verified.)

Series: Overseas Website Building