gecko-dev/webtools/lxr/robots.txt

зеркало из https://github.com/mozilla/gecko-dev.git
User-agent: *
Allow: /index.html
Disallow: /
attempt to dissuade robots from indexing the entire lxr.mozilla.org site 1999-06-15 10:34:45 +04:00			`User-agent: *`
If the application is set up at the server root (e.g. bugzilla.mozilla.org) then this robots.txt disallows robots from scanning the site except for the top level file. 2000-07-29 01:28:41 +04:00			`Allow: /index.html`
attempt to dissuade robots from indexing the entire lxr.mozilla.org site 1999-06-15 10:34:45 +04:00			`Disallow: /`