If the application is set up at the server root (e.g. bugzilla.mozilla.org) then this robots.txt disallows robots from scanning the site except for the top level file.

2000-07-28 21:28:41 +00:00 · 2000-07-28 21:28:41 +00:00 · 621c7c38ac
--- a/webtools/bonsai/robots.txt
+++ b/webtools/bonsai/robots.txt
@ -0,0 +1,4 @@
+User-agent: *
+Allow: /index.html
+Allow: /cvsqueryform.cgi
+Disallow: /
--- a/webtools/bugzilla/robots.txt
+++ b/webtools/bugzilla/robots.txt
@ -0,0 +1,3 @@
+User-agent: *
+Allow: /index.html
+Disallow: /
--- a/webtools/lxr/robots.txt
+++ b/webtools/lxr/robots.txt
@ -1,2 +1,3 @@
 User-agent: *
+Allow: /index.html
 Disallow: /
--- a/webtools/tinderbox/robots.txt
+++ b/webtools/tinderbox/robots.txt
@ -0,0 +1,3 @@
+User-agent: *
+Allow: /index.html
+Disallow: /