Wiki source for RobotsDotTxt


Show raw source

robots.txt is a file requested many times by bots like yahoo. It is assumed to be located at the root of web directory and to contain directives on which files and folders the search engine should be allowed or not to archive.
Generally, contents of /robots.txt should be something like :
%%User-agent : *
Disallow : UserSettings%%

If you want to know more about robots.txt and [[http://wikka.jsnx.com/FaviconDotIco | favicon.ico]], search these terms with Google and add the definitions in this page.

The only thing I just want to say is about ""RewriteRule"", see [[http://wikka.jsnx.com/FaviconDotIco | favicon.ico]]

----

====Problems when WikkaWiki has its own domain====

the text above does not mention that the 1.1.6.0 default install rewrites requests for favicon.ico and robots.txt into ##wikka.php?wakka=robots.txt## and ##wikka.php?wakka=favicon.ico##, respectively.

~& This is not a problem with **most** installs, because ##/robots.txt## and ##/favicon.ico## are at the top level of the domain and usually outside of the wiki's directory. But if Wikka gets its own domain (for example, mywikka.somedomain.com or www.myentiresiteisawiki.com), this problem kicks in. This also happens with 1.1.6.2. - BaxilDragon

one way to get around this is to change the last ""RewriteRule"" in wikka's .htaccess from this:

%%RewriteRule ^(.*)$ wikka.php?wakka=$1 [QSA,L]%%

To this:

%%RewriteCond %{REQUEST_FILENAME} !=/robots.txt
RewriteCond %{REQUEST_FILENAME} !=/favicon.ico
RewriteRule ^(.*)$ wikka.php?wakka=$1 [QSA,L]%%

==Favicon consolidation==

maybe you will also want to change
%%(html)
<link rel="icon" href="images/favicon.ico" type="image/x-icon" />
<link rel="shortcut icon" href="images/favicon.ico" type="image/x-icon" />%%
to
%%(html)
<link rel="icon" href="/favicon.ico" type="image/x-icon" />
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" />%%
in actions/header.php .

Then move the favicon.ico file from Wikka's images directory to the main Wikka directory. This will place it at the domain root, where browsers expect it to be, and get all Wikka's pages to look for it there.

//- Contributed by MinusF, edited by BaxilDragon//

I'd also like to point out that if either robots.txt or favicon.ico don't exist, then you'll just get kicked over to 404.shtml (or whatever page your server uses to handle 404 errors) ... which will, surprise, get redirected to ##wikka.php?wakka=404.shtml##. ;) To fix this, create a robots.txt file -- there's good example code at RobotFriendly and Mod033bRobotIndexing. The "Further consolidation" changes above will fix favicon.ico requests. - BaxilDragon

----
CategoryTroubleshooting
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki