Revision history for Mod033bRobotIndexing


Revision [23266]

Last edited on 2016-05-20 07:38:47 by JavaWoman [Replaces old-style internal links with new pipe-split links.]
Additions:
**[[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote | CharlesNepote]]** @ [[http://www.wikini.net/ | WikiNi]]
**[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel | MatthiasAppel]]**
Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N | Google]] and see the result.
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel | MatthiasAppel]]
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote | CharlesNepote]]
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel | MatthiasAppel]]
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --[[http://web.archive.org/web/20040619185958/http://www.wakkawiki.com/MoE | MoE]]
~& Yahoo! recently [[http://www.ysearchblog.com/archives/000372.html | announced wildcard support]], so the above mentioned solution works for Yahoo! Slurp as well. Up to now I haven't found a solution for msn. :( --DaC
On [[http://www.wikini.net/ | WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini | header.php]] :
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote | CharlesNepote]]
Deletions:
**[[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]** @ [[http://www.wikini.net/ WikiNi]]
**[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]**
Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N Google]] and see the result.
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --[[http://web.archive.org/web/20040619185958/http://www.wakkawiki.com/MoE MoE]]
~& Yahoo! recently [[http://www.ysearchblog.com/archives/000372.html announced wildcard support]], so the above mentioned solution works for Yahoo! Slurp as well. Up to now I haven't found a solution for msn. :( --DaC
On [[http://www.wikini.net/ WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini header.php]] :
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]

Revision [19287]

Edited on 2008-01-28 00:14:45 by JavaWoman [Modified links pointing to docs server]

No Differences

Revision [17208]

Edited on 2007-07-07 14:33:53 by JavaWoman [standardized credit links]
Additions:
**[[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]** @ [[http://www.wikini.net/ WikiNi]]
**[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]**
Deletions:
[[http://www.wikini.net/ WikiNi]] ([[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]])
[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]

Revision [17150]

Edited on 2007-07-07 04:57:07 by JavaWoman [web.archive.org links for wakkawiki]
Additions:
[[http://www.wikini.net/ WikiNi]] ([[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]])
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]
--[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --[[http://web.archive.org/web/20040619185958/http://www.wakkawiki.com/MoE MoE]]
-- [[http://web.archive.org/web/20040811012539/http://www.wakkawiki.com/CharlesNepote CharlesNepote]]
Deletions:
WikiNi (CharlesNepote)
--MatthiasAppel
-- CharlesNepote
--MatthiasAppel
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --MoE
-- CharlesNepote

Revision [17149]

Edited on 2007-07-07 04:47:06 by JavaWoman [web.archive.org links for wakkawiki]
Additions:
WikiNi (CharlesNepote)
[[http://web.archive.org/web/20040810215014/http://www.wakkawiki.com/MatthiasAppel MatthiasAppel]]
Deletions:
WikiNi (CharlesNepote) -- http://www.wakkawiki.com/MatthiasAppel

Revision [16008]

Edited on 2007-01-28 18:14:12 by BaxilDragon [pointing out fix at RobotsDotTxt]
Additions:
~& There is a fix for this problem at RobotsDotTxt. - BaxilDragon

Revision [15971]

Edited on 2007-01-19 20:17:42 by DaC [Yahoo! Wildcard support for robots.txt]
Additions:
~& Yahoo! recently [[http://www.ysearchblog.com/archives/000372.html announced wildcard support]], so the above mentioned solution works for Yahoo! Slurp as well. Up to now I haven't found a solution for msn. :( --DaC
Deletions:
~& Yahoo! recently [[http://www.ysearchblog.com/archives/000372.html announced wildcard support]] as well so the above mentioned solution works for Yahoo! Slurp as well. Up to now I haven't found a solution for msn. :( --DaC

Revision [15969]

Edited on 2007-01-19 20:15:39 by DaC [Yahoo! Wildcard support for robots.txt]
Additions:
==== Wikka Mod 033 ====
Type: Bug Fix / Feature
----
===Credit:===
WikiNi (CharlesNepote) -- http://www.wakkawiki.com/MatthiasAppel
----

This is the method used in Wikka:

In header.php :
%%(php)
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
%%

----

History from WakkaWiki........


Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N Google]] and see the result.
--MatthiasAppel

Yes. See : [[http://www.robotstxt.org/wc/exclusion.html]].
-- CharlesNepote

Does that mean, that I have to add a robots.txt with this content:
%%
User-agent: Googlebot
Disallow: /edit$
Disallow: /revisions$
Disallow: /referrers$

User-agent: *
Disallow: /edit
Disallow: /revisions
Disallow: /referrers
%%
--MatthiasAppel
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --MoE

Dont really know, but I think $ doesnt belong there. But you could use something like this:
%%
User-agent: googlebot
Disallow: /*/edit
Disallow: /*/revisions
Disallow: /*/referrers
Disallow: /*/referrers_sites
%% Googlebot understands the wildcard *. Other SE Bots won't.
But also like Charles' method noted below. --JanPiotrowski

~& Yahoo! recently [[http://www.ysearchblog.com/archives/000372.html announced wildcard support]] as well so the above mentioned solution works for Yahoo! Slurp as well. Up to now I haven't found a solution for msn. :( --DaC

On [[http://www.wikini.net/ WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini header.php]] :
%%(php)
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
%%
It seem to work quite well.
-- CharlesNepote

You can't get corrects contents of robots.txt if you install wiki at the root of your web, ie, if urls to HomePage is like my_domain.com/HomePage, because /robots.txt will be redirected to /wakka.php?wakka=robots.txt
Deletions:
==== Wikka Mod 033 ====
Type: Bug Fix / Feature
----
===Credit:===
WikiNi (CharlesNepote) -- http://www.wakkawiki.com/MatthiasAppel
----

This is the method used in Wikka:

In header.php :
%%(php)
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
%%

----

History from WakkaWiki........


Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N Google]] and see the result.
--MatthiasAppel

Yes. See : [[http://www.robotstxt.org/wc/exclusion.html]].
-- CharlesNepote

Does that mean, that I have to add a robots.txt with this content:
%%
User-agent: Googlebot
Disallow: /edit$
Disallow: /revisions$
Disallow: /referrers$

User-agent: *
Disallow: /edit
Disallow: /revisions
Disallow: /referrers
%%
--MatthiasAppel
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --MoE

Dont really know, but I think $ doesnt belong there. But you could use something like this:
%%
User-agent: googlebot
Disallow: /*/edit
Disallow: /*/revisions
Disallow: /*/referrers
Disallow: /*/referrers_sites
%% Googlebot understands the wildcard *. Other SE Bots won't.
But also like Charles' method noted below. --JanPiotrowski

On [[http://www.wikini.net/ WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini header.php]] :
%%(php)
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
%%
It seem to work quite well.
-- CharlesNepote

You can't get corrects contents of robots.txt if you install wiki at the root of your web, ie, if urls to HomePage is like my_domain.com/HomePage, because /robots.txt will be redirected to /wakka.php?wakka=robots.txt

Revision [3081]

Edited on 2004-12-10 16:45:44 by JavaWoman [put php code in code blocks]
Additions:
%%(php)
%%(php)

Revision [3080]

Edited on 2004-12-10 15:54:16 by CharlesNepote [Credits modified]
Additions:
==== Wikka Mod 033 ====
Type: Bug Fix / Feature
----
===Credit:===
WikiNi (CharlesNepote) -- http://www.wakkawiki.com/MatthiasAppel
----

This is the method used in Wikka:

In header.php :
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>

----

History from WakkaWiki........


Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N Google]] and see the result.
--MatthiasAppel

Yes. See : [[http://www.robotstxt.org/wc/exclusion.html]].
-- CharlesNepote

Does that mean, that I have to add a robots.txt with this content:
%%
User-agent: Googlebot
Disallow: /edit$
Disallow: /revisions$
Disallow: /referrers$

User-agent: *
Disallow: /edit
Disallow: /revisions
Disallow: /referrers
%%
--MatthiasAppel
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --MoE

Dont really know, but I think $ doesnt belong there. But you could use something like this:
%%
User-agent: googlebot
Disallow: /*/edit
Disallow: /*/revisions
Disallow: /*/referrers
Disallow: /*/referrers_sites
%% Googlebot understands the wildcard *. Other SE Bots won't.
But also like Charles' method noted below. --JanPiotrowski

On [[http://www.wikini.net/ WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini header.php]] :
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
It seem to work quite well.
-- CharlesNepote

You can't get corrects contents of robots.txt if you install wiki at the root of your web, ie, if urls to HomePage is like my_domain.com/HomePage, because /robots.txt will be redirected to /wakka.php?wakka=robots.txt
Deletions:
==== Wikka Mod 033 ====
Type: Bug Fix / Feature
----
===Credit:===
""CharlesNepote"" -- http://www.wakkawiki.com/MatthiasAppel
----

This is the method used in Wikka:

In header.php :
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>

----

History from WakkaWiki........


Hm, does anybody know how to make search robots not to index the ../edit ../revisions and ../referrers links on every page? You can search for WakkaWiki on [[http://www.google.de/search?q=+site:www.wakkawiki.com+wakkawiki&hl=de&lr=&ie=UTF-8&oe=UTF-8&start=50&sa=N Google]] and see the result.
--MatthiasAppel

Yes. See : [[http://www.robotstxt.org/wc/exclusion.html]].
-- CharlesNepote

Does that mean, that I have to add a robots.txt with this content:
%%
User-agent: Googlebot
Disallow: /edit$
Disallow: /revisions$
Disallow: /referrers$

User-agent: *
Disallow: /edit
Disallow: /revisions
Disallow: /referrers
%%
--MatthiasAppel
''Did it work with robots.txt? And what does $ mean?? What is with referrers_sites?'' --MoE

Dont really know, but I think $ doesnt belong there. But you could use something like this:
%%
User-agent: googlebot
Disallow: /*/edit
Disallow: /*/revisions
Disallow: /*/referrers
Disallow: /*/referrers_sites
%% Googlebot understands the wildcard *. Other SE Bots won't.
But also like Charles' method noted below. --JanPiotrowski

On [[http://www.wikini.net/ WikiNi]] we code that in [[http://cvsweb.tuxfamily.org/cvs/wikini/actions/header.php?rev=1.12&content-type=text/x-cvsweb-markup&cvsroot=wikini header.php]] :
<?php if ($this->GetMethod() != 'show') echo "<meta name=\"robots\" content=\"noindex, nofollow\" />\n";?>
It seem to work quite well.
-- CharlesNepote

You can't get corrects contents of robots.txt if you install wiki at the root of your web, ie, if urls to HomePage is like my_domain.com/HomePage, because /robots.txt will be redirected to /wakka.php?wakka=robots.txt

Revision [706]

Edited on 2004-07-08 08:59:33 by DotMG [Credits modified]
Additions:
You can't get corrects contents of robots.txt if you install wiki at the root of your web, ie, if urls to HomePage is like my_domain.com/HomePage, because /robots.txt will be redirected to /wakka.php?wakka=robots.txt
See RobotsDotTxt and FaviconDotIco. --DotMG

Revision [674]

Edited on 2004-06-17 01:50:01 by JsnX [Credits modified]
Additions:
This is the method used in Wikka:
In header.php :
History from WakkaWiki........

Revision [672]

Edited on 2004-06-17 01:45:01 by JanPiotrowski [Credits modified]
Additions:
Dont really know, but I think $ doesnt belong there. But you could use something like this:
%% Googlebot understands the wildcard *. Other SE Bots won't.
But also like Charles' method noted below. --JanPiotrowski
Deletions:
Dont know, but $ doesnt belong there. But you could use something like this:
%% Google understand the wildcard "*". --JanPiotrowski

Revision [669]

Edited on 2004-06-17 01:36:57 by JanPiotrowski [Credits modified]
Additions:

Dont know, but $ doesnt belong there. But you could use something like this:
User-agent: googlebot
Disallow: /*/edit
Disallow: /*/revisions
Disallow: /*/referrers
Disallow: /*/referrers_sites
%% Google understand the wildcard "*". --JanPiotrowski

Revision [279]

The oldest known version of this page was created on 2004-05-13 03:25:09 by JsnX [Credits modified]
Valid XHTML :: Valid CSS: :: Powered by WikkaWiki