Industrial Depot
What Is My User Agent
Change IE User Agent
Firefox User Agent
IE7 User Agent
Mail User Agent
User Agent String Utility
User Agent Header
User Agent IE
Robots Disallow
Robots.txt
.txt File
Disallow Run
Disallow Attorney Lien
Other News:
How to avoid annoying Magento URL params in search engine index
...
User
-
agent
: *
Disallow
: /index.php/
Disallow
: /*?
Disallow
: /*.js$
Disallow
: /*.css$
Disallow
: /404/
Disallow
: /admin/
Disallow
: /api/
Disallow
: /app/
Disallow
: /catalog/category/view/
Disallow
: /catalog/product/view/
Disallow
: ...
imwill.com
Disallow
: /harming/humans. A lesson in Easter Eggs from Last.fm
...
User
-
Agent
: *
Disallow
: /music?
Disallow
: /widgets/radio?
Disallow
: /show_ads.php
Disallow
: /affiliate/
Disallow
: /affiliate_redirect.php
Disallow
: /affiliate_sendto.php
Disallow
: /affiliatelink.php
Disallow
: /campaignlink.php
Disallow
: ...
creatingcode.com
Some geek humor | fredericiana
Look what last.fm has in their robots.txt file:
User
-
Agent
: * ...
Disallow
: /harming/humans
Disallow
: /ignoring/human/orders
Disallow
:
fredericiana.com
FarrukhUNPLUGGED.com » Blog Archive » Wait Googlebot! Don't Go There!
User
-
agent
: Googlebot.
Disallow
: /*sort= The above would stop Googlebot from reading any URL that includes the string 'sort=' no matter where that string occurs in the URL. Yahoo! and Ask.com also offer robots.txt wildcard support. ...
farrukhunplugged.com
Related Sections
What Is My User Agent
Change IE User Agent
Firefox User Agent
IE7 User Agent
Mail User Agent
User Agent String Utility
User Agent Header
User Agent IE
Robots Disallow
Robots.txt
.txt File
Disallow Run
Disallow Attorney Lien
©2010 Copyright Industrial Depot -
Privacy Policy