Robots.txt Answers
robots.txt *How to add to my website*?
Q. Hi, I would like to add the robots.txt to my site... I would like access to everything other than maybe one or two different files, what would I have to enter for that? any info would be great Cheers.
Asked by - Mon Oct 31 12:37:12 2011 - Search Engine Optimization - 3 Answers - Comments
A. Google webmaster tools..make account and it will give you option of creation and addition of robots.txt
Answered by - Tue Nov 1 05:11:18 2011
Q. Hi, I would like to add the robots.txt to my site... I would like access to everything other than maybe one or two different files, what would I have to enter for that? any info would be great Cheers.
Asked by - Mon Oct 31 12:37:12 2011 - Search Engine Optimization - 3 Answers - Comments
A. Google webmaster tools..make account and it will give you option of creation and addition of robots.txt
Answered by - Tue Nov 1 05:11:18 2011
What does a robots.txt file do?
Q. I had my site checked and the program said to use a robots.txt file - I dont know what it is or how to make it..can anyone help? I promise to give the best answer 10 points
Asked by cyndisource - Mon Oct 2 07:11:29 2006 - Other - Internet - 6 Answers - Comments
A. A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site. There are many good, free sites that help you create a robot.txt file: here are a couple: Make sure you… [cont.]
Answered by Kitia_98 - Mon Oct 2 07:12:23 2006
Q. I had my site checked and the program said to use a robots.txt file - I dont know what it is or how to make it..can anyone help? I promise to give the best answer 10 points
Asked by cyndisource - Mon Oct 2 07:11:29 2006 - Other - Internet - 6 Answers - Comments
A. A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. You can use it to prevent indexing totally, prevent certain areas of your site from being indexes or to issue individual indexing instructions to specific search engines. All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders or bots arrive on your site. So, even if you currently do not need to exclude the spiders from any part of your site, having a robots.txt file is still a good idea, it can act as a sort of invitation into your site. There are many good, free sites that help you create a robot.txt file: here are a couple: Make sure you… [cont.]
Answered by Kitia_98 - Mon Oct 2 07:12:23 2006
How does robots.txt handle multiple user-agent lines?
Q. In this example: User-agent: msnbot User-agent: slurp Disallow: / Does the Disallow: / apply to only slurp or does it apply to both slurp and msnbot?
Asked by - Wed Apr 20 15:58:42 2011 - Programming & Design - 1 Answers - Comments
A. That Disallow applies to both slurp and msnbot. If a User-Agent string matches the robot, the next collection of Disallow and Allow directives applies to that robot. That happens regardless of whether additional User-Agent lines appear before the Disallow and Allow lines.
Answered by om - Wed Apr 20 21:34:45 2011
Q. In this example: User-agent: msnbot User-agent: slurp Disallow: / Does the Disallow: / apply to only slurp or does it apply to both slurp and msnbot?
Asked by - Wed Apr 20 15:58:42 2011 - Programming & Design - 1 Answers - Comments
A. That Disallow applies to both slurp and msnbot. If a User-Agent string matches the robot, the next collection of Disallow and Allow directives applies to that robot. That happens regardless of whether additional User-Agent lines appear before the Disallow and Allow lines.
Answered by om - Wed Apr 20 21:34:45 2011
Do I have to make different robots.txt and sitemap.xml on every subdomain of my website?
Q. I have a website and 5 subdomains, and only my main homapage have sitemap.xml and robots.txt. I want all of my subdomains to be indexed by search engine. do i have to make robots.txt and sitemap.xml on every subdomain?
Asked by Gospelo - Fri Sep 19 15:44:00 2008 - Programming & Design - 2 Answers - Comments
A. probably so. you don't want too many subdomains, because google will consider it spam, and penalize you by not showing you on its front pages.
Answered by moneymaxer - Fri Sep 19 15:48:13 2008
Q. I have a website and 5 subdomains, and only my main homapage have sitemap.xml and robots.txt. I want all of my subdomains to be indexed by search engine. do i have to make robots.txt and sitemap.xml on every subdomain?
Asked by Gospelo - Fri Sep 19 15:44:00 2008 - Programming & Design - 2 Answers - Comments
A. probably so. you don't want too many subdomains, because google will consider it spam, and penalize you by not showing you on its front pages.
Answered by moneymaxer - Fri Sep 19 15:48:13 2008
What does the robots.txt file on whitehouse.gov say about government transparency?
Q. It's a file that tells Google, Yahoo, etc what pages they are *not* allowed to index. See:
Asked by Sciurus Reprobatia - Wed Jan 21 14:54:31 2009 - Politics - 4 Answers - Comments
A. They got robots in the White House?!??!?!
Answered by GOP 3.0 - Wed Jan 21 15:31:03 2009
Q. It's a file that tells Google, Yahoo, etc what pages they are *not* allowed to index. See:
Asked by Sciurus Reprobatia - Wed Jan 21 14:54:31 2009 - Politics - 4 Answers - Comments
A. They got robots in the White House?!??!?!
Answered by GOP 3.0 - Wed Jan 21 15:31:03 2009
what is the role for robots.txt?
Q. I am a fresher in seo. so i want to know that what is the exact role of robots.txt. what if I don't put robots.txt file in root folder... can anyone reply me...
Asked by Junior B - Wed Dec 24 08:10:37 2008 - Search Engine Optimization - 2 Answers - Comments
A. robots.txt basically tells search engine spiders not to index certain areas of your site. For example, f you have web versions and print versions of your pages, you don't want both to show up in searches, so put the forprint/ directory in robots.txt Also you can protect private areas of sites from being visible to search engines, and thus to everyone.
Answered by Martina N - Wed Dec 24 08:20:16 2008
Q. I am a fresher in seo. so i want to know that what is the exact role of robots.txt. what if I don't put robots.txt file in root folder... can anyone reply me...
Asked by Junior B - Wed Dec 24 08:10:37 2008 - Search Engine Optimization - 2 Answers - Comments
A. robots.txt basically tells search engine spiders not to index certain areas of your site. For example, f you have web versions and print versions of your pages, you don't want both to show up in searches, so put the forprint/ directory in robots.txt Also you can protect private areas of sites from being visible to search engines, and thus to everyone.
Answered by Martina N - Wed Dec 24 08:20:16 2008
is my robots.txt file stopping my pages from being indexed by all search engines?
Q. my website had never been indexed before and i was wondering is this was related to my robots.txt file. I did not create this but im wondering if it is the problem. The file contains the following: User-agent: * Disallow: (There is no / after disallow and no directories have been specified). Does this mean that i am telling all search engines to not crawl my domain then? Or is it ok?
Asked by - Thu Dec 8 19:01:29 2011 - Search Engine Optimization - 4 Answers - Comments
Q. my website had never been indexed before and i was wondering is this was related to my robots.txt file. I did not create this but im wondering if it is the problem. The file contains the following: User-agent: * Disallow: (There is no / after disallow and no directories have been specified). Does this mean that i am telling all search engines to not crawl my domain then? Or is it ok?
Asked by - Thu Dec 8 19:01:29 2011 - Search Engine Optimization - 4 Answers - Comments
What is robots txt? Why we are using these? How i can generate?
Q. Hello! I want do robots txt file how can generate these file. 1. robots file purpose? 2. Adv & Dis Adv and benefits? Thanks for advance!
Asked by Victor I - Thu Oct 23 07:53:14 2008 - Search Engine Optimization - 2 Answers - Comments
A. Two links to help you out.
Answered by Always Sunny - Thu Oct 23 11:08:53 2008
Q. Hello! I want do robots txt file how can generate these file. 1. robots file purpose? 2. Adv & Dis Adv and benefits? Thanks for advance!
Asked by Victor I - Thu Oct 23 07:53:14 2008 - Search Engine Optimization - 2 Answers - Comments
A. Two links to help you out.
Answered by Always Sunny - Thu Oct 23 11:08:53 2008
robots.txt, what is the code that is entered in a website so that search engines can find websites?
Q.
Asked by docholliday410 - Sat Nov 21 01:50:38 2009 - Search Engine Optimization - 2 Answers - Comments
A. place a meta tag on your site telling the robot to index and follow your site or whatever you may want the robot to do. for example tells the robot to index your page and its linked pages. this code will prevent robots from indexing your page. you will want to place that meta code in the head section of your html robots meta tag here hope that helps
Answered by - Sat Nov 21 04:23:16 2009
Q.
Asked by docholliday410 - Sat Nov 21 01:50:38 2009 - Search Engine Optimization - 2 Answers - Comments
A. place a meta tag on your site telling the robot to index and follow your site or whatever you may want the robot to do. for example tells the robot to index your page and its linked pages. this code will prevent robots from indexing your page. you will want to place that meta code in the head section of your html robots meta tag here hope that helps
Answered by - Sat Nov 21 04:23:16 2009
How do I get around robots.txt?
Q. I want to view a website in an archive how do I get past the robots.txt they have on the page.
Asked by Renee Lucille - Tue Nov 20 17:56:54 2007 - Programming & Design - 2 Answers - Comments
A. That depends on who wrote the archiving software. Robots.txt is just a request from the web site admin to the crawler software. Everything depends on whether the software respects the requests in robots.txt. If you cannot modify the software, you are stuck. Your only other option would be to write some kind of a network filter to return 404 when requesting robot.txt files.
Answered by DogmaBites - Tue Nov 20 18:07:12 2007
Q. I want to view a website in an archive how do I get past the robots.txt they have on the page.
Asked by Renee Lucille - Tue Nov 20 17:56:54 2007 - Programming & Design - 2 Answers - Comments
A. That depends on who wrote the archiving software. Robots.txt is just a request from the web site admin to the crawler software. Everything depends on whether the software respects the requests in robots.txt. If you cannot modify the software, you are stuck. Your only other option would be to write some kind of a network filter to return 404 when requesting robot.txt files.
Answered by DogmaBites - Tue Nov 20 18:07:12 2007
How to Bypass Robots .txt?
Q. I'm trying to use the internet archive but the site I want keeps telling me it can't get in because the site is blocked by robots.txt. How can I get around them? I looked at the robots text and it says User-agent: * Disallow: / I really want the files in the archive. They're not available anymore, please help!
Asked by - Mon Mar 1 14:41:35 2010 - Programming & Design - 1 Answers - Comments
A. What they're telling you is that the internet archive did not archive the site due to a preference set in that site s robot.txt file. The robot.txt file is checked by spiders, like the way back machine, to determine whether or not the site s owner wants that particular spider to index their site.
Answered by Doug Gunnoe - Mon Mar 1 16:07:19 2010
Q. I'm trying to use the internet archive but the site I want keeps telling me it can't get in because the site is blocked by robots.txt. How can I get around them? I looked at the robots text and it says User-agent: * Disallow: / I really want the files in the archive. They're not available anymore, please help!
Asked by - Mon Mar 1 14:41:35 2010 - Programming & Design - 1 Answers - Comments
A. What they're telling you is that the internet archive did not archive the site due to a preference set in that site s robot.txt file. The robot.txt file is checked by spiders, like the way back machine, to determine whether or not the site s owner wants that particular spider to index their site.
Answered by Doug Gunnoe - Mon Mar 1 16:07:19 2010
How do you implement in robots.txt so no folders are allowed to be crawled by search engines except top level?
Q.
Asked by Boxing A - Wed Mar 9 04:38:09 2011 - Programming & Design - 2 Answers - Comments
A. Keep in mind that only works with "well behaved" bots. If they don't check for the file it will not stop them.
Answered by - Wed Mar 9 05:11:45 2011
Q.
Asked by Boxing A - Wed Mar 9 04:38:09 2011 - Programming & Design - 2 Answers - Comments
A. Keep in mind that only works with "well behaved" bots. If they don't check for the file it will not stop them.
Answered by - Wed Mar 9 05:11:45 2011
How can I view web pages at the Wayback Machine when they're blocked by "robots.txt"? How do I bypass that?
Q. Sometimes I want to view archived web pages at the Wayback Machine but sometimes you can't view them because I get this message: Robots.txt Query Exclusion. We're sorry, access to (URL of website) has been blocked by the site owner via robots.txt. How do I bypass "robots.txt"? Is there a proxy server or some other program I can use to override "robots.txt"? Please let me know-thank you.
Asked by Jason G. - Fri Jul 30 10:46:41 2010 - Other - Internet - 1 Answers - Comments
A. The whole idea of robots.txt is to keep crawlers (and the Wayback Machine depends on crawlers) from reading a site. If a site owner has blocked crawlers, there *is* no archived page to read, because the Machine could not get in to read it. And you have no recourse. Says the Archive: "The Internet Archive is not interested in offering access to Web sites or other Internet documents whose authors do not want their materials in the collection. To remove your site from the Wayback Machine, place a robots.txt xfile at the top level of your site (e.g. www.yourdomain.com/robots.txt) and then submit your site below. "The robots.txt file will do two things: 1. It will remove all documents from your domain from the Wayback Machine. 2. It… [cont.]
Answered by - Fri Jul 30 20:22:32 2010
Q. Sometimes I want to view archived web pages at the Wayback Machine but sometimes you can't view them because I get this message: Robots.txt Query Exclusion. We're sorry, access to (URL of website) has been blocked by the site owner via robots.txt. How do I bypass "robots.txt"? Is there a proxy server or some other program I can use to override "robots.txt"? Please let me know-thank you.
Asked by Jason G. - Fri Jul 30 10:46:41 2010 - Other - Internet - 1 Answers - Comments
A. The whole idea of robots.txt is to keep crawlers (and the Wayback Machine depends on crawlers) from reading a site. If a site owner has blocked crawlers, there *is* no archived page to read, because the Machine could not get in to read it. And you have no recourse. Says the Archive: "The Internet Archive is not interested in offering access to Web sites or other Internet documents whose authors do not want their materials in the collection. To remove your site from the Wayback Machine, place a robots.txt xfile at the top level of your site (e.g. www.yourdomain.com/robots.txt) and then submit your site below. "The robots.txt file will do two things: 1. It will remove all documents from your domain from the Wayback Machine. 2. It… [cont.]
Answered by - Fri Jul 30 20:22:32 2010
Drwaback of robots.txt?
Q. I want to know the drawback of robots.txt and if we will not use robots.txt file what will be the effects on our websites ?
Asked by - Mon Oct 3 05:20:00 2011 - Google - 2 Answers - Comments
A. Draw back of robot.txt i file is when any search engine like Google, yahoo.Bing crawler come to your website then this file allow to crawler the page . Only this is the drawback of this .
Answered by - Wed Oct 5 09:09:14 2011
Q. I want to know the drawback of robots.txt and if we will not use robots.txt file what will be the effects on our websites ?
Asked by - Mon Oct 3 05:20:00 2011 - Google - 2 Answers - Comments
A. Draw back of robot.txt i file is when any search engine like Google, yahoo.Bing crawler come to your website then this file allow to crawler the page . Only this is the drawback of this .
Answered by - Wed Oct 5 09:09:14 2011
is it possible to disallow specific pages and not just entire directories, to the robots.txt file?
Q. is it possible to disallow specific pages and not just entire directories, to the robots.txt file?
Asked by ponyrunstheshow - Thu May 10 12:01:42 2007 - Programming & Design - 1 Answers - Comments
A. Yes see example below. User-agent: * Disallow: /tmp/ Disallow: /privatepage.html Disallow: /links/listing.html robots.txt are only suggestions for well-behaving spiders and robots, its been know that spammers actually use the dissallows to look for key information. To really protect pages its better to restrict control in the .htaccess file.. Go here for a more complete explanation:
Answered by acb29 - Thu May 10 12:13:28 2007
Q. is it possible to disallow specific pages and not just entire directories, to the robots.txt file?
Asked by ponyrunstheshow - Thu May 10 12:01:42 2007 - Programming & Design - 1 Answers - Comments
A. Yes see example below. User-agent: * Disallow: /tmp/ Disallow: /privatepage.html Disallow: /links/listing.html robots.txt are only suggestions for well-behaving spiders and robots, its been know that spammers actually use the dissallows to look for key information. To really protect pages its better to restrict control in the .htaccess file.. Go here for a more complete explanation:
Answered by acb29 - Thu May 10 12:13:28 2007
What will this robots.txt file do?
Q. If I add just the following to my robots.txt file... User-agent: * Disallow: / ...will Google, Bing, Yahoo, etc STILL be able to index my site and have my pages appear in their search engines?. Thanks for any help, Hope.
Asked by Hope E - Fri Oct 7 12:36:07 2011 - Programming & Design - 2 Answers - Comments
A. If you wish to take out all the search engine crawlers from your whole domain then you can utilize that tiny code... User-agent: * Disallow: / I think you should have to go though the following url to gain the knowledge regarding to robots.txt File.
Answered by - Fri Oct 7 12:58:09 2011
Q. If I add just the following to my robots.txt file... User-agent: * Disallow: / ...will Google, Bing, Yahoo, etc STILL be able to index my site and have my pages appear in their search engines?. Thanks for any help, Hope.
Asked by Hope E - Fri Oct 7 12:36:07 2011 - Programming & Design - 2 Answers - Comments
A. If you wish to take out all the search engine crawlers from your whole domain then you can utilize that tiny code... User-agent: * Disallow: / I think you should have to go though the following url to gain the knowledge regarding to robots.txt File.
Answered by - Fri Oct 7 12:58:09 2011
is it wise to generate a robots.txt file even if you don't use it?
Q. I am doing work on a website, there isn't a page that I don't want the Google Spider to crawl. Is it wise for me to generate a robots.txt file behind webmaster tools even though I don't plan on using the Robots.txt file to block any pages at the moment? Is it better SEO wise to just have on active on your site regardless? Many Thanks
Asked by - Tue Nov 1 16:05:55 2011 - Search Engine Optimization - 4 Answers - Comments
A. Are you 100% sure that you want all your pages to be crawlable? Then don't create one. But as Mitchell says it's pretty unusual. Directly from Google webmaster tools help (see the source link for details): "If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one)."
Answered by - Wed Nov 2 07:18:44 2011
Q. I am doing work on a website, there isn't a page that I don't want the Google Spider to crawl. Is it wise for me to generate a robots.txt file behind webmaster tools even though I don't plan on using the Robots.txt file to block any pages at the moment? Is it better SEO wise to just have on active on your site regardless? Many Thanks
Asked by - Tue Nov 1 16:05:55 2011 - Search Engine Optimization - 4 Answers - Comments
A. Are you 100% sure that you want all your pages to be crawlable? Then don't create one. But as Mitchell says it's pretty unusual. Directly from Google webmaster tools help (see the source link for details): "If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one)."
Answered by - Wed Nov 2 07:18:44 2011
Can using "no indexing" along with "disallow from robots.txt" cause problems for google indexing your site or?
Q. any problems in general?
Asked by - Thu Sep 1 00:57:02 2011 - Search Engine Optimization - 1 Answers - Comments
A. Meta robots "no index" is a page level option to tell search engines to not index that particular page. Robots.txt is a file where you can tell search engines what to exclude from their crawling. If done incorrectly, these can have serious side effects. Like any tool, you have to understand them thoroughly before using them. So, the tools themselves do not cause problems and can be used to improve your site profile in search engines.
Answered by - Thu Sep 1 06:44:34 2011
Q. any problems in general?
Asked by - Thu Sep 1 00:57:02 2011 - Search Engine Optimization - 1 Answers - Comments
A. Meta robots "no index" is a page level option to tell search engines to not index that particular page. Robots.txt is a file where you can tell search engines what to exclude from their crawling. If done incorrectly, these can have serious side effects. Like any tool, you have to understand them thoroughly before using them. So, the tools themselves do not cause problems and can be used to improve your site profile in search engines.
Answered by - Thu Sep 1 06:44:34 2011
Can search engine crawlers still read a domains robots.txt file if its permission is set to 600?
Q. The domain is running on a linux server, and what I'm trying to accomplish is have a robots.txt file that search engine crawlers can read but the any user can't access/view via the browser. The whole point of using a robots.txt file is to hide folders/files from ending up in search results in google, yahoo, etc, which in turn the public can access. Even though it hides folders/files from search engine crawlers, the actual robots.txt file is still accessible through which displays all the folders/files I'm trying to hide, unless you can place robots.txt file in the actual directory your trying to hide? I know you can password directories and thats what I intend to do as well.
Asked by Omnis - Sat Aug 26 11:17:25 2006 - Other - Internet - 1 Answers - Comments
A. No they can't. You could however use something like Rewrite Cond to detect the user-agent string Mozilla and then rewrite robots.txt to some other file. You would have to exclude the two or three bots that include Mozilla in the user-agent string though. Easy enough but why would you want to do this?
Answered by memetrader - Sat Aug 26 13:07:21 2006
Q. The domain is running on a linux server, and what I'm trying to accomplish is have a robots.txt file that search engine crawlers can read but the any user can't access/view via the browser. The whole point of using a robots.txt file is to hide folders/files from ending up in search results in google, yahoo, etc, which in turn the public can access. Even though it hides folders/files from search engine crawlers, the actual robots.txt file is still accessible through which displays all the folders/files I'm trying to hide, unless you can place robots.txt file in the actual directory your trying to hide? I know you can password directories and thats what I intend to do as well.
Asked by Omnis - Sat Aug 26 11:17:25 2006 - Other - Internet - 1 Answers - Comments
A. No they can't. You could however use something like Rewrite Cond to detect the user-agent string Mozilla and then rewrite robots.txt to some other file. You would have to exclude the two or three bots that include Mozilla in the user-agent string though. Easy enough but why would you want to do this?
Answered by memetrader - Sat Aug 26 13:07:21 2006
What is a sub-domain and will my robot.txt file exclude it?
Q. I uploaded a robot.txt file to exclude my personal site off of search web engines. I just went into my web-builder and it told me I have something called a sub-domain- Will the robots.txt file block crawlers & spiders from that as well?
Asked by Ilia M - Fri Mar 27 20:06:57 2009 - Programming & Design - 1 Answers - Comments
A. Just a "heads up" about using robots.txt file. While it is an often used method to deter subdomains/subfolders from being listed in search engines, it is also a well known technique by hackers to circumvent the location of any product you may be attempting to sell for profit - if that is the case. For instance... if you have an ebook named "How To Get Rich" and have it hosted on your website (but using robots.txt to hide it) a hacker can type in your URL along with robots.txt and read the location of the product. Here's a better example: www.how-to-get-rich.com/robot.txt would give the exact location of the item you are attempting to hide. Hope this helps! All the best of success, Ron B
Answered by Ron B - Fri Mar 27 20:22:02 2009
Q. I uploaded a robot.txt file to exclude my personal site off of search web engines. I just went into my web-builder and it told me I have something called a sub-domain- Will the robots.txt file block crawlers & spiders from that as well?
Asked by Ilia M - Fri Mar 27 20:06:57 2009 - Programming & Design - 1 Answers - Comments
A. Just a "heads up" about using robots.txt file. While it is an often used method to deter subdomains/subfolders from being listed in search engines, it is also a well known technique by hackers to circumvent the location of any product you may be attempting to sell for profit - if that is the case. For instance... if you have an ebook named "How To Get Rich" and have it hosted on your website (but using robots.txt to hide it) a hacker can type in your URL along with robots.txt and read the location of the product. Here's a better example: www.how-to-get-rich.com/robot.txt would give the exact location of the item you are attempting to hide. Hope this helps! All the best of success, Ron B
Answered by Ron B - Fri Mar 27 20:22:02 2009
From Yahoo Answer Search: 'robots.txt'
Wed Jan 11 13:11:56 2012
[Hide]▼
)
goginoo.com
514 x 656px
[source page]
Robots txt file screenshot:
Com News.ru
Sun, 20 Nov 2011 14:37:05 -0800
Vnov vsie ghovoriat pro otsutstviie prieslovutogho faila robots.txt . Mol, nie byla zashchishchiena im stranitsa, ghdie razmieshchalsia dokumient. Da chto za dietskiie otghovorki? Smieshno, pravo slovo. Nu, naidut blaghodaria vnutrienniemu rassliedovaniiu razziavu, zabyvshiegho postavit ...
archive.org
Archive chronicles TV's take on 9/11: Archiving every book ever published: US scholar brings ancient Balinese scripts to digital age: At ALA Midwinter, Brewster Kahle ...
www.archive.org/about/exclude.php
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
[Hide]▲