Posted by admin on

WordPress and robots.txt

WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.
I choose to use WP Robots.txt, it is simple and the least intrusive that I found, its control panel is added to Settings -> Reading dashboard panel. This is explained in the support forum.
I took advice from stackflow and spiderwebpress to add the following line
%CODE1%
The spiderweb article has some more advice about disallowing the filesystem since the content isn’t there.


Comment ( 1 )

  1. Google Tools and Analytics | wiki.davelevy.info
    […] Having created sitemaps, I want these to be advertised in the robots.txt file. See WordPress and robots.txt. […]

This site uses Akismet to reduce spam. Learn how your comment data is processed.

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close