It is not always best to have all of your pages indexed and cached by the search engines. Also not all links should be followed, therefore some editing of your wordpress header might be in order. Here is what I sometimes do.
Place the following in your wordpress themes header.php file, if the page is a single, page, or if its the home page then the robots will index and follow links on it. Otherwise search engines will not index the pages but will still follow the links.
<?php if(is_single() || is_page() || is_home()) { ?>
<meta name=”googlebot” content=”index,noarchive,follow,noodp” />
<meta name=”robots” content=”all,index,follow” />
<meta name=”msnbot” content=”all,index,follow” />
<?php } else { ?>
<meta name=”googlebot” content=”noindex,noarchive,follow,noodp” />
<meta name=”robots” content=”noindex,follow” />
<meta name=”msnbot” content=”noindex,follow” />
<?php }?>
Note that if you do not use a home.php page on your wordpress blog, it should be removes from the equasion and read as such:
<?php if(is_single() || is_page()) { ?>
