What to include and exclude using robots.txt
Posted: Wed Sep 07, 2011 8:09 pm
Hello,
I'm wondering if anyone can help me with using robots.txt and cmsms. I have never used it with a CMS before. I assume we don't want the entire site crawled by search engines, only the pages that are relevant. Does anyone have a sort of "default" set of instructions for robot.txt?
Thanks,
Adam
I'm wondering if anyone can help me with using robots.txt and cmsms. I have never used it with a CMS before. I assume we don't want the entire site crawled by search engines, only the pages that are relevant. Does anyone have a sort of "default" set of instructions for robot.txt?
Thanks,
Adam