Page 1 of 1

What to include and exclude using robots.txt

Posted: Wed Sep 07, 2011 8:09 pm
by zpeedofdark
Hello,

I'm wondering if anyone can help me with using robots.txt and cmsms. I have never used it with a CMS before. I assume we don't want the entire site crawled by search engines, only the pages that are relevant. Does anyone have a sort of "default" set of instructions for robot.txt?

Thanks,
Adam

Re: What to include and exclude using robots.txt

Posted: Wed Sep 07, 2011 8:43 pm
by paulbaker
See the example robots.txt in the doc folder of the install file structure.

Re: What to include and exclude using robots.txt

Posted: Wed Sep 07, 2011 9:24 pm
by zpeedofdark
Thank you. I moved a copy of this file to my root.

Re: What to include and exclude using robots.txt

Posted: Fri Sep 09, 2011 10:36 am
by paulbaker
You're both welcome. This forum is an excellent way to learn about CMSMS, there are plenty of helpful people here who know a lot more about it than me. ;D