Yes, install it in the demo directory.
If you have a live version of CMSMS running in the root, then install your new version with either a different database or a different table prefix. That should keep the two versions separate.
IMO, it's better to use a cms/ subdirectory instead of demo/ as then you don't have to change it when you're ready to go live. You can either write a php redirect in an index.php in the web root or if customer needs/wants a splash page, put that in the webroot. In either case, have it redirect to
http://domain.com/cms (I wouldn't like the word "demo" in the URL forever).
Also, if it is in development, you might want to add a robots.txt to keep it from getting indexed.