Some URLs redirect to home page instead of 404

[Grantovich]

A long time ago, my site used a different CMS, and its URLs looked like this:

http://site.com/?q=node/32

Now I have CMSMS, and I would like all those old URLs to go to a proper 404 error page so the search engines will remove them from their indexes. The problem is, CMSMS keeps redirecting them to the home page, causing the search bots to think they are still active, and my home page is continuously re-indexed under 10 or 20 different (meaningless) URLs. All other nonexistant pages go to a 404 message, but for some reason, URLs in the form "site.com/?q=anything" are always redirected to the home page.

Here's my .htaccess (it makes things come out as "site.com/parent/child"):

Code: Select all

Options -Indexes
Options +FollowSymLinks
RewriteEngine on
RewriteBase /

RewriteCond %{HTTP_HOST} ^www\.(.*)$
RewriteRule ^(.*)$ http://%1/$1 [R,L]

RewriteCond %{REQUEST_FILENAME} !-f [NC]
RewriteCond %{REQUEST_FILENAME} !-d [NC]
RewriteRule ^(.+)$ index.php?page=$1 [QSA]

Anybody know how I can stop the mystery redirects?

[tsw]

RewriteCond %{HTTP_HOST} ^www\.(.*)$
RewriteRule ^(.*)$ http://%1/$1 [R,L]

not sure but those rules looks like it might be the cause....

[Grantovich]

RewriteCond %{HTTP_HOST} ^www\.(.*)$
RewriteRule ^(.*)$ http://%1/$1 [R,L]

not sure but those rules looks like it might be the cause....

Hmm... what that is supposed to do is strip the "www." from the start of the address, if present. I commented out this section and tried again, but it still redirected "site.com/?q=anything" to the home page.

EDIT: I really should not be saying redirect because that's not what it is. If you enter a /?q= URL into the browser, it returns the home page, but doesn't actually redirect you to the home page's "true" address.

[swgreed]

Add this to your .htaccess

Code: Select all

ErrorDocument 404 http://www.domain.com/404.htm

[Grantovich]

Add this to your .htaccess

Code: Select all

ErrorDocument 404 http://www.domain.com/404.htm

Doesn't seem to change anything. "http://domain.com/?q=whatever" still returns the home page.

[Grantovich]

Well, I have some good news: It appears that the search engines are gradually realizing that all of the /?q= URLs return the same page, and they are removing these URLs as duplicate content. The process is glacially slow, of course.

I tried making my own RewriteRule that would return a 410 error if the URL contained "?q=" anywhere in it, but I must not have a very good understanding of the mechanism, because it didn't change anything. Then again, for all I know the problem could be in CMSMS and have nothing whatsoever to do with htaccess. Anybody have any ideas?

[Pierre M.]

HTTP "Gone" is 410 and not 404.
What about something like this :
RewriteRule ^?q=node/.* - [R=410,L]
Not quite sure, but see Apache doc.

And don't have duplicated content (same content available at multiple URLs ) : your site may be blacklisted as spammer.
Hope it helps.

PM

[Grantovich]

HTTP "Gone" is 410 and not 404.
What about something like this :
RewriteRule ^?q=node/.* - [R=410,L]
Not quite sure, but see Apache doc.

And don't have duplicated content (same content available at multiple URLs ) : your site may be blacklisted as spammer.
Hope it helps.

PM

Ow! Adding this line results in every page on the entire site giving a "500 Internal Server Error". Strange, because it looks legitimate given my extremely limited knowledge of htaccess.

By the way, I don't have any (intentionally) duplicated content. The problem is that the home page is returned, not redirected, for all of the old ?q= URLs. As far as the search engines are concerned, all of the old URLs are completely different pages, but with exactly the same content. There is no redirecting involved; the home page effectively exists at all of those locations. It's that particular attribute of the problem that's driving me batty.

[Pierre M.]

I said I was not quite sure
Here is another try :

1°)Set up a static error page for 410 :

ErrorDocument 404 /sorry_not_found.html
ErrorDocument 410 /sorry_gone.html

2°)Tell the old stuff has gone with mod_alias :

Redirect gone /q=node
according to http://httpd.apache.org/docs/2.2/mod/mo ... l#redirect

or use mod_rewrite (I've previously forgotten the '$') :
RewriteEngine On
RewriteBase /
RewriteRule ^?q=node/.*$ - [G,L]

...or may be RewriteRule ^?q=node/.*$ /sorry_gone.html [G,L]

PM

[Grantovich]

I said I was not quite sure
Here is another try :

1°)Set up a static error page for 410 :

ErrorDocument 404 /sorry_not_found.html
ErrorDocument 410 /sorry_gone.html

2°)Tell the old stuff has gone with mod_alias :

Redirect gone /q=node
according to http://httpd.apache.org/docs/2.2/mod/mo ... l#redirect

or use mod_rewrite (I've previously forgotten the '$') :
RewriteEngine On
RewriteBase /
RewriteRule ^?q=node/.*$ - [G,L]

...or may be RewriteRule ^?q=node/.*$ /sorry_gone.html [G,L]

PM

I tried every one of the following (turns out the "500 Internal Server Error" comes from not escaping the question mark):

Redirect gone /?q=
RedirectMatch gone /\?q=.*
RewriteRule ^\?q=.*$ - [G,L]
RewriteRule ^\?q=.*$ /gone.htm [G,L]
RewriteRule \?q=.* /gone.htm [G,L]
RewriteRule (.*)\?q=(.*) /gone.htm [G,L]

None of them had any effect whatsoever. It especially irritates me that not even "RedirectMatch" worked, since I'm quite positive I'm using it the right way, and the documentation agrees. Argh.

[Pierre M.]

Argh as you say.

So, there is no more any 500 ? Good point escaping the question mark. Well done, I had missed it.
What is your OS and what is your Apache ?
And what about the client side ? Have you tried with wget verbose on ?

I agree RedirectMatch should be the best. But tell us what is happening : the browser/wget gets 404, 3xx, 5xx, something else ?
(may be you should clear client side cache each time you test)

Sorry to ask for checking this, but with RewriteRule ^\?q=.*$ - [G,L] do you have RewriteEngine On and a good RewriteBase too ?
Please check the name and location of your .htaccess too.
And the files' permissions (.htaccess, gone.htm, 404.html...)

PM

[Grantovich]

I'm not sure what server information I'm looking for, so I have just attached the result of the phpinfo() function on my site. Hopefully it will be of some use. (you will need to change the extension to .htm to view it)

http://www.seoconsultants.com/tools/headers.asp
I have used this site to test the return codes for pages that start with "/?q=". Whether or not I have the RewriteRule or RedirectMatch in the htaccess, it always returns the same thing: 200 OK. I try a different URL each time to avoid caching issues (like "/?q=watermelon", "/?q=pineapple", etc).

As you can see in the original htaccess at the top of the post, I have RewriteEngine on and RewriteBase /. The file permissions must be set correctly, because the original file works and does exactly what I want it to.

[gelöscht durch Administrator]

[Pierre M.]

So, your IE answers 404 and somebody else answers 200. At least one of you is wrong. You must have good information on this to go on.

So I resuggest : Have you tried with wget instead of IE ?

wget -v -S -o log1 http://my.server.tld/known/working/static/url1.html
wget -v -S -o log2 http://my.server.tld/known/redirected/static/url2.html
wget -v -S -o log3 http://my.server.tld/?q=kiwi

Please report a matrix of your RedirectMatch/RewriteRule combinaison vs these 3 columns, with code (2xx-5xx), content result and log in each cell.

BTW, please check your username/password to the db in config.php too.

PM

[Grantovich]

So, your IE answers 404 and somebody else answers 200. At least one of you is wrong. You must have good information on this to go on.

I don't understand... what do you mean by "my IE answers 404 and somebody else answers 200"? Anyway, I was about to say I can't use wget because I'm on a shared server with no shell access, but then I remembered cron jobs. So here we go.

Key: htaccess line -- code returned by site.com/static.html -- code returned by site.com/parent/child (remapped by htaccess to site.com/index.php?page=child) -- code returned by site.com/?q=something

No change from original (see first post) -- 200 -- 200 -- 200
RedirectMatch gone /\q=.* -- 200 -- 200 -- 200
RedirectMatch gone (.*)\?q=(.*) -- 200 -- 200 -- 200
RewriteRule ^\?q=.*$ - [G,L] -- 200 -- 200 -- 200
RewriteRule ^\?q=.*$ /gone.htm [G,L] -- 200 -- 200 -- 200
RewriteRule \?q=.* /gone.htm [G,L] -- 200 -- 200 -- 200

The logs for all the static page requests look like this:

Code: Select all

--19:20:01--  http://site.com/static.html
           => `static.html'
Resolving site.com... x.x.x.x
Connecting to site.com|x.x.x.x|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Wed, 15 Nov 2006 00:20:01 GMT
  Server: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2
mod_bwlimited/1.4 PHP/4.4.3 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28
OpenSSL/0.9.7a
  Last-Modified: Tue, 31 Oct 2006 01:03:45 GMT
  ETag: "12bc028-0-4546a0f1"
  Accept-Ranges: bytes
  Content-Length: 0
  Keep-Alive: timeout=15, max=100
  Connection: Keep-Alive
  Content-Type: text/html
Length: 0 [text/html]

The logs for all the rewritten page requests look like this:

Code: Select all

--19:22:01--  http://site.com/parent/child
           => `child'
Resolving site.com... x.x.x.x
Connecting to site.com|x.x.x.x|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Wed, 15 Nov 2006 00:22:01 GMT
  Server: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2
mod_bwlimited/1.4 PHP/4.4.3 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28
OpenSSL/0.9.7a
  X-Powered-By: PHP/4.4.3
  Connection: close
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]

The logs for all the ?q= requests look like this:

Code: Select all

--19:24:01--  http://site.com/?q=kiwi
           => `index.html?q=kiwi'
Resolving site.com... x.x.x.x
Connecting to site.com|x.x.x.x|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Wed, 15 Nov 2006 00:24:02 GMT
  Server: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2
mod_bwlimited/1.4 PHP/4.4.3 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28
OpenSSL/0.9.7a
  X-Powered-By: PHP/4.4.3
  Connection: close
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]

I get the exact same logs for all the htaccess lines listed above. In every case, the static page returns itself, the remapped page returns its correct location (site.com/parent/child is correctly remapped to site.com/index.php?page=child), and the ?q= URL returns the home page (exactly the same as going to just site.com). Please correct me if I've left out some information that is important.

[Pierre M.]

If Apache allways answers "200 OK", may be your Internet Explorer (IE) is bogus when reporting 404 ?
PM