Search Module - crashes during reindexation ~43.413.600 words

[waldemar_r]

Hi there,

I've just created new module using CTLModuleMaker. I've imported more than 36000 records.

12 columns x 36000 rows = 432000 fields with TEXT data.
Each field consist of 5-100  ~ 43.413.600 words

After some time it crashes with 500 Server error. I have 46849 indexed words (each time at about the same amount).

Should I write some small search form with my own SQL Query? Or there is some solution to index this mount of text data?

[Dr.CSS]

Please paste your server settings, you may be running out of memory...

[waldemar_r]

CMSMS  1.9.1

Configuration:

php_memory_limit:
255M

process_whole_template:
false

output_compression:
false

max_upload_size:
64000000

---

I've just noticed is that during indexation it went up to 49000 INDEXEs with 90000 Redords, after that it was "reset" to 0 and started to grow up to below values (stopped due to Internal Server Error)

cms_index_search_index INDEX 41808
Records 83 616

---

What for is the "cms_module_search_items" table? I can see that there are only 5100 records.
From here, the module takes content to index?

[Jeff]

Pleas give us ALL the system settings. Including PHP timeout.

[waldemar_r]

Cms Version: 1.9.1

Installed Modules:

   * CMSMailer: 2.0
   * FileManager: 1.0.3
   * MenuManager: 1.7.2
   * ModuleManager: 1.4
   * News: 2.11
   * nuSOAP: 1.0.2
   * Printing: 1.1.1
   * Search: 1.6.7
   * ThemeManager: 1.1.3
   * TinyMCE: 2.8.1
   * ModuleMaker: 0.3.2
   * CTLModuleMaker: 2.0.3
   * Katalog: 0.1


Config Information:

   * php_memory_limit: 255M
   * process_whole_template: false
   * output_compression: false
   * max_upload_size: 64000000
   * default_upload_permission: 664
   * url_rewriting: mod_rewrite
   * page_extension: .html
   * query_var: page
   * image_manipulation_prog: GD
   * auto_alias_content: true
   * locale:
   * default_encoding: utf-8
   * admin_encoding: utf-8
   * set_names: true


Php Information:

   * phpversion: 5.2.6-ogc1
   * md5_function: On (True)
   * gd_version: 2
   * tempnam_function: On (True)
   * magic_quotes_runtime: Off (False)
   * E_STRICT: 0
   * memory_limit: 255M
   * max_execution_time: 2000
   * output_buffering: 4096
   * safe_mode: Off (False)
   * file_uploads: On (True)
   * post_max_size: 20M
   * upload_max_filesize: 20M
   * session_save_path: /tmp (1777)
   * session_use_cookies: On (True)
   * xml_function: On (True)


Server Information:

   * Server Api: cgi-fcgi
   * Server Db Type: MySQL (mysql)
   * Server Db Version: 5.0.51a

-----

mysql.allow_persistent On On
mysql.connect_timeout 60 60

[waldemar_r]

Some ideas?

[fredp]

Some ideas?

Hi,

I had similar Search module "reindex" crashes with a text-rich site, back on release 1.8.x. The site has lots of text: transcribed documents, census data in HTML tables, etc. Importing the pages worked fine, but reindex crashed. It appeared at the time that Search module reindexing didn't scale well for content such as this, but I didn't investigate the crashes much. As a short-term workaround, I marked the pages as "NotSearchable", in the page template:

Code: Select all

<!-- pageAttribute: NotSearchable --> 

That stopped the crashes, but the Search module became less effective. So, I temporarily switched to using an external search engine for searches. I'd like to revisit this sometime soon and get the Search module working for this site. If I find anything helpful, I'll post it here.

Hope this helped.

[waldemar_r]

I don't understand why the reindexaion goes up to 89000 records (in cms_module_search_index) and is reset to 0. All records are deleted. Than the reindexation starts from where it finised (not from begining). What is wrong?


I extended the memory limit up to 2000MB so the script indexed more content (not 40000 but 90000 recs). Is there any solution to manage with this topic?