Module for 20,000 products?

Have a question or a suggestion about a 3rd party addon module or plugin?
Let us know here.
Post Reply
pwg
Forum Members
Forum Members
Posts: 191
Joined: Tue Aug 01, 2006 1:48 am
Location: Sydney

Module for 20,000 products?

Post by pwg »

Hi, looking for a little advice. We have been asked to build a site for a smallish music label. The plan is to have up to 20 thousand of their releases listed. Any e-commerce probably handled off-site.
Each record will have summary and detail views
I’m a fan of LISSE, but worried about the volume of records? Would LISSE be efficient and quick?
Previous sites I’ve used CGBlog and Products for similar projects, and both work well, but to my mind have some limitations, mainly in the Admin layout and usability. That said they work.
The site will have other sections, which it would be handy to integrate to some extent, but again on previous sites, we have used CGBlog and Products for say, recipes and products, using a UDT to add recommended recipes to the Products and related products to recipe pages.
Any recommendations?
Thanks for any help.
User avatar
velden
Dev Team Member
Dev Team Member
Posts: 3483
Joined: Mon Nov 28, 2011 9:29 am
Location: The Netherlands

Re: Module for 20,000 products?

Post by velden »

Late reply though might still be useful:

I have been working on a website with those numbers of items in LISE.

In general it works well but some things to think about:

- my customer sometimes displays a list of hundreds or thousands of items on one page: bad idea, so better use the built-in pagination.
- in my case every item has several wysiwyg text fields often with large texts. This is no problem on itself but of course database will grow.
- Search: I found the CMSMS search module isn't optimized for this job. That is, when editing or adding a new item it took more than a minute to actually save the item. In the end found that the search module was the culprit and needed to do some small hacks. Initiating a re-index of the search module would better be avoided :-)
Searching an item goes very well and fast btw.
If you don't need search capabilities you can turn it off for the LISE instance btw.

So yes: LISE can handle those numbers but handle with care.

Regarding this specific website I did some feature requests:
http://dev.cmsmadesimple.org/feature_re ... ed_at+DESC

Especially the light-weight summary action would be a nice-to-have if it suits your use case too.
pwg
Forum Members
Forum Members
Posts: 191
Joined: Tue Aug 01, 2006 1:48 am
Location: Sydney

Re: Module for 20,000 products?

Post by pwg »

As always, many many thanks velden.

I'm still not completely sure which module to use, but given what you say, may go with LISE.

Thanks again.

Cheers.
deactivated010521

Re: Module for 20,000 products?

Post by deactivated010521 »

LISE is not designed for large datasets especially if you are going to use many custom fields things get slow.

Look at the table structure especially cms_module_*_field val.
Queries are resource intensive and no caching takes place, flexibility comes at a cost.

Also read the how of LISE: "An important note of warning - This is not a content construction kit, such as in Drupal. It is meant for small listings, not to store thousands of records."

20.000 products is a NO for LISE.

When you are working with a lot of products the /admin area is probally not used much. You only need good import and display functionality.

Not sure if CGProducts is already working with background tasks (CGJobMgr), importing large datasets can benefit from this module a lot.
User avatar
Jo Morg
Dev Team Member
Dev Team Member
Posts: 1921
Joined: Mon Jan 29, 2007 4:47 pm

Re: Module for 20,000 products?

Post by Jo Morg »

arnoud wrote: 20.000 products is a NO for LISE.
I beg to disagree...
Not long ago I would have agreed with you, until I had to work on a site already having multiple instances of LISE, some with over 50k records and growing, all heavily packed with custom fields an most cross linked with each other...
LISE behaved perfectly fine. As Velden described there are some issues that you'd need to pay attention to, that I'm solving in next version, but it works perfectly.
I do agree with you that Products may be a better choice particularly if handling on the order on the hundreds of thousands of records: Products is a specialized module optimized to work with very large catalogs of items linked or not with Calguy's eCommerce suite.
But I wouldn't dismiss LISE for a less than 100k records use as it brings some versatility and the possibility to cross link several instances if needed.
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).
Code of Condut | CMSMS Docs | Help Support CMSMS
My developer Page on the Forge
GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!
DevMoot 2023 in Cynwyd, Wales: I was there!
deactivated010521

Re: Module for 20,000 products?

Post by deactivated010521 »

Speed is always relative.

On shitty hosting or a low profile machine I won't bet on LISE when using large number of records.

When your server is fine the only limitation is the available ram, cpu and storage ofcourse. I would bet LISE can even store 100.000 records.

My claim is that LISE and Products both use fielddef/fieldval tables to store custom fields. As far as I know Products does a better job when importing records with images and has better caching, that is why I prefer Products in this case.

In the end a flat table is easier to query and more performant for a app like record store this might also a feasible solution. Especially when the shop will be refereshed from a seperate datastore. If there are already 20.000 records on file probally the CMS entry part is not needed at all.

Love to hear in more detail where the issues are to pay attention to when using LISE and what the improvements are you have on your wishlist?

I appreciate your effort in developing and maintaining LISE for many use-cases it is an excellent solution. For large datasets I have my doubts it is fast enhough.
calguy1000
Support Guru
Support Guru
Posts: 8169
Joined: Tue Oct 19, 2004 6:44 pm
Location: Fernie British Columbia, Canada

Re: Module for 20,000 products?

Post by calguy1000 »

The primary difference between Products and LISE is the database.

Products uses a primary table for each record and creates indexes that allow faster searching for primary filters and sorting (name, base price, categories, etc etc). People can create more indexes if needed to optimize for specific queries. Creating indexes only affects the performance of inserting and updating records, not selecting them.

It uses a flat key/value table for custom fields.

Products uses some internal caching to cache objects in memory to prevent multiple requests from the database for the same objects.

LISE uses a flat key/value table for everything. Which makes it impossible to create indexes for filtering or sorting by numeric values (i.e: price, or score, or category).

Sorting or filtering by numeric values in LISE will be relatively slow on large quantities of data because everything needs to be 'cast' to a numeric value for comparison. Even sorting or filtering by string values will be slower as you cannot (AFAIK) create an index on specific keys.

So, with LISE you are relying on CPU processing power of both the database host and the web host in order to overcome the fact that you cannot take advantage of Mysql indexes to increase performance.

YMMV
Follow me on twitter
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.
User avatar
Jo Morg
Dev Team Member
Dev Team Member
Posts: 1921
Joined: Mon Jan 29, 2007 4:47 pm

Re: Module for 20,000 products?

Post by Jo Morg »

arnoud wrote:Love to hear in more detail where the issues are to pay attention to when using LISE and what the improvements are you have on your wishlist?
As it stands LISE has a few Achilles heels:
  • - importing from csv of a large number of items (around 2k but it will depend on the allocated memory) is slow and on some servers it stalls (solved on next major version);
  • - filtering doesn't handle ranges (inclusive or exclusive) which somewhat limits its utility. Being the best way to do frontend searches and cross linking between LISE instances it needed some improvement (solved on next major version: the existing filtering parameters will be deprecated and a new number of parameters will be used so we can keep backwards compatibility for a while);
  • - allowing a filter to have an empty parameter will try to return all existing items unless there is a limit set (which will produce pagination, which is fine in most cases);
What I'm working on:
  • - from my tests, importing a huge csv file directly to a temporary db table and then from that table to the LISE instance db tables is around 100 times faster or more (no kidding) so that's what is going to be used for import/export of csv files;
  • - the queries for the extended filtering will be as optimized as possible and make use of transactions as much as possible. There is some gains in speed here, although there will be room for more improvement;
  • - templates will be moved to DM and avoid duplicates per instance. Several instances may share the same templates if needed;
  • - there will be the possibility to preview an item when editing or adding one (similar to Content Manager or News preview);
  • - something that may also make it to next version is custom instance global settings which will allow for having some preferences for an instance to be used on the templates: good for for frontend lightboxes plugins or other similar uses;
There are a few more things but I don't yet have details about them.
arnoud wrote:I appreciate your effort in developing and maintaining LISE for many use-cases it is an excellent solution. For large datasets I have my doubts it is fast enhough.
I appreciate your words and input of course. :)
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).
Code of Condut | CMSMS Docs | Help Support CMSMS
My developer Page on the Forge
GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!
DevMoot 2023 in Cynwyd, Wales: I was there!
deactivated010521

Re: Module for 20,000 products?

Post by deactivated010521 »

calguy1000 and Jo Morg both thank you for your extended explaination and effort.

Currently I don't have a project where I need to import a large dataset in LISE or Products.

In the past I created a bash script to import the whole collection of a shoestore in Products. I had todo some datamuning to get a proper csv file after that importing was easy. No promise but when I have time I will try to do the same for LISE and hope to give some feedback.

Both modules are actively mainted and like the future plans.

As said the power of a server is on great influence, some optimizations are better done on the database side. Bottom-line for a lot of data you better bring a powerful server.
Post Reply

Return to “Modules/Add-Ons”