• twitter image
  • facebook image
  • youtube image
  • linkedin image
Language: CMS Made Simple Czech CMS Made Simple France CMS Made Simple Hungary CMS Made Simple Russia CMS Made Simple Netherlands

All times are UTC




Post new topic Reply to topic  [ 8 posts ] 
Author Message
 Post subject: Module for 20,000 products?
PostPosted: Mon Nov 12, 2018 3:05 am 
Offline
Forum Members
Forum Members

Joined: Tue Aug 01, 2006 1:48 am
Posts: 179
Location: Sydney
Hi, looking for a little advice. We have been asked to build a site for a smallish music label. The plan is to have up to 20 thousand of their releases listed. Any e-commerce probably handled off-site.
Each record will have summary and detail views
I’m a fan of LISSE, but worried about the volume of records? Would LISSE be efficient and quick?
Previous sites I’ve used CGBlog and Products for similar projects, and both work well, but to my mind have some limitations, mainly in the Admin layout and usability. That said they work.
The site will have other sections, which it would be handy to integrate to some extent, but again on previous sites, we have used CGBlog and Products for say, recipes and products, using a UDT to add recommended recipes to the Products and related products to recipe pages.
Any recommendations?
Thanks for any help.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Sun Dec 09, 2018 1:32 pm 
Offline
Dev Team Member
Dev Team Member

Joined: Mon Nov 28, 2011 9:29 am
Posts: 3028
Location: The Netherlands
Late reply though might still be useful:

I have been working on a website with those numbers of items in LISE.

In general it works well but some things to think about:

- my customer sometimes displays a list of hundreds or thousands of items on one page: bad idea, so better use the built-in pagination.
- in my case every item has several wysiwyg text fields often with large texts. This is no problem on itself but of course database will grow.
- Search: I found the CMSMS search module isn't optimized for this job. That is, when editing or adding a new item it took more than a minute to actually save the item. In the end found that the search module was the culprit and needed to do some small hacks. Initiating a re-index of the search module would better be avoided :-)
Searching an item goes very well and fast btw.
If you don't need search capabilities you can turn it off for the LISE instance btw.

So yes: LISE can handle those numbers but handle with care.

Regarding this specific website I did some feature requests:
http://dev.cmsmadesimple.org/feature_re ... ed_at+DESC

Especially the light-weight summary action would be a nice-to-have if it suits your use case too.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Mon Dec 10, 2018 9:15 pm 
Offline
Forum Members
Forum Members

Joined: Tue Aug 01, 2006 1:48 am
Posts: 179
Location: Sydney
As always, many many thanks velden.

I'm still not completely sure which module to use, but given what you say, may go with LISE.

Thanks again.

Cheers.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Tue Dec 11, 2018 8:25 am 
Offline
Power Poster
Power Poster

Joined: Sun Apr 19, 2009 9:33 am
Posts: 1271
LISE is not designed for large datasets especially if you are going to use many custom fields things get slow.

Look at the table structure especially cms_module_*_field val.
Queries are resource intensive and no caching takes place, flexibility comes at a cost.

Also read the how of LISE: "An important note of warning - This is not a content construction kit, such as in Drupal. It is meant for small listings, not to store thousands of records."

20.000 products is a NO for LISE.

When you are working with a lot of products the /admin area is probally not used much. You only need good import and display functionality.

Not sure if CGProducts is already working with background tasks (CGJobMgr), importing large datasets can benefit from this module a lot.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Tue Dec 11, 2018 10:22 am 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Mon Jan 29, 2007 4:47 pm
Posts: 1737
arnoud wrote:
20.000 products is a NO for LISE.

I beg to disagree...
Not long ago I would have agreed with you, until I had to work on a site already having multiple instances of LISE, some with over 50k records and growing, all heavily packed with custom fields an most cross linked with each other...
LISE behaved perfectly fine. As Velden described there are some issues that you'd need to pay attention to, that I'm solving in next version, but it works perfectly.
I do agree with you that Products may be a better choice particularly if handling on the order on the hundreds of thousands of records: Products is a specialized module optimized to work with very large catalogs of items linked or not with Calguy's eCommerce suite.
But I wouldn't dismiss LISE for a less than 100k records use as it brings some versatility and the possibility to cross link several instances if needed.

_________________
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).


Code of Conduit | CMSMS Docs
Help Support CMSMS
My developer Page on the Forge | Yet another blog about CMSMS

GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!



Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Tue Dec 11, 2018 4:02 pm 
Offline
Power Poster
Power Poster

Joined: Sun Apr 19, 2009 9:33 am
Posts: 1271
Speed is always relative.

On shitty hosting or a low profile machine I won't bet on LISE when using large number of records.

When your server is fine the only limitation is the available ram, cpu and storage ofcourse. I would bet LISE can even store 100.000 records.

My claim is that LISE and Products both use fielddef/fieldval tables to store custom fields. As far as I know Products does a better job when importing records with images and has better caching, that is why I prefer Products in this case.

In the end a flat table is easier to query and more performant for a app like record store this might also a feasible solution. Especially when the shop will be refereshed from a seperate datastore. If there are already 20.000 records on file probally the CMS entry part is not needed at all.

Love to hear in more detail where the issues are to pay attention to when using LISE and what the improvements are you have on your wishlist?

I appreciate your effort in developing and maintaining LISE for many use-cases it is an excellent solution. For large datasets I have my doubts it is fast enhough.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Tue Dec 11, 2018 5:41 pm 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Tue Oct 19, 2004 6:44 pm
Posts: 7966
Location: Fernie British Columbia, Canada
The primary difference between Products and LISE is the database.

Products uses a primary table for each record and creates indexes that allow faster searching for primary filters and sorting (name, base price, categories, etc etc). People can create more indexes if needed to optimize for specific queries. Creating indexes only affects the performance of inserting and updating records, not selecting them.

It uses a flat key/value table for custom fields.

Products uses some internal caching to cache objects in memory to prevent multiple requests from the database for the same objects.

LISE uses a flat key/value table for everything. Which makes it impossible to create indexes for filtering or sorting by numeric values (i.e: price, or score, or category).

Sorting or filtering by numeric values in LISE will be relatively slow on large quantities of data because everything needs to be 'cast' to a numeric value for comparison. Even sorting or filtering by string values will be slower as you cannot (AFAIK) create an index on specific keys.

So, with LISE you are relying on CPU processing power of both the database host and the web host in order to overcome the fact that you cannot take advantage of Mysql indexes to increase performance.

YMMV

_________________
Follow me on twitter
Please post system information from "Extensions >> System Information" (there is a bbcode option) on all posts asking for assistance.
--------------------
If you can't bother explaining your problem well, you shouldn't expect much in the way of assistance.


Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
 Post subject: Re: Module for 20,000 products?
PostPosted: Tue Dec 11, 2018 5:42 pm 
Offline
Dev Team Member
Dev Team Member
User avatar

Joined: Mon Jan 29, 2007 4:47 pm
Posts: 1737
arnoud wrote:
Love to hear in more detail where the issues are to pay attention to when using LISE and what the improvements are you have on your wishlist?

As it stands LISE has a few Achilles heels:
  • - importing from csv of a large number of items (around 2k but it will depend on the allocated memory) is slow and on some servers it stalls (solved on next major version);
  • - filtering doesn't handle ranges (inclusive or exclusive) which somewhat limits its utility. Being the best way to do frontend searches and cross linking between LISE instances it needed some improvement (solved on next major version: the existing filtering parameters will be deprecated and a new number of parameters will be used so we can keep backwards compatibility for a while);
  • - allowing a filter to have an empty parameter will try to return all existing items unless there is a limit set (which will produce pagination, which is fine in most cases);

What I'm working on:
  • - from my tests, importing a huge csv file directly to a temporary db table and then from that table to the LISE instance db tables is around 100 times faster or more (no kidding) so that's what is going to be used for import/export of csv files;
  • - the queries for the extended filtering will be as optimized as possible and make use of transactions as much as possible. There is some gains in speed here, although there will be room for more improvement;
  • - templates will be moved to DM and avoid duplicates per instance. Several instances may share the same templates if needed;
  • - there will be the possibility to preview an item when editing or adding one (similar to Content Manager or News preview);
  • - something that may also make it to next version is custom instance global settings which will allow for having some preferences for an instance to be used on the templates: good for for frontend lightboxes plugins or other similar uses;

There are a few more things but I don't yet have details about them.

arnoud wrote:
I appreciate your effort in developing and maintaining LISE for many use-cases it is an excellent solution. For large datasets I have my doubts it is fast enhough.
I appreciate your words and input of course. :)

_________________
"There are 10 types of people in this world, those who understand binary... and those who don't."
* by the way: English is NOT my native language (sorry for any mistakes...).


Code of Conduit | CMSMS Docs
Help Support CMSMS
My developer Page on the Forge | Yet another blog about CMSMS

GeekMoot 2015 in Ghent, Belgium: I was there!
GeekMoot 2016 in Leicester, UK: I was there!



Top
  Profile  
 
Share On:
Share on Facebook Share on Twitter Share on Google+
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
A2 Hosting