Loose memory with adodb

[Piratos]

Make this test - script test.php in the root:

Code: Select all

<?php

include("config.php");
include_once($config['root_path']."/lib/adodb/adodb.inc.php");
$db = &ADONewConnection($config['dbms'], 'cms:pear');
$db->Connect($config["db_hostname"],$config["db_username"],$config["db_password"],$config["db_name"]);

echo memory_get_usage().'<br />';

$d=mysql_connect($config["db_hostname"],$config["db_username"],$config["db_password"],$config["db_name"]);
echo memory_get_usage().'<br />';
?>

The results:

First memory usage  (open database with adodb) 1357712 bytes.
Second memory usage (open database with mysql direct) 1358736.

The connection with mysql - direct needs only  1024 bytes !!

Here you can loose memory without an end  and you can loose speed.

And this you need of memory with a fresh installed 0.11.1

7333896 bytes of memory

[Ted]

I really don't want to have to rewrite every SQL query, as well as having every module be rewritten.

Can you do the same test with the latest version of adodb lite, including the pear and date modules?  That least that would be an easier fix for now.

Thanks!

[Piratos]

Test with Adodb Lite Version 1.11 with modules pear and dates


47336  - Needed memory for config.php
690416 - Starting connection with Adodb - Lite
691376 - Starting connection with mysql direct


Results

config.php 47336 Bytes
Adodb Lite 690416 - 47336 =  643080 Bytes
Mysql Direct 691376 - 690416 =  960 Bytes


The best result is mysql - direct.  i estimate a work of 15 hours

[Piratos]

I work now with Adodb Lite .
The most changes are from _Execute to Execute and RowCount() 0 to  EOF and some special changes.

And so i work that my modules and plugins are compatible with Adodb Lite and Adodb.

The latest Version of Pisearch 1.33 is my first compatible Plugin.

[Piratos]

Adodb lite is NOT the solution to solve memory problems.

Generated in 0.292727 seconds by CMS Made Simple 0.11.1 (cached) using 0 SQL queries and 7102512 bytes of memory

That are the results with following modules:

Content2pdf
Cssmenu
Ellnav
FCKeditorX
Kalender
News
Pifaq

and included Piserach

With Adodb they are using 7,888 MB

[Ted]

But what about postgres (http://dev.cmsmadesimple.org/tracker/in ... 6&atid=101)?
What about sqlite?
What about all of the existing modules out there that will break?  Or all of the date handling code that has to be changed?
And probably the biggest question...  if adodb lite is using 640k , what is using the other 6.5 megs?

Granted, I agree that would be a great way to start solving the memory issues, the ramifications of it are HUGE.

Do you know of any way of profiling the memory usage so that we can see exactly what is taking what memory?  There might be some more simple solution that we're both overlooking.

Thanks!

[Piratos]

Just another test with Adodb and Mysql direct.

I have written a pisearch version that is using mysql direct.

The results:

pisearchmysql


Get a value to start

390 Pages - view default-page

Generated in 0.349133 seconds by CMS Made Simple 0.11.1 (cached) using 14 SQL queries and 9284160 bytes of memory

searching trivial word it was in every page

Result:

Es wurden 390 Inhalte mit dem Suchbegriff ->professionellprofessionell<- gefunden.

Generated in 0.595971 seconds by CMS Made Simple 0.11.1 (cached) using 15 SQL queries and 9857352 bytes of memory

searchtime for 390 pages 0.190079

Mysql direct is 2,5 faster as Adodb and this with only ONE query !!

....................................

what is using the other 6.5 megs?

That is the structure and the code of the cms by itself.

The most modules have only overheads of 100 .. 140 KB, i have tested it.

The overhead of Adodb ca. 1,3 MB - mysql my 93 KB i have measured.

--------------------------------------------------------------------------------

If the memory grows  here from  7,88 MB to 9,28 MB only if i show the default page but i have now 390 pages and not 22 pages  is something wrong in the structure.

We must find it !

[Piratos]

390 Testpages show default page
Generated in 0.466536 seconds by CMS Made Simple 0.11.1 (cached) using 14 SQL queries and 9288656 bytes of memory

And here you can see the memory flow of the cms.

757872 - Start of include.php
757872- after fileloc.php
758768- after getlogger
759304- after Starting include.php
866320- after include /lib/misc.functions.php
870752- after /lib/classes/class.global.inc.php
872416- after define $gCms
872936- after include version.php
908032- after include /lib/config.functions.php
911808- after cms_config_load
1982616- after include /lib/adodb/adodb.inc.php
2224216- after make Adodb Connection
2517672- after include Smarty
2693952- after include /lib/page.functions.php
2971648- after include /lib/content.functions.php"
2997624- after include /lib/classes/class.content.inc.php
3323936- after include /lib/classes/class.content.inc.php
3717160- after include /lib/classes/class.module.inc.php
3753040- after include /lib/classes/class.group.inc.php
3828648after include /lib/classes/class.user.inc.php
3865072after include /lib/classes/class.bookmark.inc.php
3941016- after inbclude /lib/classes/class.htmlblob.inc.php
4031536- after include /lib/classes/class.template.inc.php
4082328- after include /lib/classes/class.stylesheet.inc.php
4097312- after include /lib/translation.functions.php
4097376- after include /lib/contenttypes
4318624- after while (($file = $ls->read()) != "")
5081744- aftre new Smarty()
5083864- after load_site_preferences
5083864- after issset($CMS_AD...
6833160- after LOAD_ALL_MODULES
6833280- after leaving include.php
6790360- after loading include.php - we are now in index.php
6790432- after $page =
6790432- after if ($page ==
6791584- before $smarty->is_cached =
9288432- after html = smarty->fetch(template:.pageinfo->template_id =
9288432- after if (isset($_GET[ =
9288432- after if (get_site_preference =
9288432- after if (!$cached) =
9288496- after foreach($gCms->modules as $key=>$value)
9288496- after pageinfo->template_encoding

[melix]

Well Piratos, if you're trying to say that ADOdb will use more memory than mysql direct, and consume less CPU, you will always be right, and there's no point discussing it. ADOdb is an abstraction library built on top of MySQL (and others), there's no comparison possible, and your problem is therefore to discuss the pros/cons of an abstraction layer over direct access to the DB.

What I can directly see is that using ADOdb :

- makes code cleaner (object model)
- makes db access more secure (no need for dirty mysql_escape_* code all the way)
- allows therefore to code much faster, concentrating on business code instead of SQL code
- allows integrating the database evolutions without having to change the code of your applications (direct success : MySQL 5, which adds a huge set of improvements, which would require you modify all your code to use them, while using ADOdb will make them available seemlessly)
- allows developing your application for several databases without core code modifications

These advantages are, for me, worth the overhead, having in mind that :

- ADOdb overhead is, in terms of algorithmic complexity, much less than twice native access, and, moreover, linear
- tweaking php.ini to add a little more memory is not that hard (granted it is not easy if you haven't direct access to the server)
- memory is cheap, much more than a developers' hours of hard work
- as CPU power increase, overhead will become less and less visible
- ADOdb exists as a PHP extension, pure C code, which allows you to have about 100% performance improvement

If these are not sufficient to you, the problem is that when you'll have tweaked things for ADOdb, you'll come to Smarty, where the problem is almost the same : having a template engine is wonderful in terms of abstraction and ease of development, but introduces an overhead to both CPU and memory usage. Then, as long as you will tweak, you're going to remove all that makes CMS easy to develop, document, and evolve.

Therefore, I'd rather work on algorithms improvements, cache tweaks and alternative data structures to make things better, but that's just my opinion

[Piratos]

I know the advantages of adodb very well, but if nobody or only a few usinf other databases  as mysql  with this cms there are no advantages.

Here we need memory and to change from adodb to mysql is one  way to get memory AND speed.

-----------------------------------------------------------------------------------------------------------------
6791584- before $smarty->is_cached =
9288432- after html = smarty->fetch(template:.pageinfo->template_id =

The reason for this is the contentcache.php

All of table content and content_props are loaded in the memory TWICE ! The whole Content !!!

a little help is to change in class.content.inc.php

from

function GetAllContent($loadprops=true)

to

function GetAllContent($loadprops=false)

and to clear the cache.

-------------------------

Another solution is to use adodb-csvlib.inc.php and to call CacheExecute  but than something must changed in the source.
------------------
In many functions is this code

global $gCms;
$db = $gCms->db;

without a use of it - this cost only memory

[melix]

This is what I say when I talk about algorithms and caching. For example, when you read :

Code: Select all

global $gCms;
$db = $gCms->db;

This is an error, which must be replaced with :

Code: Select all

global $gCms;
$db = &$gCms->db;

(note the "&"). I think there are many things like this hidden in the code. I have myself worked on a hierarchical content manager, which will include some cache mechanism. I'm sure we'll be able to cut memory usage using *really simple* things like this

[calguy1000]

I'll endeavor to go through my code and make sure that I fix all occurances of this, are there other items that can be cleaned up to help in performance.

[Ted]

In svn, I've created a few methods to give back singletons of certain objects that we're positive we only want one of.

Code: Select all

$db =& $gCms->GetDB();
$config =& gCms->GetConfig();
$smarty =& gCms->GetSmarty();

The nice thing about them is that they are lazy loaded.  Meaning that if they're never used, then they never get loaded.  Obviously, these 3 objects are used a lot...

However, I'd like to extend this pattern to other internal objects as well.  Things like sessions and such.  So that if they never get used during the course of the request, they don't get loaded.

Melix is currently on the project of making the content hierarchy more efficient as well.  I think this will follow the same kind of design, so that only as much content as necessary is loaded with the least amount of SQL statements.  It's actually pretty complicated...

[roman]

are you thinking about memory infos is right?

Code: Select all

757872 - Start of include.php
757872- after fileloc.php
758768- after getlogger

fiel fileloc.php with 4 definitions consumpt 0 bytes memory?

[Piratos]

757872 - Start of include.php
757872- after fileloc.php
758768- after getlogger

So are the results, you can test it by yourself .

Place  echo memory_get_usage().'This is Point Alpha
';  in the sources at all breakpoints you wish to look.