FEU CSV Export - 4 problems and 3 fixes

Have a question or a suggestion about a 3rd party addon module or plugin?
Let us know here.
Post Reply
apri
New Member
New Member
Posts: 5
Joined: Fri Sep 05, 2008 3:54 am

FEU CSV Export - 4 problems and 3 fixes

Post by apri »

Hi guys,

I've been using FEU and SelfReg on a production site and have run into some problems with CSV export.

Specifically:
1) I was getting a *lot* of gaps in the output meaning that when Excel opened the CSV the data was not arranged in nice columns
2) It seemed rather slow

Looking at the code in action.do_admintasks.php...

1) Turned out to be three problems:
1a) Because the table output from the query uses DISTINCT, some rows are removed which shouldn't be.  For example, a user joins and fills in two blank fields.  The DISTINCT query removes all but one of these rows from the output because the only column in the query which are unique for a particular user is the user data itself.

Code: Select all

        $query = "SELECT DISTINCT groupname,userid,username,createdate,expires,data FROM ".cms_db_prefix().
"module_feusers_groups A,".cms_db_prefix()."module_feusers_grouppropmap B, ".cms_db_prefix()."module_feusers_properties C,
 ".cms_db_prefix()."module_feusers_users D where A.id=B.group_id and C.title = B.name and C.userid = D.id 
order by groupname,userid,sort_key,name
 limit $start,$num";
Solution is either to remove the DISTINCT (I didn't do this because I wasn't sure whether it's protection for some other reason) or add the field name to make these records distinct:

Code: Select all

        $query = "SELECT DISTINCT groupname,userid,username,createdate,expires,data,name FROM ".cms_db_prefix()."module_feusers_groups A,
".cms_db_prefix()."module_feusers_grouppropmap B,
 ".cms_db_prefix()."module_feusers_properties C, 
".cms_db_prefix()."module_feusers_users D where A.id=B.group_id and C.title = B.name and 
C.userid = D.id order by groupname,userid,sort_key,name limit $start,$num";
1b) Commas in fields become field delimiters in the CSV and split field values across two cells.

Code: Select all

            $output .= ",".$row['data'];
My fix is to add quotes around each field output as so:

Code: Select all

            $output .= ",".'"'.$row['data'].'"';
1c) Now, this I don't have a solution for.  It seems that for checkbox fields an "off" value is not written as a value in the cms_module_feusers_properties.  Only "on" values are written.  (At least for the optional checkboxes I'm using).
Since the value doesn't exist in the database, it causes similar gaps in the output and messes up the resulting spreadsheet.
IDEAS ANYBODY?


2) The query is executed two rows at a time.

Code: Select all

    $start = 0;
    $num = 2; // todo, should be a preference
    $done = 0;
I realise that the comment suggests that the rows per query should be a preference but I believe that the default should be higher.  I raised it to 200 on my installation - CSV download now runs noticably and dramatically faster.

Code: Select all

    $start = 0;
    $num = 200; // todo, should be a preference
    $done = 0;
User avatar
FantomCircuit
Forum Members
Forum Members
Posts: 75
Joined: Fri Nov 10, 2006 1:34 am
Location: Gold Coast, Australia

Re: FEU CSV Export - 4 problems and 3 fixes

Post by FantomCircuit »

Thanks heaps for reporting your findings apri - seems this is still broken in the current version of FEU

the $num = 200; fix let me export my list of about 600 users in about a minute - where normally it would take 30 minutes and time out :(
jmcgin51
Power Poster
Power Poster
Posts: 1899
Joined: Mon Jun 12, 2006 9:02 pm

Re: FEU CSV Export - 4 problems and 3 fixes

Post by jmcgin51 »

please report this in the Forge for the FEU module.
apri
New Member
New Member
Posts: 5
Joined: Fri Sep 05, 2008 3:54 am

Re: FEU CSV Export - 4 problems and 3 fixes

Post by apri »

No problem!

I now have fixes for all the problems I had and I'll be posting them as patches in the tracker as soon as I can.

Cheers.
apri
New Member
New Member
Posts: 5
Joined: Fri Sep 05, 2008 3:54 am

Re: FEU CSV Export - 4 problems and 3 fixes

Post by apri »

All my FEU and SelfReg CSV fixes are in the trackers.  They are based on the HEAD version in SVN.
doodle
Forum Members
Forum Members
Posts: 10
Joined: Wed Sep 03, 2008 11:17 am

Re: FEU CSV Export - 4 problems and 3 fixes

Post by doodle »

Have you also found a solution for the export issue? When I export all users, every user seem to be member of every group...
apri
New Member
New Member
Posts: 5
Joined: Fri Sep 05, 2008 3:54 am

Re: FEU CSV Export - 4 problems and 3 fixes

Post by apri »

I wrote fixes for all the problems and uploaded a patch in the tracker.  Unfortunately these aren't available in the tracker any more.  So here they are as attachments.  I'll be generating new patches based on current SVN shortly.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

[The extension txt has been deactivated and can no longer be displayed.]

[The extension txt has been deactivated and can no longer be displayed.]

[The extension txt has been deactivated and can no longer be displayed.]

apri
New Member
New Member
Posts: 5
Joined: Fri Sep 05, 2008 3:54 am

Re: FEU CSV Export - 4 problems and 3 fixes

Post by apri »

Spoke to calguy over IRC today - he's prepping a release and asked for the fixes as patches to the latest SVN which I've done.  So hopefully all the fixes will get into the release.
Post Reply

Return to “Modules/Add-Ons”