A/B Split Testing and Multivariate Testing

[dobes_vandermeer]

What are people using for A/B Testing and/or Multivariate testing?  Is there any PHP solution that plays well with CMS Made Simple?

[nhaack]

Hi dobes_vandermeer,

indeed a very interesting topic. I haven't come across something ready yet. Do you have any conrete plans out abwhat you want to place into your A/B test? Would it be different content-blocks to be shown, different templates? Can you describe your case? Probably you can do it with smarty and either show the one or other block.

Of course, a way to track which block/template had been shown and how was the user reaction to that would have to be implemented as well.

If you do not want to make it automated, you could just run type A for one week and then type B for another and compare the results in your regular server statistics...

Very interesting problem

Best
Nils

[dobes_vandermeer]

I think AB testing is an important part of any ecommerce site - my site is http://www.clarityaccouting.com - and I'm thinking of trying some site variations to see what the responses are like.  Unfortunately, it seems like as time passes other factors like the season, economy, or whatever might be affecting our conversion rates so I was hoping to find a good package that could run multiple pages at once rather than doing this manually.  Of course, there's also a trick with getting the conversion data to show up in analytics.  Hmmm ... definitely complicated.

Google has something for this called the Google Website Optimizer which might be good - I'm just fishing around to see whether I can get a good recommendation from somewhere.

[nhaack]

Hi dobes_vandermeer,

you could create an A/B Template. With smarty we create a random number between one and zero. Further more we store the content-blocks we want to test into variables. Then we do something if our random number is one and something else when it is zero.

That could look something like that:

Code: Select all

{assign var=decision value="0"|mt_rand:1}

  {content block="testcontent_a" assign="content_block_a"}
  {content block="testcontent_b" assign="content_block_b"}

{if $decision==1}
  ...stuff...
     {$content_block_a}
  ...stuff...
{else}
  ...stuff...
     {$content_block_b}
  ...stuff...
{/if}

This would randomly provide a user with content A or Content B... if you just want to try out different layouts... just store the content-block data in variables (as in the example) and build your different template elements within the if-check.

If you want to test a version with a lower possibility do something like that:

Code: Select all

{assign var=decision value="1"|mt_rand:10}
{if $decision<=8}
80% possibility of showing this version
{else}
20% possibility of showing the other version
{/if}

Hurdle 1 taken

The next step is to track the data... a very dirty one would be:

1) create a DB table called something like "ab_testing" with you db admin tool
2) add a UDT that writes a counter of each test page to this db table ("Testpage A served X times") to the A and B section of your template.
3) On the page that you want to reach you could place a UDT that, depending on the parameter, writes a counter to a DB table  (e.g. "Target called from Testpage A X Times").

So after some time your table should like this:

Code: Select all

id    |    itemname          |    Counter
---------------------------------------------------
0         Test A Served          50
1         Test B Served          60
2         A Success              15
3         B Success              10

Now you should be able to say, which page worked better in percentage by dividing Success by Served counter.

In your target page, you place something like the following (assuming our UDT name is "Count Success" and the parameter name is "version"):

Code: Select all

{assign var=version_id value=$smarty.get.version}
{if $version_id == 'A'}
{count_success which="A"}
{/if}
{if $version_id == 'B'}
{count_success which="B"}
{/if}

You A/B template contains something like the following in the corresponding parts.

Code: Select all

{count_serve which="A"}

Hope this could inspire you a bit.... Let me know if that could be a solution for you

Best
Nils

[dobes_vandermeer]

Thanks Nils,

A/B testing generally requires that the user see the same thing if they return, but I suppose I could hack up some code that stores the random number in a cookie and re-uses it if it is there.

I'm a bit reluctant to roll my own, since it could explode into a huge project whereas there might be someone out there who made something good already.

And, I'd rather it shows up in my google analytics than creating my own database system.  I can probably submit a special pageview using analytics' javascript API and put that in my funnel to see which version converted more often.