Thanks -- PHP developers need better alternatives for A/B testing than Google Website Optimizer. Your code is about the minimum viable A/B testing library for a developer: you can use it to start getting your feet wet with A/B testing done the right way.
I think if you start using it in production you will quickly understand that some of the implementation choices you've made are going to cause you pain, but that is less important than doing A/B testing. You can always rewrite the internals later. (I've done it a few times.)
Your mileage may vary, and I don't guarantee I'm reading your code right, but:
1) You will find it useful to be able to coerce A/B test values, particularly in testing and sometimes (for CS purposes) on the live site.
2) I do not see a convenient method of having Ms. Smith get the same website at home and at her workplace. That may or may not be a problem for you.
3) You may find "control" and "experiment" to be limiting once you start doing this frequently -- they aren't nearly as self-documenting six months down the line when you're staring at your web page wondering "What possessed me to make this button teal?!"
4) I'm pretty sure you have a race condition for creation of new tests. (This has bitten A/Bingo a few times.)
A/Bingo has a notion of "identity" (a string which is unique among users of an application). If you don't have one, you get one assigned randomly. Your identity gets associated with your account as early as possible, and if you sign in on another computer, I overwrite that computer's identity with yours. (Merging would be cleaner, but this works in practice for my site.)
The way A/Bingo picks alternatives guarantees that someone with a particular identity will always see the same alternative for a particular test. See the source code for details.
Since identity management is abstracted by the A/B testing framework, you can make it as simple or complex as you want, and it hooks into user management with on the order of ~5 lines of code.
Cog is a little webdev framework of mine, a work-in-progress steal from spending couple of years in different companies.
Since A/B testing is considered pretty fundamental to most of my clients, I thought I'd share with the dear community of HN one possible implementation of it. You can see A/B testing in action on the sample page:
When used together with scriptor, an A/Btest is about 1 line in the design file (db rows, and counters are automagically set up on first page visiting). Stand-alone you might want to integrate it with a templating engine of your own :)
- Foreach is better than a for loop for iterating over arrays
- Single character variable names only make code more confusing. You write code once, you read it a dozen times.
I challenge you to rewrite this with only using classes (no global functions) and to write unit tests for your code. Remember, if you have to jump through hoops to test your code, you need to rethink what you're doing.
That resolvequery() function in cog_database.php looks interesting. Wonder what happens if you call resolvequery('INSERT INTO ... VALUES (@email, @example)', array('email' => 'test@example.com', 'example' => 123));
I try to keep most of the A/Bingo work in a cache store (Redis, by choice) because sites with traffic two orders of magnitude or so higher than mine would blow up if they used MySQL for it. At something like Facebook game scales I think it would be smart to take A/Bingo and rip the database out entirely.
Writes hurt. A/B testing requires, minimally, O(tests * participants) writes. (And more reads, but the disk will be on fire from writes first.)
However, despite the cost, A/B testing benefits minimally from ACID. If you lose 2% of writes, oh well, it comes out in the wash anyhow. Thus the huge hardware/maintenance budget you'd need for web scale A/B testing on an ACID architecture makes little sense.
I think if you start using it in production you will quickly understand that some of the implementation choices you've made are going to cause you pain, but that is less important than doing A/B testing. You can always rewrite the internals later. (I've done it a few times.)