Overly aggressive A/B testing – Beware the Jabberwork

Well that was an interesting bug. Over the last day, we had a handful of users complain that our log in, registration, and activation pages were all broken. And not just broken in a small way – the entire login form was missing! Strangely, none of our automated or manual testing had seen anything like this bug, so needless to say, as soon as we got wind we dove in to try to see what the problem was. We couldn’t reproduce it on our local machines, staging servers, or even on production, and yet the reports kept coming in.

Finally, we had a breakthrough when John Le, a developer friend of ours, told us that when he hit the signup page and pulled up the html inspector, the form was there, it had just been hidden away behind a dastardly “display: none”. Pouring over our html, css, and javascript came up empty, until we realized where the issue was.

Optimizely

Now don’t get me wrong. I have nothing against Optimizely – they’re a fantastic service that has made A/B dead simple for us. But as it turns out, a little too dead simple – in testing the layout of our home page, we decided to try hiding some elements, which happened to have the same XPath selector as our login and registration form. And since the optimizely script is run against all of our pages, but only on production, we saw the really strange behavior described.

Moral of the story: Optimizely and other such services are like power tools – great at what they do, but should be wielded carefully and with respect, because they can cut your hand off. Oh, and hiding your login form decreases conversion rate!

Read More →