Optimizely review

What Optimizely is

Optimizely started in 2010 as an A/B testing tool founded by Dan Siroker and Pete Koomen. Episerver bought it in 2020 and took the Optimizely name for the whole company in January 2021. Experimentation is now one product in a suite that also includes a CMS, personalization, analytics and commerce.

Web and feature experimentation

Testing comes in two parts. Web Experimentation runs in the browser. You add a snippet, then build variations in a visual editor or with custom code, and run A/B and multi-page tests. Feature Experimentation runs in your code through SDKs, so developers can put features behind flags and test them server-side. Developers can start with a free feature flagging plan that needs no credit card.

Stats engine

Optimizely's Stats Engine uses sequential testing with false discovery rate control. You can watch results as they come in, and you don't have to fix a sample size before you start. Optimizely also lists fixed-horizon frequentist and Bayesian methods, but both are in beta and you ask your customer success manager for access.

Opal AI agents

Optimizely's AI is called Opal. The variation development agent works inside the new visual editor. You describe a change in plain words, it finds the right page elements, applies the change and shows a live preview, falling back to custom code when it has to. Optimizely also lists an experimentation ideation agent and agents that plan and report on tests. Agent tasks use Opal credits, and Optimizely's own examples for the variation agent range from 30 to 130 credits per task.

Revenue tracking

Optimizely reports revenue per visitor and revenue per paying visitor, which is how most e-commerce tests should be judged. Getting there takes setup. You add a tracking call to your purchase confirmation page that sends the order value in cents, then add the revenue metric to each experiment. Shopify stores can use the Optimizely Data Platform app instead. Optimizely's docs also note that its revenue totals can differ from Shopify's, for example when the tracking call fires twice. There's no direct connection to Stripe or other payment processors listed.

Who it suits

Optimizely fits companies that run many experiments across several teams, have developers for server-side work, and want one vendor for content, testing and analytics. The Royal Mint and Robinson Club are among the customers it names on its experimentation page.

It fits less well if you're a small team. You can't see a price without talking to sales, usage is counted in impressions or monthly active users, and someone still has to decide what to test, build it and read the results. The AI agents speed up those steps, but they don't replace the person who owns the program.