01
The script that runs the test changes the result
A browser-side tool has to change the page before the shopper sees the original. So vendors ask you to make the page wait. Optimizely "uses a synchronous snippet to prevent flickering", which means the browser draws nothing until the script arrives. VWO's standard snippet sets the page to invisible for up to 2 seconds while its code loads. Intelligems, built for Shopify, blocks painting by default "at the cost of slightly higher FCP and LCP times", the measures Google uses for how fast a page appears.
How long does that wait last in practice? Simo Ahava measured Google Optimize's hide on real page loads: 581 milliseconds on average with the recommended snippet, 964 through Tag Manager.
Now put that next to what you are trying to measure. When Bing slowed pages on purpose, in a randomized test, every 100 milliseconds cost 0.6% of revenue. A meta-analysis of 115 published A/B tests found the median lift was close to 4%. At Bing's rate, half a second of hidden page is about the size of the win you are hunting. Your store is not Bing, and your number will differ. The point is the order of size: the tool's own cost is not small next to the effects it is built to find.
When both versions carry the cost, the test stays fair, but it measures a slower store than the one you will run after the test ends. When only the new version carries extra work, such as code that rewrites the page, its slowdown is counted against the design. Optimizely itself warns that flicker "makes the outcome of your experiment less reliable."
eLLMo's own script also runs before the page draws, so it is not free either. It does one small job, picking the version, and does no editing work after that. Everything the shopper sees is drawn by Shopify from your theme.