Does anybody know more about how Mozilla manages experiments? I'd always assumed that all users downloaded an identical binary and got the same behavior. How do they assign 10% of their users to an experimental group?
Is it possible that two users will see different functionality when they are both using the same OS and release, say Firefox from the latest version of Ubuntu?
We have several different mechanisms for running experiments. The e10s one specifically runs through what we rather poorly named a "system add-on".
e10s code is present in all builds. The system add-on contains logic that decides whether to turn it on or not. The system add-on can be updated daily (or more often), in all channels.
* I apologize for the naming, this is entirely my fault. I spent a lot of the last twelve months scheming/designing and developing experimental update mechanisms for Firefox.
Firefox generates a random number on first run. Users who are selected for the experiment are sent a signed extension from Mozilla that makes browser.tabs.remote.autostart preference return true if they are on the correct release of Firefox.
As with most native A/B tests, the binary probably ships with code for both functionalities. When the browser launches for the first time they generate a random number, and if it's below a certain threshold you get variant A, above you get variant B. The information gets persisted on disk so it remains unchanged across sessions.
It seems that there is some communication back to Mozilla, since they claim that they can increase or decrease the number of users in each group over a week; that seems unlikely to come in the form of a new binary.
Yeah, Test Pilot is more for features where Mozilla wants to experiment and see, if people even like the feature in general, not so much for bug-testing.
With e10s, it was clear that it would be included in Firefox, so they just tested it by pushing it through the normal chain of Nightly->Developer Edition->Beta->Stable.
Is it possible that two users will see different functionality when they are both using the same OS and release, say Firefox from the latest version of Ubuntu?