Ab test onboarding: check both versions first

Many onboarding tests choose the version in the browser. If the choice is not saved for the user's account, a new user who reloads at the second step, opens the confirmation link on their phone or comes back from a laptop can see the other version from that point on. Their steps are split between the two, so the counts for each version mix together and neither number means what you think. A second leak comes from the email. A user who signs up in one version and confirms in a link that opens the other version finishes in a flow they did not start.

• Sign up in one window, reload at the second step, and check you still see the same version.
• Open the confirmation email on a phone and check which version the link opens.
• Check that both versions ask for the same fields, or that the one difference is on purpose.
• Look at the first screen after signup in each version, not only the form.

A signup flow usually asks for an email, a password, a workspace or team name, and sometimes a role or the reason for signing up. It may invite teammates, connect a calendar, or ask for a first project before the product shows anything. Each of those steps is a place a new user can stop, and each version of the flow gives them a different set. Before you compare, write down the screens in order for both versions, with the fields and buttons on each. Two flows that look almost the same can differ by one screen, and that screen is often where the difference in finishing comes from.

A switch is usually a flag or a random number kept in the browser. The version is picked when the page first loads, and nothing ensures the same pick the next time unless it was stored against the user. A user who clears cookies, switches devices or opens the app in a private window gets assigned again, and their later steps can belong to a different version than their earlier ones. The leak is quiet. Each version's numbers still look reasonable, and only a user-level check shows that some people were counted in both. Check that every event recorded for one user carries the same version name.

Compare the step where each version's simulated visitors stopped, and what they said there. A version where every simulated visitor stalls on the workspace name has a fault you can fix before any real user sees it. A version that loses visitors on a screen the other one does not have needs a closer look at that screen's copy and fields. Do not treat a run as a verdict on which version is better. It tells you which one has a broken step, which is a different question from which one finishes more often, and that second question needs real signups to answer.

Each release that changes sign-up, the confirmation email or the first screen after login can split users across both versions by accident. Rerun both versions through the same visitors after every release, starting from the same email address, and compare the step each one reached. A signed-in run uses a test account you give it, so the check covers the app after onboarding too.