In the experiment simulator I built, the headline result is clean. Control converts at 8.4%, treatment at 9.1%, an 8.3% lift, p-value 0.012. Most readouts end there with the word ship.
The harder part is not the test. It is that the lift is not spread evenly. In the sample output, iOS users in the US show +14.2%. Android users in India show -3.1%. The aggregate is a weighted average of a good result and a bad one, and shipping to everyone means shipping the bad one too.
So I stopped treating significance as the output. The engine produces a decision, SHIP, DO NOT SHIP or INCONCLUSIVE, from a rule that lives in code, and then it breaks the result by platform and country and says who to ship to first.
One caveat I should be plain about: the data is simulated. This shows what the decision layer does, not a finding about a real product. A real experiment would also need a power calculation before the test, which is on my list and not built yet.
I do not think the p-value is the problem. The problem is that it is often the last thing anyone computes, so it becomes the thing the decision is made on.