Somesh Zanwar.

Work / 03

Decision Intelligence Experimentation Platform

A p-value of 0.012 on an 8.3% lift looks like a ship. Who exactly are you shipping to?

Summary

An A/B test pipeline that goes from raw events to a coded SHIP, DO NOT SHIP or INCONCLUSIVE decision, with segment breakdowns that show where the experiment worked and where it did not.

Mar – May 2026 · Public repo, live simulator

PythonPostgreSQLdbtSciPypandasStreamlit
How it flowsOne request or record, start to finish
  1. Simulated events50,000 users, 280K+ events, 3.8K+ orders
  2. PostgreSQLcontrol and treatment separated at ingestion
  3. dbt modelsstaging, intermediate, marts; conversion, AOV, segment tables
  4. Statistical enginez-test, lift, p-value, confidence interval
  5. Segment analysisplatform by country
  6. Decisioncoded rule, then a dashboard for a non-technical reader
What I built
  • A simulation of 50,000 users assigned to control and treatment, producing 280K+ events and 3.8K+ orders across platforms and countries.
  • dbt staging, intermediate and mart models, including an experiment-level base table and conversion, average order value and segment metrics.
  • A z-test for conversion with lift, p-value and confidence interval, feeding a decision rule that is written in code rather than interpreted by eye.
  • Segment analysis by platform and country that produces a plain-English recommendation on who to ship to first.
DecisionsWhat I chose, what I passed on, why
Stopping at a decision, not a p-value
over: A significance readout

Significance answers whether the result is real. The rollout question is who it works for. In the example output the aggregate says SHIP (control 8.4%, treatment 9.1%, +8.3% lift, p = 0.012), while iOS users in the US show +14.2% and Android users in India show -3.1%.

Decision logic in code
over: Analyst judgement per run

A rule can be reviewed, tested and argued with. A judgement call made on a Tuesday cannot.

Real vs simulatedWhat this does and does not show
The data is simulated, so this demonstrates the decision layer, not a finding about a real product. The example numbers above come from the simulator's sample output.
Next
  • Variance reduction (CUPED) for faster readouts.
  • Sample-size and minimum-detectable-effect calculator.
  • An experiment registry to stop overlapping tests.
  • Alerts when a guardrail metric is breached.