Introducing group sequential designs for early stopping of A/B tests

library(data.table) library(tidyverse) library(scales) library(gsDesign) library(kableExtra) Typically the sample size of a study is fixed, ideally based on a power analysis. We collect all our data then analyse it once with the appropriate statistical model. This is an ideal, which people often fall short of in practice. It’s common for scientists and analysts alike to check in on an experiment as the data is coming in (Albers, 2019, Miller, 2010). There’s a lot of potential for error here if experiments are stopped or changed based on ad hoc peeks at the data. [Read More]

Simulating A/B tests with data.table

library(tidyverse) library(data.table) set.seed(63) knitr::opts_chunk$set(warning = F, message = F) A/B tests involve comparing two different groups, typically with a randomised trial. Traditionally this might be a clinical trial where a new drug is given to a treatment group who are compared with a control group who did not receive the drug. Other early A/B tests were done in agriculture, where fields would be split into sections that were treated differently. [Read More]

Adjusting for covariates and baseline differences in A/B testing

Background A/B Testing is a term typically use in commercial settings to refer to the analysis and design of controlled experiments. Typically introductory posts on A/B Testing focus on simple between-subject randomised controlled trials (RCTs). In such experiments, participants (e.g. users, customers) are randomised into one of two groups. One group gets some treatment (e.g. a voucher code) and the other receives nothing (‘business-as-usual’), or some lesser treatment without the feature the trial is aiming to investigate. [Read More]