How does Hypothesis Testing Work 🌱

Hypothesis Testing Framework

1. Calculate a test statistic $T(X_1,...,X_n)$
2. Select a rejection region $R$ where if $T(X_1,...,X_n)\in R$ then we reject the null, otherwise we retain it

We want $power\left({\theta }^{\ast }\right)$ to be small when ${\theta }^{\ast }\in {\Theta }_0$ (to minimize the chance of a false positive) and large when ${\theta }^{\ast }\in {\Theta }_1$ (to minimize the chance of a false negative).

• We can select a value of alpha and then solve for a rejection region that would satisfy that alpha or vice versa

P-Value

The definition of the p-value is often expressed as “the probability of observing data as or more extreme than what was observed, if the null hypothesis is true.”

Suppose we have a test that rejects when $T(X_1,...,X_n)\ge c$ and we observe a sample of data $(x_1,...,x_n)$ then we define the p-value as follows:

How p-values can be used to test hypotheses

Let $c_{\alpha }$ be the cutoff at which we reject the null for an $\alpha$-level test, i.e. reject the null if $T(X_1,...X_n)\ge c_{\alpha }$ and ${max}_{{\theta }^{\ast }\in {\Theta }_0}P\left(T\right(X_1,...X_n)\ge c_{\alpha }\text{if}\theta ={\theta }^{\ast })=\alpha$. Then, if we observe $(x_1,...x_n)$ we reject the null if $T(x_1,...,x_n)\ge c_{\alpha }$.

Therefore $p(x_1,...,x_n)\le \alpha ⟺T(x_1,...,x_n)\ge c_{\alpha }$ and these rejection conditions are equivalent.

• We can interpret the p-value as the smallest alpha for which we would reject the null.

Explaining How P-Value is Area under the curve from critical point

Suppose we have that $z=T(x_1,...x_n)$ and ${\theta }^{\ast }$ is the max value for the power in the null set.

Tests such as the Z-test, T-test, and F-test assume the distribution of our test statistic, which allows us to calculate the cdf to determine p-values.

Example

Intuition: Assuming the null is true, what is the probability of getting a test statistic that equivalently or more extremely supports the alternative hypothesis?