Statistical inference - Modeling Foundations and Asymptotics

Models and Assumptions Introduction All statistical inference rests on assumptions about how data was generated. These assumptions form the foundation of statistical modeling. If the assumptions are correct, our conclusions will be valid. If they're wrong, our results can be misleading—even if our calculations are mathematically perfect. This section explores the key types of assumptions statisticians make and why getting them right matters so much. Types of Modeling Assumptions When building a statistical model, we must decide how much structure to assume about the data-generating process. Fully Parametric Models assume that the data comes from a specific, well-known family of distributions that can be completely described by a finite number of unknown parameters. For example, if we assume that observations come from a normal distribution, we only need to estimate two parameters: the mean $\mu$ and the variance $\sigma^2$. Once we know these two numbers, the entire distribution is determined. Parametric models are powerful because they allow us to make precise inferences, but they require us to make strong assumptions about the data's structure. Non-parametric Models take the opposite approach. They make minimal assumptions about the form of the data-generating distribution. Instead of assuming the data follows a normal distribution or any other specific form, non-parametric methods work with the data as it is presented. For example, non-parametric inference often relies on the sample median rather than the sample mean, and can work with virtually any distribution. The trade-off is that non-parametric methods are generally less powerful (they extract less information from the data), but they're also more robust to violations of distributional assumptions. The choice between parametric and non-parametric modeling depends on how confident you are about the data's structure and how much precision you need. The Critical Importance of Correct Assumptions Here's the key principle: valid statistical inference depends on the assumed data-generating mechanism actually reflecting reality. This is not a minor detail—it's fundamental. Your calculations might be perfectly executed, but if your assumptions are wrong, your conclusions will be wrong. Common assumptions that can lead to trouble when violated include: Random sampling: Assuming that observations are independent and randomly drawn from the population. If your sample is biased (e.g., surveying only people who visit a particular website), conclusions about the broader population will be invalid. Normality: Assuming that data follows a normal distribution. This matters when conducting t-tests or confidence intervals, and violations can distort results. Specific model forms: Assuming a particular relationship between variables. For instance, the Cox proportional hazards model (used in survival analysis) assumes that hazard ratios between groups remain constant over time. If this assumption fails, your estimates of survival differences will be incorrect. The lesson here is that before conducting any analysis, you should check whether your assumptions are plausible by examining your data and considering how it was collected. The Central Limit Theorem and Approximation One of the most important results in statistics is the Central Limit Theorem (CLT). It states that if you take repeated samples from a population and compute the sample mean $\bar{X}$ for each sample, the distribution of these sample means will approach a normal distribution as the sample size $n$ becomes large, regardless of the shape of the original population distribution. This is remarkable because it means we can use normal distribution approximations even when the original data isn't normally distributed. However, there are important conditions: The population distribution should not be heavy-tailed (meaning it shouldn't have extreme outliers that dominate the distribution) The sample size needs to be large enough for the approximation to be accurate In practice, a sample size of ten or more observations is often sufficient for the normal approximation to work reasonably well for many distributions, though larger samples are better when data is highly skewed. The histogram above illustrates this principle—the blue bars show observed data that isn't normally distributed, while the smooth curve overlaid represents a normal approximation. Even though the original data has a specific shape, the sample mean's distribution converges toward the normal curve. Limiting Distributions and Asymptotics Limiting distributions describe what happens to a statistic's distribution as the sample size approaches infinity. The Central Limit Theorem is one example of a limiting result: it tells us the limiting distribution of the sample mean is normal. Statisticians use limiting distributions for several practical reasons: Theoretical understanding: They help us understand the behavior of statistics when samples are very large. Constructing tests and confidence intervals: We often use limiting distributions to approximate the true, finite-sample distribution of our test statistics and estimators. Evaluating approximation quality: A natural question arises: how close is the limiting distribution to the actual distribution with our real (finite) sample size? To answer this question, we use Monte Carlo simulation, a computational technique where we repeatedly generate data according to our assumed model, compute our statistic, and observe the resulting distribution. By comparing the simulated distribution to the limiting distribution, we can see whether our approximation is good enough for practical use. For example, if you're planning to use a normal approximation with sample size $n = 50$, you could simulate 10,000 samples of size 50 from your assumed distribution, compute the sample mean for each, and plot the histogram. If this histogram looks very similar to the normal distribution, your approximation is reliable. If it looks quite different, you might need a larger sample or a different approach.