Heteroscedasticity in the Logit Model

The purpose of these notes is to provide an example of heteroscedasticity and to explain how we can account for the unequal variances of our residuals in a weighted least squares model. The specific example of heteroscedasticity that we will use is an analysis of proportions data in a logit model.

Suppose that we observe our dependent variable as a proportion or percentage:

$$0\le p_i\le 1$$

which reflects an underlying "true" probability:

$$0\le {\pi }_i\le 1$$

If we used the raw percentage as our dependent variable, then our predictions of the dependent variable would not be bounded by zero and one. Instead, it is possible that some of our predictions of the dependent variable would be less than zero and some would be greater than one.

Converting the raw percentage to log odds gives us an unbounded dependent variable:

$$ln\left(\frac{p_i}{1-p_i}\right)=\alpha +\beta X_i+{ϵ}_i$$

but the residuals in such a logit model do not have constant variance:

$$var\left({ϵ}_i\right)=\frac{1}{n_i{\pi }_i\left(1-{\pi }_i\right)}$$

Instead the residual variance will be larger when the true probability is closer to zero or one.

Assuming that our explanatory variable is correlated with the true probability, the residual variance will be larger at extreme values of our explanatory variable.

Because we know the nature of the heteroscedasticity, we can modify our regression model to account for it. All we need is an unbiased estimate of the true probability for each observation.

OLS will provide such an unbiased estimate if there is no correlation between the residual and the explanatory variable (i.e. if the explanatory variable only affects the residual variance, not the residual itself).

So our first step is to estimate the regression coefficients with OLS:

$$ln\left(\frac{p_i}{1-p_i}\right)=\alpha +\beta X_i+{ϵ}_i$$

and then use the estimated coefficients to predict the true log odds:

$$ln\left(\frac{\widehat{{\pi }_i}}{1-\widehat{{\pi }_i}}\right)=\hat{\alpha }+\hat{\beta }{X}_i$$

But what we really want is an unbiased prediction of the true probability:

$$\widehat{{\pi }_i}=\frac{1}{1+exp\left(-\alpha -\hat{\beta }{X}_i\right)}$$

which we can use to weight each observation:

$$w_i=\sqrt{n_i\widehat{{\pi }_i}\left(1-\widehat{{\pi }_i}\right)}$$

And in the second step, we apply the weight to each variable and estimate the regression coefficients:

$$w_{i}ln\left(\frac{p_i}{1-p_i}\right)=\alpha w_i+\beta w_i{X}_i+w_i{ϵ}_i$$

Weighting each observation in such a manner accounts for the heteroscedasticity and the residuals in our regression model should now have constant variance.