Critical issues with the Pearson's chi-square test

Abstract

Pearson's chi-square tests are among the most commonly applied statistical tools across a wide range of scientific disciplines, including medicine, engineering, biology, sociology, marketing and business. However, its usage in some areas is not correct. For example, the chi-square test for homogeneity of proportions (that is, comparing proportions across groups in a contingency table) is frequently used to verify if the rows of a given nonnegative \(m \times n\) (contingency) matrix \(A\) are proportional. The null hypothesis \(H_0\): ``\(m\) rows are proportional'' (for the whole population) is rejected with confidence level \(1 - \alpha\) if and only if \(\chi^2_{stat} > \chi^2_{crit}\), where the first term is given by Pearson's formula, while the second one depends only on \(m, n\), and \(\alpha,\) but not on the entries of \(A\).

It is immediate to notice that the Pearson's formula is not invariant. More precisely, whenever we multiply all entries of \(A\) by a constant \(c\), the value \(\chi^2_{stat}(A)\) is multiplied by \(c\), too, \(\chi^2_{stat}(cA) = c \chi^2_{stat} (A)\). Thus, if all rows of \(A\) are exactly proportional then \(\chi^2_{stat}(cA) = c \chi^2_{stat}(A) = 0\) for any \(c\) and any \(\alpha\). Otherwise, \(\chi^2_{stat} (cA)\) becomes arbitrary large or small, as positive \(c\) is increasing or decreasing. Hence, at any fixed significance level \(\alpha\), the null hypothesis \(H_0\) will be rejected with confidence \(1 - \alpha\), when \(c\) is sufficiently large and not rejected when \(c\) is sufficiently small. Yet, obviously, the rows of \(cA\) should be proportional or not for all \(c\) simultaneously. For this reason, Pearson's test certainly cannot be applied to ``physical data'', which are obtained by measurements. Indeed, in this case matrix \(A\) depends on the unit of measurement.

The test can be applied only to categorical data and even then some further limitations are required, which we consider in this paper.