First, Do No Harm

Does remediation work? Do entering college students with weak mathematical skills benefit from pre-college mathematics courses? Or are such courses merely a waste of their time and tuition dollars and the institution’s resources? Worse, might they actually impede these students’ chances of successfully completing a two-year degree? The answer, of course, depends on how such courses are taught, but the results of a 2008 study by Calcagno and Long [1] imply that—in at least one large state system—directing students to pre-college mathematics has decreased their likelihood of completing an Associate’s Degree.

One must be careful about extrapolating from a single study. The Calcagno and Long study tells us about students who entered public two-year colleges in the state of Florida during the years 1997–2000. Nevertheless, it forcefully makes the point that we cannot assume that merely offering pre-college mathematics is any guarantee that we are actually providing a service to students. In view of the fact that pre-college mathematics constitutes a substantial majority of the mathematics taught in two-year colleges, all colleges offering such courses have a responsibility to determine whether they are helping or hurting their students. The Calcagno and Long paper also offers a useful statistical approach to answering this question.

Assessing the effectiveness of remediation is difficult. The students in pre-college courses are, by definition, those with weaker mathematical backgrounds. One can expect that they are less likely to complete a two-year degree regardless of the quality of the intervention. The question is whether students who are assigned to pre-college mathematics are benefiting from the program. Would they have just as well, or perhaps even better, if they had simply gone straight into College Algebra? One approach to answering this question is to randomly assign students near the cut-off either to go straight into the college-level course or to first take the pre-college course. This has two difficulties. One is in getting a sufficient number of students so that the results are statistically significant. The other difficulty is ethical. Students who otherwise would have been allowed to proceed directly to a college-level course are held back and/or students who need help are denied it.

This is a statistical design problem that medical research encounters when studying the effectiveness of an existing treatment for which there is no alternative. Denying treatment to those who need it would be unethical, but testing the treatment on those whose need is marginal may not produce meaningful results. The statistical method that has been developed for such situations is called regression discontinuity design (RDD). It is particularly effective when there is a test to determine whether treatment is needed and when the cut-off is sharp: Those who score at or below a given level receive the treatment; those above do not.

Such is the case in the state of Florida where there is a common mathematical placement exam used by all public two-year colleges, the Florida College Entry Level Placement Test (CPT), part of the College Board’s ACCUPLACER system. There is a common cut-off score used by all colleges. On a scatterplot of test scores versus some desirable outcome, such as completion of a two-year degree, Calcagno and Long constructed a quadratic regression curve for those at or below the cut-off and a separate regression curve for those above. If the intervention is beneficial, there should be a discontinuity at the cut-off score, with the limit of the regression curve from the left higher than the limit of the curve from the right. This is illustrated in the following scatterplot that demonstrates the unsurprising insight that assigning students to remedial mathematics increases the total number of credits that they earn, including those credits that do not count toward the degree.

image 1

However, the discontinuity is reversed on the three outcomes that are meaningful: completion of a college credit bearing course in mathematics (specifically, College Algebra, MAC 1105, required for the Associate’s Degree), completion of the Associate’s Degree, and transfer to a four-year undergraduate program. The scale on the left represents the fraction of students who succeed. Each circle represents the mean value for all students with a given CPT score.

One of the problems with RDD is that the statistical power is very low. It requires a large sample size to find statistically significant results. Calcagno and Long used the records of 100,000 Florida students who enrolled in an Associate’s Degree program and took the CPT from fall 1997 through fall 2000. They tracked these students through spring 2006. They were able to adjust their analysis to account for non-compliance with the results of the placement test, which is relatively uncommon in this system, and to investigate the effect of students who took the CPT more than once.

It is worth noting that the negative effects of remediation are small: 1.4 percentage points on completion of College Algebra, 0.6 percentage points for completion of an Associate’s Degree, and 0.1 percentage points for transfer to a four-year program. In none of these cases is the difference from zero statistically significant. Nevertheless, the fact that the remediation is not demonstrably beneficial and likely is harmful should be deeply disturbing.

In an earlier column, The Problem of Persistence (January, 2010), I discussed the issue of students who arrive in college unprepared for college-level mathematics. I am encouraged by the work that is being done by organizations such as the American Mathematical Association of Two-Year College (AMATYC) and the Carnegie Foundation for the Advancement of Teaching to reconceive what mathematics these students need, to create new sequences for them such as Statway, and to assess the effectiveness of these new trajectories. The Calcagno and Long study is one more sign of the importance of this work and the need to assess the effectiveness of what we are doing.

---

[1] Calcagno, J. C. and B. T. Long. 2008. The Impact of postsecondary remediation using a regression discontinuity approach: Addressing endogenous sorting and noncompliance. National Bureau of Economic Research working paper no. 14194. http://www.nber.org/papers/w14194