My daughter, who begins the eighth grade this fall, recently informed me that many of her friends spent part of their summers preparing for the SAT either on their own or in courses. I told her that she was better off reading hard books, puzzling out the meaning of their complex sentences and looking up the words she didn’t know. I pointed her towards David Foster Wallace’s “Shipping Out” a gelastic account of his cruise on the SS Nadir, which includes words like sapropel, annular, deracinate, piacular and bosky.
Did I give my daughter good advice? How effective is studying for the SAT? How much do coaching services like Kaplan and Princeton Review improve scores? To answer these questions we have to look at what the College Board – designer of the SAT – says about the test, critically examine the claims of coaching services and look at the available research.
According to the College Board the SAT reveals how well students “use the skills and knowledge they have attained in and outside of the classroom.” They say that, “The best way for students to get ready for the SAT is by taking challenging courses and by reading and writing as much as possible.” These are the most important determinants of SAT scores.
How valuable is additional preparation? Everyone, including the College Board, thinks that some test-specific preparation is valuable. Knowledge of the kinds of questions asked, how guessing affects scores, and basic test taking strategies is probably useful. The College Board makes online preparation materials available for free and publishes a reasonably priced book with even more materials.
How effective are paid (and frequently expensive) coaching services? Let’s begin by examining their explicit claims. Coaching services often present comparisons of student scores before and after coaching. These data look impressive but testing the effectiveness of a treatment (test preparation) on an outcome (test scores) requires a comparison to a specific alternative treatment. In medical trials, for example, one cannot give a drug to a group of patients and see if they get better. One has to give the drug to some patients (treatment group) and not to other patients (control group) and compare the improvement in the two groups. Coaching services’ data are untrustworthy because they lack control group comparisons.
Coaching firms make implicit claims of effectiveness by guaranteeing improvement. Students whose scores don’t improve can retake the course, and if they still don’t improve can get their money back. This guarantee implies a high level of confidence in the coaching product. But that implication is misleading. The reason has to do with what statisticians call measurement error.
No measurement is absolutely precise but some are more precise than others. For example, at a track meet, hand held stopwatches are accurate to tenths of a second, and electronic photo finish systems are accurate to hundredths of thousandths of a second. Electronic systems are more accurate because they have smaller measurement errors, but those errors are still present.
SAT scores are a measurement so they have measurement error, and students’ scores will change from test to test just by chance. In fact, about 68% of students will see positive or negative changes of zero to thirty points and 27% will see positive or negative changes of thirty to sixty points. One consequence of measurement error is that nearly half of the people retaking a test will improve their scores just by chance and nearly 75% will improve at least once in two tries. In other words, if coaching services had no effect whatsoever, only about 25% of their clients would qualify for the refund.
Clearly, the implicit and explicit claims of coaching companies are less impressive than they seem. What does impartial research say? The best studies show average gains of about five to ten points in verbal and ten to twenty points in math. These differences are considerably smaller than the effects of measurement error. Coaching has little demonstrable impact on scores.
So I’ll stick to the advice I gave my daughter: read challenging books; watch great movies; write blogs, journals, poems and stories; go to plays and concerts. More importantly, reflect critically on all of these things. These activities will make you far more interesting to others – including colleges – than grinding away at test preparation. They might even improve your scores.