For the past couple of years, the college board has had to defend the legitimacy of the SAT. With more colleges choosing to go test-optional, accusations and statistical evidence of bias against students of color, and competition from the ACT, the college board recently made the decision to convert the test to computerized adaptive testing. A computerized adaptive test harnesses the power of machine learning to more accurately and quickly assess the ability of a student. However, is this newest evolution actually a solution to the problems the SAT is facing? Let’s delve more deeply into the pros and cons of the SAT and CAT testing to understand the implications of this decision and its impact on the students who will be taking it.
The SAT is a controversial topic that brings out a lot of emotion from those defending it and condemning it. Supporters say that the SAT is a helpful standardization tool for students who come from a wide variety of high school experiences. They claim it evaluates students on an even playing field compared to GPA. They also claim that the SAT is an accurate assessment of future college success and preparation for this exam will set up students to succeed in college.
However, detractors of the SAT will say that in practice the SAT has a racial and socioeconomic bias and is not an accurate indicator of college success. According to a study conducted by PACE the organization for Policy Analysis for California Education, a high school GPA is a better predictor of college success and is less likely to discriminate against minority groups or socioeconomic class. These findings were a factor in the University of California’s decision to not consider any ACT or SAT scores for college applicants. There have also been studies done on the racial divide on SAT scores, with findings that of those scoring between 750 to 800, 60 percent are Asian and 33 percent are white compared to 5 percent Latino and 2 percent black. Socioeconomics is also at play. A study done at the University of Pennsylvania found that wealthy students earn higher SAT scores compared to their low-income peers.
With all these problems, how will the transition to computerized adaptive testing change the landscape of college admission testing? To answer this question, it is important to look at how computerized adaptive testing works.
Computerized Adaptive Testing, or CAT, is a form of computer-based testing that is adaptive to each test taker’s ability level. Every time an individual answers an item the computer re-estimates the ability of the individual based on all the previous answers and the difficulty of those questions. The computer then selects the next item so that the individual has a fifty percent chance of answering it correctly. As a result, the individual should be answering questions that should not be too easy or too hard and with each item, the computer’s estimate of the individual’s ability becomes more precise.
There are five technical components to creating a CAT test. These are:
- Calibrated item pool
- The starting point or entry level
- Item selecting algorithm
- Scoring Procedure
- Termination criterion
For the calibrated item pool, items must be created for the exam to choose from using a psychometric model. The starting point or entry level for the CAT has to be determined for the test as the first item administered is given with no information provided to the algorithm on the individual’s ability. Next, the item selection algorithm must be created to pick the next item of the test based on the information collected previously about the individual’s abilities. This is usually determined from the conditional variance and pseudo guessing parameter. The scoring procedure of the exam is that if a student correctly answers a question the computer estimates their ability to be higher. Finally, for the termination criterion, once the individual’s standard error of measurement falls below a certain specified value the testing ends.
Though CAT testing has been praised for being efficient in confirming the skill level of an individual however according to a study in the National Library of Medicine many test takers feel discouraged after taking a CAT exam. Since the test starts out the exam with medium difficult questions, some scientists have claimed that this is too high a threshold for test takers to keep up their motivation.
Another disadvantage is those test takers cannot return to items that they have already completed. The removal of this might also contribute to a test taker’s feelings of discouragement. It could also mean that the test taker answers the first couple of questions wrong out of nervousness which then affects the items that are to be selected.
Lastly, the study points out that item exposure rate must be carefully controlled, as it may be that certain question types are given more than others, therefore affecting the overall content that is administered.
According to the college board, the most apparent changes in the new CAT SAT exam compared to the traditional format will be that the exam will be shorter (two hours instead of three hours) and conducted completely on a computer versus the more traditional paper test.
each test section. (Reading and Writing, Math) is divided into two parts called modules. Students answer a set of questions in the first module before moving on to the next. The questions that students are given in the second module depend on how they performed in the first module. Students will also have their calculator for the entire exam and reading passages will be much shorter. This new format of the exam has already been tested on a small group of students in Nov of 2021. The feedback from that test, as reported by the College Board, has been positive. Students and proctors have been happy with the shorter length of the test as well as the content of the questions. There has been no research done on the racial or socio-economic equity impact of the new format.
The college board says using the CAT format will lead to more precise measurement, tailored questions to meet students’ needs, shorter tests, and more secure testing. This new testing format will be released in 2023, but in the meantime, this big change in the SAT format begs the question of whether the new format will improve or aggravate the issues that the SAT already faces. Students, teachers, parents, and college admissions will have to wait and see.