Is the ACT the Sole Criteria?

It all started when Council on Postsecondary Education President Bob King argued in the Herald-Leader that CPE's High School Feedback Report allows the state "to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam."

King suggested that the ACT, by itself, was superior to predictions of college-readiness derived from combinations of data. Is CPE discounting graduation rates and average GPA in favor of a single test?

That drew a response from KSN&C's Skip Kifer, demonstrating the weak relationship ACT musters, and suggesting that CPE should propose placement procedures based on a robust notion of "readiness" that includes more than just the ACT and that did not violate test score use standards. It appeared to Kifer that CPE arbitrarily uses that single test score to determine whether a student is ready for regular course work in Kentucky's public universities.

Too clarify, King stated that CPE does not rely exclusively on the ACT to make college admission or placement judgments, nor does the Council on Postsecondary Education encourage such determinations.

Perhaps King should have reviewed CPE's printed material before saying that.

In last Saturday's H-L, Kifer wrote,

I am puzzled by Bob King's response to my critique of the Council on Postsecondary Education's policy of declaring a student college ready on the basis of a single test score.

King, director of the council, says: "Please allow me to clarify that Kentucky's colleges and universities do not rely exclusively on the ACT to make college admission or placement judgments, nor does the Council on Postsecondary Education encourage such determinations."

Yet when I look on the council's Web site, it says:

"The Kentucky statewide public postsecondary placement policy in English and mathematics applies to any student entering a Kentucky public college or university. The policy is based on your ACT or SAT score and determines what type of English and math classes you will need to take when you enter college."

This is what I found in my search. Notice the date posted. That's the day Kifer's piece ran. Maybe I missed something, but I didn't catch any changes in the language from when I looked at the same material in February.

StatewidePlacementPolicy: Postsecondary Placement Policy does … Postsecondary Placement Policy, please … PLACEMENT POLICY IN http://cpe.ky.gov/nr/rdonlyres/73e9a7b3-84dc-4ec2-8f1b-6a99261b5fb4/0/statewideplacementpolicy.pdf
- 120KB - kdrummond - 3/5/2011 [View duplicates]

And I found this:

The statewide placement policy is applicable to any incoming student entering a Kentucky public postsecondary institution. ACT and SAT standards form the basis of the policy because Kentucky uses the ACT (or equivalent measures) for college admissions and placement decisions.

That language says ACT forms the basis for placement decisions but stops short of saying the ACT is the only determinant. That comes next.

Kentucky Statewide Placement Policy in English
• A student earning an ACT English sub-score of 18 or higher qualifies for placement in a credit-bearing writing course at any Kentucky public postsecondary institution.

Kentucky Statewide Placement Policy in Mathematics
Three levels of readiness are identified for placement in a credit-bearing mathematics course at any Kentucky public postsecondary institution:
• Level 1: A student earning an ACT mathematics sub-score of 19 or higher qualifies for placement in a credit-bearing mathematics course, but this course may not be a requirement for many college majors or lead to subsequent coursework in mathematics. Mathematics for liberal arts is an example of such a course.
• Level 2: A student earning an ACT mathematics sub-score of 22 or higher qualifies for placement in college algebra. College algebra (or placement in more advanced courses) is required for majors such as biology, business, economics, information systems, and technology. College algebra can lead to any major.
• Level 3: A student earning an ACT mathematics sub-score of 27 or higher qualifies for placement in calculus. Calculus is required for majors such as mathematics, physics, chemistry, computer science, engineering, biology, business, and technology.

Kentucky’s statewide public postsecondary placement policy is a guarantee of
placement in credit-bearing coursework to incoming students demonstrating
specified levels of competence.

King Clarifies Stance on ACT and College Admissions

Council on Postsecondary Education President Bob King recently offered his opinions on college admissions saying,

Historically, parents are often directed to focus on graduation rates and average GPAs as evidence of how their high school is performing. Our reports allow parents and educators to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam — now required of all Kentucky students.

That drew a response from KSN&C's Skip Kifer.

Bob King, president of Kentucky's Council on Postsecondary Education applies the council's arbitrary standard of using a single test score to determine whether a student is ready for regular course work in Kentucky's public universities.

He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above-average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's readiness for higher education.

Today, King clairifed his stance in the Herald-Leader.

...Please allow me to clarify that Kentucky's colleges and universities do not rely exclusively on the ACT to make college admission or placement judgments, nor does the Council on Postsecondary Education encourage such determinations.

A student's entire record, including GPA, extracurricular activities, and other placement exams form a portfolio that allows campuses to make informed decisions on admission and placement.

The ACT serves as an important element in this consideration, but more importantly, it serves as an alarm bell in the student's secondary experience about preparation for life after high school.

This might have been a good place to stop. It acknowledges Kifer's concerns and clarifies King's stance. But King then makes allusions to "certain thresholds" in the ACT which serve to warn us if a student is not on track.

It warns of the need to take a deeper look at a student's college readiness if the scores fall below certain thresholds, but it is not the sole determinant when placing students in developmental courses... Far from arbitrary cutoff scores, there is a great deal of data from tens of millions of ACT score results upon which policy makers in Kentucky rely to set the scores used to indicate college readiness in key entry-level courses.

If the thresholds King has in mind are, in fact, a reference to ACT's benchmarks, which are inappropriately modeled, one wonders if King's effort to lay the issue to rest might draw yet another response from Kifer.

We'll see.

ACT: Not the Only Measure of College Readiness

Council on Postsecondary Education honcho Bob King recently argued in the Herald-Leader that CPE's High School Feedback Report allows the state "to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam."

King suggested that the ACT, by itself, was superior to predictions of college-readiness derived from combinations of data. Discounting graduation rates and average GPA in favor of a single test prompted our resident testing expert to retort.

NOTE: H-L seemed to struggle editing Skip's piece, so here's the unadulterated article the paper titled:

ACT not key measure of college readiness

By Skip Kifer

If I were to assert that a player who cannot make 56% of his free throws is not "ready" for the NBA, a fan would point out that there is much more to basketball than shooting free throws. An astute fan with a historic prospective would point out that Wilt Chamberlain, Shaquille O'Neal and a bevy of other current players would not be "ready" using that arbitrary standard. One facet of basketball does not make or break a player's "readiness."

Bob King, president of Kentucky's Council on Postsecondary Education, in a recent op-ed piece applies the council's arbitrary standard of using a single test score to determine whether a student is "ready" for regular course work in Kentucky's public universities. He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's "readiness" for higher education.

Decades of research indicate: performance in academic courses in high school is the single best predictor of success in higher education; a combination of the high school record and test scores predict better than the high school record alone; and, how good the prediction is and how the components are combined vary depending on the institution. Although there is a general pattern of the primacy of the high school record, there is no one-fits-all model to predict grades in different courses or different institutions.

In addition to the thoroughly suspect notion of labeling a test score readiness, the council's use of a single score for placement purposes violates standards for the proper use of tests. Those standards include the following:

In educational settings, a decision or characterization that will have major impact on a student should not be made on the basis of a single test score. Other relevant information should be taken into account if it will enhance the overall validity of the decision.

In addition:

When test scores are intended to be used as part of the process for making decisions for educational placement, promotion, or implementation of prescribed educational plans, empirical evidence documenting the relationship among particular test scores, the instructional programs, and desire student outcomes should be provided. When adequate empirical evidence is not available, users should be cautioned to weight test results accordingly in light of other relevant information about the student.

Apparently, the council determines readiness by doing statistical analyses of ACT scores and grades in first year courses without regard to institution. A certain ACT score produces a 50/50 chance of getting certain grades, say C, or better. I could find no information about this or other investigations done by the council. And, although it is possible to present information about how good a model is, I could not find any information of that kind either.

To give a sense of the power of statistical models to predict first year grades I report analyses conducted years ago on University of Kentucky student samples. The question was whether results of KIRIS, the first commonwealth assessment related to school reform, could be used for admission and placement in a university.

If a model exactly predicts grades one can say that the model accounts for 100 % of what could be known. If a model cannot at all predict grades, one can say that 0% is accounted for. One way, then, to talk about the power of a statistical model is determine what percent the model predicts.
The table below gives those percents for different courses at UK for three different statistical models: High school record only, High School record + ACT scores and High School record + KIRIS scores.

The first thing to recognize is that the models are not particularly powerful. They rarely account for 25% of what could be known leaving 75% to be explained. That 75% may be differences in students' study habits, class attendance, interests, any of a thousand other variables or simply things not explained statistically.

The pattern of results, however, is clear. Adding an ACT score to a model containing GPA makes the prediction better but not greatly so. The same is true for KIRIS scores, too. Incidentally, that was without including the KIRIS writing sample.

These are not unusual results. They point, obviously, to gathering more information about a student before making a placement decision. Here is what ACT says:

ACT offers a variety of tools to ensure postsecondary students are quickly and accurately placed in courses appropriate to their skill levels. Assessment tools from ACT offer a highly accurate and cost-effective basis for course placement. By combining students' test scores with information about their high school coursework and their needs, interests, and goals, advisors and faculty members can make placement recommendations with a high degree of validity.

To that, I would add, for obvious reasons, it is desirable for an educational agency to use tests in exemplary ways.

The op-ed piece goes on to exhort parents to ask right questions, asks an undefined "we" to fear international test results, says admissions offices should align themselves with the council's readiness standards, and the still undefined "we" to serve teachers more effectively. Such exhortations would be more convincing if, in the first instance, the council could propose placement procedures based on a robust notion of "readiness" that in addition did not violate test score use standards.