King Clarifies Stance on ACT and College Admissions

Council on Postsecondary Education President Bob King recently offered his opinions on college admissions saying,

Historically, parents are often directed to focus on graduation rates and average GPAs as evidence of how their high school is performing. Our reports allow parents and educators to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam — now required of all Kentucky students.

That drew a response from KSN&C's Skip Kifer.

Bob King, president of Kentucky's Council on Postsecondary Education applies the council's arbitrary standard of using a single test score to determine whether a student is ready for regular course work in Kentucky's public universities.

He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above-average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's readiness for higher education.

Today, King clairifed his stance in the Herald-Leader.

...Please allow me to clarify that Kentucky's colleges and universities do not rely exclusively on the ACT to make college admission or placement judgments, nor does the Council on Postsecondary Education encourage such determinations.

A student's entire record, including GPA, extracurricular activities, and other placement exams form a portfolio that allows campuses to make informed decisions on admission and placement.

The ACT serves as an important element in this consideration, but more importantly, it serves as an alarm bell in the student's secondary experience about preparation for life after high school.

This might have been a good place to stop. It acknowledges Kifer's concerns and clarifies King's stance. But King then makes allusions to "certain thresholds" in the ACT which serve to warn us if a student is not on track.

It warns of the need to take a deeper look at a student's college readiness if the scores fall below certain thresholds, but it is not the sole determinant when placing students in developmental courses... Far from arbitrary cutoff scores, there is a great deal of data from tens of millions of ACT score results upon which policy makers in Kentucky rely to set the scores used to indicate college readiness in key entry-level courses.

If the thresholds King has in mind are, in fact, a reference to ACT's benchmarks, which are inappropriately modeled, one wonders if King's effort to lay the issue to rest might draw yet another response from Kifer.

We'll see.

Famous Last Words

Amis says ACT inquiry in Perry a ‘dead issue’

This from the Hazard Herald:

An investigation into compromised test scores in Perry County is no longer an issue and districts officials have moved on, Superintendent John Paul Amis told board members during a meeting of the Perry County Board of Education last week.

In 2010, ACT, based in Iowa, launched an investigation into tests given at Perry County Central and Buckhorn High School, noting that information found during the inquiry led investigators to believe that the resulting tests were not solely the work of the students. Several scores were canceled and students were given the opportunity to retake the exam.

ACT took little action following their investigation, but did require the district to submit remediation reports and strengthen testing protocols.

Superintendent Amis took a moment during last week’s meeting to chide local media outlets for their recent coverage of ACT’s investigation, saying the now concluded inquiry is a “dead issue.”

Details of the ACT’s findings were included in two news stories in Hazard last week, first from the Hazard Herald and then from WYMT-TV after the Herald made an an open records request to the Kentucky Department of Education seeking documents related to the investigation.

According to those documents, ACT investigators found among other things high erasure counts indicating altered test responses and were told by at least one student that adult proctors had given answers to students during the test.

Amis said no students have told local officials of any impropriety, adding that there was no reason to be reporting on the story as he considers it over with. He said the investigation concluded two months ago, and there were no Perry County Schools employees implicated in the final report. He also noted that both Perry Central and Buckhorn will continue giving the ACT locally.

“It’s over,” he said. “It’s a dead issue. The only people that are keeping that issue alive is the media.”

Amis said he has never had a parent call him upset with the district about the investigation. He said there has been some anger toward ACT over their handling of the investigation, and now with the media.

“I had people today tell me that they’ll never buy another Hazard Herald or watch WYMT again because you just won’t let this issue go,” he continued. “The media and some of our school rivals are the only people that’s even mentioning this thing.”

Amis also criticized ACT for not sharing information with the district about who may have been responsible for manipulating the tests, and again noted that the final investigative report made no note of specific individuals either.

Board member Jerry Stacy said he has received calls from people about the investigation, and finds some of the allegations “troubling.”

“I was going through this today, and I couldn’t believe some of these allegations,” Stacy said during Thursday’s meeting. “I think these allegations are damning.”

Stacy said he was curious as to why the information was forwarded to other state agencies such as the attorney general’s office.

“That’s a dead issue,” Amis replied, saying that he thought ACT had a hidden agenda and wanted to hurt the Perry County District’s image.

“All ACT is trying to do ... is just to give the district a black eye. That’s all they’ve done,” he continued.

“We have done everything that they have asked us to do, and we’re going to continue to give the ACT at both of our high schools, and it’s over as far as I’m concerned,” he added.

Amis said despite charges of testing manipulation, the majority of the students scored the same or higher on the test during the make-up exam.

“We had 116 students that retook that ACT, and over 60 percent of them made the same score or a higher score than what they made the first time, but they won’t report that,” Amis said. "Nobody reports that.”

The final report issued by ACT on November 23 notes that 59 students whose scores for the 2010 ACT were canceled actually saw an average decrease in their new score by one point. But Amis said ACT cherry picked those students in an effort to ensure a negative reflection upon the school district. He said ACT’s focus on only 59 scores didn’t tell the whole story.

“They could take the 59 and make it appear like the scores went down,” he said.

“You actually think they’ve (ACT) got some kind of hidden agenda?” Stacy later asked.

“Obviously,” Amis replied.

Scott Gomer, media relations director with ACT, said the organization had no hidden agenda in its investigation, but did note that ACT’s part in the matter has concluded and offered no further comment.

According to an email from Lisa Gross with the Kentucky Department of Education (KDE) in December, the matter was referred to the Education Professional Standards Board (EPSB) for review, as well as the attorney general’s office and the U.S. Dept. of Education’s Office of Inspector General.

Gross said KDE has had no word that either of those agencies have finalized their own investigations.

“Our legal counsel remains in contact with all of the agencies mentioned, but we’re not aware that any investigations outside of ACT’s have been completed,” she wrote in an email on Monday.

The Education Professional Standards Board did receive a referral from KDE, noted Alicia Sneed, director of legal services for EPSB, but thus far the agency has taken no action.

She did note that were the EPSB to take action on any matter, it would be leveled at individuals and not at the district as a whole. The EPSB handles the issuance of certification for all teachers and administrators in Kentucky.

Shelley Johnson with the Kentucky Attorney General’s Office said her office has not yet received a referral from KDE about the matter, while a message left with the U.S. Dept. of Education’s Office of Inspector General was not returned.

Grady Varney, chairman of the Hazard Independent board of education refuted an editorial published in the January 19 edition of the Hazard Herald which includes a comment Varney made during the January 13 regular meeting. Following a discussion about the hiring of a new ACT coach at Hazard High School, Varney quipped, “We will not have any extra money for erasers or anything like that.”

ACT: Not the Only Measure of College Readiness

Council on Postsecondary Education honcho Bob King recently argued in the Herald-Leader that CPE's High School Feedback Report allows the state "to look more deeply into actual performance measured by an external, unbiased resource — the ACT exam."

King suggested that the ACT, by itself, was superior to predictions of college-readiness derived from combinations of data. Discounting graduation rates and average GPA in favor of a single test prompted our resident testing expert to retort.

NOTE: H-L seemed to struggle editing Skip's piece, so here's the unadulterated article the paper titled:

ACT not key measure of college readiness

By Skip Kifer

If I were to assert that a player who cannot make 56% of his free throws is not "ready" for the NBA, a fan would point out that there is much more to basketball than shooting free throws. An astute fan with a historic prospective would point out that Wilt Chamberlain, Shaquille O'Neal and a bevy of other current players would not be "ready" using that arbitrary standard. One facet of basketball does not make or break a player's "readiness."

Bob King, president of Kentucky's Council on Postsecondary Education, in a recent op-ed piece applies the council's arbitrary standard of using a single test score to determine whether a student is "ready" for regular course work in Kentucky's public universities. He implies a test score is a better predictor of grades in college than is a high school record. He then presents results from one high school that lump higher performing students (those with above average high school records) with lower performing ones in a misguided approach to justify his position. A test score, however, does not make or break a student's "readiness" for higher education.

Decades of research indicate: performance in academic courses in high school is the single best predictor of success in higher education; a combination of the high school record and test scores predict better than the high school record alone; and, how good the prediction is and how the components are combined vary depending on the institution. Although there is a general pattern of the primacy of the high school record, there is no one-fits-all model to predict grades in different courses or different institutions.

In addition to the thoroughly suspect notion of labeling a test score readiness, the council's use of a single score for placement purposes violates standards for the proper use of tests. Those standards include the following:

In educational settings, a decision or characterization that will have major impact on a student should not be made on the basis of a single test score. Other relevant information should be taken into account if it will enhance the overall validity of the decision.

In addition:

When test scores are intended to be used as part of the process for making decisions for educational placement, promotion, or implementation of prescribed educational plans, empirical evidence documenting the relationship among particular test scores, the instructional programs, and desire student outcomes should be provided. When adequate empirical evidence is not available, users should be cautioned to weight test results accordingly in light of other relevant information about the student.

Apparently, the council determines readiness by doing statistical analyses of ACT scores and grades in first year courses without regard to institution. A certain ACT score produces a 50/50 chance of getting certain grades, say C, or better. I could find no information about this or other investigations done by the council. And, although it is possible to present information about how good a model is, I could not find any information of that kind either.

To give a sense of the power of statistical models to predict first year grades I report analyses conducted years ago on University of Kentucky student samples. The question was whether results of KIRIS, the first commonwealth assessment related to school reform, could be used for admission and placement in a university.

If a model exactly predicts grades one can say that the model accounts for 100 % of what could be known. If a model cannot at all predict grades, one can say that 0% is accounted for. One way, then, to talk about the power of a statistical model is determine what percent the model predicts.
The table below gives those percents for different courses at UK for three different statistical models: High school record only, High School record + ACT scores and High School record + KIRIS scores.

The first thing to recognize is that the models are not particularly powerful. They rarely account for 25% of what could be known leaving 75% to be explained. That 75% may be differences in students' study habits, class attendance, interests, any of a thousand other variables or simply things not explained statistically.

The pattern of results, however, is clear. Adding an ACT score to a model containing GPA makes the prediction better but not greatly so. The same is true for KIRIS scores, too. Incidentally, that was without including the KIRIS writing sample.

These are not unusual results. They point, obviously, to gathering more information about a student before making a placement decision. Here is what ACT says:

ACT offers a variety of tools to ensure postsecondary students are quickly and accurately placed in courses appropriate to their skill levels. Assessment tools from ACT offer a highly accurate and cost-effective basis for course placement. By combining students' test scores with information about their high school coursework and their needs, interests, and goals, advisors and faculty members can make placement recommendations with a high degree of validity.

To that, I would add, for obvious reasons, it is desirable for an educational agency to use tests in exemplary ways.

The op-ed piece goes on to exhort parents to ask right questions, asks an undefined "we" to fear international test results, says admissions offices should align themselves with the council's readiness standards, and the still undefined "we" to serve teachers more effectively. Such exhortations would be more convincing if, in the first instance, the council could propose placement procedures based on a robust notion of "readiness" that in addition did not violate test score use standards.

Results of ACT Investigation in Perry Co Point to Altered Tests

Students tell ACT Investigators

Proctor Gave Answers During the Test

Supt searches for Explanations

This from the Hazard Herald:

Altered exam responses, multiple handwriting sources and statements from students were all given as evidence by a national testing organization to cancel test scores for several students at two Perry County high schools during an investigation in 2010, according to documents obtained in an open records request by the Hazard Herald.

The findings of ACT’s investigation into allegedly compromised test scores at Perry County Central and Buckhorn High School were noted in letters from ACT to Superintendent John Paul Amis and the high schools’ principals dated November 23.

Though ACT has made no finding of fault by any particular local testing administrator, it does list the organization’s key findings following the investigation which concluded in November, most notably that ACT “found evidence indicating that answer sheets for a significant portion of the students tested at PCCHS were manipulated.” ...

Fifty-nine students from Perry Central took the ACT exam again in October after their initial scores were canceled. According to ACT, the average composite scores decreased by more than one point, and 59 percent of the scores decreased from the initial exam.

ACT investigators found similar aberrations on the 2009 PLAN and ACT exams from Perry Central...

Superintendent John Paul Amis tried to explain the situation to the Herald:

ACT says there were: a high number of erasures from incorrect to correct responses.

Amis says: "...while it wouldn't happen very often, a student taking the test could go back and review answers and find that they were bubbled in out of order or on the wrong page of the answer sheet and have to change them before the time allotted ran out."

ACT says there were: slash marks on answer sheets that forensic handwriting analysts deemed were the result of someone other than the student marking responses on the test.

Amis says: “I don’t think that’s anything abnormal.”

ACT says: an adult proctor provided students with answers during the test,”

Amis says: “I could not find any student that felt like teachers provided answers to them.”

The Kentucky Department of Education forwarded ACT’s findings to three government agencies for further review: the Kentucky Attorney General’s Office, the Education Professional Standards Board and the Office of Inspector General.