Is US NEWS responsible for selling bad data?

Claremont-McKenna was among the first to admit to submitting bad data.

Thanks to data made readily available by the federal government and managers of the Common Data Set (CDS)—the College Board, Petersons, and US News—college search and evaluation have become much more numbers driven than in the past.  

An entire cottage industry of websites, college guides and software products has sprung up in the wake of freely available information used to categorize, rank, or otherwise describe and market postsecondary institutions both here and abroad.

And it doesn’t take too much effort to discover key metrics such as size of freshman class, percent of in-state students, average standardized test scores, freshman retention and/or graduation rates, or relative generosity when it comes to financial aid.  

Until fairly recently, the public believed the reams of data collected, compiled, and parsed by publications profiting from the CDS was accurate and appropriately vetted prior to being sold to college-bound students and those who advise them.

In fact, the assumption of data accuracy is part of a larger fantasy that suggests prestigious colleges and universities pay little attention to and don’t care a whit about rankings.

The truth is that they do care and are willing to falsify or manipulate data in schemes designed to improve rankings and create an image not particularly grounded in reality.

In 2012, Claremont McKenna College admitted that it submitted false admissions statistics that amounted to “modest boosting” of SAT scores, which made the school appear more competitive.  This scandal was followed by a similar revelation at Bucknell University, where officials confessed to misreporting SAT and ACT averages from 2006 through 2012.

Yet despite these embarrassments, major players in CDS data collection and sales stonewalled recommendations for improving the accuracy and reliability of information provided to the public, including

• Public dissemination of a comprehensive guide with clear definitions for each data point
• Regular opportunities for college-based IT staff training (webinars, workshops etc.)
• Easily obtainable technical assistance for staff with reporting responsibility
• Assignment of administrative responsibility for certifying data accuracy
• Random data audits conducted and funded the CDS
• Clearly stated penalties for inaccurate or deliberately falsified data

But there were no immediate changes at the Common Data Set, as even the most minimal quality assurance procedures failed to be implemented by organizations making large sums of money from the statistics provided by colleges and universities.

And so, more irregularities came to light.  Emory University, followed by George Washington University, York College of Pennsylvania, Dominican University of California, and the University of Mary Hardin-Baylor all admitted to providing fabricated information via the Common Data Set. 

Most recently, Flagler College was forced to announce that one of its senior officials altered admissions statistics for freshmen admitted for fall 2010 through fall 2013.

Eventually, US News did act on one suggestion and began imposing punishment on colleges caught falsifying data.   

GW, York College of Pennsylvania, and the University of Mary Hardin-Baylor were moved to “unranked” category in the US News guide, for submitting incorrect data that may/may not have affected their individual rankings.

When pressed about the failure to oversee a process resulting in significant profits for US News, Robert Morse, who heads the rankings, declined to take any responsibility, preferring to suggest that if a college is willing to lie to the federal government about statistics, they will lie to anyone—even US News.

“It boils down to this.  Usnews [sp] has far less power and resources than you think we have or should be able to exercise in our role,” said Morse in an email response to suggestions for improving the quality of the data product US News sells.  “We think schools have the moral and ethical responsibility to be honest in their reporting… if a school lies to all parties including the govt [sp] there is little that can be done…”

In other words, quality assurance is expensive and US News is not willing to spend the minimal amount of money it would take to ensure its product is sound.   

Morse doesn’t speak for the other huge corporate players in the Common Data Set, but given what the College Board alone makes on the deal, it seems reasonable to expect some effort on their part to ensure basic accuracy in reporting.

For the record, a survey of college admissions directors conducted by Inside Higher Ed suggests there may be more data fabrication than anyone is willing to publically admit. Asked if their institutions ever submitted false admissions data one percent of public and two percent of private admissions directors said yes.  Well over 90 percent of admissions directors believe that “others” do it.

Asked if they believed rankings producers have “reliable systems” in place to prevent this kind of fraud, 93 percent said no.

And as long as definitions are subject to interpretation, technical assistance is crowd sourced, and no training is provided, the public will continue to pay for and receive faulty information.

Tomorrow, US News releases the 2015 edition of the US News Best Colleges rankings.  The US News guidebook to colleges is set to go on sale on newsstands September 23.  

Note that both the College Board and Petersons have already used the same data to publish their guides earlier in the summer.