As I quoted last week, consumers don’t buy drills, they buy holes.
Likewise, educators don’t buy edtech software, they buy improvements in student performance.
That brings us to the Software and Information Industry Association’s (SIIA) CODiE award. (The name is derived from computer “code"; “i” is the symbol for “information.”) Does winning one tell educators what they need to know about edtech products? Can a prospective buyer rely on the award as an objective indicator that their purchase will improve student performance?
The short answer is “no.”
That is a problem for the whole school improvement industry.
(Download as a podcast here.)
How do consumers determine quality in a sea of offerings? They test drive, listen to buyers - and look for symbols. If a product is rated highly by Consumer Reports, it has credibility. If it wins an award from peers in its industry group, competitors must think it’s pretty good.
SIIA says that its CODiE is “the only-peer reviewed honor” in k-12 edtech products. Firms nominate themselves for various product categories and pay an entry fee of a few hundred dollars for each category. Nominations for the 2008 awards are open until December 3.
The Process
The first phase of the award competition is an informal ranking by representatives from the trade press, technology writers, analysts, consultants and subject-matter experts, etc. who serve as volunteer judges. Each product is reviewed by at least two judges; each judge reviews 10-15 products. Each judge provides their top five choices in each category they rank. Their votes determine the competition’s finalists.
In the second phase, SIIA member firms’ vote on the finalists.
In the third, member firms attending the association’s annual conference vote.
The informal review constitutes 50% of the total points a product may win, the sum of second and third phase voting counts for the rest. Products with the highest total score win their categories.
How the Award is Judged, Used and Perceived
(edbizbuzz.com readers may be interested in listening to my five-part podcast series “Educators as Informed Consumers” starting here. Each part runs under seven minutes.)
Finalists and winners have the right to display the CODiE label in association with their product and firm advertising.
If you are an educator reading an ad in a trade publication, walking past a booth in the vendors’ hall at a education conference, or leafing through a marketing brochure, and see the CODiE symbol associated with a product, you might be inclined to view it as evidence of efficacy. Look and feel are nice, but the taxpayer expects your “buy” decision to be about what works. So it would be reasonable to think that’s what SIIA and the firm in question are trying to tell you.
A CODiE probably does correlate with a provider’s interest in quality, broadly defined. Preparing for the competition gives companies some motivation to develop a good product. The first phase of the process involves feedback that helps refine programs. Given how judges seem to want to review products without outside assistance, the award is probably a good indicator of usability.
But if the CODiE is meant to denote the “quality” of its winners’ products; we have to ask how SIIA defines the term. What I find absolutely unfathomable as an outsider, that would certainly give me pause if I were an entrant, and floor me if I were a buyer of products with the label - is a complete lack of public information on the criteria that judges apply.
The nomination form itself suggests that reliable evidence that the product improves student academic performance is not relevant. Internal or third-party evaluation of effectiveness – scientifically based research, if you will - is not part of the competition. The CODiE reflects a bygone, input-oriented view of public schooling and SIIA does not employ the term “peer review” as it is used generally understood by professional educators.
If CODiEs were something companies kept to themselves as a means of internal professional recognition, that would be the end of it. The problem here is that an award that with no basis in evaluation is promoted by SIIA as a means for providers to increase sales in a market that views the “quality” of products as an outcome - a demonstrated contribution to student performance.
I doubt that any firm with a CODiE will correct any misimpression the educator reading the ad, walking past the booth, or leafing through the brochure might have on this point. No winner is going to append “this award says nothing about value-added to test scores” to the CODiE logo.
What is to be done?
“Buyer beware” is not the perception of industry quality we want to foster. Yet, to the extent that SIIA allows this kind of CODiE to continue, it can’t avoid complicity in misleading buyers - or allowing buyers to think something that isn’t correct if that’s what they want to think (choose your own semantics). However you describe it, this approach only contributes to the widespread perception that all the private sector cares about in k-12 education is profit.
That perception is a political problem for the whole industry, one that contributed mightily to the near-gutting of NCLB I, and one we have maybe two years to correct before we get back on that awful track to reauthorization.
• It’s time for SIIA to start making objective demonstrations of effectiveness part of the CODIE nomination process, especially in phase one - or drop the award.
• It’s time for providers whose products and services are based on evaluation to either move their trade groups in a direction where results matter, or to form one that places evaluation at the forefront of industry values.
• It’s time for the buyers’ associations – that is educators (teachers, principals, administrators, superintendents) to start 1) insisting that purchases be grounded in evidence of effectiveness based on program evaluation, and 2) recognizing providers that build evaluation into their programs with awards and, more important, contracts.
Evaluation science is far from perfect. It is helpful. It is getting better. And something beats nothing, which is where too many providers are today. We need to start expecting that providers - especially the ones with the brands we’ve always trusted to imply quality - are incorporating an advancing state of the art, instead of arguing about its faults, or ignoring it altogether. Awards are symbolic of industry values, and the school improvement industry’s values must include program evaluation.
