The evidentiary standards for i3 have stirred much conversation. On those standards, I’ve mixed feelings. On the one hand, the i3 criteria have healthfully prompted would-be applicants to think much more seriously about evaluation than has been the norm. Just in the past few weeks, I’ve heard from a number of outfits that have suddenly gotten religion on this question. This has the potential to weed out so many of the cruddy fly-by-night operators in the space and to foster a culture of performance.

At the same time, if foundations wind up mimicking federal decisions, standards of evidence could become a roadblock for promising new providers; it’s not clear how consistently or how carefully the evidentiary standard will be applied; and I have worried visions of bloated evaluation shops padding their pockets at usurious mark-ups and then taking care to produce evaluations that show sufficient promise to keep their clients happy.

But, I want to set the “evaluation” question aside for a moment and focus on the broader question of transparency. Just last month, Microsoft’s Mary Cullinane and I published What’s Next? Educational Innovation and Philadelphia’s School of the Future (Harvard Education Press 2010). For me, less compelling was any single finding than the recognition that folks in our sector hardly ever publicly look under the hood of acclaimed enterprises.

Rigorous examination of Philly’s School of the Future made clear the granular impact of important design decisions. By making the School of the Future a traditional district school, its founders adopted a course that would impose staffing constraints, shape the student body, and leave it vulnerable to district politics and leadership turnover. Understanding the import of these stumbling blocks, for this effort and future ones, is less a question of test results than scrutinizing routines, culture, and organizational practice.

As typically practiced, however, there is a “heads I win, tails you lose” dynamic to transparency. In places where test results look good, allowing researchers to dig around and potentially muck up the happy tale offers no benefits. This can only alienate funders, parents, and potential clients. If test results are not impressive, even compelling evidence of savvy operations will be met with a shrug or disbelief. In short, there is little incentive for anyone to invite this kind of careful scrutiny.

It is no coincidence that we have few accounts which objectively peer behind the curtain at admired outfits like the KIPPs, TFAs, and Green Dots, or which allow us to see what they’re doing wrong and what they’re getting right. Rather, we get statistics, the occasional business school case study, and stylized self-published accounts.

A failure to bring fresh eyes to bear or support introspection is all too rare in a sector where reformers routinely leap from one bandwagon to the next without paying much heed to how or why their last silver bullet might have disappointed.

For transparency to deliver, we cannot shrink from the messy truth that even highly effective schools, systems, and providers are going to look a lot less pretty when we peek behind the scores. Unfortunately, we’ve trained ourselves to recoil from the notion that the emperor might have forgotten even a solitary sock. Instead, reporters and writers seek out “successful” or “high-flying” schools and then write sentimental accounts that lionize the educators in question while casually whitewashing away most of the practical challenges.

This tradition makes everything look easier than it really is, even as it encourages organizations to hide their real challenges for fear of turning off potential funders or advocates. One consequence is that many who have spent time around charter schooling or education’s “new sector” have had the disconcerting sensation of seeing organizations praised for their management acumen when, behind the scenes, those organizations are desperately relying upon baling wire and duct tape.

This state of affairs is a problem. More transparency is crucial if providers are going to learn and improve. Funders need to do a better job of making it safe for applicants and grantees to be clear about their challenges and their shortcomings. Advocates need to get away from these overblown narratives in which they imbue “high-flying” or “no excuses” schools with a raft of virtues and will brook no critiques or dissent. Researchers and evaluators need to recognize that, as important as is the question of whether new ventures deliver, equally important may be that of how or why they do so. And entrepreneurs and district officials need to embrace and promote a culture of transparency in which they invite and benefit from scrutiny.