The Little Firm That Could
The state's highest court orders the dismantling of your education system. In its place, your legislature says, there should be a new, cutting-edge student-assessment system--one in which students are evaluated on what they can do with what they know rather than on how well they can regurgitate facts or guess at multiple-choice questions.
This system, the lawmakers go on to say, should be a linchpin for the state's school-reform effort. It should provide results for every school and school district. And, vital to the whole effort, schools should be rewarded or penalized on the basis of how their students score on the new assessments.
And, oh yes, it all has to be in place by the spring of 1996.
So, you go out and find a big testing company that can handle the job, one that's been around for decades, right?
Wrong. Faced with that situation in 1991, Kentucky school officials chose Advanced Systems in Measurement & Evaluation Inc., a tiny New Hampshire firm that at the time had been in business for just seven years.
The $29.5 million, five-year contract was the largest of its kind in the nation's history. To win it, Advanced Systems beat out five of the biggest, most established firms in the field. They included the Educational TestingService, the giant Princeton, N.J.-based company that administers the "nation's report card'' and other national studies for the federal government, and CTB Macmillan/McGraw-Hill, a company that even now boasts revenues a dozen times larger than Advanced Systems'.
The small firm got the job, Kentucky school officials say, for one simple reason: Its proposal was the best.
"They put forward the most comprehensive, innovative proposal, and they put forward the proposal in the most innovative way possible,'' explains Edward Reidy, a deputy commissioner in the state education department.
A Small Start
At the time, Advanced Systems had already been active in a number of states. But the Kentucky contract gave the company a major boost and a high national profile.
Lots of states, like Kentucky, were experimenting with alternatives to traditional paper-and-pencil tests. But none had placed on those assessments the kinds of stakes that Kentucky had. And the rest of the country was watching.
Now, three years into the development of Kentucky's assessment system, the question remains: Can this little firm tackle the job?
To many people in the testing field, Advanced Systems is Richard K. Hill, the company's 54-year-old president.
"He is the equivalent in this field to Steven Jobs of Apple,'' says Grant Wiggins, the president of the Center on Learning Assessment and School Structure in Geneseo, N.Y., likening Hill to the co-founder of Apple Computer Inc.
In person, however, Hill seems the opposite of a hard-charging, flamboyant chief executive officer. He is relaxed in demeanor and careful not to oversell the benefits of performance assessments, which, most experts agree, have not yet proved to be as reliable as traditional kinds of tests.
The quintessential number cruncher, Hill excelled in mathematics and science in high school and played on the chess team. Even today, pens and an eyeglass case poke out of his shirt pocket, and he professes a love for statistics, his major field of study in college.
In truth, however, Hill did not found Advanced Systems alone. He was joined in that effort by Stuart R. Kahl, his tall, redheaded vice president.
The two men met while working together at RMC Research Corporation, a Portsmouth, N.H.-based outfit. Kahl and Hill handled contracts for statewide student-testing programs there, and, although they were on good terms with their employer, they eventually decided they would be better off working on their own.
"Testing is a messier business than education research is,'' Hill explains. "There are boxes all over the place and crises all of the time, and education researchers go at a more scholarly pace.''
"We had no intention of ever becoming a big company,'' he says.
In 1984, the pair set up Advanced Systems in a three-room suite in Portsmouth. The firm had a total of four employees--including its two founders--and small testing contracts in Connecticut and Maine.
As quickly as the firm had begun, however, it was preparing to go out of business. Lawmakers in Maine and Connecticut moved to abolish their traditional student-assessment programs and, with them, went Advanced Systems' bread-and-butter contracts.
With plans to dissolve the company already in place and not much else to do, Hill and Kahl made a bid to become the architects of Maine's new student-assessment program and another program being launched in Massachusetts.
"We figured you wouldn't give a major testing contract to a company of four people,'' Hill remembers.
Wrong again. Advanced Systems won both contracts. It has been the primary contractor for the two programs ever since.
Success 'by Accident'
Over the next several years, the company added contracts for student-testing programs in Vermont, California, Pennsylvania, and Rhode Island.
Along the way, it acquired a reputation for being flexible and innovative.
"I've been frequently surprised at their proposals and the new strategies they propose,'' says Horace Maxcy, the coordinator of Maine's assessment program. "It would've been easy to continue doing what we were doing.''
With that state, for example, the company developed an "annotated holistic method'' of scoring student-writing responses that allows scorers to work more quickly but also provides students with comments on their writing. Students are given an overall grade on the quality of work plus two or more notes on the strengths or weaknesses shown in their writing sample.
Later on, the company would also help design pioneering portfolio-assessment efforts in Vermont.
Hill says Advanced Systems earned its reputation almost by accident.
"We couldn't afford to hire anybody who knew anything about testing, so we hired people who were bright and committed, most of them teachers,'' he says, "and we wound up inadvertently developing a company that was very strong in those attributes.''
"When the testing industry changed all of a sudden, we were no less qualified to do the next kinds than we were for the old kind,'' he continues, "and we really did have a mindset that we could become a new organization at a moment's notice.''
By 1989, annual revenues for the company had reached $3.1 million. Inc. magazine had named the company one of the 500 fastest-growing privately held firms in America. And Advanced Systems was building a new headquarters on a wooded 25-acre spread in Dover, a scenic community not far from the southern New Hampshire coastline.
The Kentucky contract, Hill says, was "serendipity.'' A major contract to help design the writing portion of California's new student-testing program was stalled in a dispute between that state's governor and its schools superintendent.
The delay gave Advanced Systems and California school officials time to reflect on the changing nature of student assessments and the move to evaluate students' work, rather than their responses to questions.
"We realized it wasn't practical to think about administering the new kinds of tests in the same way we could the old ones,'' Hill says. "It just took too much time.''
What was needed was a way to conduct those evaluations as part of the regular work that students do in the course of their studies, not to treat them as add-ons.
Kentucky school officials had the same idea. At a conference on assessment, Hill heard one such administrator talk about the state's plans for a "seamless, transparent'' assessment system that would be designed, in essence, from scratch.
The notion attracted him, and Advanced Systems wrote a proposal to do the job. It was, in fact, the shortest of all the proposals submitted for the project.
But it was enough to impress the five outside consultants hired by the state to evaluate the bids. They unanimously recommended choosing Advanced Systems as the primary architect of the state's new assessment system.
A separate panel of 10 educators appointed by Thomas C. Boysen, the state education commissioner, concurred.
"As I recall, the rest of the industry was very surprised that a company like Advanced Systems--which was small and, from what we understood, undercapitalized and had little track record--was selected,'' says Michael Kean, the vice president for public and governmental affairs at CTB Macmillan/McGraw-Hill. "The business world is littered with little companies that have bitten off more than they can chew.''
Some Kentucky lawmakers were also leery of the choice, but consultants and state officials say the decision simply came down to the quality of the proposal.
"My guess,'' says Wiggins of the Center on Learning Assessment and School Structure, one of the five consultants who recommended Advanced Systems, "is that we're seeing the inevitable thing in business where the large established companies are less flexible, less imaginative, and less able to respond quickly to the demands of the marketplace.''
To some degree, the field has since changed. Most major testing companies are doing some work in performance assessments and other alternative forms of assessment, and increasing numbers of states are incorporating such methods.
But, at the time, Advanced Systems looked like the only game in town.
Tense Times in Kentucky
Given to 140,000 students in grades 4, 8, and 12, Kentucky's fledgling assessment program is a complicated, massive undertaking.
Eventually, state officials hope, the new system will be entirely performance-based. Whether that will mean having students build up portfolios of their best work or complete specific hands-on tasks, the goal is the same: Students will be evaluated on what they can do with what they know rather than on just what they know.
Students might be asked, for example, to create a self-portrait in clay or to prove a mathematical theory. One exercise used last year required high school seniors to simulate a meteor impact by dropping a rock into cornstarch. Students were then asked to vary the conditions under which the rock was dropped, measure the size of the "craters'' created, and to graph their data on a chart.
For now, however, the state, with Advanced Systems' help, is using a "transitional'' testing program consisting of three parts: a written test of open-ended and multiple-choice questions; portfolios, collected throughout the year, of students' best writing; and a small number of performance tasks, which do not yet count on schools' records.
The written tests include common questions, which all students answer, and other questions that vary from student to student. Varying the questions allows educators to get information on a broader range of subject matter.
And, as specified in the Kentucky Education Reform Act, the assessments must be linked in some way to a test such as the National Assessment of Educational Progress so educators can judge how Kentucky students stack up against the rest of the nation.
Students' writing portfolios are evaluated by teachers at their own schools. Scorers at Advanced Systems' New Hampshire offices periodically recheck the results for accuracy and reliability.
None of the assessments may be used to make decisions about individual students, although every student will learn of his or her scores. But, eventually, selected results will be used to assess how well schools are doing and to reward or punish them accordingly. Schools that show improvement will receive financial rewards, while those with declining scores could ultimately be declared schools "in crisis,'' and their employees could lose their jobs.
Depending on whom you ask in Kentucky, you are likely to get a different answer on how good a job Advanced Systems is doing there.
On the one hand, educators have complained that some of the test questions have been inappropriate for the age level of children being tested; that they are getting mixed signals on how to score the portfolio samples from Advanced Systems and school administrators; and that there have been logistical problems in administering the tests.
"I believe teachers felt frustrated, confused; stress levels are unbelievably high,'' says Marnell Moorman, the president of the Kentucky Education Association, which has not been among the most vocal supporters of the sweeping changes taking place in Kentucky schools.
Some conservative parents also criticize the assessments--and, by definition, the state's learning goals--for evaluating vague "values'' and, they say, undermining traditional family values. In response, Kentucky officials have agreed to gear test questions only to strictly academic areas.
Still others have more basic concerns about the system itself.
George K. Cunningham, a professor of educational psychology and counseling at the University of Louisville, says it is unfair to judge schools according to how much their test scores change.
Moreover, he says, there is a "dissonance'' between the broad education goals the state set for itself and the very specific questions that have appeared on the assessments.
"Some of those questions nobody in the state can answer correctly,'' he charges. "Basically, it's a very flawed system.''
Technical questions have also been raised about the reliability and validity of the testing measures.
But, on the other hand, teachers like Terry Williams, a middle school teacher in the Erlanger-Elsmere school district in northern Kentucky, say the assessments are already beginning to bring about real changes in teaching.
"If people are assessed in certain things, then you'll want to be prepared for it in the classroom,'' she says. "It's a step in the right direction.''
In fact, the performance-assessment portion of the test is so popular that state officials have decided to continue to include those tasks in the transitional tests even though they do not yet produce scores that are reliable enough to be "counted'' on the formal assessment.
Over all, says Lorraine M. McDonnell, a University of Southern California researcher who interviewed educators in 20 schools in the state last year, the problems and protests have been minor compared with those of other states, such as California, that are also moving to a more performance-based measure of educational progress.
"Given the magnitude of the task, I'd say they're getting pretty high marks,'' McDonnell says of Kentucky educators and officials.
But the complaints about the testing system have been loud enough for state officials to push back the deadline by which penalties will take effect. The state is also undertaking two independent evaluations of the new system--one by the legislature's Office of Education Accountability and another by the evaluation center at Western Michigan University and the Kentucky Institute for Education Research in Frankfurt.
"This is a very high-stakes environment,'' Reidy of the Kentucky education department says. "We believe the company has done good work, but we'd like some independent folks to look at it.''
From a public-relations standpoint, it also did not help that the company was from out of state and that Kentucky students scored so poorly on the test the first time around. In 1992, 90 percent of the students scored below the "proficient'' level in reading, math, science, and social studies. The following year, student performance improved in all four subjects.
It's difficult to determine how much of the criticism--and the praise--for the effort should go to Advanced Systems and how much should go to the state.
The basic elements of all of the test questions, for example, are developed by teachers in Kentucky and fine tuned and field-tested by the staff at Advanced Systems.
Nevertheless, Hill says the company shares the blame for some of the tensions surfacing now in Kentucky.
In May 1992, for example, a Federal Express truck carrying student assessments to New Hampshire caught fire in Pennsylvania and was destroyed. Advanced Systems, unaware of the incident, did not discover and report the loss until four weeks later.
Another brouhaha occurred this year after the company audited the scores teachers had given to students' writing portfolios.
Advanced Systems rechecked portfolios from 105 schools whose average scores had varied significantly from the norm for the state or from the previous year's scores. Scorers working for the firm downgraded the scores of 101 schools.
Technically speaking, the results were good news because they showed that only 7 percent to 8 percent of the schools in the state had not gotten the message on how to score portfolios, Hill says.
But teachers across the state were insulted.
"It was a monumental public-relations blunder,'' Hill admits. "We never gave the public or teachers any explanation on why the difference might have occurred other than that teachers had either cheated or they were stupid.''
"Usually, there are stages you go through with a new testing program, and, usually, there are fairly superficial responses to it,'' he adds. "This time, we did not anticipate the kinds of problems there were.''
The 'Ben and Jerry' of Testing
Despite all the attention it has received for its work in Kentucky, Advanced Systems remains very much a small company.
The organization now occupies its own 26,000-square-foot building amid a stretch of farms, housing developments, and wooded fields. The area is zoned to be an industrial park. But, for now, Advanced Systems is the only industry in sight.
The company's part-time scorers work out of the second floor of an office building in a nearby town. The firm also rents a warehouse that is now stacked almost to the ceiling with boxes of students' test papers and performance assessments.
In their enclave, Hill and Kahl have set themselves up as a sort of "Ben and Jerry'' of the testing world. Like the socially conscious ice-cream entrepreneurs, they treat their employees well. In the basement, there is a fitness room that is open to all employees, and a basketball hoop stands as a sentry in the parking lot. A vending machine in the top-floor lounge dispenses free sodas.
"One of the things I found out rather quickly was that teachers are not treated particularly well,'' Hill says. "These sorts of things were a very powerful message to the people coming to us from the classroom.''
In addition, Hill and Kahl have designed their company so that no individual owns more than half the stock.
Hill says the company has no plans to grow as much as it did in the early years. He would, however, like to contract out some aspects of the business so that Advanced Systems can do design and development work in more states.
The owners also have pledged never to sell the company to a large corporation. They are working now on a plan to divest themselves of the company so that whoever succeeds them will not be tempted to operate it solely from a profit-making perspective.
Today, the little company that started a decade ago with four employees and revenues in six-digit figures has 80 full-time workers and 200 part-time scorers. Revenues this year will amount to $12.3 million.
"I expect to retire well set,'' Hill says, "but wealthy only in