Software Maker’s ‘Guarantee’ of Student Performance Questioned

By Peter West — December 16, 1992 12 min read

In an apparently unprecedented move, the Computer Curriculum Corporation, one of the nation’s largest developers of software for integrated learning systems, announced last month that it could “guarantee’’ that students who use one of its products would meet performance goals in reading and mathematics.

In a series of informal press briefings, company officials officially unveiled the guarantee--which it offers to buyers of a multimedia product called “SuccessMaker: The Guaranteed Learning Advantage’'--terming it an important milestone in the use of computers in education.

The guarantee, however, has been greeted with skepticism by both competitors and industry observers who question whether it will benefit those students who some say need the type of repetitive and remedial lessons that typify such systems.

The company responds that its nearly 25 years of experience in computer-based assessment of student performance, bolstered by an extensive research and development effort to produce sophisticated software, allow it to predict--and, therefore, guarantee--student performance in specific subjects.

But in making such specific claims, the company raises important questions about the goals, methods, and promise of computer-assisted instruction.

Among the first to test the guarantee will be nine Baltimore public schools that are being managed by a private, for-profit company.

Goals and Credits

Spokesmen for Computer Curriculum Corporation claim that software built into the system will allow it to guarantee that the performance of individual students in mathematics and reading will improve in a specified period of time to levels selected by teachers and parents.

The firm will offer credits on instructional costs to districts for each student who fails to meet the goals, officials said.

Observers and competitors are predictably skeptical, arguing, among other things, that the guarantee is based on the system’s ability to bring students up to speed on locally devised, criterion-referenced tests and not the more widely used, norm-referenced standardized tests.

Lynda Ellis, the vice president of field marketing for Jostens Learning, the nation’s single largest vendor of integrated learning systems, said her firm does not offer a similar guarantee. Instead, she said, Jostens prefers to aim at helping students perform better on the norm-referenced examinations that often provide benchmarks for educational improvement.

She added that the C.C.C. guarantee does little to help those students who fail to measure up.

“I don’t think giving somebody their money back is going to help a student learn,’' she said. “To say, ‘If your child isn’t learning, you get your money back, but a year of their life is already gone,’ I don’t think that is a satisfactory goal.’'

A Competitive Strategy

Few observers, however, challenged a ãŸãŸã official’s observation that the market for integrated learning systems seems to be entering a new, and far more lucrative, era.

Nor did anyone interviewed disagree that the perception that the company could guarantee student performance would provide a valuable competitive edge in a suddenly booming sector of the education-technology market.

In an interview, company officials themselves painted a rosy picture of growing revenues as budgets for remedial education and programs for at-risk students--two target populations for manufacturers of integrated learning systems--appear to be increasing nationally.

“We’re looking at extraordinary growth through the end of the century,’' said Keith Schaefer, the firm’s chief executive officer and the president of the Technology Group of Paramount Communications, its parent company. “The real growth is just beginning.’'

Indeed, shortly before company officials unveiled the SuccessMaker guarantee, they held a press conference to announce a deal to supply the software to the private company that is operating public schools under contract in several cities.

But the announcement that Educational Alternatives Inc.--which has a $147 million contract to run nine public schools in Baltimore--had signed a contract to buy $4 million worth of the SuccessMaker product was simply a harbinger of coming growth in a larger “strategic alliance’’ to equip E.A.I. schools, Mr. Schaefer said.

“We are excited to be a partner with someone who’s projecting $1 billion in [annual revenues] three years out,’' he said.

Though he did not note that E.A.I. has yet to turn a profit on its venture, Mr. Schaefer said his firm has its eyes firmly on the financial horizon.

Measuring Success

Integrated learning systems, which have been available since at least the late 1960’s, are software-based instructional-delivery systems that provide students with “drill and practice’’ style instruction under the direction of an internal management system.

Typically, the management software monitors student performance using built-in tests and attempts to match the computerized coursework with the student’s demonstrated level of ability.

Generally, the management systems also have the capacity to provide teachers with frequent, even daily, updates of an individual student’s progress.

The Computer Curriculum Corporation says that the sophistication of the SuccessMaker management system--which is said to incorporate elements of artificial intelligence that allow it to gauge, track, and predict student performance--enables the firm to guarantee the product’s success.

But P. Kenneth Komoski, the executive director of the New York-based Educational Products Information Exchange Institute, a nonprofit clearinghouse that produced a voluminous study of integrated learning systems in 1990, was skeptical of the pedagogical values underlying the C.C.C. guarantee.

“In order [to guarantee student performance], both parties have got to agree on some very clear, objective measures of what that guarantee will be,’' he said. “Now this usually means a standardized test, [but] now ... you hear more and more people questioning the value of standardized tests.’'

Mr. Komoski said that it is possible to devise a local criterion-referenced test, but that such tests may not be the best way to measure student learning or the effectiveness of the computer-based approach.

“If they can come up with a criterion-referenced test that they believe is a valid and reliable measure of the curriculum outcomes, then I think there’s a little more educational justification for it,’' he added.

“If the school district is savvy,’' he added, “they will say we want only those lessons in the I.L.S. that are in our curriculum.’'

Nevertheless, Ronald F. Fortune, the president of the company, said it is throwing its reputation behind the new guarantee, and discounted competitors’ criticism of it.

“If other people can do this, why don’t they?’' he asked.

He also said that research is under way that would allow the firm to extend its guarantee to student performance on norm-referenced exams.

A Growing Market

Mr. Komoski acknowledged that integrated learning systems are suited for certain specific purposes.

“The systems work, if you’re willing to accept the measure of a standardized test, and kids do definitely seem to like working in this manner,’' Mr. Komoski said. “My overall frustration is that they could be doing so much more than they are doing. But for many youngsters, they are an improvement over what they’re getting in class every day.’'

Nevertheless, the E.P.I.E. Institute report took the I.L.S. industry to task. The document was deliberately entitled the “Integrated Instructional Systems Report,’' Mr. Komoski noted.

“We called it that because we didn’t feel they deserved the title ‘learning systems,’ '' he added. “The majority of the systems just lock you into their prepackaged instructional programming.’'

Despite such criticism, the systems continue to grow in popularity in the educational-computing market.

An annual report soon to be released by Quality Education Data, a Denver-based educational market-research firm, indicates that only 587, or 3 percent, of the nation’s more than 16,000 districts use such systems.

But a separate survey of 217 medium and large districts that the organization says is a more accurate gauge of long-term trends indicates that 45 percent of these districts own the systems--a 12 percent increase over last year, said Greg Schwartzkopf, a company spokesman.

Targeting Basic Skills

The popularity of the systems among larger, relatively affluent districts can be traced in part to their ease of use, observers said.

Frequently sold as “turnkey systems,’' the original integrated learning systems offered school districts, many of which were unprepared for the microcomputer revolution of the early 1980’s, a means to reassure anxious parents that their instructional program was on the cutting edge.

“It was a way to alleviate the anxiety of school boards,’' Mr. Komoski said.

He noted that school boards, which were familiar with textbooks, but rarely with computers, often wanted to buy a computerized approach that was similar enough to a book to make parents happy without confusing teachers and board members.

But even as stand-alone and networked computers began to make inroads into the school market in the 1980’s, pressure increased on school boards to improve the performance of students, particularly disadvantaged ones, on standardized tests.

“Historically, the [I.L.S.] market in schools was for basic skills, Chapter 1 programs,’' Mr. Komoski said.

The approach provided a “lucrative market’’ for programs that were heavy on the drill-and-practice approach but that also provided the novelty of “computer-aided instruction,’' he added.

Critics contend that companies were--and, to a great extent, still are--selling a sophisticated, computerized version of the pencil-and-paper worksheet.

“In general, the software on these systems is not as good, imaginative, or challenging as much of the stand-alone software, yet it works,’' Mr. Komoski said. “But [the question is] it works compared to what? The ‘what’ is large-group instruction in a classroom.’'

Significantly, 19 percent of the respondents to the Q.E.D. survey indicated that they never plan to buy such a system.

‘The Next Textbooks’

Such criticisms have yet to slow the expanding market for integrated learning systems, however, and the competition in the field has become more intense.

Mergers and acquisitions have been rampant since the late 1980’s as large companies acquired the assets of smaller competitors to gain a lock on the market.

An editorial in the October edition of Technology and Learning magazine, for example, notes the burgeoning interest of educational publishers in acquiring I.L.S. companies. The editorial’s author, Holly Brady, the magazine’s editor, asked rhetorically, “Will the next textbooks be I.L.S.'s?’'

Paramount, which also owns a number of educational-publishing houses, purchased Computer Curriculum Corporation in 1990. Meanwhile, Jostens, which in 1989 acquired Education Systems Corporation, is currently negotiating with WICAT Systems, a Utah-based company.

Mr. Fortune, the president of C.C.C., was quick to note the relative strengths of his firm and Jostens. He estimates that while his competitor has 7,500 school customers, C.C.C. is available in 5,000 schools and expects to increase market share thanks to its alliance with E.A.I.

Qٌي٠ranked Jostens as the most widely used system among 192 responding districts, with a 30 percent market share. C.C.C. ranked second in the field with a 16 percent ranking.

Supporters of the I.L.S. approach also say that the growth and restructuring of the industry has helped to improve products and to move them away from the rote-learning approach.

Ms. Brady of Technology and Learning noted that the lead article of the October issue, written by Mark Sherry, explores how the systems have “grown from drill-and-kill systems of the past into packages that include [software] tools, third-party software, and even multimedia.’'

And there is some evidence that the best systems are becoming more sophisticated.

In a slickly produced promotional video for SuccessMaker, the colorful graphic displays and multimedia approach are a far cry from the gray and dull screens that were the hallmark of early systems.

But Mr. Sherry, an independent consultant who helped write the E.P.I.E. study, echoed Mr. Komoski in noting that while sophisticated integrated learning systems do exist, most systems compare unfavorably to high-quality software available for networked personal computers.

Pedagogy or Politics?

Mr. Sherry also expressed concern that systems such as SuccessMaker, which are more complex and which allow teachers more latitude to manipulate the management system, might exacerbate the I.L.S. industry’s traditional weakness in the area of training teachers to use the software.

Although Computer Curriculum Corporation touts its support for teacher training, Mr. Sherry was skeptical that many districts would be willing to support the necessary in-service time and costs associated with appropriate training.

He added that, while the industry often does provide such training, companies have also been known to sell the systems as “replacements’’ for poor teachers.

“In the early days, and C.C.C. was certainly a good example, companies would say, ‘Look, we know you’re having trouble with your teachers. Well here, buy the system, and it’ll solve your problems,’ '' Mr. Sherry said.

He also noted that the E.P.I.E. study found, for example, that while some teachers enjoyed using the systems to “shore up’’ basic skills, many others did not want their students leaving the classroom to attend laboratory-based classes.

“We also found teachers that were not really incorporating the extensive reports on student activity in the I.L.S. lab into their overall assessment of the learners,’' Mr. Komosi said.

C.C.C. executives stressed, however, that their systems are designed to “enhance’’ classroom instruction, not to replace teachers.

Mr. Sherry also noted that the best intentions on the part of manufacturers and school boards to provide a high-quality product often get lost in the hurly-burly of commerce and the rush to “update’’ the traditional instructional program.

“It’s funny how decisions are made that don’t reflect instructional decisions,’' he said, “but are political and economic decisions.’'

Frank Kuhar, a spokesman for E.A.I., said the decision to contract with Computer Curriculum Corporation was made after competitive bidding that drew proposals from several of the firm’s competitors.

He also noted that the new software is likely to be just one part of an overall technology plan for his firm’s schools.

Mr. Komoski noted that, even with the best of intentions and the best of equipment, students, particularly academically capable students, can quickly become bored with the systems.

As part of a preliminary study on integrated learning systems, he asked students whether they would recommend a particular product that they had been using for several years to other students.

They said “Yes,’' he said.

“But what we discovered was that their interpretation was a lot different from ours,’' he added. “What they were saying was, ‘Yeah. Sell it. Sell it.’ ''

A version of this article appeared in the December 16, 1992 edition of Education Week as Software Maker’s ‘Guarantee’ of Student Performance Questioned