Federal Research In Vocational Ed.Said 'Inadequate'
Panelists conducting the first "outside" assessment of the government's six-year-old, $25-million center for research in vocational education have concluded that it "lacks research objectives" and has failed to make the "significant improvements" in the nation's vocational systems envisioned by the Congress when it established the entity.
The panel found that the National Center for Research in Vocational Education (ncrve) has been well managed and has complied with federal mandates. But in a report submitted to the Education Department's office of planning, budget, and evaluation (opbe), the panelists described publications produced by the center's staff as "uncritical," "superficial," and "too narrow" in focus.
"Specifically, the panel felt that ncrve's [research publications] simply have not provided the information that is needed to isolate the factors which result in vocational-education programs that are effective and equitable," the report asserts.
The eight-member panel, assembled last June by a Washington, D.C., area consulting firm that contracted with the Education Department's evaluation office to conduct a review of the center's publications program, included state and local educators and representatives from higher education, foundations, and industry.
Created by Congress in 1976, the ncrve is administered under contract by Ohio State University and is responsible for conducting research on programmatic and policy issues of "national significance," collecting statistics, and providing technical assistance and inservice training. Since 1978, the center has received more than $25 million in federal funding; its activities are monitored by the Education Department's office of vocational and adult education (ovae).
As the national clearinghouse for the dissemination of information, the center is part of a $6-billion vocational-education enterprise that serves more than 16 million students attending secondary and postsecondary institutions nationwide.
In the evaluation report, the panel criticized the vocational office's handling of the center's current five-year contract and concluded that the responsibility for the center's problems should be shared by both agencies.
In general, according to the report, "the overall relationship between ncrve and ovae is too fraternal and not conducive to an aggressive, change-oriented program of research and development."
New Five-Year Contract
In January, the Education Department awarded Ohio State a new five-year contract, worth $4 million per year and renewable annually.
A member of the evaluation panel said the group was told at its initial briefing last spring that the assessment of the center was being undertaken because "a funding decision" would have to be made "in the near future." The panelist said he surmised that the department expected the panel to make a "harsh but fair" appraisal of ncrve's performance that could be used to "open up the selection process and avoid an automatic renewal of the contract with Ohio State."
Only two organizations, Ohio State and the University of Tennessee, bid on the contract. Sources said a "pro forma" letter from Senator Howard H. Baker Jr., the majority leader, supporting the Tennessee university's bid touched off an extensive mail campaign from supporters of Ohio State.
In addition to receiving "stacks of" letters from researchers and the vocational-education community, the department received letters from members of the both the House and Senate concerning allegations of unfairness, the sources said. Ohio's Democratic Senators, along with leaders of the House Education and Labor Committee, wrote the Secretary of Education that they were "concerned that factors other than quality and capability are entering into the award process." (See Education Week, Nov. 17, 1982.)
Faced with "intense pressure to keep the contract at Ohio State," said a source, who asked not to be identified, the department decided not to make the evaluation panel's critical assessment of ncrve (which is dated October 1982) available to the department employees and another group of outside experts, who were evaluating the proposals for the new contract, even though the assessment was available to department officials at that point.
Robert M. Worthington, assistant secretary for vocational and adult education, said last week that Ohio State was awarded the federal contract after a thorough analysis of the two proposals. He said the department's decision was based on the quality of Ohio State's proposal and its past experience in operating the center.
Ohio State's bid for the contract was higher, Mr. Worthington explained, because "Tennessee did not bid on all the functions" specified in the department's request for proposals. The department's decision, he said, was "not a question of cost, but quality."
Mr. Worthington said his office "formally" received the panel's report three weeks ago.The report is being reviewed by ovae's research staff, members of the national advisory council to the Ohio State center, and the center's staff, he said.
"We don't have any intention of moving immediately on [the panel's findings]," Mr. Worthington said. The panel was "given the contract to evaluate the center's printed products; they did not talk to us about our relationship to the center or the center's relationship to us.
"Not a single member [of the panel] visited the center or conferred with the center's staff," Mr. Worthington added. "My understanding of the report is that it was a small contract to look at the center's publications; they were not asked to comment on the center."
"There have been very few criticisms of the center's work," Mr. Worthington said, adding that he is "confident" that the center will continue to produce "the kind of quality work they have done" in the past.
Robert E. Taylor, executive director of the ncrve, said the panel's report is "disturbing" because only 10 percent of the center's publications were evaluated and those publications reflect three of six functions mandated under its contract. "Those functions constitute only 37 percent of the budget during that period," he said.
"We have been audited to death and all the evaluations have been positive," Mr. Taylor said, adding that many of the findings in the recent evaulation "do not represent a consensus on the panel."
Mr. Taylor would not comment on the political nature of the report, but he said that the lobbying efforts of several professional organizations "assured an outside peer-review panel."
Acknowledging that the center has "real problems of resource contraints," Mr. Taylor said the panel's recommendations would be difficult to implement unless changes are made in the center's contract requirements.
"In some respects, the report did not deal with the differences of opinion over what the center ought to be within the department itself,'' Mr. Taylor said.
The panel's conclusions were based on a review of work produced by the ncrve between January 1978 and January 1982, according to the report, which was prepared by Technassociates Inc., of Rockville, Md. Although the center has been evaluated in the past, the report notes that the panel's efforts represent the first time an intensive eval-uation has been conducted by a group that has no direct interest in the outcome.
The panel's conclusions are based only on ncrve's publications and do not include evaluations of other activities administered by the center, such as technical assistance, conferences, seminars, advanced study, or leadership training.
The panel nonetheless offered six recommendations intended to assure that the center is more productive in the next five years than it has been in the past. In addition to continued federal support, the panel urged that the Education Department take steps to provide the center with "greater guidance and flexibility"; narrow the center's research agenda; and allow the center to concentrate on a limited number of activities.
The panel also recommended that the center improve the quality of its staff and that it discontinue inhouse research projects and contract most of its work through a competitive process.
The panel members were Samuel Barrett,former vocational-education director, California department of education; Stanley Barnett, vocational-education chairman, Edsel Ford High School, Dearborn, Mich.; Jacqueline Danzberger, executive director, Youthwork Inc., Washington, D.C.; Hugh Gordon, personnel director, Lockheed-Georgia Corp., Marietta, Ga.; Alan Gustman, professor of economics, Dartmouth College, Hanover, N.H.; Gareth Hoachlander, president, mpr Associates, Berkeley, Calif.; James A. Kelly, president, Spring Hill Conference Center, Wayzata, Minn.; George Smith, superintendent of schools, Mesa, Ariz.; and David White, assistant state director of vocational education, District of Columbia public schools.
Other findings of the panel include the following:
Compliance. In concluding that the ncrve had not met Congressional intent, the panel explained that the reports reviewed did not "ask searching questions about the performance of the present vocational-education system" and failed to concentrate sufficiently on identifying the organizational, structural, and behavioral characteristics of effective and equitable" programs.
One panelist called the center's approach "birdshot," according to the report. "There have been enough hits to please its diverse constituencies, but no solid hits on anything," the report argues. "The system has not been sufficiently directed. The system lacks clear research objectives."
The report adds that there was "a strong consensus" that the center put "too little effort" into involving the private sector in its research process.
Target audience. The report found that the majority of the center's work has been aimed at vocational educators at the expense of groups outside of the profession.
Only about 20 percent to 30 percent of the center's work was geared toward members of Congress, state legislators, governors, and mayors. About 10 percent to 20 percent addressed the interests of corporate executives, owners of small businesses, and labor officials, according to the report.
Usefulness. The panel members unanimously agreed that the bulk of the center's products would only be helpful "in solving routine operational problems." But they concluded that the center has done an "inadequate job" on work targeted for certain audiences, such as policymakers.
The report contends that those looking for ways "to modify vocational education or to link it to other aspects of education and training in order to respond to demands for social reform, to changing technology, and to changing economic conditions are likely to find the center's products least useful."
Knowledge. The report noted that most of the panel members concluded that the center's work "added little to the existing knowledge" about vocational education.Those panelists with strong research backgrounds were "very critical"; those with nonresearch backgrounds, however, had a more positive view of the center's contributions to the field.
Quality. There was "widespread agreement that the quality of the ncrve's written works has varied greatly," according to the report. The panel members were particularly dissatisfied with the center's work in the areas of applied research, policy information, and evaluation.
The report explains that the center's research "has been conceptually and methodologically inadequate," and that it "has lagged far behind standards in almost all other fields of social science."
Improvement. The report acknowledges that much of the center's work could be helpful with routine matters, but contends that they "are unlikely to bring about significant change in the effectiveness and equity of established programs." Most of the panel members said that the chances of expanding vocational opportunities for girls and women, the handicapped, economically disadvantaged youth, and adults were either ''poor" or "fair."
"One after another, the evaluators argued that we are no closer today to knowing what makes vocational-education programs work effectively than we were when the Congress approved" the center seven years ago, the report asserts.
Management. The report contends that the Education Department's vocational office failed to establish effective compliance standards and to support "multi-year targeting of resources" on critical problems in vocational-education research and development.
"There needs to be a greater emphasis on quality and less on quantity," the report asserts.