• No results found

Lessons Learned From a Review of the Development of Selected Vaccines

N/A
N/A
Protected

Academic year: 2020

Share "Lessons Learned From a Review of the Development of Selected Vaccines"

Copied!
11
0
0

Loading.... (view fulltext now)

Full text

(1)

SPECIAL ARTICLE

Lessons Learned From a Review of the Development of Selected Vaccines

National Vaccine Advisory Committee

ABSTRACT. Background. Although the vaccine

re-search and development network in the United States remains vibrant, its continued success requires maintain-ing harmonious interaction among its many components. Changing one component is likely to affect the system overall. An examination of case studies of the develop-ment of selected vaccines would allow an examination of the network as a whole. This article presents conclusions drawn from the case study review undertaken.

Objective. Successful development of vaccines is a time-intensive process requiring years of commitment from a network of scientists and a continuum of regula-tory and manufacturing entities. We undertook this work to shed light on how well the vaccine development sys-tem in the United States performs.

Method. The National Vaccine Advisory Committee examined the research and development pathways of several vaccines that reached licensure expeditiously (hepatitis B vaccine,Haemophilus influenzaetype b con-jugate vaccines); some that became licensed only after considerable delay (oral typhoid Ty21a vaccine, varicella vaccine); some that are at the point of imminent or recent licensure (reassortant Rhesus rotavirus vaccine, which was licensed by the Food and Drug Administration on August 30, 1998) or near submission for licensure (intra-nasal cold adapted influenza vaccine); and one for which clinical development is slow because of hurdles that must be overcome (respiratory syncytial virus vaccines).

Results. Some common themes emerged from the re-views of these vaccine “case histories”: the expediting influence of a strong scientific base and rationale; the need for firm quantitation of disease burden and clear identification of target populations; the critical role played by individuals or teams who act as “champions” to overcome the inevitable obstacles; availability of rel-evant animal models, high-quality reagents and stan-dardized assays to measure immune response; the abso-lute requirement for well designed, meticulously executed clinical trials of vaccine safety, immunogenic-ity, and efficacy; postlicensure measurements of the pub-lic health impact of the vaccine and a track record of the vaccine’s safety and acceptance with large-scale use; and

the critical need for international collaborations to eval-uate vaccines against diseases of global importance that are rare in the United States (eg, typhoid fever). It was clear that the critical step-up from bench scale to pilot lots and then to large-scale production, which depends on a small group of highly trained individuals, is often a particularly vulnerable point in the development pro-cess.

Conclusions. One fundamental lesson learned is that within the varied and comprehensive US vaccine devel-opment infrastructure, multiple and rather distinct paths can be followed to reach vaccine licensure. The National Vaccine Advisory Committee review process should be conducted periodically in the future to ascertain that the US vaccine development network, which has been enor-mously productive heretofore and has played a leader-ship role globally, is adapting appropriately to ensure that new, safe, and efficacious vaccines become available in a timely manner.Pediatrics1999;104:942–950;vaccine, immunization, vaccine research, vaccine development.

ABBREVIATIONS. NIH, National Institutes of Health; FDA, US Food and Drug Administration; CDC, Centers for Disease Control and Prevention; DoD, Department of Defense; NIAID, National Institute for Allergy and Infectious Diseases; NVAC, National Vaccine Advisory Committee; HBV, hepatitis B virus; Ty21a, ty-phoid (vaccine); Hib,Haemophilus influenzaetype b (vaccine); PLA, product license application; CAIV, cold-attenuated intranasal in-fluenza vaccine; RSV, respiratory syncytial virus; ACIP, Advisory Committee on Immunization Practices.

T

he United States is recognized worldwide for the innovation and productivity of its vaccine research and development infrastructure. This loose but highly interactive network of component organizations and institutions involved in the broad vaccine development process has contributed to the development of more than two thirds of all new vaccines approved for human use worldwide in the past 20 years.

Ultimately, all licensed vaccines are produced by vaccine manufacturers. Nevertheless, the network is composed of multiple government agencies such as the National Institutes of Health (NIH), the US Food and Drug Administration (FDA), the Centers for Dis-ease Control and Prevention (CDC), the Department of Defense (DoD), and the Agency for International Development, as well as academic institutions, foun-dations, large and small vaccine-manufacturing com-panies, and small biotechnology firms.

Although these components may be involved in distinct aspects and stages of vaccine development, all are dedicated to the introduction and application of safe and effective vaccines for the control and From the National Vaccine Advisory Committee of the National Vaccine

Program Office, Centers for Disease Control and Prevention, Atlanta, Geor-gia, and the US Department of Health and Human Services, Washington, DC.

Committee members who wrote this report are listed in Appendix 1. Address correspondence to Georges Peter, MD, Chair, National Vaccine Advisory Committee, c/o Division of Pediatric Infectious Diseases, Rhode Island Hospital, 593 Eddy St, Providence, RI 02903. E-mail: georges_ [email protected]

Received for publication Jan 4, 1999; accepted Apr 6, 1999.

Reprint requests to (R.F.B.) National Vaccine Program Office, MS D-66, Centers for Disease Control and Prevention, 1600 Clifton Rd, Atlanta, GA 30333.

(2)

prevention of infectious diseases.1Some of the many

activities in which this broad network engages are outlined in Table 1.

Although the vaccine research and development network in the United States remains vibrant, its continued success requires maintaining harmonious interaction among its many components. Like any network, changing one component is likely to affect the system overall. Therefore, an examination of case studies of the development of selected vaccines al-lowed an examination of the network as a whole.

METHODS

The Goal of the Review of Selected Case Histories of Vaccine Development

Application of the tools of modern biotechnology has resulted in an array of vaccine candidates coming from many sources.2The Jordan Report,3an annual publication from the National Institute of

Allergy and Infectious Diseases (NIAID), tracks the status of a large number of vaccines in various stages of development, from those in early preclinical studies to others undergoing large-scale efficacy trials. During the past several years,The Jordan Reporthas documented the increasing number of vaccine candidates at all phases of development. In 1998, this report identified at least 350 candidates for a spectrum of infectious disease targets.3

Progres-sion through the vaccine research and development pipeline— from the laboratory bench to licensure—is necessarily methodic and deliberate to ensure that the safest and most effective vaccines are developed. The process is laborious, time-consuming, and increasingly expensive, estimated to cost $100 to $300 million to bring a successful vaccine candidate to licensure.4 Continuing

development of new candidate vaccines by basic research labora-tories will increase the pressure on the infrastructure that must prepare pilot lots and perform clinical trials of these vaccines.

With the goal of identifying components of the US vaccine research and development network that, if strengthened, may help the system to operate more efficiently, the Albert B. Sabin Foundation (now the Albert B. Sabin Vaccine Institute) convened a workshop at Cold Spring Harbor Laboratory (New York, NY), November 17–20, 1996, organized in conjunction with National Vaccine Advisory Committee’s (NVAC) Subcommittee on Future Vaccines (see Appendix 2 for participant list). The broad objective of the workshop was to review the developmental histories of several recently licensed vaccines as well as others still in the development pipeline. A more specific goal was to distill from the

reviews common features that either accelerated or inhibited the vaccine development process, from basic research to licensure and finally, to implementation. For each vaccine, a series of questions (Table 2) were discussed.

Vaccines Selected For the Initial Review

Of the many examples that could have been chosen, the diverse vaccines shown in Table 3 were examined. This selection included four vaccines that have been licensed by the FDA (hepatitis B vaccine [HBV], typhoid [Ty21a], Haemophilus influenzae type b [Hib] conjugates, and varicella),5–9 one that was at the point of

product license application (PLA) to the FDA at the time of the workshop and was subsequently licensed in August 1998 (rotavi-rus),10and another far along in the development process (a phase

3 trial of trivalent cold-attenuated intranasal influenza vaccine [CAIV] was recently completed, and the PLA has been filed).11

Finally, respiratory syncytial virus (RSV) vaccines were included in this analysis to provide an example of a much-needed vaccine

TABLE 1. Activities of the Interactive Network of Component Organizations and Institutions Involved in the Vaccine Develop-ment Process

Identification of the causative pathogens of infectious diseases and elucidation of virulence factors and antigenic diversity (eg, serotypes and serogroups), and reliable, reproducible laboratory propagation of the pathogen

Fundamental studies of pathogenesis and immune response that result in protection against disease

Development of animal models

Epidemiologic surveillance to quantify disease burden and to identify high-risk sectors of the population that would be targets for vaccine implementation

Development of vaccine candidates

Evaluation of the safety and immunogenicity profile of candidate vaccines in preclinical studies

Preparation of prototype vaccines (usually made under Good Manufacturing Practices guidelines) acceptable for testing in human subjects, to allow transition from preclinical to clinical trials

Determination of an efficient manufacturing process for a candidate vaccine

Performance of extensive clinical evaluations of candidate vaccines to assess safety and immunogenicity and, in some instances, to derive preliminary evaluations of protective efficacy

TABLE 2. Questions Asked Regarding Each Vaccine

What was the initial idea or vaccine strategy?

Where did the technologic development occur (industry, government agency, and/or academia) and how was it supported?

How were epidemiologic and disease burden data gathered?

What were the roles of industry, academia, and foundations in the different stages of development, including the

contribution of large and small companies?

What were the roles of various government agencies (eg, CDC, NIH, DoD, FDA) in the vaccine development process?

What technical difficulties needed to be overcome in the transition from a research idea into a viable vaccine candidate?

Did concerns over liability and the potential for adverse effects of vaccines delay the initiation of clinical trials?

What lessons were learned from early phase 1 (preliminary safety and immunogenicity dose response in small numbers of subjects) and phase 2 (more extensive safety and immunogenicity evaluation in larger numbers of subjects) clinical trials?

Were difficulties encountered in initiating or conducting large-scale efficacy (phase 3) trials?

What scientific and technical issues constituted obstacles to FDA licensure?

What have postlicensure (phase 4) studies of safety and effectiveness shown?

Have phase 4 studies influenced subsequent vaccine research and development?

What are the recommendations for use of the vaccine in the US population once licensure is granted and vaccine is made available?

What strategies were followed in introducing the vaccine into the market and were hurdles encountered?

What was the role of intellectual property rights in the overall vaccine development program?

What individuals, groups, or institutions served as advocates for the vaccine at different stages?

(3)

that has been in research and development for decades, yet progress has been inordinately slow and fraught with difficulty.12

This example was selected to illustrate how problems from an earlier generation of vaccine—formalin-inactivated RSV paren-teral vaccine—left a legacy of safety concerns that has had a marked impact on subsequent RSV vaccine development,13,14

slowing the pace at which newer RSV vaccine candidates are allowed to proceed to clinical testing.

The examples selected include vaccines for preventing bacterial (Ty21a and Hib conjugates) and viral infections (HBV, varicella, rotavirus, influenza, and RSV) that are administered parenterally (HBV, Hib, and varicella); orally (Ty21a and rotavirus); or intra-nasally (CAIV). One current RSV vaccine candidate that is well along in development also is administered intranasally.12 The

examples include disparate vaccine strategies including attenu-ated bacteria (Ty21a) and viruses (varicella, rotavirus, cold-atten-uated influenza) and purified antigens (HBV surface antigen, and Hib conjugate [capsular polysaccharide covalently linked to a carrier protein]). Two vaccines (varicella and Ty21a) were devel-oped in significant collaborations with other countries.

Together, these examples make up 5 of the 18 vaccines licensed by the FDA from 1980 to 1996 (Table 4). Among the licensed vaccines examined in this review, the length of time from identi-fication of the pathogenic organism to the initial phase 1 clinical trial and ultimate FDA licensure varied considerably (Table 5).

The Vaccine Development Process and the Paths to Licensure

Successful development of a vaccine results in its licensure. Licensed vaccines may become important public health tools for widespread use in the US population (eg, for universal immuni-zation of infants) or may be used in other targeted ways, such as for protection of travelers. The vaccine development process lead-ing to licensure is step-wise and bridges basic research, develop-ment, large-scale production in an approved facility, and clinical evaluation to establish safety and protective efficacy. The purpose of the phased clinical trials is to develop an adequate body of data that establishes the safety, immunogenicity, and efficacy of the well-characterized vaccine candidate in target populations. Licen-sure also requires demonstration of the ability to prepare consis-tent lots of vaccine of the same conconsis-tent and quality. The vaccines

must be produced in facilities specifically designed and approved for the production of vaccines for human use. As the result of safety considerations, certain vaccines (eg, live virus or bacterial vaccines) may require considerable additional microbiologic and clinical information relating to the vaccine strain, including com-parison of the properties of the shed vaccine isolates versus the parent vaccine strain, the pattern of shedding, transmissibility to contacts,15–17and its potential to survive in the environment or to

enter the food chain.17,18

The vaccine development process also is pyramidal and selec-tive.3 Although many vaccine candidates enter initial phase 1

clinical trials to assess their safety and immunogenicity in small numbers of subjects, only a small proportion progress to phase 2 trials in larger numbers of subjects. Phase 2 trials are designed to evaluate further the safety and immunogenicity of the product19;

to select the optimal formulation of a candidate product; and to identify the most suitable schedule for vaccine administration (including the dosage, the route of administration, and the optimal interval between primary and booster immunizations) for subse-quent evaluation in large phase 3 efficacy studies. Only a fraction of vaccines that enter phase 2 trials proceed to large-scale field trials in which their efficacy can be assessed. Because sample sizes for these pivotal studies are large, they often require several years to complete enrollment and follow-up.20Thus, although many

vaccine candidates enter the early phases of clinical trials, only a handful achieve FDA licensure.

RESULTS

Common Themes Identified Among Successful Vaccine Development Programs

Eight common themes emerged from this review of vaccine development case histories. These themes demonstrated the dynamics of the multifaceted US vaccine development network and provided insights into where it worked well and where it floundered.

A Solid Scientific Base Is Critical For Success

Large gaps still exist in our understanding of the precise immunologic mechanisms that mediate the protection conferred by a number of old and new vaccines, thus empiric approaches still must be ap-plied. Similarly, the biologic mechanisms that lead to certain adverse events from vaccination are not well understood. The case histories argue that although an element of empiricism in vaccine development, by necessity, is inevitable, the likelihood that new can-didates will succeed as safe and effective vaccines depends on the scientific foundation. The case stud-ies demonstrated that vaccine development moved forward expeditiously when the scientific base was well established. For example, both Hib conjugate vaccines and HBV vaccines aimed to elicit specific immune responses (serum IgG antibodies) against defined antigens (Hib capsular polysaccharide and HBV surface antigen, respectively). In fact, one of the Hib conjugate vaccines was licensed based on the demonstration of serologic equivalence in its immu-nogenicity.7

Development of some vaccines requires a key sci-entific discovery before development can proceed rapidly. Hib vaccine research accelerated when Hib capsular polysaccharide was demonstrated to be ef-fectively converted from a independent to a T-dependent antigen by conjugation to a carrier pro-tein.21

Where the science is less mature, vaccine develop-ment efforts often stall because they lack clear direc-tion and endpoints. For both the attenuated bacterial

TABLE 3. Vaccines Selected for Review

Ty21a live oral typhoid

Haemophilus influenzaetype b (Hib) conjugate Attenuated varicella

Hepatitis B

Reassortant Rhesus rotavirus

Intranasal cold-adapted attenuated influenza virus Respiratory syncytial virus

TABLE 4. Vaccines Licensed by the FDA Between 1980 and 1996

Year Licensed

Adenovirus (military use only) 1980

Pneumococcal polysaccharide (14-valent) 1980

Rabies (human diploid cell culture) 1980

Hepatitis B (plasma derived) 1981

Meningococcal polysaccharide (4-valent) 1981 Pneumococcal polysaccharide (23-valent) 1983

Hib polysaccharide 1985

Recombinant hepatitis B vaccine 1986

Enhanced inactivated polio vaccine 1987

Ty21a 1989

Hib conjugates (for young infants) 1990

DTaP (for booster doses) 1991

Salmonella typhivi polysaccharide (typhoid) 1992

DTP-Hib 1993

Japanese encephalitis (purified inactivated) 1993

Hepatitis A 1994

Varicella 1995

(4)

strain Ty21a and the attenuated varicella virus vac-cines, development and evaluation were slowed be-cause immunologic correlates of protection were not available when efficacy trials were undertaken. In contrast, availability of correlates of immunity forH influenzae b and hepatitis B has facilitated develop-ment of vaccines (including combination vaccines) against these pathogens. These observations under-score the need to continue to invest in the basic sciences that underpin vaccinology, including microbiology, molecular biology, genetics, and immunology.

Epidemiologic Studies, Disease Surveillance, and Economic Data Are Critical For Priority Setting and Guide the Vaccine Development Process in Both the Public and Private Sectors

Because available resources are insufficient to meet the demands of all vaccine development projects, establishment of priorities to guide alloca-tions among competing programs is critical. The in-formation for formulating priorities includes a quan-titation of the disease burden to be prevented, including mortality, acute morbidity, long-term se-quelae, and the direct and indirect economic costs associated with this disease burden.22,23

The review of the case histories demonstrated the importance—for both public health and commercial decision-makers— of timely surveillance of disease and adverse events in setting goals for vaccine de-velopment and immunization programs. Systematic surveillance helps to identify high-risk populations and provides insights into changing epidemiologic trends. For example, the initiation of CDC popula-tion-based surveillance activities in the early 1970s focused attention on the importance of invasive in-fections caused byHaemophilus influenzaetype b as a cause of hospitalization and death among infants and young children in the United States.24,25

How-ever, vaccine licensure antedated uniform reporting and actually provided a stimulus to report. A similar analysis demonstrated that most cases of typhoid fever among US residents occurred among those who traveled, and it identified them as a high-risk group that would benefit from immunoprophy-laxis.26,27In the case of rotavirus infection, mortality

in the United States was found to be low, but the incidence of diarrhea among infants is high and the economic costs of hospitalization for diarrheal dehy-dration are considerable.28 –30In addition to the

mis-ery they cause, these illnesses keep infants out of child care and cause parents to be absent from work, thereby having an impact on national productivity.

The case studies highlighted the importance of well-conducted cost-effectiveness and cost-benefit studies to demonstrate the value of vaccines and their eco-nomic impact.23,29,31,32Vaccines have been highlighted

as among the most cost-effective health interven-tions.2

The Importance of Champions

Those vaccines that survived the difficulties en-countered early on the path of development, and that progressed to licensure and implementation, were fostered vigorously by an effective, energetic, and dedicated champion—an individual or team—that functioned as the persistent advocate for the vac-cine’s development. The case histories highlighted the importance of supporting research environments that foster such individuals and teams in settings where scientific debates are encouraged. Often champions exist for each aspect of vaccine develop-ment. The leading advocates, who typically perse-vered during multiple stages of the vaccine’s devel-opment to ensure continuing progress, were from different areas of work, including government insti-tutions (rotavirus vaccine),33,34academia (Ty21a

vac-cine),35–37 industry (HBV vaccine),38,39 or a

combina-tion thereof.40 – 42 This theme was noted in almost all

the vaccine case histories and was evident in indus-try as well as in the public sector and academia.

The case history analyses disclosed that the role of a champion (advocate) is particularly important for so-called orphan vaccines. By definition, the market potential of these products is limited. For example, in the United States, typhoid vaccine is almost entirely limited to use in travelers.

Animal Models, Reagent Availability, Assay Development and Standardization, and the Identification of Immunologic Correlates of Protection Are Essential to the Vaccine Development Process

Although the ultimate test of a vaccine’s effective-ness must await evaluation of its performance when used in the population at large, the bridge from basic vaccine research to clinical development, and from production of pilot lots to consistent manufacture of large-scale lots, depends on the ability to make indi-rect laboratory assessments to ensure that develop-ment is on track. The validation of each progressive step in the process is a fundamental part of vaccine development. Measurements of vaccine potency or its effect on the immune system depend on the avail-ability of reliable assays, which often are developed in parallel with the vaccine itself. For example, the

TABLE 5. A Comparison in the Time Between Identification of the Pathogen, Initiation of the First Clinical Trial, and the Year of Licensure of the Vaccine by the FDA

Vaccine Isolation of

the Pathogen

First Phase 1 Trial

FDA Licensure

Interval, Phase 1 to Licensure

Ty21a 1884 1973 1989* 16 years

Varicella 1953 1974 1995 21 years

HBV (plasma) 1965 1975 1981 6 years

HBV (recomb) 1965 1985 1986 2 years

Hib conjugate 1889 1981 1990† 9 years

* Licensed earlier in many other countries.

(5)

lack of a simple test of potency to monitor the ade-quacy of different lots of varicella vaccine impeded its development. Failure to address adequately these technical aspects of vaccine development may derail development efforts. Given the complicated nature and expense of large, clinical efficacy trials to deter-mine the potential impact of a candidate vaccine, the development of laboratory-based clues to a vaccine’s performance could streamline the evaluation pro-cess.

Carefully Designed, Conducted, and Analyzed Clinical Trials, Including Challenge Studies and Studies in High-risk Populations, Are Essential to Vaccine Development

The review of the case histories of the various vaccines underscored the importance of the design and analysis of phase 1 and 2 clinical studies (includ-ing challenge studies) and phase 3 efficacy trials. The results of these investigations are the best predictors of the vaccine’s ultimate success, after licensure and implementation. Thus, the sequential phases of clin-ical trials must be designed to answer well-defined questions, including an appreciation of a variety of outcomes that would be acceptable and appropriate. Surveillance systems to detect adverse events should be incorporated into large prelicensure clinical studies.

The reviews revealed that inadequate or poorly designed trials can have a detrimental effect on vac-cine development and may lead to flawed decision-making. Overall, the sophistication of clinical vacci-nology research continues to improve. Nevertheless, ensuring that trials are well designed, conducted, analyzed, and communicated is fundamental to the expeditious progress of vaccine development and has broad implications for the field of vaccine devel-opment in general.

The Transition From Research to Full-scale Production: Process Development and Pilot Lots

Process engineers and industrial chemists are un-sung heroes and heroines of the vaccine develop-ment network. They transfer technology from bio-logic scientists to biobio-logic engineers. Their expertise, primarily based in the manufacturing sector, ensures that production-sized lots of vaccine maintain the required degree of safety and efficacy. Scale-up or changes in formulation can result in a loss of potency or diminished efficacy. For example, the efficacy of Ty21a vaccine is strongly correlated with the formu-lation in which this oral vaccine is presented. Liquid formulations (reconstituted lyophilates) provide a high level of protection, enteric-coated capsules con-fer a moderate level, and gelatin capsule/sodium bicarbonate formulations provide poor protec-tion.35,37,43

Postlicensure Studies: Monitoring Public Health Impact and Vaccine Safety

Ensuring the safety and efficacy of new vaccines as they become licensed and are given to larger num-bers of persons is critical to ensuring the trust of the public and stability of the immunization program. Postlicensure data may indicate that some vaccines

are more effective than predicted before licensure,44,45

whereas others have been disappointing.46

Paradox-ically, when effective vaccination controls disease, apathy and complacency about the disease pre-vented alters risk– benefit perceptions about the vac-cination, especially among young parents, who may have no knowledge of the disease. Consequently, the temporal association of extremely rare adverse events and vaccination, even if causation is not proven, can result in decreased immunization rates and outbreaks of disease. Experience with killed whole-cell pertussis vaccine in the 1970s and 1980s and with measles vaccine in the 1990s provides good examples. In countries where the coverage with killed whole-cell pertussis vaccine was high, the in-cidence of pertussis disease was 10 to 100 times lower than that in countries in which vaccine use was discontinued or diminished because of controversy about its safety.47 In several European countries,

measles vaccine coverage is low, at least in part as a reflection of public controversy about unsubstanti-ated allegations that measles vaccine may predispose to inflammatory bowel disease and other chronic illnesses.48 –52Incidence of measles in those countries

remains high. By contrast, in the Americas and in Finland, where measles vaccine coverage is extraor-dinarily high, measles has been nearly eliminat-ed.53–56

Accelerating the Evaluation of Vaccines in Endemic Areas: The Importance of International Collaboration

Vaccines in development to protect against dis-eases that are not endemic in the United States (eg, typhoid, Japanese B encephalitis, and dengue fever) must be tested primarily in countries with high inci-dence of disease. This testing is especially important for vaccines for which no immunologic correlate of protection exists. For most vaccines, international collaboration will, over time, become an increasingly essential part of developmental efforts. This aspect underscores the need to support effective interna-tional collaborations now, among academic and gov-ernment investigators, regulators, manufacturers, and national governments. Moreover, the effort will require strong leadership in both vaccine sciences and vaccine policy at the national level as well as in the international arena.

RECOMMENDATIONS

Based on the reviews of examples in vaccine de-velopment, the NVAC strongly recommends that the following actions (Table 6) be taken to foster devel-opment of additional safe and effective vaccines that

TABLE 6. Recommendations of the NVAC

Ensure support for the nation’s vaccine research and development effort.

Strengthen education and training in vaccinology. Foster public-private collaborations.

Strengthen international collaborations for vaccine research, development, and evaluation.

Support educational efforts to improve awareness of the value of vaccines.

(6)

can reduce further the burden of infectious diseases and provide tools to respond to the threat posed by emerging infections.

Ensure Support For the Nation’s Vaccine Research and Development Effort

Centers of excellence in both the public and the private sectors have been fundamental to successful vaccine development in past years. Ensuring the con-tinued strength of these programs is critical to a healthy US vaccine development network. Specific elements of this process are to:

1. Explore the feasibility and implementation of a Vaccine Study Section at the NIH that would in-clude all aspects of vaccinology, from basic re-search through process development and clinical testing of candidate vaccines.

2. Examine the regulatory process with the goal of ensuring continued efficiency of the recently streamlined licensing application review proce-dures, while also ensuring that safety and efficacy considerations are addressed. To sustain an effi-cient, rapidly responsive, regulatory review of vaccine PLAs, appropriation of adequate addi-tional funding to the FDA specifically to support this process would be helpful.

3. Assess the capability of the network of vaccine manufacturers to develop public health priority vaccines that may have limited commercial ap-peal. An example of such a vaccine might be one against an early emerging influenza virus that because of its antigenic make-up (eg, H5:N1) is potentially capable of pandemic spread, although firm epidemiologic evidence of such a propensity is pending.

4. Conduct reviews of the progress being made in the development and introduction of new vac-cines with periodic reports to NVAC and the Ad-visory Committee on Immunization Practices (ACIP) for consideration of policy recommenda-tions.

Strengthen Education and Training in Vaccinology The term vaccinology is used increasingly to de-scribe the full gamut of vaccine development and application activities, including basic research to study pathogenesis and identify protective antigens; construction of vaccine candidates (attenuated strains, purified antigens, and DNA vaccine plas-mids) and identification of new adjuvants; clinical trials to evaluate the safety and immunogenicity of vaccines, combinations thereof, and new routes of immunization (eg, mucosal vaccines); large-scale field trials to measure the efficacy of vaccines; de-tailed studies of human immune responses to vac-cine antigens; epidemiologic studies to quantify the burden of specific diseases and to identify target populations for immunization; postlicensure moni-toring of vaccine safety and effectiveness; and global policy issues such as expediting the introduction of new vaccines into developing countries, regional co-operation to eliminate certain vaccine-preventible in-fections (eg, polio and measles from the Americas),

and global eradication programs (eg, smallpox and polio). The multifaceted burgeoning discipline of vaccinology within the United States is a national resource that should be strengthened. One facet that particularly needs nurture is the transition from bio-science to bioengineering. The expertise and skills required for pilot lot manufacturing and scale-up thereafter of vaccines are not obtained easily through current academic disciplines. The following are indi-cated:

1. Support centers of excellence for training the next generation of vaccinologists, which should clude training in academic, government, and in-dustrial vaccine development programs. Few such formal programs currently exist.

2. Develop, support, and create incentives for post-doctoral fellowships in both basic and clinical vac-cinology, including linkages to allied fields such as bioengineering.

Foster Public–Private Collaborations

Collaborations among academia, industry, and government have had a primarily positive impact in expediting the progression of several vaccines to the point of licensure or PLAs. The recommendations to ensure that these collaborations are strengthened and that novel mechanisms are developed are:

1. Explore the feasibility of novel incentives for vac-cine development that are particularly relevant to small companies that rely on venture capital rather than on sales.

2. Ensure that the price of new vaccines is not a barrier to successful introduction and implemen-tation in target, high-risk populations in the United States and globally.

3. Ensure that the need for indemnification does not serve as a barrier to vaccine development.

Strengthen International Collaborations for Vaccine Research, Development, and Evaluation

Given the demonstrated importance of interna-tional collaborations, encouragement, and support of US scientists to participate in vaccine research, de-velopment, and evaluation throughout the world is critical, including:

1. Evaluating the opportunities for participation by US scientists in vaccine-related research globally; 2. Recognizing the crucial role of a strong World Health Organization to foster global vaccine re-search and immunization efforts, including tech-nology transfer where appropriate, and to facili-tate US participation in these projects;

3. Enhancing surveillance of infectious diseases, both in the United States and in other countries globally, to ensure availability of critical data for priority setting, site selection for clinical trials, and appropriate introduction of new vaccines; 4. Considering support for undertaking an

(7)

Med-icine study on domestic priorities for vaccine de-velopment.

Support Educational Efforts to Improve Awareness of the Value of Vaccines

The risk-versus-benefit of a vaccine is evaluated continuously in the context of changing epidemiol-ogy, control and eradication efforts, novel therapies, and new data on risks that may emerge from phase 4 trials or postlicensure surveillance. Coupled with an increasingly complex childhood immunization schedule, appropriate education of all parties, in-cluding parents, health care providers, and vaccinees is needed. This effort should include:

1. Improving graduate medical education on vac-cines;

2. Implementing the recommendations of the Task Force on Safer Childhood Vaccines57; and

Undertaking Reviews of Additional Vaccine Case Histories

Case histories of the development of other vac-cines should be reviewed periodically to ascertain that the US vaccine development network is adapt-ing adequately in a changadapt-ing environment, so that additional safe, effective vaccines will become li-censed in a timely manner to protect populations at risk of disease.

Some Useful Lessons Learned and Limitations of the Review

Examination of the case studies of vaccine devel-opment highlighted both common and unique as-pects of the vaccine development process. This ex-amination revealed that in the era of molecular vaccinology and immunology, understanding the precise mechanisms by which most vaccines work remains inadequate. Consequently, the process of vaccine research and development continues to be, at least in part, an empiric, methodic science. The case histories demonstrated clearly the need to maintain a wide array of partnerships that include various tal-ents and expertise to ensure continued momentum within the development process. One fundamental lesson is that within the varied and comprehensive US vaccine development infrastructure, multiple and rather distinct paths can be followed to achieve vac-cine licensure.

Reviewing vaccine case histories also revealed lim-itations of this approach in attempting to draw con-clusions and to make recommendations for the de-velopment of the vaccines of the future. For example, because development takes so long, much of the history of the cases examined occurred in the 1960s, 1970s, or early 1980s, when the landscape was dif-ferent. Moreover, because the case histories consti-tute only a sample of the many vaccines that could have been selected, the lessons they teach are limited to the experiences they represent. Examples of other vaccines could be examined similarly to address ad-ditional specific questions such as: 1) What are the barriers to testing of novel vaccine formulations in young infants? 2) What are the barriers to the

devel-opment of a maternal immunization program that would have the goal of protecting infants from dis-ease in the first few months of life? 3) Is product liability a barrier to conducting clinical trials? and 4) How do intellectual property rights affect vaccine development?

Implementation of NVAC’s six recommendations will require the concerted effort of many of the ele-ments of the “fragile fabric” of vaccine development in the United States, ie, the loose network of federal, state, academic, and private groups that has created vaccines and orchestrated their use so successfully.1

Some recommendations are discrete and easily im-plemented by a single agency (eg, creation of a Vac-cine Study Section at NIH). Others, such as improved education about immunization, will require inte-grated multiagency and ongoing efforts. The Na-tional Vaccine Program Office should continue to play a prominent supportive role in ensuring imple-mentation of the recommendations.

APPENDIX 1

Members of the National Vaccine Advisory Committee who have authorship responsibility for this article are Georges Peter, MD, (Chair), Rhode Island Hospital, Providence, RI; Mary des-Vignes-Kendrick, MD, City of Houston, Department of Health and Human Services, Houston, TX; Theodore, C. Eickhoff, MD, Uni-versity of Colorado, Health Sciences Center, Denver, CO; Amy Fine, MPH, BSN, Health Policy/Consultant, Washington, DC; Virginia Galvin, MD, MPHa, Cobb/Douglas Health District, Mari-etta, GA; Myron M. Levine, MD, University of Maryland, Balti-more, MD; Yvonne A. Maldonado, MD, Stanford University School of Medicine, Stanford, CA; Edgar K. Marcuse, MD, MPHa (Past Chair), Children’s Hospital & Medical Center, Seattle, WA; Thomas P.C. Monath, MD, OraVax, Inc, Cambridge, MA; June E. Osborn, MD, Josiah Macy, Jr, Foundation, New York; Stanley Plotkin, MD, Pasteur Merieux Connaught, Doylestown, PA; Greg-ory A. Poland, MD, Mayo Clinic and Foundation, Rochester, MN; M. Patricia Quinlisk, MD, Iowa Department of Public Health, Des Moines, IA; David R. Smith, MDa, Texas Tech Health Sciences Center, Lubbock, TX; Marian Sokol, PhD, Any Baby Can, Inc, San Antonio, TX; Daniel B. Soland, SmithKline Beecham Pharmaceu-ticals, Belgium, The Netherlands; Patricia N. Whitley-Williams, MD, Robert Wood Johnson Medical School, New Brunswick, NJ; Donald E. Williamson, MD, Alabama Department of Public Health, Montgomery, AL; Robert F. Breiman, MD, (Executive Secretary), Director, National Vaccine Program Office, Atlanta, GA.

APPENDIX 2. PARTICIPANTS: CASE STUDIES IN VACCINE DEVELOPMENT

Dr Ann Arvin, Stanford University School of Medicine; Dr Norman Baylor, Food and Drug Administration; Dr Robert Breiman, National Vaccine Program Office; Dr Mary Lou Clem-ents-Mann†, Johns Hopkins University School of Hygiene and Public Health; Dr Robert Couch, Baylor College of Medicine; Dr R. Gordon Douglasa, Merck Vaccine Division; Professor Louis Ga-lambos, Johns Hopkins University; Dr Bruce Gellin, National In-stitute of Allergy and Infectious Diseases/NIH; Dr Ann Gershon, Columbia University College of Physicians and Surgeons; Dr Roger Glass, Centers for Disease Control and Prevention; Dr Lance Gordon, Oravax, Inc; Dr Mary Glode, Children’s Hospital of Denver; Dr Harry Greenberg, Stanford University Medical Center; Dr Carolyn Hardegreec, Food and Drug Administration; Dr Carole Heilman, National Institutes of Allergy and Infectious Diseases/ NIH; Dr Maurice Hilleman, Merck Institute for Therapeutic

Re-aFormer NVAC members. bNVAC members.

cLiaison representatives to NVAC.

(8)

search; Dr William Jordan, National Institutes of Allergy and Infectious Diseases/NIH; Dr Myron M. Levinea, University of Maryland Center for Vaccine Development; Dr Edgar Marcusea, University of Washington School of Medicine; Dr Peter Paradiso, Wyeth-Lederle Vaccines and Pediatrics; Dr Paul Parkman; Dr Stanley Plotkinb, Pasteur Merieux Connaught; Dr Gerald Quinnan, Uniformed Services University of Health Sciences; Dr Regina Rabinovichc, National Institute of Allergy and Infectious Diseas-es/NIH; Dr Leighton Read, Aviron, Inc; Dr Philip Russell, Johns Hopkins University School of Hygiene and Public Health; Profes-sor Jane Sewell, Johns Hopkins University; Mr H.R. Shepherd, Albert Sabin Vaccine Foundation; Dr George Siber, Wyeth-Lederle Vaccines and Pediatrics; Mr Daniel Solanda, Smith Klein Beecham Pharmaceuticals; Dr Joel Ward, UCLA Center for Vaccine Re-search; Dr Jo White, Aviron, Inc; Dr Peter Wright, Vanderbilt University Medical Center.

ACKNOWLEDGMENTS

NVAC acknowledges the work of Drs Myron M. Levine, Bruce Gellin, and Regina Rabinovich in the preparation of this manu-script.

REFERENCES

1. National Vaccine Advisory Committee. United States vaccine research: a delicate fabric of public and private collaboration.Pediatrics. 1997;100: 1015–1020

2. Levine MM. The legacy of Edward Jenner. Br Med J. 1996;312: 1177–11783

3. Division of Microbiology and Infectious Diseases, National Institute of Allergy and Infectious Diseases.The Jordan Report: Accelerated Develop-ment of Vaccines, 1998. Bethesda, MD: NIAID; 1998

4. Gregersen J. Vaccine development: the long road from initial idea to product licensure. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.New Generation Vaccines. New York, NY: Marcel Dekker; 1997: 1165–1183

5. Centers for Disease Control and Prevention. Food and Drug Adminis-tration approval of use ofHaemophilusb conjugate vaccine for infants.

MMWR Morb Mortal Wkly Rep. 1990;39:698 – 699

6. Centers for Disease Control and Prevention. Food and Drug Adminis-tration approval of use of aHaemophilusb conjugate vaccine for infants.

MMWR Morb Mortal Wkly Rep. 1990;39:925–926

7. Centers for Disease Control and Prevention. FDA approval of use of a new Haemophilus b conjugate vaccine and a combined diphtheria–tetanus–pertussis andHaemophilusb conjugate vaccine for infants and children.MMWR Morb Mortal Wkly Rep. 1993;42:296 –298 8. Centers for Disease Control and Prevention. Typhoid immunization:

recommendations of the Advisory Committee on Immunization Prac-tices.MMWR Morb Mortal Wkly Rep. 1990;39:1–5

9. Centers for Disease Control and Prevention. Prevention of varicella: recommendations of the Advisory Committee on Immunization Prac-tices.MMWR Morb Mortal Wkly Rep. 1996;45:1–25

10. Rennels MB, Glass RI, Dennehy PH, US Rotavirus Vaccine Efficacy Group. Safety and efficacy of high-dose rhesus-human reassortant ro-tavirus vaccines: report of the National Multicenter Trial.Pediatrics. 1996;97:7–13

11. Belshe RB, Mendelman PM, Treanor J, et al. The efficacy of live atten-uated, cold-adapted, trivalent, intranasal influenzavirus vaccine in chil-dren.N Engl J Med. 1998;338:1405–1412

12. Karron RA, Wright PF, Crowe JE, Jr, et al. Evaluation of two live, cold-passaged, temperature-sensitive respiratory syncytial virus vac-cines in chimpanzees and in human adults, infants, and children.J Infect Dis. 1997;176:1428 –1436

13. Kapikian AZ, Mitchell RH, Chanock RM, Shvedoff RA, Stewart CE. An epidemiologic study of altered clinical reactivity to respiratory syncytial (RS) virus infection in children previously vaccinated with an inacti-vated RS virus vaccine.Am J Epidemiol. 1969;89:405– 421

14. Kim HW, Canchola JG, Brandt CD, et al. Respiratory syncytial virus disease in infants despite prior administration of antigenic inactivated vaccine.Am J Epidemiol. 1969;89:422– 434

15. Levine MM, Barry EM. Inadvertent nontarget effects of live vaccines. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.New Generation Vaccines.New York, NY: Marcel Dekker; 1997:1179 –1184

16. Benyesh-Melnick M, Melnick JL, Rawls WE, et al. Studies of the immu-nogenicity, communicability and genetic stability of oral poliovaccine administered during the winter.Am J Epidemiol. 1967;86:112–136 17. Simanjuntak CH, O’Hanley P, Punjabi NH, et al. The safety,

immuno-genicity, and transmissibility of single-dose live oral cholera vaccine

CVD 103-HgR in 24 to 59 month old Indonesian children.J Infect Dis. 1993;168:1169 –1176

18. Wallis C, Melnick JL. Concentration of viruses from sewage by adsorp-tion on millipore membranes.Bull WHO. 1967;36:219 –225

19. Tacket CO, Mattheis M, Rennels MB. Initial clinical evaluation of new candidate vaccines: phase 1 and 2 clinical trials of safety, immunoge-nicity and preliminary efficacy. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.New Generation Vaccines.New York, NY: Marcel Dekker; 1997:35– 46

20. Clemens JD, Naficy A, Rao MR. Long-term evaluation of vaccine protection: methodological issues for phase 3 trials and phase 4 studies. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.New Generation Vaccines. New York, NY: Marcel Dekker; 1997:47– 67

21. Schneerson R, Barrera O, Sutton A, Robbins JB. Preparation, character-ization, and immunogenicity ofHaemophilus influenzaetype b polysac-charide-protein conjugates.J Exp Med. 1980;152:361–376

22. Levine MM, Levine OS. Influence of disease burden, public perception, and other factors on new vaccine development, implementation, and continued use.Lancet. 1997;350:1386 –1392

23. Levine OS, Schwartz B. Economic analyses of vaccines and vaccination programs: evaluating the economic rationale for immunization. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.New Generation Vaccines. New York, NY: Marcel Dekker; 1997:69 –78

24. Fraser DW, Geil CC, Feldman RA. Bacterial meningitis in Bernalillo County, New Mexico: a comparison with three other American popu-lations.Am J Epidemiol. 1974;100:29 –34

25. Fraser DW, Darby CP, Koehler RE, Jacobs CF, Feldman RA. Risk factors in bacterial meningitis: Charleston County, South Carolina.J Infect Dis. 1973;127:271–277

26. Ryan CA, Hargrett-Bean NT, Blake PA.Salmonella typhiinfections in the United States, 1975–1984: increasing role of foreign travel.Rev Infect Dis. 1989;II:1– 8

27. Taylor DN, Pollard RA, Blake PA. Typhoid in the United States and risk to the international traveler.J Infect Dis. 1983;148:599 – 602

28. Jin S, Kilgore PE, Holman RC, Clarke MJ, Gangarosa EJ, Glass RI. Trends in hospitalizations for diarrhea in United States children from 1979 through 1992: estimates of the morbidity associated with rotavirus.

Pediatr Infect Dis J. 1996;15:397– 404

29. Smith JC, Haddix AC, Teutsch SM, Glass RI. Cost-effectiveness analysis of a rotavirus immunization program for the United States.Pediatrics. 1995;96:609 – 615

30. Glass RI, Kilgore PE, Holman RC, et al. The epidemiology of rotavirus diarrhea in the United States: surveillance and estimates of disease burden.J Infect Dis. 1996;174(suppl 1):S5–S11

31. Huse DM, Meissner HC, Lacey MJ, Oster G. Childhood vaccination against chickenpox: an analysis of benefits and costs.J Pediatr. 1994;124: 869 – 874

32. Hay JW, Daum RS. Cost-benefit analysis ofHaemophilus influenzaetype b prevention: conjugate vaccination at eighteen months of age.Pediatr Infect Dis J. 1990;9:246 –252

33. Kapikian AZ, Flores J, Vesikari T, et al. Recent advances in development of a rotavirus vaccine for prevention of severe diarrheal illness of infants and young children.Adv Exp Med Biol. 1991;310:255–264 34. Kapikian AZ, Hoshino Y, Chanock RM, Perez-Schael I. Efficacy of a

quadrivalent rhesus rotavirus-based human rotavirus vaccine aimed at preventing severe rotavirus diarrhea in infants and young children.

J Infect Dis. 1996;174(suppl 1):S65–S72

35. Levine MM, Ferreccio C, Black RE, Germanier R, Chilean Typhoid Committee. Large-scale field trial of Ty21a live oral typhoid vaccine in enteric-coated capsule formulation.Lancet. 1987;1:1049 –1052 36. Ferreccio C, Levine MM, Rodriguez H, Contreras R. Comparative

effi-cacy of two, three, or four doses of Ty21a live oral typhoid vaccine in enteric-coated capsules: a field trial in an endemic area.J Infect Dis. 1989;159:766 –769

37. Levine MM, Ferreccio C, Cryz S, Ortiz E. Comparison of enteric-coated capsules and liquid formulation of Ty21a typhoid vaccine in random-ised controlled field trial.Lancet. 1990;336:891– 894

38. McAleer WJ, Buynak EB, Maigetter RZ, Wampler DE, Miller WJ, Hill-eman MR. Human hepatitis B vaccine from recombinant yeast.Nature. 1984;307:178 –180

39. Hilleman MR. Yeast recombinant hepatitis B vaccine.Infection. 1987;15: 3–7

40. Gershon AA, LaRussa P, Steinberg S, Mervish N, Lo SH, Meier P. The protective effect of immunologic boosting against zoster: an analysis in leukemic children who were vaccinated against chickenpox.J Infect Dis. 1996;173:450 – 453

(9)

42. Gershon AA. Varicella vaccine: its past, present and future.Pediatr Infect Dis J. 1995;14:742–744

43. Wahdan MH, Serie C, Cerisier Y, Sallam S, Germanier R. A controlled field trial of live Salmonella typhistrain Ty21a oral vaccine against typhoid: three year results.J Infect Dis. 1982;145:292–296

44. Wenger JD, Heath PT, Moxon R, Booy R. Epidemiological impact of conjugate vaccines against invasive disease caused byHaemophilus in-fluenzaetype b. In: Levine MM, Woodrow GC, Kaper JB, Cobon GS, eds.

New Generation Vaccines. New York, NY: Marcel Dekker; 1997:489 –502 45. Adams WG, Deaver KA, Cochi SL, et al. Decline of childhood Haemophi-lus influenzaetype b (Hib) disease in the Hib vaccine era.JAMA. 1993; 269:221–226

46. Daum RS, Marcuse EK, Giebink GS, et al.Haemophilus influenzaetype b vaccines: lessons from the past.Pediatrics. 1988;81:893– 897

47. Gangarosa EJ, Galazka AM, Wolfe CR, et al. Impact of anti-vaccine movements on pertussis control: the untold story.Lancet. 1998;351: 356 –361

48. Thompson NP, Montgomery SM, Pounder RE, Wakefield AJ. Is measles vaccination a risk factor for inflammatory bowel disease?Lancet. 1995; 345:1071–1074

49. Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder

in children.Lancet. 1998;351:637– 641

50. Feeney M, Ciegg A, Winwood P, Snook J, East Dorset Gastroenterology Group. A case– control study of measles vaccination and inflammatory bowel disease.Lancet. 1997;350:764 –766

51. Chen RT, DeStefano F. Vaccine adverse events: casual or coincidental?

Lancet. 1998;351:611– 612

52. Thomas DR, Salmon RL, King J. Rates of first measles-mumps-rubella immunisation in Wales (UK).Lancet. 1998;351:1927

53. De Quadros CA, Olive JM, Hersh BS, et al. Measles elimination in the Americas. Evolving strategies.JAMA. 1996;275:224 –229

54. Watson JC, Redd SC, Rhodes PH, Hadler SC. The interruption of transmission of indigenous measles in the United States during 1993.

Pediatr Infect Dis J. 1998;17:363–367

55. Peltola H, Davidkin I, Valle M, et al. No measles in Finland.Lancet. 1997;350:1364 –1365

56. Peltola H, Heinonen OP, Valle M, et al. The elimination of indigenous measles, mumps, and rubella from Finland by a 12-year, two-dose vaccination program.N Engl J Med. 1994;331:1397–1402

(10)

DOI: 10.1542/peds.104.4.942

1999;104;942

Pediatrics

National Vaccine Advisory Committee

Lessons Learned From a Review of the Development of Selected Vaccines

Services

Updated Information &

http://pediatrics.aappublications.org/content/104/4/942 including high resolution figures, can be found at:

References

http://pediatrics.aappublications.org/content/104/4/942#BIBL This article cites 49 articles, 6 of which you can access for free at:

Subspecialty Collections

_sub

http://www.aappublications.org/cgi/collection/vaccine:immunization

Vaccine/Immunization

b

http://www.aappublications.org/cgi/collection/infectious_diseases_su

Infectious Disease

following collection(s):

This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtml in its entirety can be found online at:

Information about reproducing this article in parts (figures, tables) or

Reprints

(11)

DOI: 10.1542/peds.104.4.942

1999;104;942

Pediatrics

National Vaccine Advisory Committee

Lessons Learned From a Review of the Development of Selected Vaccines

http://pediatrics.aappublications.org/content/104/4/942

located on the World Wide Web at:

The online version of this article, along with updated information and services, is

by the American Academy of Pediatrics. All rights reserved. Print ISSN: 1073-0397.

Figure

TABLE 1.Activities of the Interactive Network of ComponentOrganizations and Institutions Involved in the Vaccine Develop-ment Process
TABLE 4.Vaccines Licensed by the FDA Between 1980 and1996
TABLE 5.A Comparison in the Time Between Identification of the Pathogen, Initiation of the First Clinical Trial, and the Year ofLicensure of the Vaccine by the FDA

References

Related documents

It is also revealed that traffic from the mobile nodes causes increase in the energy consumption, overall network throughput, average end-to-end delay and average jitter, compared

APRNs reported improved compliance with best practice recommendations including initial PDMP review prior to prescribing opioid medication for non-cancer, non-terminal pain,

The primary purpose of this quantitative study was to determine the perceived level of technology leadership preparation of Utah elementary principals using the International

Starr/Zapata Counties, Texas had no multiple-race reported. The most frequent race combinations reported were American Indian and White; Asian and White; Black and White; and

In this study, the Standardized Precipitation Evaporation Index (SPEI) and Normalized Difference Vegetation Index (NDVI) and Vegetation Condition Index (VCI), were applied

We then extend our approach to tackle the clothing parsing problem using a data-driven methodology: for a query image, we find similar styles from a large database of tagged

Electronic funds transfers (“EFTs”) are electronically initiated transfers of money involving a deposit account at the credit union and multiple access options including

Curriculum planners at some institutions have now started to see sophisticated electronic mapping as an opportunity to share the vision of the curriculum, as a tool to