• No results found

HOMELAND SECURITY STUDIES AND

N/A
N/A
Protected

Academic year: 2021

Share "HOMELAND SECURITY STUDIES AND"

Copied!
79
0
0

Loading.... (view fulltext now)

Full text

(1)
(2)

The Homeland Security Act of 2002 (Section 305 of PL 107-296, as codified in 6 U.S.C. 185) , herein referred to as the “Act,” authorizes the Secretary of the Department of Homeland Security (DHS), acting through the Under Secretary for Science and Technology, to establish one or more federally funded research and development centers or FFRDCs to provide independent analysis of homeland security issues. Analytic Services Inc. operates the Homeland Security Studies and Analysis Institute (HSsaI) as a FFRDC for DHS under contract HSHQDC-09-D-00003.

HSsaI provides the government with the necessary expertise to conduct: cross-cutting mission analysis, strategic studies and assessments, development of models that baseline current capabilities, development of simulations and technical evaluations to evaluate mission trade-offs, creation and evolution of high-level operational and system concepts, development of top-level system and operational requirements and performance metrics, operational analysis across the homeland security enterprise, and analytic support for operational testing evaluation in tandem with the government’s acquisition process. HSsaI also works with supports other federal, state, local, tribal, public and private sector organizations that make up the homeland security enterprise.

HSsaI’s research is undertaken by mutual consent with DHS and is organized by Tasks in the annual HSsaI Research Plan. This report presents the results of research and analysis conducted under

Task 09-01.03.02.13, System Readiness Assessment

of HSsaI’s Fiscal Year 2009 Research Plan. The purpose of the task is to (i) update and document a technology readiness level calculator; (ii) document a technology readiness assessment methodology, and (iii) explore how HSSAI can apply system readiness assessment approaches to support DHS in the future. This report presents the results of research and analysis conducted in fulfillment of (iii).

The results presented in this report do not necessarily reflect official DHS opinion or policy.

(3)

SECURITY STUDIES AND ANALYSIS INSTITUTE

D

EPARTMENT OF

H

OMELAND

S

ECURITY

S

CIENCE AND

T

ECHNOLOGY

R

EADINESS

L

EVEL

C

ALCULATOR

(

VER

1.1)

Final Report and User‟s Manual

September 30, 2009

Prepared for

Department of Homeland Security Science and

Technology Directorate

Dr. David McGarvey Mr. Jim Olson Dr. Scott Savitz Task lead Dr. Gerald Diaz Division Manager Mr. George Thompson Deputy Director

(4)

ii

A

CKNOWLEDGMENTS

This is a minor revision to a report published in 2008 by the Homeland Security Institute,

the predecessor organization to HSSAI. We wish to acknowledge, with thanks, the very

considerable contributions of Tavis Steenbeke, the co-leader of the task that lead to that report and of Megan McHugh and Laura Parker who were co-authors of that report.

We would like to extend our gratitude to former HSSAI member Ric Blacksten and to

HSSAI member Melanie Cummings for their talent, support, and dedication to various

aspects of this project.

We would also like to thank William Nolte and James Bilbro for sharing their insight and information about Technology Readiness Levels and the AFRL TRL Calculator (ver 2.2). It was an invaluable education for us, and critical to the development of the DHS S&T RL Calculator (Ver1.1).

Finally, we would like to thank Randy Long, Doug Drabkowski, Angela Ervin and Samuel Francis at the Department of Homeland Security, and Christopher Smith and his staff at the Transportation Security Laboratory, for working with us to develop user requirements for the modified calculator and providing feedback on the content of the DHS S&T RL Calculator.

For information about this publication or other HSSAIresearch, contact

Homeland Security Studies and Analysis Institute

Analytic Services Incorporated 2900 S. Quincy Street

Arlington, VA 22206

Tel (703) 416-3550 • Fax (703) 416-3530

http://www.homelandsecurity.org

HS

SA

I Publication Number: 09-01.03.02.13-02

This publication updates and supersedes

(5)

T

ABLE OF

C

ONTENTS

Executive Summary ... v

Introduction ... 1

Objective ... 2

Methodology and Scope ... 2

Background ... 3

Technology Readiness Levels (TRLs) ... 3

Modification of TRL scale ... 4

AFRL TRL Calculator (ver 2.2) ... 7

Overview of the DHS S&T RL Calculator (Ver1.1) ... 9

Summary and Recommendations ... 13

Appendix A: User’s Manual for the DHS S&T RL Calculator (Ver1.1) ... 15

Starting Up ... 15

For users with Excel 2003 and compatible versions of Excel ... 15

For users with Excel 2007 ... 15

Quick Start Instructions ... 16

Detailed Operating Instructions ... 16

DHS S&T RL Calculator Worksheets ... 16

START HERE Worksheet ... 16

RL Calculator Worksheet ... 18

Summary Report Worksheet ... 21

Appendix B: TRL, MRL, and PRL Definitions and Descriptions used in the DHS S&T RL Calculator (Ver1.1)... 23

Appendix C: Glossary of Terms for the DHS S&T RL Calculator (Ver1.1) and Acronyms ... 27

Acronyms ... 55

Glossary References ... 57

Appendix D: Decision algorithms, Change Log for the DHS S&T RL Calculator (Ver1.1), and Explanation of Administrative Functions ... 59

Decision Algorithms ... 59

Log of modifications ... 59

Administrative Functions ... 60

(6)

iv

L

IST OF

T

ABLES

TABLE 1:NASATRL SCALE ... 3

TABLE 2:DHSS&TRLCALCULATOR (VER1.1) HIGH-LEVEL TRL DEFINITIONS ... 10

TABLE 3:DHSS&TRLCALCULATOR (VER1.1) HIGH-LEVEL MRL DEFINITIONS ... 10

TABLE 4:DHSS&TRLCALCULATOR (VER1.1) HIGH-LEVEL PRL DEFINITIONS... 10

TABLE 5:TECHNOLOGY READINESS LEVEL DEFINITIONS AND DESCRIPTIONS ... 23

TABLE 6:MANUFACTURING READINESS LEVEL DEFINITIONS AND DESCRIPTIONS ... 24

TABLE 7:PROGRAM READINESS LEVELS DEFINITIONS AND DESCRIPTIONS* ... 26

TABLE 8:DHSS&TTECHNOLOGY READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ... 63

TABLE 9:DHSS&TMANUFACTURING READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ... 66

TABLE 10:DHSS&TPROGRAMMATIC READINESS LEVEL QUESTIONS FOR THE MODIFIED CALCULATOR ... 68

L

IST OF

F

IGURES

FIGURE 1:DHSS&TRLCALCULATOR STARTHERE WORKSHEET. ... 17

FIGURE 2:COLOR-CODE KEY FOR RL SCALES IN RLCALCULATOR AND SUMMARY REPORT WORKSHEETS. ... 18

FIGURE 3:GREEN AND YELLOW SET POINTS. ... 19

FIGURE 4:EXAMPLE OF TRL QUESTIONS FOR RL1. ... 19

FIGURE 5:EXAMPLE OF RL LAYOUT. ... 20

(7)

E

XECUTIVE

S

UMMARY

Technology Readiness Assessments (TRAs) are important procedures for organizations managing resource allocation for technology development programs. A metric commonly employed in TRAs for approximating the degree of maturity of a technology is the Technology Readiness Level (TRL) scale first developed by the National Aeronautics and Space Administration (NASA). This scale has been implemented and modified since the early 1990‘s in government programs. Ultimately, this work resulted in a calculator, the Air Force Research Laboratory (AFRL) TRL Calculator, that helps a user assess the TRL, Programmatic Readiness Level (PRL), and Manufacturing Readiness Level (MRL) of a given technology or system. This calculator is oriented in its terminology and structure to the Department of Defense (DoD) research, development, and acquisition process.

In 2008 the Homeland Security Institute (HSI), the predecessor to the HSSAI modified the existing AFRL

TRL Calculator for use within the Department of Homeland Security Science and Technology

Directorate. That new calculator, the Department of Homeland Security (DHS) Science and Technology (S&T) Readiness Level (RL) Calculator Version 1.0, allowed users to assess the RL of technology, as well as the RLs of manufacturing, and programmatics, independently of one another.

This document presents Version 1.1, an updated version of the calculator that is compatible with either Excel 2003 or Excel 2007. (Version 1.0 was developed using Excel 2003.) It revises and supersedes the previous Final Report and Users Manual. Although we believe the calculator is a useful tool for S&T Program Managers, its full potential is yet to be realized. Based on our research and analysis to date,

HSSAI recommends that DHS S&T take appropriate steps to

Develop entrance and exit criteria for each RL category. Develop guidelines for specific applications of the calculator. Continue refining concepts and terms specific to DHS S&T. Develop a DHS S&T-specific TRA methodology.

Validate DHS S&T RL Calculator results.

Apply lessons learned by other organizations to modification of RL methodologies.

As described in the body of the report, these efforts will entail a combination of research and analysis (e.g., identifying new questions and methods) and management actions (e.g., developing a management directive to govern the use of TRAs in S&T processes).

(8)
(9)

I

NTRODUCTION

Technology Readiness Assessments (TRAs) are established tools used to qualify technology development and help make investment decisions within those programs in order to deploy systems or elements of technology to an end user in a timely fashion. The National Aeronautics and Space Administration (NASA) and the Department of Defense (DoD) have taken the lead among government agencies in incorporating TRAs into their technology development programs and refining the TRA process

specifically to their organizations in order to produce operational systems on schedule and within budget. TRAs are vital to the process of maturing technologies to the point where they can be operationally produced and deployed. They support that process by

Providing metrics for technology maturity. TRAs help guide evaluation and tracking of technology maturity levels and program milestones.1, 2

Identifying risk associated with technologies and investment requirements. TRAs help inform decisions associated with allocating resources and funds for a given technology development. Identifying potential problems early in a development process when solutions are less expensive and easier to execute. TRAs provide a systematic method for ensuring the success of a project by tracking completion of various steps as a project develops.

Identifying gaps in testing, demonstration, and knowledge of a technology‘s current readiness

level and the information and steps necessary to reach the required technology readiness level.2

In the DoD, TRAs are a requirement of all acquisition programs. The DoD Technology Readiness Assessment Deskbook defines TRAs as ―a systematic, metric-based process and accompanying report that assesses the maturity of certain technologies.‖1

Although the Department of Homeland Security (DHS) Science and Technology Directorate (S&T) does not have acquisition programs, per se, it does have the requirement to identify and develop technologies for use in homeland security.

A metric commonly employed in TRAs for approximating and summarizing the degree of maturity of a technology is the Technology Readiness Level (TRL) scale first developed by the National Aeronautics and Space Administration (NASA) and since adapted and adopted by the Department of Defense (DoD) and other agencies. The TRL scale begins at 1 (―basic principles observed and reported‖) and goes through 9 (―actual system flight proven through successful mission operations‖). While a TRA does much more than assign a TRL, the TRL provides a useful summary index.

Technology Readiness Levels (TRLs) can facilitate a cost-effective, systematic process of transitioning technology from development to an operational environment by identifying a program‘s technical risk in areas such as design, architecture, cost, schedule, and manufacturing. As a result, TRLs can be a useful tool for DHS Program Managers (PMs) and other decision makers in assessing technology development programs.

1

Deputy Under Secretary of Defense for Science and Technology (DUSD(S&T)). (May 2005). Technology Readiness Assessment (TRA) Deskbook.

2 U.S. Department of Energy, Office of Environmental Management. (March 2008). Technology Readiness Assessment (TRA)/Technology Maturation Plan (TMP) Process Guide.

(10)

2

In 2001, William Nolte and coworkers at the Air Force Research Laboratory (AFRL) developed a TRL Calculator tool to aid in assigning and documenting TRLs in technology development programs. This calculator was used to help create a standard repeatable method for determining the TRLs. Initially released in January 2002, it is a Microsoft Excel workbook that presents a user with a series of questions to assess the maturity or readiness level (RL) of the technology. Because TRLs by themselves do not provide a full picture of risk associated with a program/project or the difficultly required to advance a program/project, Nolte further refined his calculator by May 2004 to include Manufacturing Readiness Levels (MRLs) and Programmatic Readiness Levels (PRLs) (AFRL TRL Calculator (ver 2.2)).

Objective

In 2008, the Homeland Security Institute (HSI) adapted the AFRL TRL Calculator for use within the DHS S&T Directorate by PMs or others interested in tracking the progress of technology development. The objective was to improve the transition of technologies into and out of development within S&T, provide more uniformity in program management, improve the documentation of technology readiness decisions, and facilitate development of TRA methodologies, including entrance and exit criteria for a given maturity level throughout the development life cycle of a technology within a DHS program.

Version 1.0 of the DHS S&T Readiness Level (RL) Calculator was released in December 2008. 3 It was

developed using Excel 2003.

In 2009, HSI‘s successor—the HSSAI–revised the calculator. Version 1.1 is now compatible with Excel

2007. The revised User‘s Manual provides instructions for use with either version of Excel. In addition, the revised Final Report updates the Institute‘s recommendations for continued development and use of the calculator.

Methodology and Scope

The DHS S&T RL calculator was originally developed as follows. Using open-source information, HSI analysts gathered information specific to TRL scales and historical modifications of those scales. They met with various DHS S&T stakeholders to facilitate development of end user requirements, and then attempted to clarify ambiguous RL questions, define all terms used in a DHS-specific orientation, and apply the lessons learned from historical modifications. As a result, the modified calculator agreed with DHS S&T terminology and documented management procedures. In addition, modifications included advanced user functionality, such as the ability to update the calculator‘s questions as new DHS lexicon and/or management processes are adopted and generate RL-specific reports for DHS PM analysis. How RL categories impact one another, however, or the importance of various questions in each of the RLs was not addressed in the modified calculator. Also outside the scope of this task of calculator

development were RL entrance and exit criteria.

Potential users were assumed to reside within the Chemical/Biological Division (CBD) of DHS S&T, although it was anticipated that the product would be useful in other divisions tasked with technology development.

3

McGarvey, D. et.al. (31 December 2008). Department of Homeland Security, Science and Technology Directorate, Readiness Level Calculator (HSI Publication 08-02.01.04-01). Version 1.0 of the calculator was developed under contract W81XWH-04-D011, ―Homeland Security Institute.‖ The Homeland Security Institute was established in 2004 as DHS‘s Federally Funded Research and Development Center (FFRDC) for homeland security studies and analyses. In March 2009, DHS established a successor organization, the Homeland Security Studies and Analysis Institute (HSSAI).

(11)

Background

Technology Readiness Levels (TRLs)

TRLs were originally developed by NASA. In the early 1980s, NASA observed delays and costs overruns of its programs. Analysis by Werner Gruel, NASA comptroller, concluded that immature technologies resulted in unpredictable development costs and schedules, and the resultant cost growth and schedule

slip.4 To address this cause-and-effect, the NASA instituted TRLs, a methodical system that provides a

consistent framework for assessing technology maturity and maintaining cost and schedule within a

program.5 TRLs are relevant to both simple and complex technological systems, as well as to their

component subsystems. They are applicable to software and/or hardware or to systems encompassing both software and hardware elements. TRLs can also be used as exit criteria for program life cycle phases. In terms of program management, knowing a program‘s TRL can prescribe an action plan of activities that still need to be accomplished in order to facilitate transition for a particular technology to an operational end user.

TRLs provide measures that can indicate a program‘s risk and potential for success of transitioning a technology to an end user, in other words, the value of investment. For example, a low TRL, e.g. TRL 1, represents high programmatic risk because the technology requires investment in more developmental milestones; while a high TRL, e.g., TRL 7, indicates the technology has matured further in development, achieved more milestones, and is associated with less programmatic risk.

By the 1990s, TRLs had been modified to address nine readiness levels used across NASA today. The NASA TRL levels and definitions are included below in Table 1: NASA TRL scale.

Table 1: NASA TRL scale5 NASA

TRL

Definition 1 Basic principles observed and reported

2 Technology concept and/or application formulated

3 Analytical and experimental critical function and/or characteristic proof-of-concept 4 Component and/or breadboard validation in laboratory environment

5 Component and/or breadboard validation in relevant environment

6 System/subsystem model or prototype demonstration in a relevant environment (ground or space) 7 System prototype demonstration in a space environment

8 Actual system completed and ―flight qualified‖ through test and demonstration (ground or space) 9 Actual system ―flight proven‖ through successful mission operations

In 1999, the DoD adopted the use of TRLs during its acquisition phase to aid in the decisions made

during technology development.6,7

4 Bilbro, J. JB Consulting International-Technology Readiness Levels. Last accessed on 12/19/08 at http://jbconsultinginternational.com/TechnologyReadinessLevel.aspx,

5 Mankins, J. (April 6, 1995). Technology Readiness Levels—A White Paper. Advanced Concepts Office, Office of Space Access and Technology, NASA.

(12)

4

Modification of TRL scale

Because the TRL scale is applicable to many different technologies and can be interpreted or modified for specific types of technologies, several groups have adapted TRL definitions for their own use. The Army has developed a mapping of the TRLs to software and the Army Medical Research and Materiel

Command has defined corollaries for biomedical TRLs.8 In addition, TRL scales have since been adopted

internationally in Canada, the United Kingdom, and Japan.

The TRL scale, however, only provides a snapshot of the maturity of a technology or system at a given

point in time. 9 In fact, in an ongoing development program, the technology will invariably be in constant

flux. And depending on the type of technology, the TRL may decrease or increase as a result of availability of components, changes in funding, or mission directives. Other shortcomings of the TRL

scale, pointed out by Jim Smith,10 include blurring contributions to readiness (e.g., how programmatics

influence TRLs), product criticality (e.g., an increased importance for developing a technology may push necessity for skipping TRLs or assuming completion of steps to reach self-imposed or mandated

deadlines), software aging (e.g., passé or critical software components of systems may require updating, impacting the overall systems TRL), and readiness in context (e.g., who is really looking at the RL and why?).

In general, readiness is a measure of the suitability of a technology or product for use within a larger system in a particular context. It is also a measure of the risks associated with developing or investing in a program associated with developing the technology or system. But TRLs do not provide a full assessment of the difficulty of integrating technology into an operational system, provide no guidance of the potential uncertainty in moving through the maturation of the technology, and offer no comparative analysis techniques for alternative TRLs.

Alternative methodologies for assessing these RLs are being developed. Some combine the desirable aspects of TRLs with additional readiness attributes, such as PRLs and MRLs, to better assess program risk. These new methodologies include developing evaluation criterion, or milestones, tailored to the context of the RL assessment. In short, improvements in existing methodologies and development of alternative methodologies to assess RLs is an ongoing field of study. In an effort to address readiness attributes not captured when using the NASA or DoD TRL scale, numerous types of readiness levels have been created since the inception of TRLs, including MRLs, PRLs, Integration Readiness Levels (IRLs), and System Readiness Levels (SRLs). Further, some of these new methodologies suggest assessing a combination of scales in order to generate a more accurate picture of a technology‘s maturity.

Integration Readiness Levels (IRLs)

The existing TRL scale does not accurately address integration of a component technology into a

complete system. In other words, component technologies may advance at different speeds along the TRL

7 Sauser, B., et.al. (April 7-8, 2006). From TRL to SRL: The Concept of System Readiness Levels. Conference on Systems Engineering Research, Los Angeles, CA.

8 Graettinger, C. et.al. (August 2002). Using the ―Technology Readiness Levels‖ Scale to Support Technology Management in the DoD‘s ATD/STO Environments: A Findings and Recommendations Report by the Software Engineering Institute Conducted for Army CECOM. Carnegie Mellon Software Engineering Institute, Pittsburgh, PA.

9 Nolte, W. Technology Readiness Level Calculator (ver 2.2).

10 Smith, J. (2005). An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. Proceedings of the 38th Hawaii

(13)

scale, however, as a system, an adequate TRL is difficult to assess. Developers of IRLs contend most development of complex systems fail at these ―integration points.‖ IRLs were developed for an accurate

assessment of interface maturity between developing technologies.7

At this time IRLs are not formally a part of the DHS acquisition process, and hence are not included in the DHS S&T RL Calculator.

System Readiness Levels (SRLs)

TRLs most accurately apply to individual technologies or system components. It becomes more complex to apply the existing TRL scale to a system. SRLs indicate the level of maturity applied at the system-level. SRLs are determined by using the current concept of TRLs combined with IRLs. The SRL of a given system is a function of individual component TRL maturities and the links between them, as

indicated by the IRL.7 SRLs are useful when going from the individual technology to a system context

which may involve multiple technologies, as is the case with most technologies in the operational environment.

At this time SRLs are not formally a part of the DHS acquisition process, and hence are not included in the DHS S&T RL Calculator.

Manufacturing Readiness Levels (MRLs)

In 2003, the Government Accountability Office (GAO) recommended in GAO Report 03-476, establishing cost, schedule, and quality targets for product manufacturing early on in technology

development in order to obtain process maturity.11 The report suggests that design and manufacturing

knowledge should be obtained early in product development for a product to be successful. In response, the Joint Defense Manufacturing Technology Panel developed MRL definitions as well as Manufacturing

Readiness Assessments (MRAs)12. This MRL scale helps program managers assess manufacturing risks,

which will facilitate identification of areas that require additional management attention or investment. ―Manufacturing readiness and producibility are as important to the successful development of a system as are readiness and the capabilities for the system.‖1

Though MRLs were created from the manufacturing perspective to evaluate ―manufacturing readiness‖ of a product and supplement existing TRLs, they, too, have limitations. A limitation of the MRLs is that the lower MRL levels can be difficult to correlate to corresponding TRL numbers due to the technology immaturity (i.e., it is difficult to know what types of manufacturing steps are required when a technology concept hasn‘t yet been proven). As a result of these limitations, MRL levels 1 and 2 are not used in the DHS Readiness Level Calculator.

Programmatic Readiness Levels (PRLs)

PRLs were developed to address program management concerns, such as documentation of programmatic

milestones seen as vital to successful technology product development.13 A PRL scale was developed by

HSI, in 2008, to align with the TRLs. The PRL scale follows basic systems engineering steps and is discussed further in the following sections.

11 GAO Report 03-476. (May 2003). Defense Acquisitions, Assessments of Major Weapons Programs.

12 Joint Defense Manufacturing Technology Panel Manufacturing Readiness Level Working Group. (February 2007). MRL Guide. 13 Nolte, W. (October 20, 2003 ). Technology Readiness Level Calculator. NDIA Systems Engineering Conference.

(14)
(15)

AFRL

TRL

C

ALCULATOR

(

VER

2.2)

As described in the Introduction, William Nolte of the AFRL developed a TRL Calculator in 2001 to create a standard, repeatable method for determining TRLs. This original TRL calculator (ver 1.0), released in January 2002, is a Microsoft Excel spreadsheet application that presents a user with a series of questions by TRL about the technology. The methodology for the TRL assessment used in the calculator was refined by May 2004, to include questions relating to TRLs, MRLs, and PRLs (TRL Calculator (ver 2.2). The ―user‖ has the option of assessing the ―overall TRL‖ based on three combinations of questions: 1) only TRL questions,

2) TRL questions and PRL or MRL questions, or

3) all three categories of questions together. In any case, TRL questions are always required.14

The AFRL TRL Calculator (ver 2.2) calculates the overall TRL of the technology or system in question by averaging the responses to all selected categories in a RL. In other words, PRLs and MRLs are ultimately additional TRL questions, if included. If the user cannot answer those questions associated with manufacturing, for example, the overall TRL of the technology or system for that RL will be low, even if the TRL questions alone can be adequately addressed.

The user always has the option of looking at the results if only the technology readiness questions are considered, but not the option of looking at results for only manufacturing or programmatic readiness.

(16)
(17)

O

VERVIEW OF THE

DHS

S&T

RL

C

ALCULATOR

(V

ER

1.1)

The DHS S&T RL Calculator Version 1.0 was developed by HSI, in 2008, for the DHS S&T Chem/Bio Division PMs and others interested in assessing technology and associated programs. HSI facilitated developing end user requirements with DHS S&T potential users.

Like its predecessor, Version 1.1 of the calculator provides the user options to assess Technological,

Programmatic, and/or Manufacturing Readiness Levels (herein, referred to as categories) for a given

technology or system. The calculator allows a user to assess any or all of the RLs required for successful technology development and transition: TRLs, MRL, and PRLs. The user has the ability to do these

assessments separately, and at the same time. Each of these assessments is independent of the other. In

other words, it is essentially three calculators in one workbook. All of these categories have been defined along a scale of 1-9. (However, the lowest MRL a user can achieve or begin to address readiness is 3, since it is assumed that a technology or system at a TRL 1 or 2 does not allow for a meaningful ability to address manufacturability.)

In general, the levels can be grouped into three higher-level activities:

1. RLs 1-3: Research and Development (R&D): these activities most likely occur in a basic

laboratory setting, prior to identification of a sponsoring organization.

2. RLs 4-6: Technology Demonstration: these activities occur as a result of funding provided by

a sponsoring organization, such as DHS S&T.

3. RLs 7-9: Production and Deployment: these activities occur once the technology has been

transferred from the sponsoring organization to the customer or end user.

Deviations from this grouping occur. For instance in some cases systems are taken into development when some of the required technologies are only at TRL 2 and in some cases the sponsoring organization supports development through TRL 7 or higher before transition to the customer or end user.

These basic groups begin to show how all three RL categories (technology, manufacturing,

programmatic) are intimately linked, generally requiring information and interaction with one another. How RL categories impact one another or the importance of various questions in each of the RLs are not addressed in the DHS S&T RL Calculator. Nor does the calculator address the question of what TRL, MRL, or PRL levels are required for different entrance and exit criteria. This aspect is left for future modifications based on DHS S&T programmatic directives.

As with the AFRL TRL Calculator (ver 2.2), the DHS S&T RL Calculator (Ver1.1) is an Excel

workbook.15 The user specifies what categories of RLs are to be calculated (TRL, MRL, or PRL

separately, or any combination of these). In addition to a visual scale for each category that indicates the readiness levels achieved and not achieved, this calculator also has the capability to generate category-specific summary reports. Each summary report details each RL and the responses provided by the user. These reports are intended to facilitate discussions about the results, in addition to preparing a PM for next steps in order to progress to the next readiness level.

15

Two variants are provided, one for use with Excel 2003 and compatible versions of Excel and one for use with Excel 2007.

(18)

10

The following tables provide the top-level definition for each level in the RL categories (as modified by HSI) (Tables 2-4). Definitions of specific terms can be found in Appendix C: Glossary of Terms. Full explanations of the RLs are provided in Appendix B: TRL, MRL, and PRL Definitions and Descriptions used in the DHS S&T RL Calculator (Ver1.1).

Table 2: DHS S&T RL Calculator (Ver1.1) high-level TRL definitions

TRL TRL Definition

1 Basic principles observed and reported.

2 Technology concept and/or application formulated.

3 Analytical and experimental critical function and/or characteristic proof-of-concept. 4 Component and/or breadboard validation in laboratory environment.

5 Component and/or breadboard validation in relevant environment.

6 System/subsystem model or prototype demonstration in a relevant environment. 7 System prototype demonstration in an operational environment.

8 Actual system completed and qualified through test and demonstration. 9 Actual system proven through successful mission operations.

Table 3: DHS S&T RL Calculator (Ver1.1) high-level MRL definitions

MRL MRL Definition

3 Manufacturing process development. 4 Critical manufacturing processes prototyped. 5 Prototype manufacturing system.

6 Manufacturing process maturity demonstration. 7 Manufacturing processes proven.

8 Manufacturing concepts identified.

9 Laboratory manufacturing process demonstration.

Table 4: DHS S&T RL Calculator (Ver1.1) high-level PRL definitions

PRL PRL Definition

1 Identification of basic scientific concepts and Performers.

2 Establishment of program with identified customer and technology.

3 Program risk, requirements, and performance characteristics and measures are determined.

4 Integrated Product Teams and working groups for developing and transitioning technology are established.

5 Systems engineering methodology, system architecture and end user involvement are established.

6 Formal requirement documents, final Test and Evaluation Master Plan, and Systems Engineering Plan are complete.

7 Finalized Verification, Validation and Accreditation of system. 8 Training and Test and Evaluation Documentation are complete. 9 Safety and Training is complete.

(19)

Appendix A provides operating instructions for the calculator. While there are several worksheets within the workbook, only two require a user to physically enter information in order to generate TRL, MRL, and PRL-specific summary reports: the Start Here and RL Calculator workbooks.

(20)
(21)

S

UMMARY AND

R

ECOMMENDATIONS

HSSAIhas modified the existing AFRL TRL Calculator for use within the DHS S&T. This new calculator,

the DHS S&T RL Calculator (Ver 1.1), allows users to assess the RLs of technology, manufacturing, and programmatic, independently of one another. This calculator is a step towards developing a standardized TRA methodology within DHS for assessing and tracking progress in technology development programs. While this calculator should be useful as a guide for Program Managers (PMs) and for conducting

program- and technology-related budget allocations, there is still work to be done. HSSAImakes the

following recommendations based on its efforts associated with the research into TRAs, RLs, and the resultant DHS S&T RL Calculator (Ver1.1).

1. Develop entrance and exit criteria for each RL category. Currently, the DHS S&T RL

Calculator has generic types of milestones presented as questions that have no more weight than a previous or following question. Further, a user can achieve any RL without achieving lower RLs because there is no requirement that the level below, or specific steps need to be achieved

first. HSSAIrecommends that certain questions be identified as mandatory for passing to the

next RL. In this way, PMs would be assured that all projects are addressing the same criterion for a given level and when a technology has been deemed RL 5, for example, have an

understanding of exactly what has been accomplished with that classification.

2. Develop guidelines for specific applications of the calculator. While this task focused on

DHS S&T Chem/Bio Division PMs, the results of an assessment may vary by user. A laboratory PM, for example, may require different information than a sponsoring organization (e.g.,

manufacturing questions may not impact a laboratory‘s information collection in RLs 1-3, but may impact a sponsoring organization‘s willingness to assume responsibility for a technology development).

3. Continue refining concepts and terms specific to DHS S&T. Having common understanding

on the meaning of terms is important for ensuring minimal confusion among those assessing RLs and those making policy decisions based on them.

4. Develop a DHS S&T-specific TRA methodology. TRAs historically have been modified to

apply to specific organizations. To facilitate understanding between DHS, and other

organizations (i.e., laboratories, manufacturers), a DHS TRA methodology should be vetted and made into a management directive. This would also allow a user to add questions specific to documented DHS programmatic milestones.

5. Validate DHS S&T RL Calculator results. Consider having an outside organization assess a

given project‘s RL using the calculator and compare the results to the same assessment

performed by DHS S&T PMs. Differences between the assessments would be valuable starting points for discussions and setting expectations.

6. Apply lessons learned by other organizations to modification of RL methodologies. TRLs

have shortfalls, For instance, they do not tell us what efforts have preceded or what efforts are expected to follow the achievement of the TRL. In addition, they do not account for system integration issues. There is precedence, however, for modifying the RL methodology in such a way that accounts for ―other factors‖ that should be considered, in order to more accurately understand what is required to move from one RL to the next or address integration issues. For example, James Bilbro developed the Advancement Degree of Difficultly (AD2) as another

(22)

14

assessment scale for technology maturity. AD2 is the description of what is required to move a system, subsystem, or a component from one TRL to another taking into account the cost, schedule, risk, people and tools available, and organizational aspects, such as the ability of an organization to reproduce existing technology.

(23)

A

PPENDIX

A:

U

SER

S

M

ANUAL FOR THE

DHS

S&T

RL

C

ALCULATOR

(V

ER

1.1)

Starting Up

There is no ―installation‖ of the DHS RL Calculator. The calculator is a Microsoft (MS) Excel workbook template. Two variants are provided. One, DHS RL Calculator for Excel 2003 1.1.xls is for use with Excel 2003 and some compatible versions of Excel. The other, DHS RL Calculator for Excel 2007 1.1.xlsm, is for use with Excel 2007.

For users with Excel 2003 and compatible versions of Excel

Use the version of the calculator titled DHS RL Calculator for Excel 2003 1.1.xls.

Before attempting to use the calculator, make sure that Excel is set to Medium security so that the macros in the workbook will run. To do this, open Microsoft Excel. Select "Tools", then "Macros" and

"Security." Pick the "Medium" radio button.

Open the calculator workbook - DHS RL Calculator for Excel 2003 1.1.xls. A dialog box will pop up asking whether or not you wish to allow macros to run. The default selection is "Disable Macros."

Change this to "Enable Macros" or the calculator won't work.

The calculator will be open to the START HERE worksheet.

For users with Excel 2007

Use the version of the calculator titled DHS RL Calculator for Excel 2007 1.1.xlsm.

When using Excel 2007 it is necessary that macros be enabled each time the calculator is opened.

To do this click on the ―Options‖ box next to the ―Security Warning Some active content has been disabled‖ warning that appears when the calculator is first opened. Choose ―Enable this content‖ from the options that appear.

(24)

16

Quick Start Instructions

1 Enter identification information (Start Here) 2 Select RL categories to be used (Start Here)

3 If desired, select level to begin RL questions for each RL from drop down scales (Start Here)

4 Click ―Continue to Calculator‖ 5 Save workbook with unique identifier

6 Set green and yellow set points (RL Calculator)

7 Begin answering questions until ready to generate summary report(s) 8 Save workbook with different name

9 Click ―Generate Report‖

10 Save workbook with yet another name 11 Print summary reports

Detailed Operating Instructions

The DHS S&T RL Calculator (Ver1.1) is an Excel workbook containing six worksheets visible to a user:

1. Start Here: This is the default opening worksheet. The user should enter identification

information, select RL categories and, if desired, the level for each category to begin assessing specific RLs on the next worksheet.

2. RL Calculator: The user should set threshold values for green and yellow set points and answer

questions until desired stopping point (see RL Calculator worksheet explanation below). Also, the user can generate reports from this worksheet.

3. Three (3) summary report worksheets: A RL summary report worksheet will only appear if

that RL category (TRL, MRL, and PRL) has been selected on the Start Here worksheet.

4. Glossary: Definitions for terms used throughout the calculator. This can be accessed from each

worksheet or by directly clicking on the tab. See Appendix C: Glossary of Terms.

DHS S&T RL Calculator Worksheets

START HERE Worksheet

It is essential that the user begin by selecting categories desired on the START HERE worksheet. Deviation from this will cause the calculator to function incorrectly. Figure 1 shows the Start Here worksheet.

On the START HERE worksheet the user should enter the Project Name, Project Manager, and Date the RL is being calculated in the spaces provided at the top of the worksheet.

(25)

Figure 1: DHS S&T RL Calculator START HERE worksheet.

Glossary: This button opens the glossary of terms worksheet. These terms are used in the calculator and/or related to concepts addressed in the questions. The glossary may be consulted at any point in the operation of the calculator.

Select technology types to be included: The user selects whether the project to be evaluated consists of hardware only, software only, or a combination of hardware and software.

(26)

18

Select types of RLs to evaluate: The user selects which types of RLs are to be included in the evaluation – Technology, Manufacturing, and/or Programmatic. For each RL selected, the user has the option of choosing a specific level to begin the questions. In addition, full definitions for each scale are provided. Assuming a given RL has already been achieved: The user also has the option of assuming that a given RL has been achieved for each of the RLs to be included in the evaluation. This allows the user to avoid answering the questions for that and lower RLs. Note: If the user decides to begin assessing RLs at a given level, all levels below the one selected will be assumed 100% completed for that category. The user will not be able to change this once ―Continue to Calculator‖ is clicked.

“Continue to Calculator”: Once the user is satisfied with the answers supplied on the START HERE

work sheet, but not before, the user must select the ―Continue to Calculator‖ button. This is an irreversible step that prompts the user to now save the workbook with a suitable name in a suitable location and takes the user to the TRL Calculator worksheet, which is the calculator proper. The newly named workbook can now be saved and opened at will while the questions on the TRL Calculator worksheet are being answered. Note: Once the file is saved after ―continue to calculator‖ is selected, the user cannot go back to the START HERE worksheet and change or enter new information to generate new reports without causing malfunction. The only way to change information on the START HERE page, at this point, is to completely start over by closing the document and reopening the template.

RL Calculator Worksheet

Upper portion of the worksheet

Program Name, Program Manager, and Date will have been copied from the START HERE worksheet. There are two buttons on the top of the worksheet that will be of interest only if parts of the bottom portion of the worksheet have been completed.

1. ―Start Over on this Page‖ button unassumes all Levels, unchecks all boxes, but does not show

any RL selected as omitted on the START HERE page.

2. ―Create Report‖ button generates RL summary reports of questions answered. See RL

Summary Reports below.

RL scales appear for each of the RL categories selected on the START HERE worksheet. Figure 2 shows the key for those scales based on the answers to the questions in the lower part of the worksheet (see detailed explanation in the following section, Lower portion of the worksheet).

Figure 2: Color-code key for RL scales in RL Calculator and Summary Report worksheets.

Colors are calculated based on two ―set points.‖ Set points are percentages of questions required by the user to be answered in order to achieve a ―green‖ or ―yellow‖ grade (Figure 3 shows default values of

Most, if not all, tasks required for this TRL have been achieved. This TRL has not been achieved.

(27)

100% and 75%, respectively). If a set of questions has not been answered to a combined total percentage of the yellow set point, the level will be red.

Figure 3: Green and yellow set points. In this example, the user must answer 100% of the questions for a given level to achieve “green.” If the user answers between 99.9% and 75%, the level will be yellow. Any percentage below 75 will show red.

Lower portion of the worksheet

For each RL category selected on the START HERE worksheet, the calculator then poses a set of questions for each level on its related scale of 1-9. See Appendix E for list of questions by category and level. Figure 4 provides an example of how the questions appear in the calculator.

Figure 4: Example of TRL questions for RL 1.

Questions are preceded by several data fields. Figure 5 shows these fields and explanations are provided below.

Apply? (Y/N): There is a drop-down menu here for ‗Y‘ or ‗N‘. The default setting is that all

questions apply. The user can decide that certain questions do not apply to the project/technology being evaluated. Entering an ‗N‘ in this field will remove a question from the evaluation (green ‗Y‘ turns to red ‗N‘). The question will still appear in the list, but any answer supplied will be ignored in the calculations. If ‗N‘ is selected for a question, it will also appear on the appropriate RL Summary Report worksheets in a field named ―Questions marked Not Applicable.‖.

H/SW Both: In this field, ‗H‘ indicates hardware, ‗S‘ indicates software, and ‗B‘ indicates both

hardware and software. The user should not try to change these.

Ques Catgry: In the Question Category field, ‗T‘ indicates a technology question, ‗M‘ indicates

manufacturing, and ‗P‘ indicates programmatic. The user should not try to change these.

% Complete: The user ―answers‖ each question according to a percentage of 100 completed. For

example, if the question is successfully answered as ‗yes‘, the user will check the box that indicates 100%. If the answer is a partial ‗yes‘, but not 100%, a percentage of 100 can also be inserted manually by changing the green 100 or by using the sliding bar associated with the question. Default for all questions is ‗no‘ or 0%. Answers provided will appear in the Summary

Report. The green cell displays 100 by default and is technically counted as “0%” or not

completed unless the checkbox is manually clicked or a different value is added.

Do you want to assume completion checkbox: Above the questions for each level there is a red

check box that allows the user to assume completion of this RL. Note: This box applies to all the RLs being considered at that level (technology, manufacturing, and/or programmatics). The user

Green Set Point:

100%

Yellow Set Point:

75%

Both Catgry B T 0 B T 0 S T 0 S T 0 B T 0 S T 0 S T 0

Have mathematical formulations of concepts that might be realizable in software been developed? Have the basic principles of a possible algorithm been formulated?

Are physical laws/assumptions for new technology defined? Do rough calculations support the concept?

% Complete

Do paper studies confirm basic scientific principles of new technology? Know what software needs to do in general terms?

Level 1 (Check all that apply or use slider for % complete)

(28)

20

cannot assume completion of individual RLs on this page, only on the START HERE worksheet using the drop-down scales.

Figure 5: Example of RL layout.

The background of the questions is color-coded to indicate whether the question refers to technology, manufacturing, or programmatics, information that is also included in the Ques Catgry field (Blue: Technology, Green: Programmatic, and Pink: Manufacturing).

RL Questions

Questions should be answered beginning with those at the lowest RL and proceeding down the page. As the questions are being answered the calculator continually recomputes the RLs achieved and displays the results in the scales in the upper portion of the worksheet.

At its core, the calculator sums responses to questions for a given category and RL from the list of questions and follows a few pre-defined rules: summation of all questions answered ―yes,‖ some percentage of ―yes,‖ ―not applicable,‖ and ―not answered.‖ As the questions posed are answered, the calculator displays a red, green, yellow color-coded scale associated with each category of question (this will appear in the upper section of the worksheet).

The calculator contains two user-defined thresholds or ―set points‖: green and yellow. As questions are answered, the calculator displays a color-coded scale for that RL according to the set points.

The calculator displays a ―green‖ status for a given RL if the percentage of questions answered equals or exceeds the green set point. The default setting for the green set point is 100% (i.e., each question for a given category must be answered 100% to achieve green), but this can be changed by the user.

The calculator displays a ―yellow‖ status for a given RL if the percentage of questions answered equals or exceeds the yellow set point, but is below the green set point. The default setting for the yellow set point is 75% (i.e., the combined totals of all the questions in that category for that RL must equal 75% or greater to achieve yellow). NOTE: The calculator has been modified to calculate the portion of 100% each question contributes. No questions within a category carry more value than another. For example, if there are 10 technology questions for RL 1, each question has a value of 10% of the total. If a percentage of 100 is inserted for a given question, that percentage is multiplied by the value of that question which is linked to the total number of questions for that category and RL. Answered questions are automatically summed for that RL in real time. If the total percentage of combined answers meets or exceeds a given

Ques Apply? H/SW Ques (Y/N) Both Catgry

Y H P 100 Y H M 100 Y B P 29 Y B P 100 Y H M 100 Y B P 100 Y B P 100 Y H M 100 Y H M 50 Y B P 100 Y B P 100 Y B P 100 Y B P 100

Is customer participating in requirements generation?

Has customer representative to work with R&D team been identified?

Have programmatic risk mitigation strategies been documented? Have design techniques been identified and/or developed?

Has preliminary value analysis been performed? Have programmatic risks been identified?

Can key components needed for breadboard be produced? % Complete

Does basic laboratory research equipment verify physical principles?

Readiness Level 3 (Check all that apply or use slider for % complete)

Have scaling studies been started?

Have system performance characteristics and measures been documented?

Have current manufacturability concepts been assessed? Has analysis of alternatives been completed?

Has Technology Transition Agreement (TTA) including possible TRL for transition been drafted? Do you want to assume completion of Level 3?

(29)

threshold, it will display the appropriate color. If the total does not meet either green or yellow ranges, the scale will display red for that RL.

Green indicates a satisfactory completion of the given level. Yellow indicates a percentage lower than green, and red indicates the user has not completed the level to a satisfactory degree. Questions marked ―not applicable‖ do not count against this calculation.

NOTE: If the user wanted to start at TRL 3 on the START HERE worksheet, then all questions pertaining to Technology at levels 2 and below would automatically be answered ―yes‖ (and would be depicted as green on the scale related to that RL as shown below for TRL SCALE). In addition, if more than one category was selected on the START HERE page, also shown in the example below, TRL questions for levels 1 and 2 will be automatically checked 100%.

Figure 6: Example of real-time color coded readiness levels. In the example provided, there is a TRL of 2, an MRL of 3, and a PRL of 2.

„Comments‟ text box: The user has the option to enter information concerning the RL under

consideration in this text box. These comments will automatically show up on the Summary Report worksheets.

“Generate Report”: At any time during the process of answering the questions the user may press the

Create Report button on the upper portion of the worksheet to generate reports of the status of each RL under consideration. Pressing this button will open a spreadsheet showing the status of one of the RLs under consideration. Responses in the RL Calculator worksheet can be changed, however, at a later date by saving the file once before generating a report and once after with different names. If the user desires to change or add responses, he/she should open the file saved before generating reports and go to RL Calculator worksheet after opening the file (worksheet tab at the bottom of the document).

Summary Report Worksheet

At the top of a report there is a summary of the RL achieved or partially achieved, A list of any question marked as Not Applicable will be in the next section of the report. These question are listed regardless of the level achieved or the RL of the report.

Lists for each readiness level of all the questions for this level are sorted in descending order by the degree of completion provided appear next. Those answered as completed to meet the green standard are presented with a green background; those not meeting the green standard but satisfying the yellow

standard are presented with a yellow background, and those not meeting the yellow standard are presented with a red background. If comments have been added on the RL Calculator worksheet, they will show up under each associated RL summary. The user has the option to edit or add to this text. Finally, each RL

1 2 3 4 5 6 7 8 9 G G 1 2 3 4 5 6 7 8 9 G Y 1 2 3 4 5 6 7 8 9 G G Y

PRL SCALE both Hardware and Software TRL SCALE both Hardware and Software

(30)

22

will show the percentage of questions answered. The user may print copies of these reports. (Use Print Preview to adjust the printout).

(31)

A

PPENDIX

B:

TRL,

MRL,

AND

PRL

D

EFINITIONS AND

D

ESCRIPTIONS USED IN THE

DHS

S&T

RL

C

ALCULATOR

(V

ER

1.1)

Table 5: Technology Readiness Level Definitions and Descriptions

Level TRL Definitions DoD TRL Level Descriptions* DHS TRL Descriptions

Re search an d Dev el opment 1

Basic principles observed and reported.

Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology's basic properties.

Scientific research begins the first steps toward applied research and development. Examples include paper studies of a technology‘s basic properties, exploration of a technical phenomenon, and definition of a technical concept. This level represents the origin of technology readiness.

2

Technology concept and/or application formulated.

Invention begins. Once basic principles are observed, practical applications can be invented.

Applications are speculative and there may be no proof or detailed analysis to support the assumptions. Examples are limited to analytic studies.

Once basic principles are observed and proven repeatable, practical applications can be formulated. Applications are speculative and there may be no proof or detailed analysis to support the assumptions. Examples are limited to analytic studies, device phenomenology, and experimentation.

3

Analytical and experimental critical function and/or characteristic proof of concept.

Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative.

Active research and development is initiated. This includes analytical and laboratory studies to physically validate analytical predications of separate elements of the technology. Examples include components that are not yet integrated.

Test ing a nd De monstrati on 4

Component and/or breadboard validation in laboratory environment.

Basic technological components are integrated to establish that they will work together. This is relatively "low fidelity" compared to the eventual system. Examples include integration of "ad hoc" hardware in the laboratory.

Basic technological components are integrated to establish that they will work together. Examples include integration of modules and components in the laboratory.

5

Component and/or breadboard validation in relevant environment.

Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so it can be tested in a simulated environment. Examples include "high fidelity" laboratory integration of components.

The basic technological components are integrated with reasonably realistic supporting elements so it can be tested in simulated environment. Examples include ―high-fidelity‖ laboratory integration of

components and software.

6

System/subsystem model or prototype demonstration in a relevant

environment.

Representative model or prototype system, which is well beyond that of TRL 5, is tested in a relevant environment. Represents a major step up in a technology's demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in simulated operational environment.

Representative model or prototype system is tested in a relevant environment. Represents a major step up in a technology‘s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated operational environment.

7

System prototype demonstration in an operational environment.

Prototype near, or at, planned operational system. Represents a major step up from TRL 6, requiring demonstration of an actual system prototype in an operational environment such as an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft.

Prototype near, or at, planned operational system level. Represents a major step up from TRL 6, requiring demonstration of an actual system prototype in an operational environment.

Pr od uctio n and Dep loyment 8

Actual system completed and qualified through test and demonstration.

Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications.

Technology had been proven to work in its final form and under expected operational deployment conditions. In almost all cases, this TRL represents completion of system development. Examples include test and evaluation of the system in its intended system configuration and operational requirement.

9

Actual system proven through successful mission operations.

Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluation. Examples include using the system under operational mission conditions.

Actual application of the technology in its final form and under mission conditions, in accordance with the user‘s Concept of Operations.

(32)

24

Table 6: Manufacturing Readiness Level Definitions and Descriptions

Level MRL Definitions Original DoD MRL Descriptions* AFRL Calculator MRL Descriptions¥ DHS MRL Descriptions

Re search and Dev el opment 1* Manufacturing feasibility assessed.

This is the lowest level of manufacturing readiness. The focus is on a top level assessment of feasibility and manufacturing shortfalls. Basic manufacturing principles are defined and observed. Begin basic research in the form of studies (i.e. 6.1 funds) to identify producibility and material solutions.

n/a n/a

2*

Manufacturing concepts defined.

This level is characterized by developing new manufacturing approaches or capabilities. Applied Research translates basic research into solutions for broadly defined military needs. Begin demonstrating the feasibility of producing a prototype product/ component with very little support/data available. Typically this level of readiness is associated with Applied Research (i.e. 6.2 funds) in the S&T environment and includes identification and study of material and process approaches, including modeling and simulation.

n/a n/a

3

Manufacturing concepts identified.

This begins the first real demonstration of the manufacturing concepts. This level of readiness is typical of technologies in the S&T funding categories of 6.2 and 6.3. Within these levels, identification of current manufacturing concepts or producibility has occurred and is based on laboratory studies. Materials have been characterized for manufacturability and availability but further evaluation and demonstration is required. Models have been developed in a lab environment that may possess limited functionality.

Assessment of current manufacturability concepts or producibility needs for key breadboard components.

This begins the first real demonstrations of the manufacturing concepts. Identification of current manufacturing concepts or producibility has occurred and is based on laboratory studies. Key components needed for breadboards can be produced.

Test ing a nd De monstrati on 4 Laboratory manufacturing process demonstration.

This level of readiness is typical for S&T Programs in the 6.2 and 6.3 Advanced Development categories, and acts as an exit criteria for the Concept Refinement (CR) phase approaching a Milestone (MS) A decision. This level indicates that the technologies are ready for the Technology Development phase of acquisition.

Technologies should have matured to at least TRL 4. At this point, required investments, such as manufacturing technology development have been identified; processes to ensure manufacturability, producibility and quality are in place; and manufacturing risks have been identified for prototype build. Manufacturing cost drivers have also been identified. Producibility assessments of design concepts have been completed. Design Key Performance Parameters (KPPs) have been identified as well as any special needs for tooling, facilities, material handling and skills.

Key processes identified and assessed in lab. Mitigation strategies identified to address manufacturing/producibility shortfalls. Cost as an independent variable (CAIV) targets set and initial cost drivers identified.

Technologies have matured sufficiently to determine required manufacturing technology development investments;

processes to ensure manufacturability, producibility and quality are in place; and manufacturing risks have been identified for prototype build. Key manufacturing processes have been identified, assessed in the laboratory and potential manufacturing problems have been documents.

5

Manufacturing process development.

This level of maturity is typical of the mid-point in the Technology Development phase of acquisition, or in the case of key technologies, near the mid-point of an ATD program. Technologies should have matured to at least TRL 5. The Industrial Base has been assessed to identify potential manufacturing sources. A manufacturing strategy has been refined and integrated with the Risk Management Plan. Identification of enabling/critical technologies and components is complete. Prototype materials, tooling and test equipment, as well as personnel skills have been demonstrated on components in a production relevant environment, but many manufacturing processes and procedures are still in development. Manufacturing technology development efforts have been initiated or are ongoing. Producibility assessments of key technologies and components are ongoing. A cost model has been constructed which is based upon a detailed end-to-end value stream map.

Trade studies and lab experiments define key manufacturing processes and sigma levels needed to satisfy CAIV targets. Initial assessment of assembly needs conducted. Process, tooling, inspection, and test equipment in development. Significant engineering and design changes. Quality and reliability levels not yet established. Tooling and machines

demonstrated in lab. Physical and functional interfaces have not been completely defined.

The Industrial Base has been assessed to identify potential manufacturing sources and yields. Identification of

enabling/critical technologies and components is complete and prototypes have been created. Producibility assessments of key technologies and components are ongoing and tooling and machines have been demonstrated in the lab. A

manufacturing strategy has been refined and a risk management plan has been incorporated. A cost model has been constructed.

6

Critical manufacturing processes prototyped.

This MRL is associated with readiness for a MS B decision to initiate an acquisition program by entering into the System Development and Demonstration (SDD) phase of acquisition. It is normally seen as the level of manufacturing readiness that denotes completion of S&T development and acceptance into a baseline system design. Technologies should have matured to at least TRL 6. An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined and characterized, but there are still significant engineering and/or design changes. However, preliminary design of critical components has been completed so that producibility assessments of key technologies can be complete. Prototype materials, tooling and test equipment, as well as personnel skills have been demonstrated on systems and/or subsystems in a production relevant environment. Detailed cost analysis includes design trades, and all cost targets have been allocated. Producibility considerations shape system development plans. Industrial Capabilities Assessment (ICA) for MS B has been completed and long lead and key supply chain elements are identified.

Critical manufacturing processes prototyped, targets for improved yield established. Process and tooling mature. Frequent design changes still occur. Investments in machining and tooling identified. Quality and reliability levels

identified. Design to cost goals identified.

An initial manufacturing approach has been developed. The majority of manufacturing processes have been defined, characterized and are available, but there are still significant engineering and/or design changes. Producibility assessments of key technologies are complete. Production demonstrations on systems/subsystems in a relevant environment are complete to include prototype materials, tooling and test equipment, and personnel skills.

7

Prototype

manufacturing system.

This level of manufacturing readiness is typical for the mid-point of the System Development and

Demonstration (SDD) Phase leading to the Design Readiness Review. Technologies should be maturing to at least TRL 7. System detailed design activity is underway. Material specifications have been approved and materials are available to meet the planned pilot line build schedule. Manufacturing processes and procedures have been demonstrated in a production representative environment. Detailed producibility trade studies and risk assessments are underway. Cost model have been updated with detailed designs, rolled up to system level

Prototype system built on soft tooling, initial sigma levels established. Ready for low rate initial production (LRIP). Design changes decrease significantly. Process tooling and inspection and test equipment demonstrated in production environment. Manufacturing processes generally well understood. Machines

System detailed design activity is underway. Material

specifications have been approved and materials are available. Few design changes are occurring. Production planning is complete and prototypes represent actual form, fit and function. Production tooling and test equipment design and development have been initiated. Low-rate initial production

References

Related documents

First, our results converge to indi- cate that Latin American countries are influenced by foreign variables, either the real one for Brazil and Chile, or the financial one

The negative excess burden of the proportional tax system arises primarily from two related factors: (i) the desirable effects capital gains taxation has on stockholding risk (even

This paper sets up an image compression experiment that tests two shal- low sparse autoencoders against three common sparse cod- ing algorithms by evaluating both

Use of robotic-assisted THA allowed for accurate and reproducible estimation of the postoperative radiographic mea- surements of cup inclination, cup ante- version, leg-length

     IT Health Check      (508) 756‐9300 |  support@akuity.com    Backup  Category: Time to Restore Backup 

Os animais que apresentaram estes sinais pertenciam aos tanques onde a dose de bactéria inoculada foi superior (T1-T3), tornando-se esta situação mais evidente aos 51

In other words, the RDD methodology is applicable when the agents (workers and employers in this setting) have no incentive to act before the discontinuity actually

Sponsored by Nordic Industrial Fund the Nordic and Baltic electronics industry associations 1 organised on 4 April 2002 a workshop in Tallinn in order to discuss how an advanced