Analyzing and Reporting Patent Quality Data

(1)

Worcester Polytechnic Institute

Digital WPI

Interactive Qualifying Projects (All Years) Interactive Qualifying Projects

December 2016

Analyzing and Reporting Patent Quality Data

Alex Riley

Worcester Polytechnic Institute Alex Witkin

Worcester Polytechnic Institute Brian McCarthy

Worcester Polytechnic Institute Evan Stelly

Worcester Polytechnic Institute

Follow this and additional works at:https://digitalcommons.wpi.edu/iqp-all

This Unrestricted is brought to you for free and open access by the Interactive Qualifying Projects at Digital WPI. It has been accepted for inclusion in Interactive Qualifying Projects (All Years) by an authorized administrator of Digital WPI. For more information, please [email protected]. Repository Citation

Riley, A., Witkin, A., McCarthy, B., & Stelly, E. (2016). Analyzing and Reporting Patent Quality Data. Retrieved from

(2)

Date Submitted: December 13, 2016

Analyzing and Reporting

Patent Quality Data:

Examining the Master Review Form

An Interactive Qualifying Project for the United States Patent and

Trademark Office

Submitted to the Faculty of

WORCESTER POLYTECHNIC INSTITUTE

In partial fulfillment of the requirements for the Degree of Bachelor of Science

By

Alex Riley, Mechanical Engineering Evan Stelly, Mechanical Engineering

Brian McCarthy, Computer Science Alex Witkin, Biomedical Engineering

Submitted to:

Liaison: Daniel Sullivan, Director, Technology Center 1600: Biotechnology and Organic Chemistry

Liaison: Martin Rater, Acting Director, Office of Patent Quality Assurance Liaison: David Fitzpatrick, Project Analyst, Office of Patent Quality Assurance

Project Advisor: James Hanlan, WPI Professor Project Co-Advisor: Stephen McCauley, WPI Professor

(3)

i

Abstract

Our team worked with the United States Patent and Trademark Office (USPTO) to improve the quality of patent examinations. We analyzed Master Review Form (MRF) responses to identify potential quality issues in order to recommend areas for further analysis. Based on interviews with managers regarding internal reporting, we also created a decision matrix to assist the Office of Patent Quality Assurance to implement an Ad Hoc analysis tool. Our

(4)

ii

Acknowledgements

To complete our project, the team worked with many individuals who provided valuable resources and information to help structure our final outcome. We would like to give a huge thanks to our liaisons at the United States Patent and Trademark Office (USPTO), Martin Rater, David Fitzpatrick and Daniel Sullivan. Without the countless hours they provided to us through email and in person meetings, we wouldn't have been able to construct and format our project as we did. We would also like to thank all USPTO employees who were able to meet with us for a half-hour interview. Information collected from these interviews helped us to form recommendations for models to display data. We would also like to thank our project advisors, James Hanlan and Stephen McCauley. They have worked endlessly with our group the past two terms and have been there to help us with issues regarding our paper, learning how to construct an interview and providing feedback based on our team work.

Finally, we would like to acknowledge and thank all others who have helped this project in various and important ways:

Mary Kitlowski- USPTO Employee Steven Ricks - USPTO Employee Sara Ringer- WPI Employee

(5)

iii

Executive Summary

The United States Patent and Trademark Office (USPTO) is essential to fostering the spirit of competitiveness and innovation across the country. On average, the USPTO examines 580,000 patent applications and processes 1.6 million office actions annually from various organizations and industries nationwide (USPTO, 2016c). Because of the number of organizations that depend on this system, it is important that patent examinations are conducted fairly and equally. In addition, when a patent application is rejected it can often lead to a lengthy and expensive appeal process. Therefore, it is vital that the patent examination process is as consistent and accurate as possible, rejecting applications only when necessary. Accordingly, the USPTO established the Office of Patent Quality Assurance (OPQA), an office within the USPTO which handles assessment, measurement and improvement of patent examination quality (USPTO, 2015a). Our project assisted the OPQA in improving the quality and consistency of patent reviews by identifying key trends underlying patent quality and mapping potential quality issues within the patent review process.

Background

In early 2015, the USPTO launched the Enhanced Patent Quality Initiative (EPQI) in order to improve the quality and consistency of the patent examination process (Camarota, 2016). As a part of this initiative, the OPQA created the Master Review Form (MRF), which measures the overall correctness and clarity of the work of patent examiners (Spyrou & Rater, 2016).

The MRF is broken down by what statute of patent law is being cited by the examiner as the reason for rejection. The most common statutes cited are 102 and 103. Statute 102 refers to prior art; if a claimed invention was patented or available to the public before the filing date, it will be rejected due to prior art (USPTO, 2015b). Statute 103 refers to non-obviousness; if a claimed invention is similar enough to previously patented work to be considered obvious “to a person having ordinary skill in the art to which the claimed invention pertains”, it will be rejected under statute 103 (USPTO, 2015c). Reviewers rate individual sections of the examination as either No Issues Found (OK), Needs Attention (AT), or Significantly Deficient (SD), for both the correctness and the clarity of the examiner’s work. In addition, each section

(6)

iv

contains a number of sub-questions which help identify exactly which portions of the examiner’s decision are not meeting the USPTO standards of quality (Spyrou & Rater, 2016).

Methods

To begin, we determined the needs of USPTO managers regarding the MRF data. In order to accomplish this, we conducted structured interviews with Quality Leads (QLs), Training Quality Assurance Specialists (TQASs) and Supervisory Patent Examiners (SPEs), utilizing a snowball sampling method to identify relevant interviewees in each category. We began interviews with QLs because they hold managerial positions and work closely with the MRF. Interviewees were questioned about what MRF data they would find most useful, and about various ways they would like to have the data presented to them. After each interview, responses taken from our written and audio notes and further analyzed to determine key patterns.

After determining employees’ MRF data needs, we analyzed the MRF data to find patterns related to the consistency and accuracy of patent examinations. We started our comparisons by looking at the overall clarity and correctness scores for each statute, as well of the specific sub-questions for these statutes. In addition, we filtered data by examiner grouping based on subject matter, referred to as Technology Centers (TC), as well as the Work Groups and Art Units that make up these TCs. This allowed us to determine whether there are patterns in correctness and clarity among groups of examiners.

Finally, we used the data we gathered throughout the interviews to identify the most popular models for reporting MRF data. The three reporting models that were discussed are Canned Reports, Snapshots, and Ad Hoc analysis. A Canned Report is a detailed, widely distributed report which focuses on data and analysis deemed to be most important. A Snapshot is a brief display of key data points that allows users to quickly gauge important information. Ad Hoc analysis allows the user to interactively examine trends and information they determine is most important. We then identified the advantages and disadvantages of each model based on interviewee responses. We organized these features, based on importance, by coding interviewee responses. This information was used to create a decision matrix to help guide implementation of the selected reporting method in order to help the OPQA meet the data reporting needs of USPTO managers.

(7)

v Results

One of the first questions in our interview script (see Appendix A) asked employees about what information from the MRF would be most useful to them. Their responses were coded and displayed in the word cloud seen in Summary Figure 1. In this figure, terms appear larger based on the number of interviewees who mentioned the term. The word cloud shows that many employees were concerned with results based on statutes, specifically statutes 101 (usefulness), 102 (prior art) and 103 (obviousness). Also seen from the results, employees are most concerned with AT and SD rated examinations, which are examinations that had errors somewhere in the process. These examinations are also referred to as non-compliant.

Summary Figure 1- Word Cloud of Interviewee Responses to Question 1.3

We asked interviewees to rate the three reporting methods (Canned Reports, Snapshots and Ad Hoc analysis) on a scale of one to ten. On average, Canned Reports received a rating of 5.3, Snapshots received 6.3, and Ad Hoc analysis received 8.6. These results clearly show that Ad Hoc analysis is the most popular data reporting model among interviewees.

In addition, we collected employee opinions on the advantages and disadvantages of each model. In Summary Figure 2, we can see that employees identified customizability as the chief advantage of Ad Hoc analysis. Likewise, the employees’ major concern was that the chosen Ad Hoc tool would have a steep learning curve.

(8)

vi

Summary Figure 2- Advantages and Disadvantages of Ad Hoc Analysis Based on Interview Responses

Our team organized the data collected by the MRF, then analyzed and drew conclusions from the information related to patent quality trends. While the MRF contains 395 questions concerning several different sections of patent law, we chose to focus on only the questions concerning statutes 102 (prior art) and 103 (obviousness) of the US Code, as the majority of reviews cited these statutes.

Our first analysis compared overall clarity to overall correctness. We looked at the percentage of total reviews that received the same rating in clarity and correctness, seen in Summary Table 1. Summing the percentages together, it can be seen that 85% of reviews received the same rating in both clarity and correctness, suggesting that there is a strong relationship between overall clarity and correctness.

(9)

vii 102 Comparison of Overall Scores (From a sample of 3,711) Correctness Clarity OK AT SD OK 2,790 (75% of total) 173 70 AT 259 280 (8%) 63 SD 1 4 71 (2%)

Summary Table 1- Comparison of 102 Overall Correctness and Overall Clarity

After comparing overall clarity and correctness scores, we examined whether the inclusion of specific features of an examination, such as clear explanations, affected that examination’s overall rating. We did this by graphing the percentage of examinations that received an OK rating in statute 102 (prior art) clarity that also included each of that statute’s features, compared to those that received an AT or SD rating in clarity that included that feature. This comparison can be seen in Sample Figure 3. By observing features that were included in most OK reviews but excluded from most SD and AT reviews, we can identify these features as driving the overall clarity for 102 (prior art). An example of this type of driver is clear annotations. Conversely, features that were included or excluded in the majority of all reviews, regardless of ranking, such as Effective Date OK, likely do not affect the overall clarity score or overall quality. This suggests that these questions could be removed from the MRF to save time and simplify the output, without affecting the overall results. These types of comparisons were also conducted for 102 correctness, as well as for 103 clarity and correctness as can be seen in Appendix C.

(10)

viii

Summary Figure 3- Feature Inclusion Percentage for 102 Overall Clarity

In addition to understanding trends for the entire USPTO, it was important to examine how different groups of examiners were performing. Specifically, we examined the overall correctness and clarity of reviews for 102 and 103 rejections for each TC (TC). By seeing how many of these reviews were rated as OK or AT/SD for each TC, we can quickly view which TCs have the most room for improvement and in which areas. The example in Summary Figure 4 below looks at TC performance for the clarity of statute 102. Using the sample size estimation equation we find that meaningful conclusions can only be drawn from groupings with a sample of 300 or more reviews.

Conclusions can only be made about TCs with a percentage of AT and SD reviews that is above or below a 95% confidence interval, indicated by the blue dotted line. A 95% confidence interval means that, if the results were recreated 100 times, it would be expected that a result would fall within this confidence range 95% of the time (Lane, n.d). Results outside this interval are considered statistically significant. TCs above this interval have statistically significantly more AT and SD reviews than other TCs and therefore may be the target of future training to

(11)

ix

improve quality concerning statute 102. In the future, further analysis can be done to analyze Work Groups and Art Units that make up any TC that could be having a problem with a given statute. Currently no conclusions can be made from looking at these groupings, because the sample sizes are too small to find statistically significant results.

Summary Figure 4- 102 Overall Correctness SD or AT

Proposed Modeling for Reporting Needs

To assist the OPQA with identifying data reporting models that will meet the needs of USPTO managers, we created a Decision Matrix for Ad Hoc Analysis implementation, as seen in Summary Figure 5. The OPQA can use this Matrix to determine exactly what implementation of the Ad Hoc model will best fit the needs of USPTO employees. The categories, features, and weights were all chosen based on three main areas: the interview responses we collected, our discussions with USPTO liaisons, and our own experience with Ad Hoc tools like Microsoft Excel or Microsoft Analysis. Implementations of Ad Hoc analysis can be rated for each feature on a scale of 1-10; these scores are then multiplied by that feature’s weight to find the weighted

(12)

x

score. Finally, the weighted scores are summed together to find the total score. The implementation which receives the highest total score is best suited to fit the needs of the USPTO.

Summary Figure 5- Ad Hoc Decision Matrix

Recommendations and Conclusions

Our project focused on performing a preliminary analysis on MRF data with a focus on statues 102 (prior art) and 103 (obviousness). We believe that this will provide a valuable basis from which the OPQA can complete more detailed and specific analysis of this data involving all 12 statutes cited by the MRF. There are many areas of the MRF where analysis could allow employees to improve the quality of patent examination. However, in order to understand the results, employees need the proper tools to generate and communicate results.

(13)

xi

We offer the following recommendations:

● Utilize the decision matrix to implement a form of Ad Hoc analysis and distribute this implementation, integrated with MRF data, to USPTO employees

● Consider other forms of data reporting for MRF data, such as canned reports or snapshots, using interview responses to guide the design of these models

● Use MRF data to identify areas of USPTO experiencing quality difficulties

● Improve validity of MRF conclusions by examining the methods used for form sample selection

● Examine MRF data based on reviewers who complete the form to find and eliminate any potential biases that could skew results

● Reevaluate relevance of questions asked by MRF based on main drivers of quality, by focusing on questions regarded as drivers of quality while considering eliminating questions that are not seen as drivers of quality

Our project will allow OPQA employees to better interpret and utilize results they receive from the MRF. This will allow them to better identify and eliminate quality issues involved with the patent examination process. Effectively eliminating issues regarding patent quality will not only increase the reliability of examination results produced by the USPTO, but also improve the trust between the USPTO and the community of inventors and innovators whom they serve.

(14)

xii

Table of Authorship

Chapter and

Section Numbers Section Name Primary Author(s)

Primary Editor(s)

Title Page Team Team

Abstract Team Team

Acknowledgments Team Team

Executive Summary Team Team

Table of Authorship Team Team

Table of Contents Team Team

List of Figures Team Team

List of Tables Team Team

Acronyms Team Team

1.0 Introduction

Brian McCarthy, Alex Witkin

Alex Riley, Evan Stelly 2.0 Background Brian McCarthy Alex Riley

2.1

Patent Application and

Examination Process Alex Witkin Evan Stelly

2.2

History of Quality

Assurance Alex Riley Brian McCarthy

2.2.1

Evaluation by Quality Assurance Specialists

(QAS)

Evan Stelly Alex Witkin

2.2.2

In-Process Review

(IPR) Alex Riley

Evan Stelly, Alex Witkin

(15)

xiii

2.2.3

Supervisory Patent Examiner Internal

Process Review

Alex Witkin Brian McCarthy

2.3

Current Patent Quality Assessment

Brian McCarthy Alex Riley

2.3.1

US Code: Made 102

and 103 Alex Riley Brian McCarthy

2.4

Analysis of Patent

Quality Data Evan Stelly Alex Witkin 2.4.1 USPTO Data Analysis Alex Witkin Alex Riley

2.42

Relevant Statistical Data

Analysis Techniques _{Brian McCarthy} _{Evan Stelly}

3.0 Methods Alex Riley Evan Stelly

3.1

Objective 1: Determine Needs of USPTO Employees Regarding

MRF Data

Evan Stelly Alex Riley

3.2

Objective 2: Analyze Master Review Form

Data

Alex Witkin Brian McCarthy

3.3

Objective 3: Develop

Data Reporting Model Brian McCarthy Alex Witkin

3.3.1 Snapshot Alex Riley Brian McCarthy

3.3.2 Canned Report Alex Witkin Evan Stelly 3.3.3 Ad Hoc Analysis Brian McCarthy Alex Witkin

3.4 Project Timeline Evan Stelly Brian McCarthy 4.0 Findings and Results Brian McCarthy Alex Riley

(16)

xiv

4.1

Objective 1: Needs of USPTO Management Regarding Collected Data

Alex Witkin Alex Riley

4.1.1

Master Review Form

Data Reporting Needs Alex Riley Brian McCarthy

4.1.2

Internal Report Design

Needs Alex Witkin Evan Stelly 4.1.3 Limitations Evan Stelly Alex Riley

4.2

Objective 2: Analysis Master Review Form

Data

Brian McCarthy Alex Riley

4.2.1

Comparison of Overall Correctness and Clarity

for 102 and 103 Alex Riley Brian McCarthy

4.2.2

Analysis of Individual Questions Based on

Clarity and Correctness Alex Witkin Evan Stelly

4.2.2.1

Combining the Needs Attention and Significantly Deficient

Fields

Evan Stelly Alex Riley

4.2.3 Context-Based Analysis Brian McCarthy Evan Stelly

5.0

Proposed Modeling for

Reporting Needs Alex Riley Alex Witkin

(17)

xv

5.2 Robustness Alex Riley Evan Stelly

5.3 Ease of Use Alex Witkin Brian McCarthy

5.4 Utility Evan Stelly Alex Riley

5.5 Rankings Alex Riley Brian McCarthy

6.0

Recommendations and

Conclusions Evan Stelly Alex Riley

6.1

Internal Report

Generation Alex Witkin Evan Stelly 6.1.1 Ad Hoc Analysis Alex Riley Brian McCarthy 6.1.2 Other Reporting Methods Brian McCarthy Evan Stelly

6.2 Data Analysis Brian McCarthy Alex Witkin

6.2.1

Master Review Form

Data Alex Witkin Alex Riley

6.2.2 Other Data Analysis Evan Stelly Brian McCarthy

6.3

Changes to the Master

Review Form Alex Riley Evan Stelly 6.4 Conclusion Brian McCarthy Alex Riley

Appendix A Team Team

Appendix B

Team

(18)

xvi

Appendix D Team Team

Appendix E Team Team

(19)

xvii

List of Figures

Figure 1 – Example Patent Quality Map ...11

Figure 2 – Gantt Chart, Project Timeline ...17

Figure 3 – Word Cloud ...20

Figure 4 – Preferred Method of Receiving Data ...21

Figure 5 – Average Rating of the Three Models ...22

Figure 6 – Advantages of Ad Hoc Analysis ...23

Figure 7 – Disadvantages of Ad Hoc Analysis ...23

Figure 8 – Advantages of Canned Report ...24

Figure 9 – Disadvantages of Canned Report ...25

Figure 10 – Advantages of Snapshots ...25

Figure 11 – Disadvantages of Snapshots ...26

Figure 12 – Percentage of Reviews Rejected by Statute ...27

Figure 13 – Feature Inclusion Percentage for 102 Overall Clarity Reviews ...32

Figure 14 – 102 Overall Correctness Significantly Deficient or Needs Attention Reviews ...34

Figure 15 – 103 Overall Correctness Significantly Deficient or Needs Attention Reviews ...35

Figure 16 – 102 Overall Clarity Significantly Deficient and Needs Attention Reviews ...36

Figure 17 – 103 Overall Clarity Significantly Deficient and Needs Attention Reviews ...37

Figure 18 – TC2400: 103 Overall Correctness SD or AT Reviews by Work Group ...38

Figure 19 – Work Group 2440: 103 Overall Correctness SD or AT Reviews by Art Unit ...39

Figure 20 – Ad Hoc Analysis Decision Matrix ...41

Figure 21 – Ad Hoc Analysis Decision Matrix featuring example Implementation ...41

List of Tables

Table 1 – Definition of Overall Ranking ...10

Table 2 – 102 Comparison of Overall Scores ...29

(22)

xx

Acronyms

EPQI – Enhanced Patent Quality Initiative IPR – In- Process Review

MRF – Master Review Form AT – Needs Attention

OPQA – Office of Patent Quality Assurance OK – No Issues Found

PATI – Patent Application Text Initiative QAS – Quality Assurance Specialists QIR – Quality Index Report

QL – Quality Leads

RQAS – Review Quality Assurance Specialists SD – Significantly Deficient

SPE – Supervisory Patent Examiner TC – Technology Center

TQAS – Training Quality Assurance Specialists USPTO – United States Patent and Trademark Office

(23)

1

Chapter 1: Introduction

The United States Patent & Trademark Office (USPTO) is essential to fostering the spirit of competitiveness and innovation across the country. It is responsible for maintaining a high quality, productive and responsive network that supports a vast intellectual property system (USPTO, 2009). On average, the USPTO examines 580,000 patent applications and processes 1.6 million office actions annually (USPTO, 2016c); with such a large number of applications being processed each year, systemic issues can impact many organizations and industries nation-wide (Camarota, 2016). Because of the number of organizations that depend on this system, it is vitally important that patent examinations are conducted fairly and equally. To achieve this, the USPTO established the Office of Patent Quality Assurance (OPQA), an office within the USPTO which handles assessment, measurement and improvement of patent examination quality

(USPTO, 2015a).

A rigorous inspection is required to ensure that an application deserves a patent; this process can be extremely time consuming. Examiners must thoroughly consider all parts of each application in order to ensure that the proposed patent does not infringe on a previous patent, referred to as prior art. The application is also checked to make sure it is not considered obvious for its field, since an application must be considered "non-obvious" to be eligible for a patent, along with many other areas of patent law (Purvis, 2013). A massive backlog of pending patent reviews has built up due to the time-consuming review process the examiners have to follow. The backlog consists of over 540,000 applications and contains patent applications which have been filed up to two years ago (USPTO, 2016b). In addition, communication and motivation problems exist in the current examination process. The process discourages examiners from correcting mistakes because admitting to a mistake counts against them in the workplace (Weiler et. al, 2014). Along with the pressure that is put on from mistakes and appeals, the examiners are given a quota of applications that they must process each quarter; they must “maintain an average minimum production level … to avoid disciplinary action by the USPTO” (Penny, 2014). New patent examiners tend to take longer on each application, taking more time to review all past information. Patent examiners with more experience tend to cite less prior art and are more likely to grant patents than less experienced examiners, which suggests that a patent’s chance of being approved can be greatly impacted by which examiner reviews it (Lemley & Sampat, 2012).

(24)

2

There have been a number of attempts by the USPTO to improve the quality of work produced by the examination process. In early 2015, the Enhanced Patent Quality Initiative (EPQI) was launched to improve the process the OPQA had used to ensure patent quality (Camarota, 2016). The EPQI introduced several changes to the metrics used by the USPTO and the OPQA to more accurately quantify the quality of the patent review process. With the EPQI, communications were started with patent applicants and other stakeholders through initiatives such as a webinar series and a patent summit hosted in 2015. The webinar series went in depth on a number of the changes introduced by the EPQI. The summit was designed to allow all stakeholders the opportunity to give feedback on the EPQI and on patent quality assessment as a whole (Lee, M, 2015).

As part of the EPQI, the Master Review Form (MRF) was introduced to improve the metrics used by the OPQA when evaluating patent examination quality. The MRF is a 250 point form that is completed by patent examiner supervisors and OPQA quality reviewers to check the quality of decisions made on patent applications. While the MRF collects vital data on patent applications and the review process, the data has not yet been analyzed. In order to increase overall efficiency and accuracy at the USPTO, this raw data should be compiled into reports that are available to supervisors (Lee, M, 2016).

Our project assists the OPQA in improving the quality and consistency of patent reviews by identifying key topics linked to patent quality and mapping potential quality issues within those topics. We accomplished this goal by first analyzing the MRF for potential improvements using software such as Microsoft Excel and Microsoft Access to examine data collected from the form. In addition, our project aims to improve the techniques used to report the form’s data. This was accomplished by interviewing employees about opinions on internal reporting and then working with the OPQA to select a reporting style that reflects the gathered opinions. These features will allow the USPTO to collect and use data in the most efficient way possible. Efficient data usage allows issues in the OPQA’s current process to be identified and remedied, thus improving patent quality throughout all of the USPTO.

(25)

3

Chapter 2: Background

This section discusses the information we have gathered about the United States Patent and Trademark Office (USPTO) and data analysis. This includes an overview of current patent examination techniques as well as a history of improvements and changes that the examination process has undergone. In addition, this section reviews challenges that the USPTO has encountered in analyzing the quality of patent examinations and what has been done to meet these challenges. In this section we will also explore the best practices for the methods used in analyzing and reporting of data.

2.1 Patent Application and Examination Process

The patent application process is important to ensuring that the approved patents are novel and non-obvious. This section will explore the various steps of this process, as well as both positive and negative effects the process can have on the applicants and the reviewers.

There are three different forms of patents in the United States (USPTO, 2014): ● Design patents such as those for ornamental characteristics

● Plant patents such as those for new asexual plants

● Utility patents such as those for processes, machines, and manufactured items Utility and plant applications can be submitted in a provisional or nonprovisional format, while design patents can only be submitted as nonprovisional. Provisional applications do not undergo examination within the USPTO (Norek, 2015). Provisional patents are used to protect a product while testing feasibility of an idea and determining if the patent would be beneficial. During this period, it allows the innovator to begin working on the nonprovisional applications. Nonprovisional applications fully undergo examination.

Patent applicants commonly use a registered attorney or agent in order to legally support the invention. While attorneys can cost roughly $2,000 - $10,000 per application, they can be useful to fully protect both the invention and the applicant (Pressman, 2012). The application must be filed electronically through the Electronic Filing System. Upon filing the application, the applicant receives a customer number and a digital certificate. The customer number allows the applicant to organize all filings under a single mailing address, and a digital certificate allows them to be uniquely identified and provides secure access to vulnerable data (USPTO, 2014).

(26)

4

Once the application has been written, it can either be submitted normally or expedited using one of several different programs: Prioritized Examination, Accelerated Examination Program, First Action Interview, or Patent Prosecution Highway. Prioritized Examination can be used for utility and plant applications, and, if accepted, “the application will be accorded special status during prosecution before the patent examiner.” This program halves the application process to 12 months; however, only 10,000 applications total can be accepted per year on a first come first serve basis, with additional fees attached. A similar process exists for Design Patents, known as the Design Rocket Docket (Signore, 2010). The Accelerated Examination Program is similar to Prioritized Examination in that it will be completed within 12 months, but is specific to groundbreaking environmental, medical or counter-terrorism research. This program is meant to expedite inventions that have a major impact on the world, such as life saving medication. On a more personal level, First Action Interviews can be set up between the examiner and the applicant. Interviews speed up prosecution and provide a personal interaction between the applicant and the examiner, giving the applicant the chance to discuss, and possibly fix, issues with the application in front of the examiner. Finally, Patent Prosecution Highway is a program that speeds the process of an application once it has been transferred between offices. These four programs are an effective way for relevant and well written patent applications to speed up the normal two-year backup (USPTO, 2014).

After the full application has been sent in electronically, it is received and sorted into a class and subclass, and then assigned to a specific examiner by a Supervisory Patent Examiner (SPE) (Lemley & Sampat, 2012). The class and subclass allows the application to be assigned to a Technology Center (TC). The TC will review each application and distribute them out to examiners based on each examiner’s expertise. Once the application reaches the examiner, it will go through many checks, including prior art, non-obviousness, and usefulness to make sure the application meets the basic patent requirements. If an application fails any part of the examination initially, the applicant is given a chance to make corrections and resubmit. After this, if a final rejection is made, the applicant is given the option to request reconsideration or attempt to appeal the decision in court. If the application is allowed, the applicant must pay issue fees and publication fees to receive an official patent. The only upkeep after the patent has been granted are maintenance fees that are due 3.5, 7.5 and 11.5 years after the patent has been granted. Patents expire after 20 years, but can be reapplied for if proper maintenance fees are

(27)

5

paid. If the holder of the patent does not pay proper fees, the patent becomes abandoned. (USPTO, 2014).

2.2 History of Quality Assurance

The USPTO keeps a strict quality assurance program in order to ensure patent quality. The public has spoken through many surveys, sent out by the USPTO, requesting better quality in patent examinations. Prior to the recent Enhanced Patent Quality Initiative (EPQI), the USPTO improved their Quality Assurance Program in 2012 (USPTO, 2016a). This program improved connections between data collection and training development. Using various forms of data, such as that collected from the Master Review Form (MRF), training programs can be established to maximize instruction efficiency. To improve patent quality, USPTO has implemented the following review processes.

2.2.1 Evaluation by Quality Assurance Specialists

Quality Assurance Specialists (QAS) measure and review data from past examiner reports year round and make sure this data is available and current. Examiner reports are based on past applications and whether they were rejected or accepted. When reviews are completed, employees upload raw data from these reports to a shared database (USPTO, 2016a). Reviews and measurements are based on the following items:

● Was the rejection clearly wrong?

● Were claim limitations correctly matched to the prior art? ● Was the examiner’s position properly supported?

● Were the differences from prior art stated?

2.2.2 In-Process Review

Based on applicant and attorney feedback, the USPTO established its current in-process review (IPR) of examiners’ work. This review looks at the quality of an examiner’s intermediate decisions as the application goes through the examination process. QASs from each TC are assigned to review a random sample of roughly 380 office actions each year. QASs in technology centers can perform focused IPR in addition to the random reviews. With focused

(28)

6

IPR, specific training needs of a single examiner can be developed

(USPTO, 2016a)

. The data that is collected during in-process reviews can lead to:

● Training sessions on both an individual level and for an overall department ● Examiner certification or recertification

● Quality measurements

2.2.3 Supervisory Patent Examiner Internal Process Review

A SPE is expected to review four applications per year, one application per quarter year, from each one of their primary examiners. Every supervisory patent examiner is responsible for the overview of roughly 13-14 patent examiners. The purpose of the SPE reviewing their primary examiners is to keep them directly connected with the work produced within their area of specific art (USPTO, 2016a). It is likely that having past applications examined keeps the examiner’s work quality from decreasing, which in turn increases patent quality.

2.3 Current Patent Quality Assessment

Patent examination quality is important in preventing uncertainty and unfairness in the patent application system. If businesses become uncertain of the validity and scope of patents, they will be less likely to file new patents, which could slow the process of innovation. Uncertainty in patents also leads to more patent litigation, which is expensive and time consuming. Poor quality often comes in the form of both inappropriate grants and inappropriate rejections (Wagner, 2009). This section will explore the efforts of the entire USPTO to solve quality concerns, as well as critiques of their methods and recent innovations made to the quality review process.

A number of different Office of Patent Quality Assurance (OPQA) employees are involved with both general quality review as well as the MRF. The QAS’ main duties are the quality control of examiner work, as well as the conduct of various quality programs such as training. Training Quality Assurance Specialists (TQAS) work more closely with training programs and are stationed at individual technology centers. Review Quality Assurance Specialists (RQAS) work primarily with reviewing the work done by patent examiners. They are stationed in the OPQA office. All QASs are assigned work focused on a specific technology and

(29)

7

report to Quality Leads. Quality Leads are experienced patent examiners whose primary duty is to review the Patent Examiner and QAS work product (USPTO, 2013; Spyrou & Rater, 2016).

In February, 2015, the Acting Head of the USPTO, Michelle K. Lee, launched the EPQI. This initiative encourages collaboration between many offices of the USPTO, as well as members of the public involved with the patent application process, in order to help achieve excellence in three main categories: Work Products, Measuring Patent Quality, and Customer Service. “Excellence in Work Products” refers to the performance of patent examiners, as well as the quality of the processes and programs they follow and the training they have received. “Excellence in Measuring Patent Quality” involves the improvement of the metrics used by the USPTO to analyze patent quality. Lastly, “Excellence in Customer Service” relates to the relationship between the patent office and customers such as patent applicants and holders; this includes interviews between examiners and applicants (Camarota, 2016).

The USPTO has received many recommendations for improvement in the past; these recommendations may have prompted the launch of the EPQI (USPTO, 2016d). A past Worcester Polytechnic Institute Interactive Qualifying Project recommended that the quality of the patent examination process be improved by giving clearer deadlines to examiners, as well as improving collaboration between examiners and the patent applicants themselves. This study also suggested changes to the feedback system used to evaluate patent examiners, as the current system discouraged the correction of mistakes by viewing actions such as reopening cases as errors which count against the examiner (Weiler et. al., 2014).

Since launching the EPQI, the USPTO has made many changes to the patent quality metrics and data collection process used by the Office of Patent Quality Assurance. Before these changes, several metrics were combined to create a Quality Composite Score, which was used to assess patent quality.

(30)

8

The metrics were as follows (USPTO): 1. Final Disposition Compliance Rate; 2. In-Process Compliance Rate;

3. Complete First Action on the Merits Review; 4. Search Review;

5. Quality Index Report (QIR); 6. External Quality Survey; 7. Internal Quality Survey.

These metrics have now been sorted into three categories, Product Indicators, Process Indicators, and Perception Indicators. These three categories have replaced the Quality Composite (Spyrou & Rater, 2016).

The Product Indicators metric focuses on the correctness and clarity of the examiner’s work. Clarity is especially important to patent filers and inventors who need to understand what rights their patent gives them or why the patent has been rejected. It is also important for those who plan on using past patents as part of their research. Correctness and clarity are measured through the MRF. This form is designed to be used by many of the USPTO organizations. The form adapts to individual circumstances as it is being completed, automatically filling out certain sections and directing the examiners to the relevant sections. The Process Indicators are used to help reduce how often examiners have to reopen and rework patent applications. The Perception Indicators are a combination of two metrics: The Internal and External Surveys. These are used to get opinions from both the public and from examiners on patent quality and help verify that the patent quality measurement methods are working (Caputa & Rater, 2015; USPTO, 2014).

2.3.1 US Code: Statutes 102 and 103

The two main forms of patent application rejections are prior art (102) and non-obviousness (103). While writing a patent, the patentee makes certain claims about their invention that they wish to own. The claims then define the limit that the patent will cover if approved. If a claim is already patented, the new patent application will be rejected due to prior art (USPTO, 2015b). A patent examiner has many resources he or she can use while determining

(31)

9

prior art. Online patent databases make searching old patents easy with advance searches. Non-obviousness deals with patent applications that have claims similar to previous patents and their similarities are considered obvious “to a person having ordinary skill in the art to which the claimed invention pertains” (USPTO, 2015c). For more patent related statutes referenced in the MRF see appendix D.

2.4 Analysis of Patent Quality Data

David Fitzpatrick, Office of Patent Planning and Data Analysis, works closely with data and numbers that deal with patent applications and patent quality. According to Fitzpatrick, the OPQA handles roughly 102 applications a day. In particular, the OPQA works directly with significant patent quality data to determine the quality and consistency of patent reviews (USPTO, 2015a). In this section, we will examine the best practices of analyzing and presenting large amounts of data, as well as how the USPTO analyzes their data.

2.4.1 USPTO Data Analysis

The USPTO utilizes several methods to handle analysis of their data. To determine the quality of work performed by the USPTO, the OPQA collects multiple types of data. The MRF primarily collects nominal data. This type of data is organized and labeled based on categories which have no quantitative value. The MRF rates the correctness and clarity of examiner decisions as No Issues Found (OK), Needs Attention (AT), or Significantly Deficient (SD), as seen in Table 1 below (Spyrou & Rater, 2016). This data will later need to be quantified in order to conduct analysis (Duignan, 2016).

(32)

10

Definition of Overall Rankings

Rank Explanation

No Issues Found (OK) Everything was Correct and Clear

Needs Attention (AT) Coaching and mentoring issues are present Significant Deficiency (SD) Corrections are needed AND prosecution is

adversely affected

Table 1- Definition of Overall Rankings

Another form of categorical data that could be used for quality analysis is ordinal data. Unlike nominal data, ordinal data is organized by categories that do have quantitative values. These categories, such as least preferred and most preferred, are easier to order and rank. However, it is not as effective as nominal data for measuring the frequency of responses (Duignan, 2016).

Recently, steps have been taken by the USPTO to make it easier for patent examiners to quickly search through and analyze patent data. Patent applications are stored as TIFF image files, in a database that is at least 78 TB large (Hamer et. al, 2012). Because these applications are stored as images, they must be manually examined and sorted with no way to automatically search through text. The Patent Application Text Initiative (PATI) aims to transcribe three key documents in each application into text format, so that patent examiners can quickly search through these documents to find and analyze the relevant data.

Loet Leydesdorff and Lutz Bornmann (2012) developed an additional method to present patent data. By creating a Google Maps overlay using published USPTO data, they created a map which indicates the quantity and quality of patents at the city level, as shown in Figure 1.

(33)

11

Figure 1- Example of the Patent Quality Map Created by Leydesdorff and Bornmann

Leydesdorff and Bornmann primarily use the number of times a patent is cited as a way to determine its quality; the more a patent is cited, the higher quality it is likely to be. This map assists the public by indicating where a high volume of high-quality patents are being filed, which would indicate to a new patent applicant where to go for assistance. This also makes technological analysis based on economic geography and innovational studies easier by centralizing all of the relevant data.

2.4.2 Relevant Statistical Data Analysis Techniques

Analyzing large sets of data presents a unique challenge. With a large amount of variables, comparing data sets to find valid relationships can be difficult (Reshef et. al, 2011). Therefore, it is vital to perform statistical analysis of the data in order to determine what points contain significant or valuable information. For our particular needs, we analyzed data mostly through the use of descriptive statistics. This type of analysis was used to better summarize data in order to locate patterns. An important tool for this type of analysis is percentiles. Percentiles are used to show where one data point is in relation to other relatable data points. Specifically,

(34)

12

the xth_{percentile is defined as the smallest organized data point that is greater than or equal to} x% of the data points (Lane, n.d).

Findings from any statistical analyses must be tested for statistical significance. Statistical significance means that the perceived findings actually correlate to a relationship in the data and are not caused by random chance. This is extremely vital for all statistical findings because, without statistical significance, conclusions are not necessarily valid. In order to test for statistical significance, a confidence interval can be constructed. A confidence interval is an interval estimate of a population parameter that differs from sample to sample based on sample size and characteristic. The characteristic of a sample is a measurable average usually calculated as a percent. In applied statistical practices, 95% confidence intervals are typically used. A 95% confidence interval means that, if the results were recreated 100 times, it would be expected that a result would fall within this confidence range 95% of the time (Lane, n.d). The confidence interval equation found in Appendix F is used to calculate these intervals for sets of data. If a data point is outside of the calculated confidence interval, then it is statistically significantly different than other data points at that level of confidence. After the confidence interval is created, an appropriate sample size can be estimated using the characteristic and the desired tolerance, or how wide the confidence interval is. This is useful for excluding groupings that are too small to be statistically significant. This equation can also be found in Appendix F.

In addition to analyzing the large scale data, it is important that the finding be presented correctly. Edward Tufte’s “The Visual Display of Quantitative Information” describes several core philosophies for designing various methods for displaying such analysis. Most significantly, Tufte argues that graphical displays should reveal all of the data without distorting it at all; they should induce the viewer to think about the substance of the data, rather than the methodology or the presentation itself. Tufte defines the idea of “data-ink”, which is the portion of graphical representations of data which display the data itself. He argues that designers should maximize data-ink while removing non-data-ink as much as possible, within reason. This prevents the graphic from becoming cluttered or distracting, and facilitates the presentation of data directly with as little distortion or confusion as possible. These guidelines guarantee that the data and analysis displayed is clear, concise, and informative.

(35)

13

Chapter 3: Methods

Our project assists the Office of Patent Quality Assurance (OPQA) in improving the quality and consistency of patent reviews. This is accomplished by identifying specific areas of analysis that may drive the overall quality of patent examiner work and mapping potential quality issues within these areas. In addition, we developed recommendations for a method of communicating both current and future Master Review Form (MRF) results to patent examiners, supervisors, and OPQA employees. To accomplish our goals, we identified three objectives:

•

Determine the needs of United States Patent and Trademark Office (USPTO) managers and supervisors regarding analysis of the MRF data

•

Analyze MRF data to find patterns related to the consistency and accuracy of patent examinations

•

Develop recommendations regarding internal reports of MRF data based on USPTO employee preferences

3.1 Objective 1: Determine Needs of USPTO Employees Regarding MRF Data

Before we began analyzing the MRF data, it was important to understand how the USPTO employees would use our findings. To do this, we had to determine the reporting needs of these employees. We accomplished this by conducting interviews with employees at three different management levels: Quality Leads (QL), Training Quality Assurance Specialist (TQAS) and Supervisory Patent Examiners (SPE). With the data from the interviews, we were then able to determine employee needs from the overall MRF data, which allowed us to focus our attention in more detailed areas.

For our interviews, we initially considered a semi-structured interview format, aiming to learn stories of past employee experiences with internal reporting. However, after more consideration we decided that a more structured interview format would provide a better picture of employee internal reporting needs. This format better allowed us to ask the specific questions we had concerning both the types of data that need reporting, as well as the design choices that lead to an effective report. The more structured interview format also allowed results to easily be compared quantitatively. The questions for these interviews, along with our oral consent script can be found in Appendix B.

(36)

14

We began our interview process by conducting interviews with QLs. From there, we interviewed TQASs and finally SPEs who were recommended by previous interviewees through the snowball method of sample selection. The snowball selection method is performed by asking interviewees for other employees whom they believe could benefit our project through an interview. Choosing this method allowed us to interview employees we wouldn't have considered otherwise and helped make sure that our interviewees had relevant information for our project.

Interviewees were questioned about what data they wanted and would find most useful, and about various ways they would like to have the data presented to them. To limit biasing opinions within the interviews, interviewees were first asked to provide suggestions on what types of data they were looking for along with what presentation methods they found useful. We then went into more detail on what data and presentation methods we believe they would find useful from ideas that were provided to us by our liaison, Martin Rater, the Acting Director of the OPQA. In this way, the interviewee’s ideas of how data should be presented would not be skewed to match one of the methods we were already considering. Qualitative responses were then coded by keywords and phrases and sorted based on frequency. In the first section of the interview we looked for mentions of fields from the MRF on which to focus our analysis, as well as for specific types of analysis that may need to be conducted. In the second section we looked for mentions of certain reporting types, as well as positive and negative characteristics of these reports. In addition to coding, ordinal data concerning the styles of reporting models was collected by having interviewees rank reports from 1 to 10, where 1 is not useful at all and 10 is most useful. This combination of data types allows us to get a comprehensive picture of USPTO employee needs.

3.2 Objective 2: Analyze Master Review Form Data

To accomplish our goal of discovering potential patent quality issues, our team analyzed data that had already been collected on the quality of patent examinations through the MRF. We looked for relationships between the overall clarity and correctness of patent reviews and individual examination components, such as explanations of rejections. In addition, we began a

(37)

15

preliminary exploration of trends seen in Technology Centers (TCs), Work Group and Art Units, as well as trends based on individual reviewers.

To begin, we looked at office action reviews collected by versions 1.0 to 2.01 of the MRF, provided to us by Martin Rater. We first separated the data into different categories based on which areas of patent law that a question concerns. Each section contains an overall correctness and overall clarity field, as well as additional sub-questions which help identify exactly which portions of the examiner’s decisions are not meeting the USPTO standards of quality. We began our comparisons by looking at the overall clarity and correctness scores for each statute. This helped us see any relationships that may exist between the two fields. We then compared the results of the statute’s sub-questions to the overall correctness and clarity fields. These comparisons were used to help identify which specific areas of the examination process seem to result in actions of insufficient quality. In addition, we filtered data by various examiner groupings, such as the TCs that examiners are a part of, or the Work Groups and Art Units within these TCs. This allowed us to see which groups of examiners have the most significant issues in terms of quality and clarity. We used two main tools to filter the data, Microsoft Excel’s pivot tables and Microsoft Access. These allowed us to compare a large number of data fields relatively quickly to find results.

After the results were organized, statistical significance testing was performed on appropriate data sets in order to determine whether or not the differences in data warranted additional attention. General significance testing using the average sample size and characteristic was done in order to visualize statistically significant results (see Section 2.4.2). For future analysis, individual confidence intervals can be calculated for each grouping, allowing more accurate statistical significance. Significance testing was necessary for the TC, Work Group, and Art Unit breakdowns for statute overall correctness and clarity. This was done by calculating the average sample size of each grouping, along with the characteristic, or the average percent of Significant Deficiency (SD) or Needs Attention (AT) reviews found. Using the sample size and characteristic, a general 95% confidence interval was created for each finding. If a perceived difference is outside the confidence interval, it is considered a statistically significant difference. This then allowed us to use the characteristic and the allowed tolerance to calculate an estimation of the valid sample size needed for a difference to be significant under a 95% confidence interval.

(38)

16

3.3 Objective 3: Develop Data Reporting Model

Using the data, we gathered throughout our interviews found within objective 1, we were able to identify the most popular forms for reporting the collected data. We then identified the features that made this form of reporting most popular, as well as characteristics that employees feared may become potential problems. We also organized these features based on importance by coding interviewee responses. This information could then later be used to create a decision matrix to help guide the development of the selected reporting method.

3.3.1 Snapshot

One form of presenting data is the Snapshot. The goal of the Snapshot is to present data in a format that is easily readable and that can be understood quickly. Snapshots often use graphs, such as tables or charts, to display data in a format that can be read and interpreted quickly without the need for further analysis. In the context of MRF data, a Snapshot would likely consist of individual dashboards for TCs, Work Groups and Art Units reporting on the number of examinations that were rated No Issues Found (OK), AT or SD. This way only the most relevant data would be seen by the user, preventing unnecessary analysis.

3.3.2 Canned Report

The team also considered a Canned Report to present the collected data. The purpose of the report is to allow the user to read and understand data that has already been analyzed, with the use of graphs, tables and bar charts so no additional manipulation of data is required. In addition to graphs and figures, text explanations are provided to better explain the conclusions that were made by analysis. For MRF analysis, a single report would likely be used to communicate data to all employees in every TC including as much information as possible. Due to this, proper labeling and organization of reports are needed to ensure employees can easily find the data they need.

(39)

17

3.3.3 Ad Hoc Analysis

The last form of data reporting that we looked at was Ad Hoc analysis. This form of data presentation requires additional manipulation of data on the user end of the platform. Using an Ad Hoc tool allows employees to customize the data they are looking at by filtering it on an individual level. Because Ad Hoc analysis would involve introducing a new tool to employees, it would likely require training sessions to ensure that the users could fully take advantage of every feature. Using Ad Hoc at the USPTO would mean that employees would generate their own reports, ensuring that they get exactly what they need.

3.4 Project Timeline

During our time at the USPTO, our team followed a structured schedule to make sure all tasks were completed. To display our timeline, we created a Gantt chart. On the chart, each row represents a certain task where each column represents how long the certain task took to complete as shown in Figure 1.

Figure 2- Gantt Chart, Project Timeline

During WPI’s first terms of classes in the 2016 academic year, starting August 25th and ending October 13th, our team worked on constructing the Introduction, Background and starting the Methods section of the report. We also began analyzing data sent by Martin Rater to begin familiarizing ourselves with the MRF. When we first arrived at the USPTO, our team reached out to QLs to set up interview dates and worked on analyzing data. At the start of the second

(40)

18

week we began interviews with QLs and sent emails to both SPEs and TQASs to set up dates for interviews. The specifics and details of all interviews are discussed in depth in the following chapter. After all data was gathered, we compiled the data and begin creating reports to provide to the USPTO.

(41)

19

Chapter 4: Findings and Results

This chapter contains the results of our objectives outlined in Chapter 3. We begin by presenting the results of the interviews conducted with United States Patent and Trademark Office (USPTO) employees. This is followed by analysis of Master Review Form (MRF) data and related findings.

4.1 Objective 1: Needs of USPTO Management Regarding Collected Data

After conducting interviews with eight Quality Leads (QLs), six Training Quality Assurance Specialists (TQASs) and two Supervisory Patent Examiners (SPEs), their responses were recorded, reviewed and summarized. Initially, responses were separated into two categories, data needs and reporting design needs. After the data was separated into these categories, the responses were coded and analyzed. Responses were organized into specific groups by keywords and the response rates were then added.

4.1.1 Master Review Form Data Reporting Needs

There were three questions classified as concerning data needs (for interview questions and interview script, see Appendix B). The first two questions, 1.1 and 1.2, were primarily icebreaker questions used to give context to the following discussion. Question 1.3 regarded the use of MRF Data by the employee's position and by the USPTO as a whole. These questions were used to determine what data collected by the MRF would be most useful to be reported back to the interviewee. These two questions were coded based on the frequency of responses. Figure 3 below shows a word cloud of interviewee responses to question 1. In the word cloud, responses that are displayed bolder and larger are responses that occurred more frequently within the interview.

(42)

20

Figure 3- Word Cloud of Interviewee Responses to Question 1.3

From the word cloud it is clear that employees care most about which statutes the data relates to as well as whether or not the review was ranked Needs Attention (AT) or Significantly Deficient (SD). In particular, employees cared most about statutes 101 (usefulness), 102 (prior art) and 103 (obviousness). The interviewees also mentioned statute compliance, the terminology used in the most recent version of the MRF to describe review ratings. Compliant refers to No Issues Found (OK) reviews, partially compliant refers to AT reviews and non-compliant refers to SD reviews. This allows them to see exactly where the issue lied in the examination. These responses embody what employees are looking for in the MRF data and what they want to be reported to them.

4.1.2 Internal Report Design Needs

Thirteen interview questions were drafted concerning the design of reports. Of these questions, 2.1 and 2.5 a-c were the primary focus of our coding and analysis. Question 2.1 asked the interviewee for an example of a way that data had been reported to them in the past that was clear and effective. The most popular answer to this question was that employees preferred receiving data directly in an Excel spreadsheet, allowing them to perform their own analysis, as seen in figure 4 below.

(43)

21

Figure 4- Preferred Method of Receiving Data Based on Interview Responses

While questions 2.2 through 2.4 asked them specific advantages and disadvantages of this example. Question 2.5 a-c asked for opinions about three types of reports that were being considered: Snapshots, Canned Reports and Ad Hoc analysis. Questions 2.8, 2.9 and 2.10 responses were recorded as ordinal data and therefore did not need to be coded before analysis. The remaining questions were used to provide context to the coded responses and were summarized to better understand and explain the results of the analysis of the coded responses.

Questions 2.8 through 2.10 discuss the three reporting methods by having interviewees score each report from 1 to 10 based on usefulness, as seen in figure 5 below. This graph indicates that most employees found the Ad Hoc Analysis style of reporting to be highly useful. Canned Reports were ranked lowest on average suggesting that they would be the least useful report. However, all three reporting models were ranked as having above average usefulness. This could mean that employees have a use for all three forms of reporting, or that all three methods are desirable in comparison to what they are currently receiving.

(44)

22

Figure 5- Average Rating of the Three Models Presented During Interviews

Because Ad-Hoc reporting received the highest rating, we prioritized analysis of question 2.5c response, which concerned the pros and cons of Ad-Hoc analysis. Figures 6 and 7 below show the results of 2.5c. From these figures we can see that employees identified customizability as the chief advantage of Ad Hoc analysis, while also identifying the ability to sort or filter results and the familiar format as benefits of using this model. Several employees also mentioned the usefulness of being able to perform the analysis themselves without relying on anyone else. However, there were also concerns about the learning curve of using this type of tool, as well about the process of training sessions being time consuming and results looking cluttered. Some employees also felt that Ad-Hoc software was not an appropriate form of reporting because using it was not a part of their job. Both the positive and negative attributes suggested were taken into consideration when developing a decision matrix for selection of an Ad-Hoc tool as discussed in section 5.

(45)

23

Figure 6- Advantages of Ad Hoc Analysis Based on Interview Responses

(46)

24

We also analyzed questions 2.5a and 2.5b in order to see if employees had similar compliments and complaints about Canned Reports and Snapshots. The purpose of analyzing these questions after deciding to go with Ad Hoc is for future reports. The results are shown in figures 8 and 9, respectively. The figures show that employees frequently mentioned the inclusion of necessary info and the usage of tables as advantages of a Canned Report model; however, they cite the lack of customizability, as well as the potential problem with including too much information, as disadvantages of the model.