2021 Oracle Artificial Intelligence/Machine Learning Survey Report. February 2021

(1)

1

2021 Oracle Artificial Intelligence/Machine Learning Survey Report

February 2021

(2)

Introduction

The Oracle Artificial Intelligence/Machine Learning Survey conducted by WCG CenterWatch was initiated in the fourth quarter of 2020 to evaluate the clinical trials industry’s attitudes toward and plans for using AI/ML. Respondents were asked to indicate their level of experience with the technologies – business understanding, technical understanding, a combination of the two or no understanding.

Executive Summary

Experience

When asked to indicate their level of experience with AI/ML, 41% said “no understanding,” a response that shows a need for more education and awareness around the topic.

The next largest group to the “no understanding” group included those who said they had a “business understanding”

of AI/ML at 26%. Respondents who said they had “technical understanding” of the topics and a combination of both business and technical understanding were nearly even at 16% and 17%, respectively.

Only respondents who indicated they had some level of understanding of the topics were asked to answer subsequent survey questions about the benefits, prerequisites, uses, challenges and implementation of AI/ML. However, survey questions about the impact of AI/ML on eight different characteristics of trial operations included the entire pool of respondents — both those who said they had no understanding and those who said they had some level of experience with the technologies.

(3)

3

Executive Summary (continued)

Benefits

A substantial number of respondents (62%) indicated that increased trial speed was the top benefit of AI/ML; 65% of those with business understanding put speed at the top of the list. Improved quality came in second at 50%.

Reduced cost was the top choice among 54% of respondents with both business and technical understanding.

Increased innovation and increased safety were most popular among the technical understanding group at 50% and 44%, respectively.

“Other” responses to the benefits question included better representation of rural regions (telemedicine), greater efficiency in meeting trial participation efforts for patients and sites, identifying trends, easing inclusion, better and quicker aggregation of trial outcomes and results, and advancements in data collection.

Ways of Using AI/ML

While 17% said they were not currently using AI/ML (the top answer), 16% said they were using it for data

management. Regulatory/IRB approval came in at the bottom of the list at 7%, tying with central and local testing.

Prerequisites

Asked what the most important characteristics needed to use AI/ML successfully were, quality data topped the list for all respondents at 23%. Availability of staff with technical knowledge came in a close second at 22%. Availability of real-time data came in third at 16%.

(4)

Executive Summary (continued)

Challenges

Lack of knowledge was the most important challenge to adopting AI/ML at 21%, followed by funding at 17% and changes in procedures and systems at 17%. Not surprisingly, 20% of respondents who said they had a business understanding of AI/ML ranked funding as the top challenge. Changes in procedures and systems came in third overall at 17%, although 20% of respondents with technical understanding of AI/ML ranked it as the top challenge.

Unclear regulatory acceptance ranked fourth overall with 16% of all respondents but was ranked as the No. 2 challenge among respondents with technical understanding at 18%.

Implementation

The largest number of respondents, 39%, said they planned to implement AI/ML through in-house initiatives, followed by strategic partnerships with 26% of all respondents. Vendors and industry collaborations came in at the bottom of the ranking at 14% and 13%, respectively.

Impact of AI/ML

Faster site activation was cited as AI/ML’s most positive impact by 40% of all respondents, although respondents with both business and technical understanding of the topics (76%) rated that the highest. Reduced risk came in second overall at 32%. As for negative impact, 57% of respondents expected AI/ML to lead to an increase in system

complexity. Few respondents overall (7%) expected AI/ML to have a negative impact on trial staff levels.

(5)

5

Executive Summary (continued)

Conclusion

Although the three different level of understanding groups provided similar responses in most cases, there were some clear standouts in each of the groups. Respondents in the technical understanding group said they primarily use AI/ML for trial and document management. They also found increased innovation to be its biggest benefit and expressed the most concern about system changes.

Respondents with business understanding focused mostly on the benefit of increased speed and were most concerned about the cost of the technology.

Respondents with a combination of business and technical understanding valued the promise of increased speed and improved quality most highly, expected more changes in patient engagement and loyalty, and were most concerned about lack of institutional knowledge and technical capabilities.

All three groups, however, agreed that increased trial speed and improved quality would be the most valuable outcomes of using AI/ML.

(6)

Survey Methodology

Survey period: Oct. 27 to Dec. 16, 2020 Number of responses: 311

Respondents by type: ' Sites (academic and independent), 72%; sponsors, 12%; CROs and nonclinical CROs, 9%;

consulting companies, 3%; other, 4% (including software provider, nonprofit organization, market research organization, data analytics provider and government entity.

Respondents by functional

area of responsibility: ' Clinical operations, 43%; senior management, 12%; regulatory, 8%; quality, 7%;

data management, 6%; medical affairs, 6%; information technology/management, 3%; pharmacovigilance, 1%; statistical sciences, 1%; other, 12% (including study/site coordination or management, regulatory or compliance, contracting, financial

management, facilities, business development and outsourcing).

(7)

7

Section 1 – Respondent Profile

7

(8)

Respondents by Type of Organization.

80%

72%

Percent%

70%4 60%4 50%4 40%4 30%4 20%4

9% 12%

10% 3% 4%

0%

Sites (academic and CRO/Nonclinical Sponsor (pharma, Consulting Other*

independent) CRO biotech and medical Company device)

*Other types of organizations responding included software provider, nonprofit organization, market research organization,*

data analytics provider and government entity.*

Sample size: Total (n= 311)

Sites (academic and independent) (n= 224), CRO/Nonclinical CRO (n= 28), Sponsor (pharma, biotech and medical device) (n= 36), Consulting Company (n= 11), Other (n= 12)*

(9)

Section 1 – Respondent Profile

9

Mean Number of Trials Sponsored in 2020, by Type of Organization

140 120 117

Mean%

100 80 60

40 41 30

20

0 0

0

Sites (academic and CRO/Nonclinical CRO; Sponsor (pharma, Consulting Other; 4% of independent); 72% of 9% of respondents biotech and medical Company; 3% of respondents

respondents device); 12% of respondents

respondents

Note: 53 site respondents indicated that they sponsored trials in 2020, which likely is a misinterpretation of the original*

question.*

Sample size: Total (n= 84)*

Sites (academic and independent) (n= 53), CRO/Nonclinical CRO (n= 0), Sponsor (pharma, biotech and medical device) (n= 30), Consulting Company (n= 0), Other (n= 1)*

(10)

Mean Number of Trials Conducted in 2020, by Type of Organization

Sites (academic and independent) (n= 128), CRO/Nonclinical CRO (n= 18), Sponsor (pharma, biotech and medical device) (n= 0), Consulting Company (n= 8), Other (n= 9)

42

47

0

30

37

0 5 10 15 20 25 30 35 40 45 50

Sites (academic and independent); 72% of

respondents

CRO/Nonclinical CRO;

9% of respondents Sponsor (pharma, biotech and medical

device); 12% of respondents

Consulting Company; 3% of

respondents

Other; 4% of respondents

Mean

(11)

11

Respondents by Functional Area of Responsibility

Section 1 – Respondent Profile

*Other areas of responsibility respondents cited included study/site coordination or management, regulatory or compliance, contracting, financial management, facilities, business development and outsourcing.

1%

12%

8%

7%

1%

13%

6%

3%

6%

43%

Statistical Sciences Senior Management Regulatory Quality Pharmacovigilance Other*

Medical Affairs Information Technology/Management Data Management Clinical Operations

(12)

Respondents’ Level of Experience with Machine Learning

Business Understanding (n= 63), Technical Understanding (n= 38), Business and Technical Understanding (n= 42), No Understanding (n= 99)

41%

26%

17% 16%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

No Understanding Business

Understanding Business and

Technical Understanding Technical Understanding

Percent

(13)

13

Section 2 – Survey Results

13

(14)

Question 1: What are the top three benefits you think machine learning will have on clinical trials in the next 10 years?

19%

17%

13%

15%

10% 10%

14%

2%

22%

13%

14%

9% 9%

14%

16%

3%

20%

14%

18%

11% 11%

7%

20%

1%

21%

15% 15%

11%

10%

11%

16%

2%

0%

5%

10%

15%

20%

25%

Increased speed Increased

innovation Reduced cost Increased safety Reduction in

failures Enhanced risk

management Improved quality Other*

Percent

Technical understanding Business understanding Business and technical understanding Average of all respondents

*Other responses included better representation of rural regions (telemedicine), greater efficiency in meeting trial participation efforts for patients and sites, identifying trends, easing inclusion, better aggregation of patient-reported outcomes and all outcomes and results more quickly, and advancements in data collection.

Note: The 41 percent of respondents who indicated they have “No Understanding” of machine learning did not answer questions 1 through 5.

(15)

Section 2 – Survey Results

15

What are the top benefits you think machine learning will have on clinical trials in the next 10 years (select top three)?

62%

50%

45%43%

33% 32%

30%

6%

0%

10%

20%

30%

40%

50%

60%

70%

Percent

58%

65%

58%

47%

42%

54%

43%

41% 39%

39%

50%

22%

43%

32% 30%

26%

44%

32%

28% 30%

2%

8%

5%

0%

10%

20%

30%

40%

50%

60%

70%

Business and technical

understanding Business understanding Technical understanding

Percent

Increased speed Improved quailty Reduced cost Increased innovation

Enhanced risk management Increased safety Reduction in faliures Other

(16)

Question 2: In what areas are you using machine learning?

4%

17%

10%

11%

16%

9%

7%

9%

10%

Other*

We are not currently using machine learning.

Quality Management Statistics Data Management Safety Reporting Central and Local Testing/Third Parties Regulatory/IRB or IEC, and other approvals Site Management/IP and Trial Supplies Trial Management/Central Trial Documents

Average of all respondents

*Other responses included patient identification, research on predictive analytics, labs and conducting industrywide discussions.

(17)

Section 2 – Survey Results

17

Question 2 (cont.): Uses of Machine Learning by Respondents’

Level of Understanding

3%

10%

20%

10%

6%

9%

11%

9%

5%

29%

10%

11%

8%

6%

7%

4%

12%

11%

12%

18%

7%

6%

8%

15%

Other We are not currently using machine learning.

Quality Management Statistics Data Management Safety Reporting Central and Local Testing/Third Parties Regulatory/IRB or IEC, and other approvals Site Management/IP and Trial Supplies Trial Management/Central Trial Documents

Technical understanding Business understanding Business and technical understanding

(18)

Question 3: How have you implemented or are planning to implement your machine learning capabilities?

44%

7%

28%

9%

12%

36%

14%

27%

12% 11%

41%

17%

23%

17%

1%

39%

14%

26%

13%

8%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

In-house initiative Vendor project Strategic partnerships Industry consortium

collaboration Other*

Percent

*Other responses included none, not implementing and “I don’t know.”

(19)

Section 2 – Survey Results

19

How have you implemented or are planning to implement your machine learning capabilities? (select all that apply)

39%

26%

14% 13%

8%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Percent

41%

36%

44%

23%

27% 28%

17%

14%

7%

17%

12%

9%

1%

11% 12%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Percent

In-house initiative Strategic partnerships Vendor project Industry consortium collaboration Other

(20)

Question 4: In your opinion, what are the three most important prerequisites for making machine learning a reality?

23%

7%

17% 18%

25%

8%

2% 2%

21%

9%

14% 14%

21%

11%

5% 5%

23%

12%

10%

17%

23%

8%

4% 3%

22%

9%

14%

16%

23%

10%

4% 3%

0%

5%

10%

15%

20%

25%

30%

Staff with technical knowledge

Staff with business knowledge

Large data sets Real-time data Quality data Applicable industry standards

Granular metrics Other*

Percent

*Respondents with technical understanding cited cost and collaboration. Respondents with business understanding cited granular business use cases to test the technology, regulators’ blessing, compatibility with current software systems, investing in the database infrastructure and programs, safety, focus on user interface design, subject matter expertise in therapeutic areas partnered with technical knowledge of how to normalize volumes of qualitative data, and ethical principles. Respondents with both business and technical understanding cited willingness of early adopters to understand and apply the potential benefits of AI and ML in clinical research, broad multicenter collaborations, and data and privacy security.

(21)

Section 2 – Survey Results

21

In your opinion, what are the most important prerequisites for making machine learning a reality? (select top three)

23%22%

16%

14%

9% 9%

4% 3%

0%

5%

10%

15%

20%

25%

Percent

23%

21%

24%

22%

20%

22%

17%

14%

18%

10%

15%

17%

12%

9%

7%

8%

11%

8%

4% 5%

2%

3%

4%

2%

0%

5%

10%

15%

20%

25%

Percent

Quality data Staff with technical knowledge Real-time data

Large data sets Staff with business knowledge Applicable industry standards

Granular metrics Other

(22)

Question 5: In your opinion, what are the three biggest challenges facing the adoption of machine learning?

20%

17%

15%

18%

16%

5%

7%

1%

17%

13%

20%

16%

22%

4%

7%

3%

16%

15% 16%

15%

23%

4%

7%

4%

18%

14%

18%

16%

21%

4%

7%

3%

0%

5%

10%

15%

20%

25%

Changes required to procedures processes and/or

systems

Change management (i.e., internal

resistance)

Funding Unclear regulatory acceptance

Lack of institutional knowledge and/or technical capabilities

Unable to meet prerequisites

Management support Other*

Percent

*Respondents with business understanding cited taking into account human factors, commercialization of sites causing reduced quality of data management, unclear methods for detecting and correcting AI errors and biases and effective comparators for its conclusions, generalizability and validation standards. Respondents with business and technical understanding cited the need for planning, resistance from individuals – both direct users and management, unproven value proposition, IT security and institutional HIPAA compliance, and lack of IT skills.

(23)

Section 2 – Survey Results

23

In your opinion, what are the biggest challenges facing the adoption of machine learning (select top three)?

21%

18%17%

16%

14%

7%

4% 3%

0%

5%

10%

15%

20%

25%

Percent

23%

22%

16%

20%

15%

16%

17%

20%

14%

15%

18%

14%

13%

17%

7% 7% 7%

4%

3%

5%

4%

2%

1%

0%

5%

10%

15%

20%

25%

Percent

Lack of knowledge capabilities Funding

Change required to procedures, processes, systems Unclear regulatory acceptance

Change management Management support

Unable to meet prerequisites Other

(24)

Question 6: In your opinion, how would machine learning affect clinical staff?

19%

25%

56%

20%

16%

64%

32%

13%

55%

27%

17%

56%

0%

10%

20%

30%

40%

50%

60%

70%

More changes Fewer changes No changes

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(25)

Section 2 – Survey Results

25

Question 7: In your opinion, how would machine learning affect quality issues?

44% 44%

12%

25%

61%

14%

45% 45%

11%

37%

31% 32%

0%

10%

20%

30%

40%

50%

60%

70%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(26)

Question 8: In your opinion, how would machine learning affect identified risks?

56%

25%

19%

43%

27% 30%

61%

21% 18%

53%

15%

32%

0%

10%

20%

30%

40%

50%

60%

70%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(27)

Section 2 – Survey Results

27

Question 9: In your opinion, how would machine learning affect customer loyalty (patient engagement)?

19%

6%

75%

34%

12%

55% 54%

3%

42%

24%

19%

57%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(28)

Question 10: In your opinion, how would machine learning affect patients?

44%

12%

44%

37%

9%

54%

50%

8%

42%

28%

15%

57%

0%

10%

20%

30%

40%

50%

60%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(29)

Section 2 – Survey Results

29

Question 11: In your opinion, how would machine learning affect costs?

44%

34%

22%

43%

32%

25%

42%

32%

26%

47%

24%

29%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(30)

Question 12: In your opinion, how would machine learning affect system complexities?

66%

12%

22%

73%

13% 14%

66%

18% 16%

69%

9%

22%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(31)

Section 2 – Survey Results

31

Question 13: In your opinion, how would machine learning affect faster site activations?

50%

16%

34%

43%

5%

52%

76%

6%

18%

41%

10%

49%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Percent

Technical understanding Business understanding

Business and technical understanding No understanding

(32)

In your opinion, would machine learning result in more/no change/or fewer…?

40%

32%

25%

19%

6%

-7%

-15%

-57%

-80% -60% -40% -20% 0% 20% 40% 60%

Percent

More/less system complexities

More/less costs

More/fewer clinical staff

More/fewer quality issues

Less/more patient

engagement/customer loyalty Fewer/more patients

Fewer/more identified risks

Slower/faster site activations Positive impact

Negative impact

(33)

Section 2 – Survey Results

33

300 N. Washington St., Suite 200 · Falls Church, VA 22046 · www.centerwatch.com