The Integration of Secure Programming Education into the IDE

(1)

The Integration of Secure

Programming Education into the

IDE

Bill Chu

Heather Lipford

Department of Software and Information Systems

University of North Carolina at Charlotte

1 02/15/13

(2)

Motivation



Software vulnerabilities are a major contributor to information

security problems



Many vulnerabilities can be avoided by adhering to secure

programming practices, such as input validation/sanitization



Example: SQL injection



Discovered a long time ago



Easily preventable (e.g. using prepared SQL statements)



Yet it is still a problem for about 7% of enterprise web applications



We are not doing enough for secure programming

education

2 02/15/13

(3)

Challenges



Most CS courses do not talk about secure programming



Courses focus on subject matters (e.g. OS, Database, AI)



Not enough time for all primary topics



Difficulties of providing a secure programming course



May be elective, and thus reach only a subset of computing

students



There may not be room in the degree program



Students may not take them until they are juniors or seniors,

and thus may have learned bad secure programming habits

without being aware

3 02/15/13

(4)

Our Approach



To provide training in, and support for, good secure

programming practices as part of the tools that students

use to program

throughout all courses



Serves as a continuous educational opportunity that

adds to or reinforces the students’ secure programming

training while they are performing their coding

activities.

(5)

Integrative Learning Theory



_{Our brains connect stored information based on}

use patterns.



_{Stronger connections make it easier to recall and}

apply information.



_{When the learning process facilitates these}

(6)

ESIDE Demo



ESIDE stands for Education Security in Integrated

Development Environment



Based on Eclipse Java Development Tooling (JDT)



https://www.youtube.com/watch?v=VjzlpccMjTM



Major features



Instant security warnings (e.g. input validation/output

encoding/dynamic SQL statements)



Code generation



Interactive annotation



Explanations webpages

(7)

(8)

(9)

Seven Mixed Method Studies

Early Student Studies

Elon

CS1 Classroom Activity

JCSU CS1 Focus Group

JCSU CS1 Interactive Walkthrough / Interview

Elon

CS1 Points Incentive on 10 Day Assignment

Advanced Student Studies

UNCC NBAD Assignment Deployment

UNCC NBAD Semester Deployment

Intermediate Student Study

(10)

ESIDE Research Questions

RQ1 - Can ESIDE positively influence a student's secure

coding mindset?

RQ3 - Can ESIDE motivate and incentivize students to

implement secure programming practices?

RQ2 - Can ESIDE motivate and incentivize students to learn

about secure programming (vulnerabilities, coding, etc)?

(11)

Advanced Students Study I



Students enrolled in a web-programming course



20 students with no prior secure programming training

 Worked on their programming assignments in the lab using Eclipse/Java for 3 hours  Two students in the lab at same time, based on their sign ups

 Students are at different stages of project implementation



Study setup



Screen recording & ESIDE logs



Pre- and post-tests

 On secure programming knowledge

 Two exams each with 18 true/false questions  Counterbalanced



Semi-structured interview

 Perceptions about warnings, explanations pages  Actions they took with ESIDE

 What they learned

(12)

Sample Test questions



(Choose ALL that apply) For the following statements, which of the

declared variables will require subsequent validation?

 Integer time = System.getTime();

 String title = request.getParameter(“Title”);

 String message = in.readLine(); (in is an instance of BufferedReader)  Boolean complete = myOrder.checkComplete(orderData);



(Choose ALL that apply) Which of the following statements may be

subject to SQL injection?

 String sql = "SELECT composer, date FROM symphony WHERE conductor = ?";

prest = con.prepareStatement(sql);

prest.setString(1,request.getParameter(“conductor”)); ResultSet rs1 = prest.executeQuery();

 statement.executeUpdate (“UPDATE users SET item=“ + user.getItem() + “””);

(13)

User Study Results



20 participants



12 male Master’s students, 6 female Master’s students and 2 male

undergraduate students



4 reported they know a few but very limited security concepts



None of them had secure programming experience before the study



Test scores



Student behavior observations



Student perception interviews

(14)

Test Results Analysis



10.3 increased in average scores after using ESIDE



Average pre-test score: 53.03



Average post-test score: 63.33



The differences between pre- and post test scores are statistically

significant



Wilcoxon Signed Ranks Test on raw test scores t (Z = -2.931, P = 0.003)

14 02/15/13

Mean

Std.

Deviation

Min

Max

Pre- test

percentage

53.03%

12.81 33.33%

83.33%

Post- test

percentage

63.33%

10.86 44.44%

77.78%

(15)

Student Behavior



461 distinctive warnings generated overall



70%

clicked (321/461

) 

47%

resolved (217/461)

15 02/15/13

(16)

Advanced Student Study II



Same population of students as Advanced Student

Study I



Observed 64 students use ESIDE over the course of

a semester

02/15/13

(17)

Interaction Findings

Study Length (days) Avg Days ESIDE Ran Interaction avg per day Interacted ≥1 Interacted >10 NBAD F12 10 3.57 1-7 range 7.3 6 6 NBAD Sp13 95 5.56 1-34 range <1 32 9

Usage

Interaction / Impression

[p 86] – “it wasn't too disruptive or in the way” [p183] – “easy to use – just had to click in one spot” [p190] – “the popups are great and super helpful”

(18)

Advanced Findings

RQ1- Can ESIDE positively influence a student's

secure coding mindset?

[p 9] – “I started to be able to predict when the icon would flag my code before I

even wrote it”

[p10] – “I hadn’t realized my code was insecure”

[p11] – “I found it helpful and enlightening to see points of insecurity in my code”

[p190] – “the explanations helped me understand why certain code practices are

(19)

Advanced Findings

RQ2 - Can ESIDE motivate and incentivize students to

learn about secure programming?

[p 26] – “the explanations helped me learn about dangerous code practices” [p189] – “I learned why my code was unsafe”

Pre (SD)

Post (SD)

Change

NBAD F12

(n8)

60.4 (16.8)

65.4 (13.5)

5.0 ran

≥

4 days (n4)

54.4 (19.2)

69.1 (18.3)

14.7*

ran < 4 days (n4)

66.4 (13.9)

61.8 ( 7.4)

-4.63

NBAD Sp13

(n18)

30.6 (25.1)

38.9 (23.5)

8.3 ran

≥

5 days (n9)

27.8 (26.8)

43.3 (30.0)

15.5*

ran < 5 days (n9)

33.3 (24.5)

34.4 (15.1)

1.1

(20)

Advanced Findings

RQ3 - Can ESIDE motivate and incentivize students to

implement secure programming practices?

[p50] – “I was always concerned about getting the project done and not necessarily the quality (in regards to security) of the finished product”

[p11] – “I did not use ESIDE to correct insecurities as that was not my primary concern”

[p16] – “I have not had a chance to work with ESIDE because my main focus was getting [the] project done with full functionality before the due date”

Assignments Reviewed auto-gen secure code Secure code in homework NBAD F12 8 5 2 NBAD Sp13 256 4 0

(21)

Advanced Themes

Functionality Mindset

Timing

Students do learn secure programming from

ESIDE

(22)

Early / Intermediate Overview

School / Class Date Level Participants Female / Male Study Type

Elon CS1 F12 early 61 25 / 36 Classroom Activity

Elon CS2 Sp13 intermediate 22 9 /13 Classroom Activity

JCSU CS1 Sp13 early 5 1 / 4 Focus Group

JCSU CS1 F13 early 4 1 / 3 Walkthrough

(23)

Study #1: Early: Elon CS1

Goal:

Explore receptiveness to ESIDE support, assess support level alignment

to student knowledge and abilities, obtain student perceptions of ESIDE

interaction

Study:

Single lab setting, three sections, 61 participants (25F / 36M). Students

wrote a program that accepted a username and averaged grades.

Data:

Pre/post

surveys

, assignment code, interaction logs, instructional page

visits, group interaction notes, informal interviews, open ended survey

questions.

Quick Findings:

Appreciated ESIDE, code modification willingness, course

content confusion, encouraged to ignore Eclipse warnings –

[p56] “Eclipse pop-up windows are always kind of vague”.

(24)

Study #4: Early: Elon CS1

Goal:

Examine the influence of a 5% points incentive has on coding behavior

and knowledge gain and explore the receptiveness and appropriateness of

the ESIDE support.

Study:

Assignment study with 57 (17F/40M) students from 3 sections. Students

completed a review assignment focused on array manipulation with simple

i/o data acquisition.

Data:

Pre/post

surveys

, assignment code, interaction logs, informal interviews,

open ended survey questions.

Quick Findings

:

Students un-expectantly struggled with the assignment, 34/43

completed the assignment, those with no incentive

(25)

Study #5: Intermediate: Elon CS2

Goal:

Assess how well ESIDE’s materials and evaluation survey aligned with

intermediate student knowledge and obtain intermediate student

impressions of their ESIDE interaction.

Study

:

Single lab setting, one section, 22 (9F/13M) participants. Students worked

on an app which took a name from a user, checked it against a female list

and a male list (as found in a separate file), then return the number of

occurrences.

Data:

Post-

survey

, interaction logs, ethnographic notes.

Quick Findings:

Two learning topic difficulty e.g., course instruction vs.

ESIDE instruction (Directories - Path Traversal) .

(26)

Early / Intermediate Findings

RQ1- Can ESIDE positively influence a student's

secure coding mindset?

[p 54] – “it allowed me to be aware of security risks”

[p 93] – “It helped me to learn more about how to make better, more secure code” [p134] – “it shows me that though my code will run with proper user inputs, its still

vulnerable“

[p137] – “showed me what exactly insecure code was”

[p124] – “I believe it gave some good information about something I did not know about” [p103] – “as a beginning programmer it was nice for me to learn more about this for future

(27)

Early / Intermediate Findings

RQ2 - Can ESIDE motivate and incentivize students to

learn about secure programming?

n pre Pre SD post Post SD Gain

Section 1: Points: V2 11 62.8% 9.05 73.6% 6.1 10.7%* Section 2: No Points: V1 15 69.5% 14.42 71.4% 13.68 1.9% Section 3: No points: V2 (exam) 9 67.7% 9.69 78.8% 4.3 11.1%*

*Significant at the p <0.05 level. Note small sample size

[p 99] – “taught me a little about code security”

[p134] – “I wish I had more practice using ESIDE… I only had a week”

[p124] – “it gave some good information about something I did not know about” [p117] – “the help page taught me things about secure code that I didn't know before”

(28)

Early / Intermediate Findings

RQ3 - Can ESIDE motivate and incentivize students to

implement secure programming practices?

[p134] – “I was too focused on getting the assignment done”

[p143] – “I was more focused on getting my program to run correctly”

[p 90] – “for a class assignment I am not worried about the security of my code”

[p 89] – “this was a CSC130 homework assignment - I wasn't concerned about security”

CS1 Assignment Study Assignments collected Complete assignments Wrote secure code Section 1: Points 16 14 11 Section 2: No Points 14 11 2