Boston University
OpenBU
http://open.bu.edu
Metropolitan College BU Open Access Articles
2019-11-20
Case study - improving teaching
and learning effectiveness of
computer science courses
This work was made openly accessible by BU Faculty. Please
share
how this access benefits you.
Your story matters.
Version
Accepted manuscript
Citation (published version): Alex Elentukh, Vijay Kanabar. 2019. "Case Study - Improving
Teaching and Learning Effectiveness of Computer Science Courses."
Computer Science and Education in Computer Science. CSECS. Fulda,
Germany, 2019-06-28 - 2019-07-01.
https://hdl.handle.net/2144/40825
COMPUTER SCIENCE AND EDUCATION
IN COMPUTER SCIENCE
(CSECS)
15
thANNUAL INTERNATIONAL CONFERENCE
June 28th – July 1st, 2019
Fulda, Germany
COMPUTER SCIENCE AND EDUCATION
IN COMPUTER SCIENCE
(CSECS)
15
th
ANNUAL INTERNATIONAL CONFERENCE
June 28th – July 1st, 2019
Fulda, Germany
Chairmen:
Peter Peinl (Germany), Ivan Landjev
(Bulgaria), Tanya Zlateva (USA)
General Secretaries:
Petya Asenova (Bulgaria),
Eugene Pinsky (USA)
Editors: Petya Assenova, Eugene Pinsky, Peter Peinl
Copyright © 2019
New Bulgarian University 21 Montevideo Str., 1618 Sofia, Bulgaria
University of Applied Sciences Fulda 123 Leipziger Str., 36037 Fulda, Germany
Boston University, MET 1010 Commonwealth Avenue, 02215 Boston, USA
Improving Teaching and Learning Effectiveness of
Computer Science Courses – Case Study
Alex Elentukh Metropolitan College, BU Boston, US [email protected] Vijay Kanabar Metropolitan College, BU Boston, US [email protected]
Abstract — This paper summarizes our experience teaching several courses at Metropolitan College of Boston University Computer Science department over five years. A number of innovative teaching techniques are presented in this paper. We specifically address the role of a project archive, when designing a course. This research paper explores survey results from every running of courses, from 2014 to 2019. During each class, students participated in two distinct surveys: first, dealing with key learning outcomes, and, second, with teaching techniques used. This paper makes several practical recommendations based on the analysis of collected data. The research validates the value of a sound repository of technical term projects and the role such repository plays in effective teaching and learning of computer science courses.
Keywords — K.3.2 Computer & Information Science Education, Software Quality, Project Based Learning
I. INTRODUCTION AND CONTEXT
The reason we are here today because we want to learn, because we are, in fact, learning. It is not coincidental that verbs 'learn' and 'teach' have the same root in many languages, e.g. Chinese, Hebrew, and Russian. This paper could be quite useful for both, instructors (improving their teaching mastery) and students (selecting the most suitable learning method).
During past five years, BU course CS633 “Software Quality, Testing and Security Management” is offered in its current format. Four hundred students with a variety of professional interests and backgrounds have completed the course. The lecture track with theories and methodologies - parallels the practical track with implementation of an actual software system. A semester-long project is accounted for thirty percent of the grade.
Fig. 1. Cumulative Number of Retained Project Reports
The whole class is divided into teams of five, with each team building a useful software application. Each next project is able to benefit from experience of previous projects. In fact, the focus of the whole course has shifted. During initial year of running the course, the largest portion of learning had attributed to theoretical track. This is contrary to the report from current students, who confirmed that most significant part of their learning came from the term project. Seventy-six completed projects (along with their best and worst parts) represent an extensive body of knowledge, from which any student is able to learn.
As the repository with completed projects grows, the next projects 'stand on the shoulders of their predecessors' to become increasingly sophisticated and effective. Such positive feedback loop is expressed in the concept pictorial below.
Fig. 2. Effectiveness of Term Project Grows
Over the years, as the course matured, the share of learning attributed to a term project has increased.
It is a misconception to assume that improved effectiveness of Term Projects results in other learning avenues, e.g. Lectures, being squeezed out. On a contrary, the depth of the whole class is increasing, which triggers a very different conversation. Hence, Lectures become more effective. It is the same tide that lifts all boats, Lectures, Assignments, and Quizzes.
II. KEY LEARNING AVENUES
There are many ways to get familiar with a topic. We shall explore several avenues available to us. A well-balanced course explores a variety of avenues. A single method (for example,
Computer Science and Education in Computer Science 15th Annual International Conference ISSN 2603-4794. June 28th to July 1st 2019. Fulda, Germany
lectures-only or term project-only) does not work well for a diverse group of students.
A. Listen to a Lecture
Provides a good introduction to a subject. Heavily dependent on a proficiency of a lecturer. In most cases, this method is too passive for a student to deepen understanding. The beauty of this avenue is that a student receives an instant response to any question he/she might have. In fact, this avenue is deceptively simple. Since very few students have the trait to raise questions and fully digest responses - at the same time. A professor should be keenly aware of folks, who get irritated from a protracted lecture. Students might feel disempowered and restless. They want to do something tangible, to actively participate instead of observe on a sideline.
B. Reiterate
One of the ways to get to the "bottom of it" is to repeat lecture several times, going over the recording and notes that still have some hidden angles. Finding new patterns, refreshing and reviewing the same topics are effective methods to re-associate your memories and to recover some long-forgotten knowledge. This is particularly true in view of the line "nothing is new under the sun". The best part of reiteration is that you can re-examine material that is less familiar and skip material that is well-understood.
C. Complete assignments
It is not enough to repeat word-by-word your instructor. Paraphrasing is a great exercise that involves your own experience. Responding questions about grey-area scenarios forces a student to think independently.
D. Score Quizzes
Yes/No questions bear no judgment. They are impersonal. They are offered with a time restriction. A student must really know the answers to continue responding unmistakably to a brisk flow of questions. Answers should be rooted deeply in a student's psyche to come out without a hesitation. If you wake up in the middle of the night, you should respond to these questions the same way. Quizzes offer no time for a research or a discussion, so unless one really confident in a correct response, the quiz is lost.
An important finding resulting from this study is as follows. While reviewing quizzes with students, do not hesitate revealing the expected answer. The benefit of making the point clear - outweighs the negative of oversimplifying the quiz. Do not be afraid to give away the answer. Avoid the situation, when students do not comprehend a question or an answer. The context here is that quizzes are designed to be non-trivial. They include pictorials and up to a dozen possible responses. Hence, even if professor reveals responses, there are too many of them to remember. A student usually takes the same quiz several times and the final score is averaged from all attempts. There is an additional side effect from an uninhibited discussion about quizzes. It facilitates an atmosphere of openness and encourages focusing on learning and nothing else. In any case, quizzes should never be a straight replication of a covered material. Quizzes should enable a student for an additional leap forward.
E. Provide / receive comments to / from peers
People learn best from their own mistakes. Still, a better alternative is to learn from mistakes made by others. To find an error in someone else's work requires a higher level of understanding.
F. Implement independent term project
It is not enough to talk about a subject. If you are able to implement these ideas, to make them actually work - you are coming to a very different level of understanding.
G. Collaborate during term project
If you are able to tailor your own techniques and be flexible, in order to move along the development path together with other folks. If you are able to help someone else and to accept an alternative direction - you are coming to a higher level.
H. Reference well-known practitioners
It brings an additional respect, inspiration and appreciation to the subject.
I. Develop an argument
Being able to distinguish a strong argument based on facts from a weak argument coming from a hearsay - requires an in-depth understanding of the subject. It is not enough to know what to do, one needs to understand why we are doing it. To be able to think on your own feet, while supporting an argument against a strong opponent, does involve a certain confidence. J. Research a variety of books and papers
It creates a body of knowledge re-confirming and expanding your own experiences.
K. Tell Stories.
Finding analogous situation and articulating it with a well-defined beginning, apex and conclusion is a powerful method to acquire knowledge. Software teams commonly use the storytelling technique, to remind each other about the 'juiciest bugs that brought the house down' or about an 'elegant dozen-lines routine that replaced an obsolete thousand-line code'. Keep in mind the main purpose of storytelling, which is not to communicate to others, but to enhance your own understanding. In the beginning of adoption, storytelling should be scheduled and facilitated.
These avenues are not hierarchical. It seems logical to start with a lecture (see point A). Although it is quite possible to get into a context by engaging with a real project (see point F) and then search for various answers. Here is a logical order of learning for a new student.
First, scan through the information at the course's site. This will give you a taste for the topic. Make sure you sketch few questions to be investigated further about particularly unfamiliar parts.
Then participate in a live session. Listen to what other folks are asking about. Do not hesitate raising your hand. Even a simple social interaction goes a long way in appreciating the subject.
Improving Teaching and Learning Effectiveness of Computer Science Courses - Case Study
Examine the slides posted after the lecture. Professor usually keeps enhancing lectures. Pay attention to details that were added after a live session. One of the misconceptions is that professors keep repeating the same topics for many years. In fact, any reasonable instructor enhances a course based on various inputs and new ideas.
Within few days from the lecture, while the topic is fresh in your memory, make a first attempt at the quiz. It is important that all responses are clear. How does the quiz fit into the overall course? What are the learning objectives of each question? A wrong strategy would be to take the first attempt unprepared. Consider that all responses will be given to you after you hit the 'submit' button. Still, you do not want to miss researching lecture material, asking questions, etc. Another unavoidable task is to re-run through all quizzes prior to the final exam. While doing it, you might be discovering how much you do not know.
Listen to recording of a live session. Go right to the place of your specific interest and bypass introductory material.
It worth mentioning 'essays', as a commonly used technique. Students are asked to write a research paper on a particular relevant topic. Apparently, if you can clearly reflect on a certain notion and derive logical conclusions, it means the notion is, in fact, clear in your mind. Unfortunately, it does not always work the other way around. Namely, folks who are unable to express themselves clearly, could have various reasons for their inefficiencies. For example, English might be not their language of choice. They are unable to express nuances. Hence, even with a great degree of clarity and mental understanding, their writing could come out incoherent. Also, it puts a significant burden on an instructor to grade essays fairly. Since extensive essays (even with a well-documented criteria) are prone to a personal judgment.
III. STUDENT SURVEY -LEARNING METHOD
Since we have several avenues to follow, students are requested to rate their individual preferences at the table below. What methods for acquiring the skills and knowledge in the context of this course - are most effective?
TABLE I. EXAMPLE DATA - STUDENT SURVEY
A chart resulted from survey of one class is shown on Figure 3.
Fig. 3. Survey Result - Learning Method
Horizontal axis of the chart is matching the column Learning Method. The height of each bar is weighted by its effectiveness. For example, the bar 'Assignments' has a height of:
41 = 1*3+2*4 + 6*5
The first conclusion that becomes apparent from looking at the chart is that different people pick different methods. This speaks for the diversity of our students. There is no one method that fits perfectly all students, hence the selection of an appropriate individual avenue is important.
The benefit from administering this survey is for a student to discover and then focus on the specific method that suits him / her best. On another hand, without such a survey, students do not identify and do not apply the most powerful learning method in their possession.
IV. STUDENT SURVEY -TOPICS COVERED
The second survey administered to students represents a different slice of the course. It delves into topics covered in the class. Following is the exact language of the survey, which is the last question on the final exam. As one of the goals of this course is for you to adopt at your day job - some techniques covered in the class. Which of the best practices from the course do you plan to adopt?
Here are several examples of actual student responses.
I plan adopting the “All Pairs" tool. In my job as a Systems Verification Engineer at the company called Scientific Drilling, I run many data-driven automation scripts. I currently select test cases manually without the use of a tool. This has worked for the most part, but I can think of several instances where there are hundreds or thousands of possible combination. The use of All Pairs tool would definitely come in handy. It will allow me to visualize all possible combinations, as well as, reduce the
Alex Elentukh and Vijay Kanabar
number of test cases to a more manageable size. It is a great tool that can easily be incorporated.
In my role as a Cloud Architect at PwC, the estimation methods (t-shirt sizing) that we discussed in this course, will be helpful, when discussing my recommendations to clients.
There will be an increased focus on peer reviews. For too long applications have been delivered at my company without a second set of eyes looking upon what was done; not even at a high-level! The iterative peer review process that we employed during our term project was hugely helpful in being able to stay focused. I will be personally making it one of my fiscal-year goals this upcoming FY to employ at least ONE extra set of eyes on all applications that are delivered by our team. Figure 4 shows an example of a chart resulted from one class.
Fig. 4. Survey Result - Topics Covered
While examining trends of this survey, professor is able to re-orient the class toward those topics that will be actually applied. In other words, the survey confirms the usefulness of the class.
I witnessed at conferences as professors struggle with a notion of applicability of their courses. One way to resolve it is to simply ask students through a survey. In this course, such survey is elevated to an actual question on the final exam. Very few students ever failed this question. A wrong answer would be to provide a long (and unrealistic) list of techniques covered in the class that will be adopted within a shortest timeframe. All of us know that organizational adoption could take years.
V. HUMAN BRAIN IS THE BEST TOOL
Coincidentally, the title of this section matches the title of the Static Testing part of the course. Human brain works in many mysterious ways. How often do I go to bed with a puzzle in my head, only to wake up in a morning with a perfect solution waiting for me.
If you are not particularly good with quizzes, you should definitely prepare to your first attempt and then learn from results of each following attempt, making sure the score shows an upward trend.
The anatomical and somewhat humorous perspective, shown at Figure 5, gives a sense of each part of a human brain responsible for a different learning avenue. To mature into a well-rounded person, one has to advance all avenues. The reason this pictorial has a humorous side to it, since the learning process is infinitely more complicated.
Fig. 5. Human Brain Context
(Human Brain, MS Power Point Clip Art, retrieved April 2019)
As various people have very different talents. You are expected to recognize your own strong and weak traits to optimize the learning process.
VI. EXAMPLES -AQUIZ IS JUST AQUIZ
Following is an example of a quiz along with its evolution. __________________________________________________
Suppose you have two practices, (a) Defect Management and (b) Test Case Management. Which one of these two practices is more mature?
Practice 1. Defect Management is less mature Practice 2. Test Case Management is more mature
__________________________________________________
After staring at this quiz for few minutes a student called me to her place and whispered into my ear, so everyone in the class could hear ... This is mean... really mean !! What was going through her mind? Apparently, she assumed that the only thing I was trying to do is to catch her. Yet, I assured her, that catching anyone is a last thought I would ever have.
The goal of an instructor and the goal of a student are aligned. This is all about learning, nothing else but learning. In this example, a student automatically assumes that her goal is to get a good grade. Then she derives that, if the question is difficult, then the purpose of the question is to prevent her from getting a good grade.
Hence, my logical reaction was to have an immediate brainstorming session with the whole class, to solicit various constructive suggestions and to change the quiz, as 'the iron is
Improving Teaching and Learning Effectiveness of Computer Science Courses - Case Study
hot' so to speak. The improved quiz came out even more involved.
Consider a quiz integrating several learning methods. First, this is the meta quiz, as it has a question inside a question. Second, it alternates two options of 'less mature' for Practice 1 and 'more mature' for Practice 2, while both of these options, expect an identical response. The resulting language of the quiz is as follows.
__________________________________________________
Consider a student while taking the final exam is coming to the following question,
Suppose you have two practices, (a) Defect Management and (b) Test Case Management. Which one of these two practices is more mature?
Consider that a mature practice usually requires an advanced coordination and more difficult to implement in an organization.
Practice 1. Defect Management is less mature Practice 2. Test Case Management is more mature
What is the appropriate thinking going through the mind of a student while reading this question, select all that apply,
Student assumes that there is no right answer to the question and whatever he/she responds will result in a zero score
Student assumes that this is the tricky question that has the only purpose to mislead him/her
Student thinks that Practice 1 Defect management is less mature
Student thinks that Practice 2 Test Cases Management is less mature
Student appreciates the question and is going to tell his friends about Practice 1 Defect Management being more mature and Practice 2 Test Cases Management being less mature.
__________________________________________________
An additional goal here is to dissuade students from a common misconception that a quiz is a way for a professor to catch a student. Nothing could be further from the truth. Quiz represents an alternative method to learn; this is not the way to get back at a student. Question is, what happens if we do not alternate quiz options and we avoid any difficulties? You can be certain that the whole topic will be forgotten tomorrow. There will be no reward (read 'learning') without an appropriate challenge.
Following is another example of a quiz. It airs the notion that a student and a professor are partners collaborating on creation of a course material.
__________________________________________________
If you are a university professor trying to set up a quiz for your students. Consider the attribute of being ‘unambiguous’ is the only criteria for goodness of a quiz. Among the following two statements, select the best one.
Both, peer reviews and refactoring are group activities.
Both, peer reviews and refactoring are individual activities.
__________________________________________________
Fig. 6. Sweet Spot of a Quiz
The second option is the correct one, since it is unequivocally wrong, always.
VII. THE VERY FINAL EXAM
Preparation and completion of the final exam is a topic in itself. Students survey highlights this learning avenue among most effective, which might be a surprise for an outsider. Reality is that students work extra hard toward an exam. This is the point of no-return, where all delinquencies and missed classes are going to bite you. There is no way to postpone it, to ask for another week or another night. Final exam is the punch line. It’s the very last thing we do. From all perspectives, it is, in fact, ‘final’. Exam always presents an additional time pressure. Consider 80 questions with 5 possible responses that you have to digest within 3 hours. This gives you approximately a minute per response, causing a natural adrenalin rush, along with a huge stimulus for your thinking.
Only a small part of the final exam is an exercise in memorization. One is unable to get a high score without an underlying knowledge of the subject. The note from a student right after the exam reflecting on its dynamics, "it was brutal, but I love every minute of it", does sound as a compliment to the whole course and not just to the exam.
Many people said that everything they learned (in life) they learned during the night before final exam. As any exaggeration, this one has a grain of truth to it.
Should note that for all the years we are running this class, there has been a single person with a perfect score of one hundred percent. He scored a perfect score on all assignments and all quizzes ... for all classes he ever took at university. He grew up in India and he is currently working for Fidelity as a software lead engineer. Having said this, to get an 'A' for this class, one does not need to score a one hundred percent.
Sweet spot The quiz is unambiguous individual group refactoring
X
X
peer reviewsX
Alex Elentukh and Vijay Kanabar
VIII. LEVELS OF LEARNING
One method is apparently missing from the long list of learning avenues discussed previously. Namely the 'Teaching' should be added to the list. The following paragraphs expand this fundamental thought.
One might find the following sequence somewhat repetitive. It drives home the point of a comprehensiveness of a learning experience.
Unless you are able to articulate a subject in your own words and relate it to your own experience - you do not know.
Unless you are able to solve a problem and respond to a grey-area situation - you do not know.
Unless you are able to implement the idea, to make it actually work in brickNmortar - you do not know Unless you are able to find real defects in other people
work - you do not know
Unless you are able to effectively teach other people - you do not know
Unless you are able to bring the whole organization to a different level of understanding - you do not know.
Fig. 7. Maturity Paradigm - Levels of Learning
Apparently, there are many levels of learning, as Figure 7 pictorial provides a logical direction for our efforts.
IX. ALEARNING ORGANIZATION
Concepts discussed in this paper are relevant to both, university and business environments. Establishing an organizational training program is an imperative step in bringing everyone to the same page. Figure 8 shows a sequence of events necessary to adopt a certain technique.
In General Training, we invite a lecturer with extensive experience in this subject, to jumpstart the learning and assure consistency with latest industry standards. Working Group - (then) Pilots - (then) Adoption Metrics
demonstrate the degree of spreading of the practice. Who is doing, who is not doing it. For example, a Working Group develops a template and five teams out of ten are, in fact, using the template, then we have a
50% adoption. The same folks, who participated in a working group, will spearhead the concept and will play a role of change agents within their respective teams. Specific Training is based on examples from this
environment. It is usually recorded, e.g. with a Brainshark, and is available from a company site. It is common for a senior executive to record an introduction to such a training and to encourage adoption.
Fig. 8. A Learning Organization
X. PROJECT ARCHIVE
Screenshot at Figure 9 shows the menu of Project Archive. This is a durable repository of past projects readily available to students. It has been built by a team of students, while following a prudent software process, as a semester-long term project.
Fig. 9. Project Archive - Application UI Screenshot
Any student is able to easily find a project that is similar to the one he/she is contemplating, by searching on predefined keywords. To load a new project, mandatory fields include, Project Objectives, links to GitHub and Google Doc.
Project repository is fundamental for this course. On one hand, it enables a professor to bubble up best practices and establish standards between projects. On another hand, building a repository itself - represents a showcase of how a term project should be implemented.
XI. CONCLUSIONS
Here are several directions toward an effective course design. An attempt is made to generalize and apply these conclusions to any technical course.
Articulate key ideas Solve grey area issue Implement this idea Find defect for others Teach a class Adopt an org practice
Improving Teaching and Learning Effectiveness of Computer Science Courses - Case Study
A. Utilize a variety of methods
The same topic is explored through different angles to advance a consistent message. Our course is using five grading criteria. It is important to relate to students that these are not five ways to kick them. These are five avenues for all of us to learn new material.
While developing this thought further, should note the deliberate structure of this paper. Section II eleborates on eleven learning avenues, and section XI covers nine teaching techniques. Apparently, both lists are aligned, as both, students and instructors should utilize a variety of methods for the most effective learning and teaching.
B. Sharpen the message with a consistency of style
A balanced course has a well-defined pattern (routine, cadence) about it. Even such a minor detail as the 'number of questions on each quiz', should be consistent.
C. Assure course structure supports its content
If, as part of the class, you are covering a certain technique, e.g. peer reviews, then you should use the same technique to design the course. Namely you should review course material with students using peer reviews. In another example, we use 'simplicity' as a strong motivation for various refactorings of code. It is logical to extend the notion and review the course itself, to make sure it has no redundant parts and it fits into the category of being 'simple'.
D. Open all parts of a course
Students should be absolutely sure that additional effort they put into study is bound to result in a greater learning effect. The wrong thing to imply is that professor is not willing to share. This paper about teaching methods is openly discussed with students, as part of the course.
E. Actively solicit students’ feedback
Facilitating class synergy is much easier said than done. Make sure you document a student’s comment, then incorporate the comment into material and publicly reward the student. Both, course structure and content are influenced by students' comments throughout multiple runnings of a course. F. Slice a course two-ways
First slice shows the teaching methods you are using. Second slice shows the disciplines you are covering. Make sure you are soliciting feedback along both these slices. Make both slices as transparent as possible by publicly reviewing survey results. G. Define maturity paradigm
The notion of a logical progression from a ‘basic’ to an ‘advanced’, should be explicitly defined. All techniques covered in the class are decomposed and presented as part of a maturity ladder. Students should be clear where to start and how to make the second step. In fact, the same maturity paradigm should apply, whenever students adopt these techniques at their organizations and relate these concepts to others.
H. Engage students of all levels
The biggest misguidance is to underestimate students’ talent. Referring to the famous TV show, when a person glances at two huge rotating boards full of random numbers. Then he turns away and reads all numbers back. This is so much above an average human capability! The point of this example is that a course should apply to all levels, to junior and to experienced folks, at the same time. The message should have depth to it. The real challenge is to engage students with diverse backgrounds within the same class. Considering some students have little programming experience, how to engage them with developing a full blown software application? A clear definition of team roles goes a long way. At the end of a project, expert software developers, as well as folks with alternative inclinations, achieve the same goal - contribute to a successful implementation of a real-life application.
I. Unleash individual potential of every student
A wrong thing to do is to over-control and to stifle students' creativity. I am thinking of two avenues, for an instructor not to travel. First, an instructor should never tell students their precise roles, which they play within a team. And second, should not tell them the exact scope of their term project.
ACKNOWLEDGMENT
I would like to specifically thank Vijay Kanabar, Program Director of Project Management at MET CS, creating an environment conducive to most effective teaching and research. Acknowledgements of appreciation to Robert Schudy, who led the science of an in-depth analysis of students' grades.
Appreciations to Anatoly Temkin, Chair of Computer Science Department for his leadership, expert advice and encouragement, that are most valuable.
We would like to express our gratitude to Lou Chitkushev, Associate Dean for Academic Affairs for many years support of professional development, fostering an active learning and teaching.
Gratitude to Center for Teaching and Learning (CTL) for sponsoring and inspiring the most effective teaching that reshapes the future of pedagogy.
REFERENCES
[1] Alex Elentukh, "Software Quality, Testing and Security Management", BU MET CS 633 , https://www.bu.edu/csmet/cs633/
[2] Morgan McCall, Robert Eichinger and Michael Lombardo, "The Formula 70-20-10", Center for Creative Leadership. It proposes that on average, 70% of person's learning is experience-based, 20% from interacting with fellow employees and 10% is the result of formal training and reading. [3] Sallie Gordon, "Systematic Training Program Design", Prentice Hall,
ISBN-13: 978-0131003897 . Excellent book, written by a professional instructional designer, that taps into areas of cognitive psychology and ergonomics.
[4] Alex Elentukh, "Systems Reliability Policy at Motorola", ISSRE, 1994. An experience report of adopting an effective practice throughout a large organization.
[5] Dwi Sari, Alex Elentukh, "Systematic Implementation of an Assets Management System", CITSM 2018. Reflection of one of MET BU CS633 term projects presented at Jakarta yearly conference.
Alex Elentukh and Vijay Kanabar
