Implications of the Flat World for Evaluation
in Instructional Design and Technology
David D. Williams
Brigham Young University
Abstract
This article and an associated AECT presentation explore several implications of technology innovations for instructional design and technology evaluation. Questions raised about changes in society associated with technology, based on a reading of Friedman’s World is Flat, will be discussed by participants. It is expected that they will be able to use what is discussed to build evaluation more systematically into instructional design and technology projects and processes.
Introduction
In his book, The World Is Flat, (Friedman, 2006) reviews several technological innovations that have impacted political, economic, social, cultural, and many other dimensions of the world we live in. He explores implications of these changes for businesses, governments, countries, disciplines such as education, and individuals on various fronts. Many of these implications raise questions about the role of technology in instructional design and associated evaluations of quality.
After identifying several potential implications from Friedman’s work, this article explores three groups of questions raised and considers their value for guiding research projects and identifying alternative modes of designing and evaluating instruction. During the AECT session some of these questions will be presented to participants and they will be invited to discuss possible responses and to identify other questions they would like to answer through subsequent research and practice. Implications will be explored for building evaluation more systematically into instructional design and technology projects and processes.
This AECT session will be highly interactive and participants will be invited to think aloud with the presenter to answer the questions raised, raise other questions, and begin to articulate ways to answer these questions in terms of their own instructional design and technology projects and processes.
Implications from Friedman’s Flat World
When reading through Friedman’s book, I folded down scores of pages where implications for evaluation in instructional design and technology came to mind. Summaries of several of the points on those pages illustrate how thinking about a technologically flattened world raises questions worth exploring for the improvement of evaluation, which might also lead to improving instructional design and the use of technology to enhance learning. Several such summaries follow.
Open-Source Evaluation
In describing the open-source invention of the Apache Web server architecture, Friedman points out in several ways that the participants valued evaluation of their freely offered work by trusted experts provided quickly and for free. The open-source context provided this atmosphere and Apache grew so strong and so fast that IBM acknowledged the quality of their work and asked to join their community. They were allowed to do so but “the one thing the Apache demanded in return for their collaboration with IBM was that IBM assign its best engineers to join the Apache open-source group and contribute, like everyone else, for free” (p. 103). This late 1990’s development points out the power of open-source collaboration which utilizes peer evaluation delivered quickly by multiple experts who value giving and receiving feedback as a way to improve common projects that have implications for proprietary projects as well.
Friedman notes (p. 112) that the principle of “tap[ping] the innovative power of the community” can be and is being applied by many others in addition to software engineers. Examples include inviting the community scientists, engineers, and geologists to help a gold mining company find ore, involving citizens in evaluating community problems and giving feedback to their political leaders, and opening the newsroom and judgment of
news-worthiness of issues through blogging. Friedman summarizes the point this way, “The new model in business [as well as other areas] is that you involve your community and customers in an ongoing conversation about every aspect of your business, from the moment you conceive a product, to how you design it, to the supply chain that builds it and delivers it, to the way you collect and absorb customer feedback and respond more quickly to changing tastes” (p. 116). As with the other stories from Friedman’s “Brief History of the Twenty-First Century,” what are the implications for how evaluation is conceived as part of the instructional design process? What could be done to use open-source communities to provide rapid and expert feedback to teachers, learners and people trying to support them?
Technological Feedback Immediacy
Using Wal-Mart as a key example, Friedman points out several ways technology has been used to speed up feedback to employees and suppliers so they can more easily and cheaply provide their services, driving down the price of the supply chain, and ultimately the products customers want. He describes (on pages 159-162) the use of radios and satellites to guide drivers so they rarely drive an empty truck, computerized instructions and rate
feedback through headphones to pallet movers, opening sales and inventory databases to suppliers to enhance a just-in-time inventory program in which suppliers are viewed as partners, and the use of radio frequency identification microchips to replace barcodes which allows “Wal-Mart to track any pallet or box at each stage in its supply chain and know exactly what product from which manufacturer is inside, with what expiration date” (p. 161). These innovations provide evaluative feedback to people who need it to produce and ship products customers want and will buy. How would instructional design be different if similar technologies were used to enhance the speed and quality of evaluative feedback from learners, teachers, and other stakeholders designers want to serve?
Insourcing Means Trusting Your Evaluator
Using the United Postal Service (UPS) as a model, Friedman notes that by building trust through basic delivery services over years, this company has been allowed “deep inside” several companies, large and small, to evaluate what is wrong with supply-chains. They then provide recommendations and services that save the
companies money and even take over many of these functions so the companies they serve can focus on only a few of their original business tasks. UPS has used wireless technology to enhance accurate and efficient delivery and the Internet to encourage customers to track their own packages. They have found ways to use technology to allow them to collaborate more intimately and extensively than ever before. “In many cases today, UPS and its employees are so deep inside their clients’ infrastructure that it is almost impossible to determine where one stops and the other starts. The UPS people are not just synchronizing your packages—they are synchronizing your whole company and its interaction with both customers and suppliers” (p. 175). Is that level of trust possible in instructional design? Will teachers and learners trust instructional designers “deep inside” their learning experiences and are designers prepared to evaluate the needs and experiences of learners and teachers in sensitive but accurate and rapid ways so their “customers” can see that they have provided them with information they didn’t have without this help? Are there ways technology could help designers do this?
Everyone Can Be Their Own Evaluator
Using the Internet, wireless services, TiVo, Google, digitization, and other technological innovations, Friedman points out that more and more people are able to access and explore incredible amounts of information as they choose to do so and they are also able to evaluate programs, products, and people associated with that
information using whatever criteria they personally want to employ. In other words, everyone is an evaluator and everyone and everything they do or produce is more easily evaluable by them and everyone else. As Friedman summarizes, “In a flat world, you can’t run, you can’t hide, and smaller and smaller rocks are turned over. Live your life honestly, because whatever you do, whatever mistakes you make, will be searchable one day” (p. 185). What are the implications for learners and teachers? Should instructional designers be helping learners evaluate themselves and all the potential aids to their learning more efficiently and effectively through use of these technologies? Could designers be evaluating their design efforts formatively and through the evaluative eyes of their clients better if they took advantage of these technologies? Whether they do or not, will learners, teachers, and other clients find ways to make those evaluations themselves?
These examples and several others in Friedman’s book, various newsletters and technology updates remind me that evaluation is a central part of what we do as humans and technology can help us perform our decision
making, information processing, moral judging better if we will allow it to. If we don’t, we are likely to be judged and evaluated by others who have this same power, before we can make the adjustments we want to make.
Three Groups of Questions
Three groups of questions raised by a reading of The World is Flat and their potential value for guiding research and adjusting modes of instructional design and evaluation are presented below. These questions along with those raised earlier will be used to stimulate discussion during the AECT convention session. Participants will be invited to identify possible responses and additional questions they would like to address through future research and practice.
Agency
Given that technology is making the world more accessible to individuals, their evaluation skills,
responsibilities, and powers are growing as well. In a business sense, technology is helping the customer, who wants to, become better informed and more discriminating or selective in what they consume. Likewise, in terms of learning, taking advantage of the privileges available through so many sources to anyone with access to the Internet and the world it opens is becoming easier for individuals everywhere. Because more people have more options or choices, they are more and more free to choose what they want to learn, how they want to learn it, and how to know if they have learned enough. In short, technology is making it possible for more of us to be full agents over our own learning.
With this power comes a huge responsibility as well. More and more, learners can be viewed as responsible for their own learning and not so dependent on teachers and others to guide them. In terms of evaluation, this means that individuals are more clearly responsible for evaluating potential information, experiences, learning
opportunities, and teachers. They are also more responsible, therefore, to clarify what their own values are, what criteria they will use in judging potential learning experiences, to gather evaluative information about various resources, and to make their own evaluative judgments and decisions they will live by.
But to really have evaluation agency, individuals have to be aware that they have this responsibility. They need to develop skills of critical thinking and bias control. They need to become disciplined in their evaluation skills if they are to fully realize their evaluation powers and appropriately use their evaluation responsibilities. Simply having access to tools doesn’t guarantee people can appropriately use those tools.
These realities about human agencies have huge implications for learners, teachers, and instructional designers who want to assist them. Several questions we ought to explore in understanding and dealing with these implications are listed below.
Questions Raised
Are we recognizing the role of agency and how technological innovations are impacting learners’ agency? How are instructional design and evaluation theories and approaches taking this individual agency into account, if at all? Are evaluations associated with instructional design projects taking agency into account? How? How might they do this better? How are instructional technologists addressing issues of responsibility, self-interest, self-confidence, and the balance between individuals and the groups/organizations/collaboratives they are part of? What are we doing to develop theories and practices in instructional design, technology, and evaluation that respect and build upon the agency of the learners and teachers we serve?
Value for Guiding Research, Design, and Evaluation
Is it important to answer these questions about agency? Based on my reading of Freidman book and many other sources, taking human agency into account is one of the most important things we should be doing as we pursue the learning sciences, learning theories, instructional design practices, and evaluation of interventions. If we do not take into account who the people are that we want to encourage to learn better and if we do not build upon their responsibilities as well as their access and other powers that are enhanced by technological progress, we are morally responsible for violating or at least ignoring that agency. We will spend time in this session exploring that possibility and what we are doing with OUR agency to respect the agency of learners.
Sharing Values
As Friedman has documented, advances in technology have made it easier for everyone who has access to them to be exposed to a wider and wider variety of ideas and values. Living in a world flattened by technologies such as the Internet, cell phones, satellite television, etc. potentially exposes many more people than ever before to other cultures, other lifestyles, other religious views, other experiences, and other values.
This exposure makes it possible for evaluators and all the people they serve (designers, clients, learners, teachers, and so on) to better understand others’ values and perhaps to reconsider their own views. However, people may still resist understanding others’ values for a variety of reasons because, as mentioned earlier, they are agents and can choose to attend to whatever values they want to attend to.
A key issue in all evaluation studies is the role of stakeholder values. For many years only selected stakeholders and their values have guided many evaluations. For example, in schools, tests have traditionally been built by teachers to reflect the parts of the curriculum they valued most and felt students should also value. In businesses, the values of business owners and leaders usually take precedence over the values of the workers if they are at odds. In instructional design, the values of the people sponsoring the development of an instructional product and the values of the designer often take precedence over the values of the consumers or learners or the people they might serve with whatever they learn from the product.
However, in recent years evaluation theorists, such as Patton (1997) and Cousins & Earl (1995) have pointed out convincingly that for evaluations to be truly useful and worth the effort, all the stakeholders who are impacted by the thing being evaluated and thus, by the evaluation, should have a role to play in voicing their values and having their perspectives addressed in evaluations with which they are associated. In other words, the field of evaluation has followed the trends illustrated by Friedman and seeks to account more equitably for the multiple and often conflicting values of all the stakeholders involved.
Technology might make this process easier but it certainly makes it clear that there are many more values to be taken into account than evaluations used to attend to. Having more ready access to multiple value perspectives provides a challenge to traditions that may impact instructional designers as well. How they use evaluation to address those values with or without using technological innovations is an issue that ought to be addressed by theoreticians as well as practitioners.
Questions Raised
How does technology help reveal the perspectives of multiple stakeholders? Is it used to conceal or ignore certain values as well? How do instructional design and evaluation theories take multiple values into account? How do they ignore or inappropriately balance alternative perspectives? How are stakeholders’ views and values shared and built into evaluations and associated instructional design projects? How could they be better shared? Could using technology help with this task? What should we be doing about the use of technology to understand and share values to enhance instructional design and its evaluation?
Value for Guiding Research, Design, and Evaluation
The lessons learned by evaluation experts indicate that if all the stakeholders’ views are not acknowledged, the evaluation results are less important to many stakeholders. They are less likely to value evaluation conclusions and more likely to ignore them instead of using them to improve their practices. This is likely the case for
instructional design as well. If the learners’ values are not included in formative and summative evaluations of an instructional product, why would they value the product or conclusions about it? Thus, for practical reasons, as well as moral imperatives, the people who have an interest in or who are impacted by an instructional process or product should also have their values included in the evaluation and shaping of that instruction. Research into these issues must take the participants’ views and values into account or it is not worth doing.
Trust Issues
The example cited earlier of UPS using technology to get deep inside various organizations to evaluate their financial, supply-chain, and other processes raises several trust issues. In contrast the many examples given by Friedman regarding how much information is available on the Internet regarding organizations and individuals suggest some different but related issues. In all the cases Friedman explores, it is clear that people can and will use technology to gather and interpret information that may be considered private or sensitive by others. However, often
people are willing to share that information if they believe they can trust others to use that information for their mutual benefit. In contrast, many individuals may feel powerless to protect information about themselves and thus it is possible they will develop distrust of various technologies and the people using them. Implications for
instructional designers who use technology and for evaluators who seek to help instructional designers improve their practices might include the following:
1. Designers cannot simply assume that the clients or consumers of their products will use technology built into the designs. Although various technologies may provide the most efficient means for sharing the products, they may not be the most effective if the users distrust and refuse to use them.
2. To get to know the clients and consumers and their needs, designers may be tempted to use technologies that these people may not trust. Sensitivity in how needs assessment information is gathered and interpreted is essential.
3. Designers and evaluators who are helping them may also want to use technologies or assume the utility of technologies in evaluating the implementation and impact of various products. Exploring how stakeholders view the technologies should be part of the evaluation process.
4. Clients and others to be served by designers and evaluators may be using technologies to gather their own evaluative information about the designers and evaluators in more sophisticated ways than expected. Trust goes both ways and all parties’ trustworthiness is more easily gauged through evolving technological sources.
These trust issues highlight another growing movement in the professional evaluation field that might be important to consider by both instructional designers and evaluators. Promoted by David Fetterman (2001), empowerment evaluation has been evolving as a practiced approach for the last decade or so. This approach to evaluation involves evaluation experts serving as coaches to guide, encourage, and train various stakeholders (usually those who feel they have been in some way disenfranchised as stakeholders by an organization, group, or culture in which they function) to do their own evaluations and to use evaluation as a tool to improve their situations socio-economically, politically, and/or in other ways. Possibly, evaluation from this perspective is based on a distrust among some stakeholders that their values are going to be represented well by traditional evaluators. If instructional designers were to work with clients who wanted to use empowerment evaluation of their instructional products, they too would want to be sensitive to the trust issues involved. Finally, empowerment evaluation
stakeholders may be particularly interested in using technologies that would allow them to gain the power necessary to conduct their own social agenda-oriented evaluations.
This example of empowerment evaluation and the trust issues associated with technology and other influences suggest the importance of examining the basic process that evaluations usually follow to see what other implications there may be for trust issues associated with design and evaluation practices that involve various technologies. The evaluation process is rather straightforward, though often politically charged and includes the following steps as summarized by Williams (2006):
1. Clarify who cares about the thing to be evaluated; who are the stakeholders and who’s values should be included (e.g., are the consumers’ and the funders’ views equally important)?
2. Help the stakeholders clarify what they care about; what is the evaluand or thing to be evaluated and what aspects of it are they most interested in (e.g., an instructional product)?
3. Help the stakeholders clarify what their values are and what criteria and standards or levels of
performance on the criteria those values imply for judging the quality of the evaluand (e.g., how much and what learners should learn and at what level of performance when using the instructional product). 4. Use steps 1-3 to identify the evaluation questions stakeholders want answered.
5. Develop data collection, analysis, and reporting strategies to answer the questions, focusing primarily on the main question, which is “how well does the evaluand meet the specified criteria?”
Reflection on this process reveals the point that stakeholders and their values guide the entire process. Therefore, trust levels between stakeholders and evaluators and among stakeholders can greatly impact the identification of criteria, definitions of the evaluands, questions to be asked, and methods for conducting the data gathering, interpretation, and reporting. If use of technologies by the stakeholders, by the instructional designers of the evaluands being evaluated, or by the evaluators introduce trust or reduce trust among the stakeholders at any of these stages, the entire enterprise will be affected.
Questions Raised
Is something like what UPS is doing desirable for design organizations? What would it take to do this? What could the role of technology be in doing this? Will instructional design and technology organizations allow the
kind of deep involvement by evaluators that businesses have shown with UPS as their evaluator? What are the trust issues involved and how can trust be built? How do stakeholders use technology to do their own evaluations of their own operations and of potential instructional designers and/or evaluators? How does the individual empowerment of stakeholders through the use of technologies such as Internet search engines and/or the use of empowerment evaluation approaches impact their trust of professional instructional designers and evaluators and vice versa? How is the evaluation process being used in instructional design projects that use some of the new technologies? Could that process be more effective if it were refined? If so, how? What would be the interaction between a refined process, use of technologies, and trust?
Value for Guiding Research, Design, and Evaluation
Are the issues discussed earlier regarding trust, technology, and various approaches to evaluation important for guiding research on and the practice of instructional design and evaluation? The relationships among the various approaches to evaluation and how they are applied to instructional design are complex. They are further complicated when designers use technological innovations that involve the trust issues described earlier. Likewise, even if designers use more traditional tools, many of their consumers and clients may be using technologies that empower them as individuals and organizations. And these uses of technologies may complicate the trust needed between designers, clients, consumers, and evaluators in ways not well understood. Research into this phenomenon would be worth pursuing.
Conclusions and Implications
Although many other ideas will likely be stimulated by the discussion during AECT, tentative conclusions and implications of this analysis are summarized below.
First, the fields of instructional design and evaluation have always been related; but technological
innovations and their influences on both fields could stimulate more thoughtful attention to how these related fields could be more integrated. We should examine more thoroughly how instructional designers can and could use formal internal and external evaluation processes to assess needs, weigh the strengths and weaknesses of alternative ways to meet identified needs, formatively evaluate emerging designs and development efforts, evaluate
implementation fidelity of developed programs and products, as well as summatively judge the ultimate effectiveness of these programs and products in achieving their objectives.
Likewise, how are the various approaches to evaluation that have been developed over the last 40 years being shared with and used by instructional designers to enhance their work? Do designers know about
empowerment (Fetterman, 2001), participant (Cousins & Earl, 1995), responsive (Stake, 2003), utilization-focused (Patton, 1997), fourth-generation (Guba & Lincoln, 1989), and other approaches to evaluation (see Stufflebeam, 2001 for a review of several approaches)? If they do know about them, are they using them? If so, how? If not, why? How could these approaches be modified to be more helpful to instructional designers and developers?
Second, innovative ways of using and developing technology explored by Friedman come from essentially every discipline and field of interest in the world today. Reading about them raises questions regarding how open instructional designers and evaluators are to using what others are discovering from other fields AND what we are doing to discover and innovate new uses for technology ourselves. Are we using the lessons learned by others? Are we willing to explore uncharted territory as well? What are we doing to take advantage of the new tools being shared by people throughout the world? What conversations are we having with people in various fields that would allow us to be stimulated in our creativity and to help them help us develop our own tools?
Third, agency, sharing, and trust are key issues raised by this analysis of the flat world. How are we dealing with these issues in our research and practices associated with instructional design and evaluation? Should we be addressing these issues? What difference would it make if we did? The analyses in this article conclude that these are important concepts that have powerful implications for work we are doing and could do. How can we encourage researchers and practitioners alike to take these issues seriously? How do instructional designers and evaluators think about people (one another and their clients and consumers) in terms of their agency, in terms of their
trustworthiness and trustfulness, and/or in terms of their willingness and ability to understand and share their values? On the other hand, how do clients and consumers of design and evaluation projects think about each other and their project designers and evaluators in terms of agency, trust, and sharing? Does it matter how these questions are answered? If so, what can be done about obtaining answers and what can be done with results? What should we be doing as members of AECT and as an organization to address these issues?
Fourth, are the implications raised in this article from a reading of Friedman’s book worth further consideration? This article claims that they should be explored further and perhaps instructional designers and evaluators should experiment with these ideas in their research and practices. In particular, lessons learned from open-source communities could be synthesized. Instructional designers and evaluators could experiment with using open-source procedures for enhancing both the design of instruction and the evaluation of those design efforts.
Similarly, could the immediacy of feedback available through new technologies that various industries have begun to use to continually inform their performance help instructional designers and evaluators in educational settings too? Although people and their learning are very different from products sold at Wal-Mart, are there practical ways to use the electronic feedback and evaluation systems Wal-Mart uses in instructional designs, schools, and other learning settings? Creatively exploring ethical and humane ways of using technology to monitor student interest, learning, performance, attention, and so on seems like a powerful way for AECT members to transfer lessons learned from others to their settings. Use of student feedback clickers (such as TurningPoint) is one example that some innovators have been exploring. What others might we consider if we took the experiences of entities such as Wal-Mart seriously in our own contexts?
Trust-building between evaluators, instructional designers, and those they serve is another issue that was addressed somewhat in this article. What else should be done to explore the role of trust in the design process? How would viewing each other as agents modify our views of and approaches to trust-building? How do stakeholders view evaluation and evaluators and how does that affect trust? What could be done using technology innovations to overcome distrust? How can instructional designers and evaluators come to see that trust-building is essential and not something that can be accomplished simply through typical evaluation methods?
Finally, are instructional designers and evaluators aware of the power individual clients and consumers are gaining through access to information newly available by use of innovative technologies? Do they see how that power can be used effectively to assess consumer needs, design solutions, deliver instructional services and products, and evaluate the entire process and outcomes? Do they see the interface of participants’ agency, trust, and power with the instructional design and evaluation objectives? What differences could be achieved in learning and growth if these issues were more explicitly addressed by professionals represented by AECT?
This AECT session will provide an opportunity for participants to explore a few of these many questions. However, you are invited to join in this conversation as well by contacting me and sharing your thoughts,
suggestions, and additional questions. Please contact me at [email protected]. References
Cousins, J. B., & Earl, L. M. (Eds.). (1995). Participatory evaluation in education: Studies in evaluation use and organizational learning. London, England: Falmer Press.
Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.
Friedman, T. L. (2006). The world is flat: A brief history of the twenty-first century (2nd ed.). New York: Farrar, Straus and Giroux.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.
Patton, M. Q. (1997). Utilization focused evaluation: The new century text (Vol. 3rd). Thousand Oaks, CA: Sage. Stake, R. E. (2003). Standards-based and responsive evaluation. Thousand Oaks, CA: Sage.
Stufflebeam, D. L. (2001). Evaluation models. New Directions for Evaluation, 89, 7-99.
Williams, D. D. (2006). Measurement and Assessment Supporting Evaluation in Online Settings. In D. D. Williams, M. Hricko & S. Howell (Eds.), Online assessment, measurement, and evaluation: Emerging practices. Hershey, PA: Idea Group, Inc.
