Many times when we start out on a project, we have a clear vision, and with a little forethought, we establish a course of action that enables us to achi eve our desired results. Of course, as Robert Burns once said: “Sometimes the best-laid plans of mice and men oft go awry.” Mice, men, or even social media analysts—it’s all the same! Sometimes the ideas we have, the questions we’ve posed, or the plan of attack that we’ve devised needs to be modified. We have to be flexible because sometimes we’re just plain wrong, or we’ve picked up a screwdriver to drive a nail. Things just aren’t right.
An iterative method of working, be it the creation of a model, the writing of software, or even of writing a book, is one in which we do not attempt to start with a full specification of requirements or plan. We don’t define steps 1 through 100 and then proceed in order creating our solution. Instead, we
ptg16373464
begin by specifying and implementing just part of our solution, which can then be reviewed and evaluated in order to identify further requirements or changes. This process is then repeated, producing a new version that is hopefully an improved version of the previous release. We continue cycling through iterations of our product, software, or solution until we believe we’ve achieved success.
As we work iteratively on a project, we create a rough draft or rough set of results in a single iteration. We then review it, decide on changes to (hope- fully) improve it in next iteration, and continue until we’ve finished. Figure 7.5 visually describes the iterative method.
Create Social Media Analysis Model Final Solution Rough Draft Make Course Corrections Complete? Yes Evaluate Results Initial Idea
Figure 7.5 Visual depiction of the iterative method.
In the iterative model, we are building and improving the final analysis step-by-step. In this way, we can address deficiencies in our model or analy- sis early on in the process. This allows us to perhaps change our model, look for additional data, or perhaps change the question in a way that is more relevant. The most important aspect is the ability to obtain feedback as we progress.
A feedback loop is something we use to gather feedback about what we’re doing, learn from the feedback, and then make changes based on that feed- back. The sole purpose of a feedback loop is to improve a project based on
ptg16373464
7: Reading Tea Leaves: Discovering Themes, Topics, or Trends 119
the current plan of attack. These loops are important because they allow us a systemized approach to observing our results and learning from them.
Using this approach helps us to avoid those awkward moments that can happen when we present our analysis and the customers simply look on with a blank stare and comment, “That’s not what I asked for.” If they can see the progress (and perhaps help shape the final analysis), there are no surprises at the end.
Summary
For projects in which we are validating a previously formed hypothe- sis, the analysis is complete as soon as we find enough evidence that either proves or disproves the hypothesis. For projects in which we are discovering the themes, the end is not so clear. The team that is performing the analy- sis has to work with the stakeholders iteratively to determine if the level of detail analyzed and the types of conclusions derived are in line with stake- holder expectations. There are also projects in which, even after validating a hypothesis, there is considerable business value to be gained by discovering other themes that the team had not considered before.
Endnotes
[1] Spaulding, Jonathan. Ansel Adams and the American Landscape: A Biography. University of California Press, 1998.
[2] Mila Gessner is an analyst who works at IBM. Many of the examples used in this chapter are based on the specific work she did on these projects. These examples are reproduced here with her knowledge and permission.
[3] The Heritage Foundation, “2015 Index of Economic Freedom.” Retrieved from http://www.heritage.org/index/country/france. To locate a specific country, use the fol- lowing URL, supplying the name of country where indicated: http://www.heritage.org/ index/country/name of country.
ptg16373464
This chapter delves into the concept of real-time and near real-time data analysis. We look at it from a few perspectives. For example, we look at real-time tweets during a televised presidential debate, and analysis of data in what we call near real time (think of feedback from attendees at a live conference), or any simple yet powerful analytics that can be computed and presented in a short timeframe for analysis despite the limited amount of time afforded for analysis.
There are two main reasons to consider analytics systems when dealing with evolving information in real time and near real time. One reason is to sense and respond. In these situations, we want to monitor data to get a sense of what is happening in order to respond to it in a timely fashion. A second reason is to implement an early warning system. This includes timely collection and analysis of data, resulting in prompt interventions. We look at examples of both of these scenarios in the following sections. We also describe the concept of stream computing and describe a specific analytics system that we built using the principles of stream computing to harness the analytics from real-time and near real-time systems.