Analysis

researchers to focus on the development and deployment of such gestures that perceived more interactions (e.g. due to their simplicity) [HRD11].

2.6

Analysis

The related work described spans across foundational research on audience interaction models for describing viewer behaviour in the immediate vicinity of a display, data capture and reporting techniques such as video analytics, and work that automatically utilises analytics data to adapt novel forms of interaction modalities. In this section, we provide an overview of the coverage of the body of related work regarding the three areas of challenge (analytics data, reporting and the automated use of analytics data), and subsequently provide an analysis of the suitability of the work for use in the context of open pervasive display networks by identifying and highlighting limitations for each of the characteristics (openness, pervasiveness, and networked).

2.6.1 Evolution and Coverage

In order to allow us to highlight the coverage of the body of related work and the changing research focus over time, we first categorised each piece of work into one of the three areas of challenge. To provide a better understanding of our categorisation, we defined each of the areas as follows:

Analytics Data Capture Systems, probes or solutions that are related to capturing ana- lytical insights about viewers / audience, users of a system, and the digital sign itself. Additionally, we consider work regarding the development of novel capture techniques such as visual computing algorithms.

Reporting Systems, probes or solutions that specifically focus on the creation and dis- semination of relevant analytics reports, including work that focuses on the presentation of relevant or novel metrics.

Automated Use of Analytics Data Systems, probes or solutions that utilise analytics insights in real-time to dynamically adapt the content shown or the available inter- action modalities to viewers present in the vicinity of the display. Additionally, we include work on retrospectively informing the design of (interactive) content or display deployments based on analytics.

Due to the large amount of related work that was specifically conducted on investigating and enabling novel interaction modalities (e.g. via gaze), we additionally considerinteraction as a fourth area – allowing us to provide further insight into the research focus of the related work. Of course, novel interaction modalities can be used as a driver for capturing relevant analytics data (i.e. viewer interactions and engagement with the display). However, we assign

2.6 Analysis 44 1997 1999 2000 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017

year

Figure 2.3: Distribution of related and relevant digital signage and Web analytics work by their publication date grouped by the main focus of the paper into four categories:capturingof analytics data,interactingof users,reportingof relevant analytics insights and the use of analytics foractuation.

related work to the ‘interaction’ category only if it was not specifically focusing on enabling analytics.

Figure2.3shows the coverage of the related work for each year as a stacked bar chart (due to the scope of some of the introduced research papers, a subset of papers may be assigned to multiple categories at the same time). We observe that the majority of the work focuses on areas of data capture and interaction – i.e. enabling the capture of novel interaction events from the audience. Little work has been carried out on the creation of analytical insights and reports, or that utilises such reports for the actuation of digital signs. We note that the increased trend of developing novel forms of interaction modalities – likely to be grounded in the opportunities that have emerged with technical advancements. For example, gaze [SHT10], gesture [Gil+14] and mobile phones emerged as new ways for viewers to interact with displays in recent years. Additionally, we observe a constant tendency to focus on analytics data capture techniques (e.g. user mobility tracking in spaces [Jay+16;WW14] or the detection of attention states of viewers [WBG12]). The common thread across the related work lies in the focus on single or small scale display deployments with analytical insights typically provided about the audience (e.g. demographics and age ranges) and their behaviour (e.g. interaction phases). For example, the introduced audience models and metrics (Section2.2, p.14) define a detailed language and understanding of how viewers navigate and behave in the immediate vicinity of a display (e.g. “passing by” and “viewing and reacting” [MM11]), perform certain forms of interactions (e.g. implicit and subtle interaction [VB04], and direct interactions [MM11]) and the consideration of potential follow-up interactions (e.g. “follow-up actions” [MM11] or “conatation” [She+14]).

2.6 Analysis 45

In the following section, we will address the shortcomings and limitations of related work in the context of open pervasive display networks.

2.6.2 Suitability for Pervasive Display Networks

Our work is motivated by the shift from closed networks of displays towardsopen pervasive

display networks, drawing on the scenario introduced in Section1.2(p.3). Specifically, such

networks of displays feature a unique set of characteristics regarding their ‘openness’, ‘perva- siveness’ and ‘networked’ aspects. Considering the existing body of work in digital signage analytics in terms of these characterisations highlights a set of limitations and shortcomings.

Openness

With pervasive display networks becoming moreopen, the number of potential stakeholders who contribute to the display network significantly increases compared to closed display networks. Previous work has identified such stakeholders to include display owners, content providers, space owners and specifically also include viewers as a fourth stakeholder [VO11; AMS12; Cli+14]. Each of these stakeholder groups are likely to be composed of a high number of individual stakeholders.

The existing body of work, however, typically focusses on capturing and reporting analyt- ics for single stakeholder entities only. As an example for the body of work that focuses on data capture and reporting, the system-based monitoring tools introduced are predominately designed to provide display owners with an overview of the entire network of deployed digital signs (e.g. Esprida [Esp17] and CAYIN [CAY]). More sophisticated analytics tools (e.g. Intel AVA [Cav11] and Fraunhofer SHORE [Frab]) additionally provide insights into the audience demographics (including age ranges and gender using cameras mounted on displays). Such audience analytics reports, however, are targeted for display owners as the single stakeholder entity. A similar trend can be observed in work on feeding (analytics) data back into the sign, e.g. to support novel forms of interactions. For example, systems that utilise insights on viewer demographics [Tia+12], audience attention levels and locations [MG12] and interactions with physical objects in the vicinity of the display [Sla16] have been designed for use with either a single display deployments (e.g. proximity-based interactions [MG15]) or in the context of environments that are controlled by a single stakeholder entity (e.g. [Tia+12]). However, with displays becoming open ecosystems to which a range of stakeholders can contribute, the complexity increases in accommodating requirements and constraints imposed by individual stakeholders and the high number of potential contextual changes to be fed into the sign. It will likely become highly challenging to, for example, create a definite display content schedule that accommodates all these requirements and is capable of dynamically responding to analytics events in the vicinity of a display.

We were unable to identify prior work that considersopennessas a characteristic across the data capture, reporting and automated use of analytics data areas of challenge. In particular, work that considers capturing and combine relevant datasets across a range of stakeholders

2.6 Analysis 46

(also including viewers as an equal stakeholder) leveraging the potentially rich ecosystems through which viewers navigate and in which viewers interact. Drawing on the introductory scenario, analytics systems that consider events from multiple data points, likely to be owned by multiple stakeholders, will be required in futureopendisplay networks. Additionally, such systems will need to include mechanisms for resolving potentially conflicting requirements and constraints when feeding analytical insights from various stakeholders back into the sign.

Networked

Existing work focussed on capturing and reporting analytics regarding individual displays or networks owned by a single stakeholder entity. In particular, the techniques described focus on capturing and analysing data from the perspective of a display, i.e. provide insights about displays and their audience. For example, commercial solutions capture a set of system- related information from displays that are part of a single network such as logs of content played and the state of display players [CAY;Rem17;OnS17]. Additionally, more advanced solutions feature visual analytics techniques to capture (anonymised) audience numbers and demographics from individual displays including gender, age ranges and in some cases even the mood of members of viewers [Int18;IBM13;Tia+12;Alt+12].

In contrast, with the emergence of interconnected networks of displays we will be able to observe potentially complex and spatially distributed viewer interactions and engagements with displays and the content shown that will be difficult to describe with conventional analytics techniques. Instead, a shift towards aviewer-centric perspectiveon analytics will be required to enable the creation of reports that describe how viewers experience digital signs and content when moving across spaces. For example, instead of capturing and reporting the average demographic group, analytics may provide further insights on the content viewers have previously seen, and the potential impact of displays on their behaviour and movement patterns – almost providing an equivalent to a click-through event given in Web analytics. We were unable to identify prior work that focussed on providing a viewer-centric perspective in the context of open pervasive display networks.

Pervasiveness

The final characteristic of future display networks is the pervasiveness: public displays appear embedded into urban environments and are becoming omnipresent to the viewer. With displays becoming more pervasive and ubiquitous, the amounts of data, data stakeholders and users is likely to grow consistently – creating a need for novel analytics systems that facilitate the requirements emerging from the unique characteristics from future open display networks. Such novel analytics systems, however, impose risk of revealing highly sensitive and potentially privacy-invasive insights about individuals. For example, researchers have focussed on developing accurate indoor location tracking technologies that work within a limited spatial area, e.g. by utilising Wi-Fi hotspots [GJC04;MB13;Air13] with the overall goal to gain a better understanding about the behaviour and navigational patterns of individuals. Some