DRO
Deakin Research Online,
Deakin University’s Research Repository Deakin University CRICOS Provider Code: 00113B
Tangible user interface design for climate change education in interactive
installation art
Citation:
de Bérigny, Caitilin, Gough, Phillip, Faleh, Majdi and Woolsey, Erika 2014, Tangible user
interface design for climate change education in interactive installation art
, Leonardo
, vol.
47, no. 5, pp. 451-456.
DOI:
10.1162/LEON_a_00710
©2014, ISAST
Reproduced with permission.
Downloaded from DRO:
Caitilin de Bérigny (artist, lecturer), University of Sydney, Design Lab, Architecture, Design and Planning, Wilkinson Building (G04), 148 City Road, University of Sydney, Australia, NSW 2006. Email: <[email protected]>.
Phillip Gough (artist), University of Sydney, Design, Architecture, Design and Planning, Wilkinson Building (G04), 148 City Road, University of Sydney, Australia, NSW 2006. Email: <[email protected]>.
Majdi Faleh (architect), The University of Western Australia (UWA), Architecture, Landscape, and Visual Arts (ALVA), 35 Stirling Highway, Crawley, Perth, Australia, WA 6009. Email: <[email protected]>, <[email protected]>. Erika Woolsey (marine biologist), Australian Research Council Centre of Excellence for Coral Reef Studies, James Cook University, Townsville, Australia, QLD 4811. Email: <[email protected]>.
See <www.mitpressjournals.org/toc/leon/47/5> for supplemental files associated with this issue.
A r t i s t s ’ A r t i c l e
Tangible User Interface Design
for Climate Change Education
in Interactive Installation Art
Caitilin de Bérigny and
Phillip Gough with Majdi
Faleh and Erika Woolsey
I
nteractive art installation exhibitions have thepotential to attract large public audiences. Thus, such instal-lations can be significant tools for communicating environ-mental effects of climate change. Reefs on the Edge (Fig. 1) is an interactive artwork that utilizes a cross-media platform to experiment with scientific data and underwater photography, video and sound, much of which was collected at One Tree Island Reef, located on the Great Barrier Reef (GBR) off the northeastern coast of Australia. The installation uses tangible user interface (TUI) technologies, multiple-channel video, painting and sound. This collaborative artwork is informed by Erika Woolsey’s project that investigates the survival of young corals on the Great Barrier Reef [1].
Reefs on the Edge engages users with the novelty of being able
to control an artistically designed computer simulation, or vi-sualization, of coral spawning. Coral spawning is a seasonal reproductive event where reef-building corals simultaneously release pink and orange bundles filled with eggs and sperm. This natural phenomenon is visually stunning as well as cru-cial for replenishment of coral reefs, particularly those that are stressed or damaged. In Reefs on the Edge, users exercise control using physical “objects” designed not only to represent the spawning corals but also to embody the objects’ intended use as a set of computer controls. The installation includes a TUI table that incorporates a projection screen on its surface that also provides part of the visualization, using an animated simulation of the spawning and early coral growth. This sys-tem allows multiple users to interact with the simulation and conceptually to connect their actions with environmental con-sequences experienced by corals in tropical Australian waters. Analogically, Hiroshi Ishii—a Japanese-born computer scien-tist and pioneer of the TUI—developed a work titled
genie-Bottles (2001). This work presents
a story that is told by three genies that live in glass bottles. The bottles embody the tangible interface; as a bottle is opened, it releases a genie that starts talking to the user [2]. Ishii’s genieBottles highlights the fact that objects designed as TUIs should relate to their function in
en vI r o n m en T 2 .0 A b s t r A c t
T
he authors discuss how tangible user interface objects can be important educational and entertainment tools in environmental education. The authors describe their interac-tive installation artwork Reefson the Edge, which incorporates
tangible user interface objects and combines environmental science and multiple art forms to explore coral reef ecosys-tems that are threatened by the effects of climate change. The authors/artists argue that the use of tangible user interface in an installation-art setting can help engage and inform the public about crucial environmen-tal issues.
Fig. 1. onacloV and Phillip Gough, Reefs on the Edge, 2012. (a) Detail. (b) close-up of tangible user interface (tUi) objects and film pro-jection. (© caitilin de bérigny/onacloV and Phillip Gough. Photo: Phillip Gough.)
452
de Bérigny et al., TUI for Climate Change Education en vI r o n m en T 2 .0order to help the user understand the work as quickly as possible.
The artwork Reefs on the Edge was de-signed by Australian artist Caitilin de Bérigny, known as onacloV, and built by a team of artists and designers: Phillip Gough and Adityo Pratomo, who de-signed and built the TUI objects and table; Ge Wu, who edited the multiple-channel video installation; and Michael Bates, who created the soundscape. To develop an affordance or semantic that would allow users to change objects’ ro-tation and position, the objects were de-signed as small spherical shapes, so that the average adult hand would be able to grasp them easily [3].
Climate Change Education
Currently there is a growing interna-tional movement of artwork being cre-ated in response to climate change in multidisciplinary contexts such as art, design, science and architecture [4–8]. The effects of global climate change, such as ocean warming, ocean acidifica-tion, more intense storm events and sea level rise, threaten the long-term health of coral reef ecosystems [9–11]. Higher ocean temperatures can cause coral bleaching, so called because stressed cor-als turn white when they lose their algal symbionts. Mass bleaching events are as-sociated with high mortality and reduced reproductive potential and are becom-ing more frequent worldwide [10–11]. Ocean acidification, caused by
absorp-tion of excess atmospheric CO2 into the
ocean, weakens the skeletal structures of coral, making them more susceptible to disturbance [11]. In addition, such large-scale factors are compounded by local, human-induced stressors such as over-fishing, habitat destruction and pollution [9]. As a result, coral reefs throughout
the world are degrading rapidly and have uncertain futures [11].
“Edutainment,” a portmanteau of edu-cation and entertainment, can be effec-tive to raise awareness and engage users. In interactive museums, hands-on inter-action can be combined with digital dis-plays to create tangible user interfaces, which serve as effective intuitive tools for conveying complex scientific and/or en-vironmental data [3,12].
R
eefsonthee
dge: Tangible
User Interface Platform
The tangible user interface is an emerg-ing experimental tool that is beemerg-ing devel-oped as an alternative to the graphical user interface (GUI). A GUI, typically a computer display that is manipulated by abstract controls such as a keyboard and mouse, can be replaced by a set of controls designed to control a specific set of com-puter functions. This set of controls is de-signed to represent that which they control as well as to provide user feedback [12]. Ishii refers to this concept as Tangible Bits [13]. The goal of Tangible Bits is to bridge the gap between cyberspace and the physical environment. The features of Tangible Bits, as proposed by Ishii, are:
1. Interactive Surfaces—the transforma-tion of a surface in an architectural space into an interface between the user and information.
2. Coupling of Bits and
Atoms—combin-ing everyday objects with the digi-tal information that they are being used to represent.
3. Ambient Media—light, airflow or other environmental factors as a medium for a subtle interface, at the periph-ery of human perception [13]. In order to couple the “bits” of infor-mation with the “atoms” of the interface,
the form of the “object” that controls a TUI must represent its intended use. This affordance allows users to quickly and efficiently engage with the visualiza-tion, allowing them to find the informa-tion they are looking for and making the visualization more successful.
Other Works Employing a
Tangible User Interface
One other well-known work that includes a TUI is the reacTable. ReacTIVision, the computer-vision framework powering the
reacTable, was created in order to enable
the design of interactive surfaces and was used to create the TUI for Reefs on the
Edge [14]. The reacTable is a digital
musi-cal instrument. It incorporates simplistic, minimalist objects, which can be used for real-time interaction with music loops as a flow-controlled modular synthesizer [14]. Reefs on the Edge relied heavily on the implementation of reacTIVision to allow a multi-user approach to explor-ing an artwork, as well as to uniquely identify each object as it is placed on the screen [3].
Another pertinent illustration of an artwork that uses both TUI and cli-mate change visualization is onacloV’s
InterANTARCTICA (Fig. 2) [15].
Enter-ing the exhibition space housEnter-ing
Inter-ANTARCTICA, the viewer is surrounded
by a three-screen video installation of the Antarctic landscape. The viewer hears Antarctic compositions, created by other viewers via real-time audio in-teractions. There are two levels of inter-action with the TUI objects: By moving the objects the viewer both creates sound and modifies a visualization.
InterANT-ARCTICA helped viewers understand
critical scientific data about global warm-ing in Australia through a multi-sensory experience.
Fig. 2. onacloV, InterANTARCTICA, 2010. (a) close-up of tUi object. (© caitilin de bérigny/onacloV) (b) Users moving objects in the exhibition, creat-ing sound and transformcreat-ing the data visualization. shown at Australian National University in an exhibition titled beginning, Middle, end, 2010. (© caitilin de bérigny/onacloV)
en vI r o n m en T 2 .0
Scientific Perspectives
The TUI technologies we used in Reefs
on the Edge are based on different stages
of the scientific process, providing a range of perspectives to illustrate the importance of scientific observation and experimentation. The research project associated with the Reefs on the Edge art/ science collaboration investigates the ef-fects of temperature on young corals at One Tree Island. This scientific study was designed to help us all to understand the effects of ocean warming on early coral development and thus to anticipate ways that climate change is likely to affect coral reef environments [1]. Tempera-ture strongly affects the development and survival of young corals [1,16]. Contemporary ocean warming, which is contributing to habitat degradation and population declines that result from coral bleaching, is therefore expected to interrupt reproductive success and long-term survival of coral species.
The title Reefs on the Edge refers to the uncertain future of coral reefs as well as the geographical location of One Tree Island, which sits at the edge of the trop-ics (at 23.5°S latitude, on the Tropic of Capricorn) and at the southern limit of the GBR Marine Park. Most of the sound and photographs for this project were collected at One Tree Island.
Visualization
Our user observation showed that the TUI successfully acted as a novel system to draw people into the interactive in-stallation and engage the audience in an intuitive way. This initial contact and engagement allowed the visualization to communicate its message.
The visualization for Reefs on the Edge was developed in a programming
envi-cessing program controls the projection onto the surface of the TUI table; it also changes the color of the TUI objects to provide additional feedback to the user. Designed as a programming envi-ronment and language for artists and designers—rather than professional software developers—Processing allows the creation of animations, images and meaningful and interactive displays that can be delivered to the public. Process-ing has been used for a broad range of purposes—from artistic to analytical— to communicate data [17,18] and is an ideal platform for interactive installa-tions, such as Reefs on the Edge, to enhance the artistic and scientific display digitally and help users relate to information.
British designer David McCandless ar-gues that many people suffer from “infor-mation overload, or data glut” [8]. The goal of mapping and visualizing informa-tion interpreted from raw scientific data is to create an understanding of scien-tific research for the general public. In 1995, Nobel laureate Herbert A. Simon pointed out the need to refine how we communicate not only scientific research but all information:
What information consumes is rather obvious: It consumes the attention of its recipients. Hence a wealth of informa-tion creates a poverty of atteninforma-tion and a need to allocate that attention efficiently among the overabundance of informa-tion sources that might consume it [19]. McCandless also states that by visual-izing information we turn it into a land-scape that one can explore with one’s eyes, a sort of information map. When one is lost in information, an informa-tion map is useful [8]. With this setting a precedent, artistic information visualiza-tions such as Reefs on the Edge are helpful in disseminating information to a
gen-Tangible User
Interface Table
Reefs on the Edge gives gallery users the
opportunity to learn about coral reefs and climate change in a compelling and interactive way. Our method is at the intersection of emerging technological tools and scientific observation. The sci-entific study conducted in coordination with this project was converted into an interactive visualization controlled by the TUI table and objects (Color Plate A). The installation’s interactive table helps viewers understand the effects of rising temperatures and ocean acidification. As previously mentioned, the TUI’s col-ors change in tandem with the changes made to the temperature and acidifica-tion projecacidifica-tion displayed on the table.
The use of reacTIVision allows the computer running the simulation to pinpoint the location, identity and ori-entation of the objects on the surface of the table, in real time, as the users move them. The information from the posi-tion, rotation and presence of objects is then sent to Processing, which animates the visualization accordingly.
Other Installation
Art Components of
R
eefsonthee
dgeVideo installation by Ge Wu
In the installation, a multiple-channel video is projected around the installa-tion environment. The photographic material (Fig. 3) was edited into numer-ous sequences and projected onto the sculptural surfaces of the installation. The concept of the video installation is to play with notions of vibrant colors found on the reef, juxtaposing them with the destruction of the reef environment Fig. 3. onacloV and Phillip Gough, Reefs on the Edge, 2012. (a) Multiple-channel video screens and detail of tUi table. (© caitilin de bérigny/ onacloV and Phillip Gough. Photo: Phillip Gough.) (b) Users interacting with the tUi table at Web Directions south conference, sydney, Australia, 2012. (© caitilin de bérigny/onacloV and Phillip Gough. Photo: Phillip Gough.)
454
de Bérigny et al., TUI for Climate Change Education en vI r o n m en T 2.0 Photographic imagery was collected at
One Tree Island or provided by Charlie Veron [20].
Visualization of scientific
information by Phillip Gough
and Adityo Pratomo
In “On Beauty,” a chapter in the book
Beautiful Visualization, Noah Iliinsky
out-lines what makes a visualization both beautiful and successful. He writes that visualizations that are beautiful are novel, informative, efficient and aesthetic [21]. We considered Iliinsky’s concepts when designing the TUI objects and their func-tion, as well as the visualization. The data used specifically for this project was con-verted into artistic visualizations. This mix of art and science in a cross-disci-plinary approach helps viewers to better understand the message.
soundscape by Michael bates
Artistic works can emphasize envi-ronmental audio to create poetic experiences. Following his work on
Inter ANTARCTICA [15], sound designer
Michael Bates composed soundscapes us-ing samples taken from the natural reef environment. These sounds were trans-formed and recontextualized in various ways to resonate with images, blurring the distinction between organic marine atmospheres and music. This created a sonic architecture for the installation.
Paintings by onacloV
Visual artist onacloV designed and de-veloped a series of paintings based on images taken at One Tree Island Reef during coral spawning season (Fig. 4). She created these works using an ex-perimental dot painting screen printing technique to represent large underwater photographs.
Interaction and User
Experience Design
After initially choosing to develop a TUI using the model of an interactive surface, as described by Ishii [22], we needed to develop the TUI objects. We created the objects from epoxy resin using a soft silicone mold with space to house both the multi-colored LEDs and a microcon-troller. Though epoxy resin is not the most desirable material for use in an en-vironmental installation, it was chosen for its exceptional durability, transpar-ency and the ease with which it can be molded to any form (Fig. 5). The artists Phil Gough and Adityo Pratomo decided on the initial concept of the form based on user testing, abstraction of control objects and prototyping.
Fig. 4. (a) caitilin de bérigny/onacloV, Astrocoeniina #3. (© caitilin de bérigny/onacloV) (b) coral Acropora formosa, one of the many species found on One tree island reef in the Great barrier reef. image from Reefs on the Edge. (© caitilin de bérigny/onacloV and Phillip Gough. Photo: Phillip Gough.)
en vI r o n m en T 2 .0
addition of the video allowed users to successfully make a connection between the TUI and the coral, which was not readily apparent without the video.
Conclusion
Experimentation with innovative forms of interactive technology can play an important role in global education sur-rounding the issue of climate change [25]. Tangible user interface technolo-gies have shaped and informed new modes of interactive installation art. These advances have allowed the mes-sage of the effects of rising sea surface temperatures on young corals to be trans-lated seamlessly across multiple techno-logical platforms, creating a new user experience.
Reefs on the Edge presents coral reefs as
beautiful and threatened environments and encourages viewers to learn and ex-plore via interactive TUI technologies. This hybrid of art and science promotes understanding of natural environments and the challenges they face while en-hancing scientific comprehension and curiosity.
references
Unedited references as provided by the authors. 1. Woolsey, E.S.; Byrne, M.; and Baird, A.H., “The effects of temperature on embryonic development and larval survival in two scleractinian corals,” Marine
Ecology Progress Series 493: 179–184 (2013).
2. Mazalek, A.; Wood, A.; and Ishii, H., “genieBottles: An Interactive Narrative in Bottles,” Proceedings of SIGGRAPH, 12–17 August 2001, ACM Press. 3. Kaltenbrunner, M. and Bencina, R., “reacTIVi-sion: A Computer-Vision Framework for Table-Based Tangible Interaction,” Proceedings of the First In-ternational Conference on Tangible and Embedded Interaction (TEI’07), Louisiana, U.S.A. (2007). 4. Coral, Art, Science, Life. Exhibition, Macleay Mu-seum, University of Sydney, 2012: <http://sydney. edu.au/news/science/418.html?eventid=9274> (ac-cessed June 2012).
5. Wall, C. and Wang, X., “InterANTARCTICA: Tan-gible User Interface for Museum Based Interaction,”
The International Journal of Virtual Reality, Vol. 8, No.
1 (2009), pp. 17–26.
6. Fox, M. and Kemp, M., Interactive Architecture, New Jersey: Princeton University Press, p. 142 (2009). 7. Koblin, A., “Artfully visualizing our humanity,” Video on TED.com 2011. Retrieved 28 May 2011 from: TED.com <www.ted.com/talks/lang/eng/ aaron_koblin.html> (accessed 29 May 2012). 8. McCandless, D., “The Beauty of Data Visualiza-tion,” presented at TED conference 2010, <www.ted. com/talks/david_mccandless_the_beauty_of_data_ visualization.html> (accessed 29 May 2012). 9. Hughes, T.P. et al., “Climate Change, Human Im-pacts, and the Resilience of Coral Reefs,” Science 301: 929–933 (2003).
10. Hoegh-Guldberg, O. et al., “Coral reefs under rapid climate change and ocean acidification,”
Sci-ence 318: 1737–1742 (2007).
Object Design
The objects were also designed to pro-vide feedback to the user by changing color. This is another reason we selected a moldable material with a high level of transparency. When the system detects the rotation of an object, it sends a sig-nal wirelessly to the microcontroller to change the color of the object. With a simulated rise in sea temperature, the ob-ject turns red, and the visualization is also affected. This feedback also indicates to the user that he or she is affecting the visualization.
User testing
Through ethnographic observation and one-on-one interviews, we gained
feed-back from a prototype exhibition in Oc-tober 2011. The TUI table was initially displayed at the Web Directions South Conference in 2011. Users’ reactions were positive and user exploration of the interface was tremendously success-ful, as users were able to control the objects without needing instructions. Users’ experimentation with the objects confirmed the success of the design of the objects, and we were able to use user feedback to guide revisions to the inter-face design.
The second exhibition of Reefs on the
Edge [23–24] included a multi-screen
video produced for Reefs on the Edge by Goldy Wu with sound by Michael Bates, in addition to the interactive table. The Fig. 5. onacloV and Phillip Gough, Reefs on the Edge, 2012. (a) the illuminated tUi objects. (© caitilin de bérigny/onacloV and Phillip Gough. Photo: Phillip Gough.) (b) Detail show-ing the reactiVision markers on the undersides of the tUi objects and the internal Arduino microcontrollers. (© caitilin de bérigny/onacloV and Phillip Gough. Photo: Phillip Gough.)
456
de Bérigny et al., TUI for Climate Change Education en vI r o n m en T 2.0 11. Pandolfi, J.M.; Connolly, S.R.; Marshall, D.J.; and Cohen, A.L., “Projecting coral reef futures un-der global warming and ocean acidification,” Science 333: 418–422 (2011).
12. Shaer, O. and Hornecker, E., “Tangible User In-terfaces: Past, Present, and Future Directions,”
Foun-dations and Trends in Human-Computer Interaction Vol.
3, 2009, pp. 35–36.
13. Ishii, H., “Tangible Bits: Beyond Pixels,” Pro-ceedings of the Second International Conference on Tangible and Embedded Interaction (TEI’08), Bonn, Germany (2008).
14. Pratomo, A.; Gough, P.; and Wall, C., “Designing Objects and Interaction of Tangible User Interface for an Interactive Data Visualization in a Transmedia Artwork,” Hawaii International Conference on Arts and Humanities, Honolulu, HI, U.S.A. (2012). 15. InterANTARCTICA: <www.interantarctica.com> (accessed 29 May 2012).
16. Negri, A.; Marshall, P.; and Heyward, A., “Differ-ing effects of thermal stress on coral fertilization and early embryogenesis in four Indo Pacific species,”
Coral Reefs 26: 759–763 (2007).
17. Pearson, M. (2011). “Generative Art,” Manning Publications.
18. Processing.org, Exhibition Archives/Processing, 2012. Retrieved 29 May 2012 from Processing.org: <http://processing.org/exhibition/>.
19. Varian, H., “The Information Economy,” Scientific
American, 273 (3) 200–201 (1995).
20. Veron, J.E.N., Corals of the World, Townsville, QLD: Australia, Australian Institute of Marine Sci-ence (2000).
21. Iliinsky, Noah, “On Beauty,” Beautiful
Visualiza-tion, edited by Steele, J. and Iliinsky, N. (Canada:
O’Reilly Media Inc.), 1–14 (2010).
22. Ishii, H. and Ulmer, B., “Tangible bits: Towards seamless interfaces between people, bits and atoms,” Proceedings of CHI 97, pp. 234–241, NY: ACM (1997). 23. The second exhibition of Reefs on the Edge was held at Attract::Relate::Sustain, an exhibition at the Verge Gallery in December 2011 at the University of Sydney.
24. The interactive table and multi-screen video from
Reefs on the Edge was exhibited at the Macleay
Mu-seum, Sydney, from September 2012 to March 2013. 25. Wall, C. and Erika Woolsey, “Climate Change Education, Fostering Dialogues between Art and Science,” The International Journal of Climate Change,
Impacts and Responses, Volume 3, Issue 2 (2011). Glossary
affordance—a feature of an object that communi-cates the intended function of the object with a user. An example is a volume control button with a (+) or (–) symbol to represent increasing or decreasing the sound volume.
Arduino—a commercially available brand of micro-electronics controllers.
graspable objects—objects that can be used as a representation of digital content, giving people the possibility of being in control through the tangibility of these physical objects.
GUi (Graphical User interface)—a typical interface between a person and a computer, where a screen provides feedback from the actions of the user.
Typi-cally, the GUI is manipulated with an abstract input system, such as a keyboard or mouse.
tUi (tangible User interface)—an atypical interface where the user manipulates an object that has been designed to relate to its function and also provide user feedback.
Manuscript received 13 August 2012.
Caitilin de Bérigny, also known as the artist onacloV, is a new media artist and designer. She earned her Ph.D. from the University of Canberra. In addition to being an artist, de Bérigny is also a lecturer at the University of Sydney.
Phillip Gough is a Ph.D. student at the Uni-versity of Sydney. He earned his Master of In-teraction Design and Electronic Media Arts from the University of Sydney.
Majdi Faleh is an architect and member of the AIA/OAT, American/Tunisian Institute of Architects. He earned his Master of Archi-tecture from Ball State University, Muncie, Indiana, U.S.A. He is currently a Fulbright scholar and Ph.D. candidate.
Erika Woolsey is a marine biologist and Ph.D. candidate at the ARC Centre of Excellence for Coral Reef Studies at James Cook University in Townsville, QLD.
Call for PaPers
environment 2.0
Guest Editor: Drew Hemment
In urban environments we are separated from the consequences of our actions as surely as the tarmac
of the road cuts us off from the earth beneath. But between the cracks in the pavement, another world
flourishes—local activism, recycling, environmental collectives, permaculture, urban gardening. Artistic
and social projects can widen the cracks in the pavements. Such creative innovations might be artworks,
social entrepreneurship, scientific intervention or innovations that harness everyday creativity.
Leonardo is soliciting texts that document the works of artists, researchers and scholars involved in the
exploration of sustainability in urban environments.
Published authors include: Marcos Lutyens and Andrew Manning; Dermot McMeel and Chris Speed; Ruth
Wallen; Gabriella Giannachi; Helen Mayer Harrison and Newton Harrison; Chris Welsby; Drew Hemment;
Rebecca Ellis; Brian Wynne; Carlo Buontempo; Alfie Dennen and others.
Authors are encouraged to submit manuscripts or proposals to <[email protected]>.