T TE M
TECHNICS TECHNOLOGIES EDUCATION MANAGEMENT
Version for pr
eview
AbstractTouchscreens are hardly accessible to the blind. Assistive technologies offered by mobile device manufacturers do not eliminate all the obstacles. We propose new solutions that would improve the acces-sibility of touch interfaces in Android-based devices. We present the concept of services that provide func-tions and resources of the touchable Android-based
devices for blind with Braille notetakers.
Key words: User Interfaces – Input devices and strategies, User-centered design, assistive technol-ogy, mobile accessibility, touch screens, blind users
1. Introduction
Advanced functions of Android-based de-vices are hardly accessible to the blind using a touchscreen. One example of barrier is the virtual QWERTY keyboard, other examples include lack of haptic points on the screen (these would im-prove spatial orientation) and poor suitability of touchscreen gestures for the blind (preferred ges-tures start on the edge of the screen or in its close vicinity). The concept of our solution improving the accessibility is based on two assumptions, test-ed positively for example in [1] and [2]:
– most of the tested blind smartphone users deem notetakers to be indispensable, especially for taking notes quickly;
– the use of a linear-sequential Braille
interface by the blind, especially for entering
text using a physical Braille keyboard, is much more efficient than the use of a virtual keyboard (QWERTY or Braille).
Advanced smartphone functions are made avail-able through software that we developed as part of concept implementation. This part operates under the Android system and provides services for the notetaker operating under Windows CE like most
ones. These services are called by specific software
operated under Windows CE and developed as
sec-ond part of the concept. Both of these parts of soft -ware also support the application layer protocols
and transmission layer. The Bluetooth it is trans -mission layer protocol and it is supported by the
lowest level of communication software. Bluetooth
protocol software runs on both devices and con-nects either the smartphone to the notetaker or the other way. And the application layer XML-based protocol has been developed by us. The protocol that transmits data from the notetaker of the service to be performed with the necessary parameters. As well as transfers to notetaker result of the service.
Pilot implementation of the concept involved a Samsung Galaxy S III smartphone and a Polish notetaker Kajetek SD, see [3]; it included access
to contact management and message composing services. Results of pilot implementation tests
conducted on 5 blind users with different Braille
and technical experience indicate a need for fur-ther development of the concept.
2. Related Work
There are many different ways to interact with interfaces of touchscreen-based devices: a simple single tap, several simultaneous or successive taps, directional and scanning gestures as well as
fixed and adaptive layouts of function fields com -prising the screen. However, software developers often neglect to adapt their methods to the needs of a blind user.
This is because researches are focused on
over-coming the two greatest difficulties faced by the
blind when using touchscreen-based mobile
devic-es: enabling efficient navigation through graphical
elements and their correct selection and develop-ing a method for quick and correct text input.
Findings of Kane et al. presented in [4] point to differences in preferred gestures and ways of mak-ing them between sighted and blind users. Oliviera
The services of Android-based devices
increasing their accessibility for the blind
Jolanta Brzostek-Pawlowska, Daniel Kocielinski Institute of Mathematical Machines, Warsaw, Poland.
et al. demonstrated in [5] that the blind input text
using different methods more or less efficiently
depending on their personality traits and personal
experience. D’Andrea found out that most of the
tested blind students prefer using smartphones with
Braille notetakers and value their Braille skills, es -pecially for the possibility of taking notes, see [1]. Southern et al. in turn noticed lower erroneousness
in case of entering text in Braille, using a physical
or virtual keyboard, see [2] and [5]; according to 4.,
the use of a physical Braille keyboard is the most efficient method. Azenkot et al. confirmed these findings in [6] and [2]. The virtual QWERTY key -board has been enhanced e.g. by Findlater et al. in
[7], applying extensions available through specific
user gestures. In [8] Costagliola and Capua suggest implementing into the virtual keyboard an addition-al menu displayed upon making certain gestures around a given character. In [9] Ruamviboonsuk et al. suggest entering numbers sightlessly (to enable
quicker dialling), using specific multi-touch combi
-nations. Improvements suggested in [7], [8] and [9]
have not been tested by the blind nor designed for this group of users.
In [4] the authors suggest a fixed virtual Braille
keyboard that would have 6 keys: 3 on the left and 3 on the right side of the screen, near its edges .[6]
in turn suggests a dynamic virtual Braille keyboard appearing where 3 fingers of one hand or 6 fingers of two hands are placed. In [10] Kocieliński and Brzostek-Pawłowska have proposed multi-touch
gestures preferred by blind to speed up navigation through the lists of items.
3. Concept of increasing the accessibility through the facilitated access to specialized services
Our goal is allowing blind people to easily use such advanced functions of smartphones as speech recognition, cloud data (e.g. contacts) management,
file sharing (e.g. Dropbox), web-based search en -gines, instant messaging, phonecalls or GSM and
GPS navigation. Our concept makes it possible
blind user to effective use all the functions of a smartphone using developed specialized services
launched with Braille notetaker. Our solution elimi -nates the need for development aimed at imple-menting smartphone functions into notetakers. This
way the blind would be able to use common mobile devices, not only to selected functions of
smart-phones developed in Braille notetaker.
This goal we will achieve using integration of the two environments – Windows CE of the no-tetaker and Android of the smartphone.
In this research we have applied the results of
our earlier researches presented in[10]. Their aim was to find ways of transforming the graphical ob
-jects on a Braille interface. As a results we have
formulated basic assumptions concerning Versa-tile Multimodal Linear Interface (VMLI) and we have tested its pilot version. VMLI transforms a graphical objects and their features into the hierar-chy of linear lists.
In research in on the integration of smart-phones and notetakers presented in this paper we have used the VMLI to be able generate a lists of items transmitted to the notetaker as an output pa-rameters of executed services. We developed set of Android services and the protocol of access to services (application layer protocol) and of pro-viding results of services.
By integrating user will have easier access to
advanced functionalities of smartphone using the
physical Braille keyboard. Operation of notetakers is based mainly on Braille keystrokes and sequen -tial-hierarchical access to smartphone functions.
Preferred interaction methods, which speed up
and facilitate blind users’ action, include:
– function selection lists activated by pressing appropriate navigation keys,
– handy menus available through hierarchical navigation or called using keystrokes, – sets of Braille keystrokes that provide access
to all the important notetaker functions at any time,
– editing functions that allow using six- and
eight-dot Braille and navigating through the
text quickly.
4. Description of Braille notekers
The Braille notetakers are provided with Braille keyboard (6 or 8 keys) and with synthesis
speech module which allow a user to access no-tetaker’s functionalities. Less advanced notetakers are mainly used to edit text notes with the aid of
Version for pr
eview
advanced notetakers also have Braille displayswhich allow a blind user to read and edit notes in a more comfortable way. Notetakers can also be provided with complex editor, calculator, media player, Internet browser, mail client etc. Those notetakers are similar to the smartphones consid-ering their functionality. The most advanced ones
are similar to laptops of sighted users. On figure
1 there are examples of the notetakers: from sim-plest to most complex ones.
Figure 1. a. A simple Braille notetaker which al-lows to edit text with the aid of Braille keyboard. b. A Braille notetaker which is provided with Braille keyboard and Braille display.
c. A complex Braille notetaker which is provided with advanced functionality, such as Internet browser, media player etc.
In general the Braille notetakers are operated
under mobile operating systems (such as Windows CE, Windows Mobile), they can operate external
SD/SDHC memory cards and (through build-in USB ports) another external memory drivers, printers etc. Simple Braille notetakers usually have
limited editing functions; on the other hand more
complex devices can be provided with MS Office
package (Word, Excel), Internet browser, mail
cli-ent, synchronizing functions with PC (for example
to synchronize personal documents and contacts)
etc. Braille notetakers can have a number of addi -tional attractive functions such as e-book’s player, dictaphone, speaking colour tester, stopwatch etc..
As opposed to the case of graphical smartphone interfaces, where users interact with applications by making touch gestures on items displayed on the screen, operation of notetakers is based mainly
on Braille keystrokes and sequential-hierarchical access to functions. Preferred interaction methods,
which speed up and facilitate blind users’ actions and are usually integrated in notetakers, include:
– function selection lists activated by pressing appropriate navigation keys,
– handy menus available through hierarchical navigation or called using keystrokes, – sets of Braille keystrokes that provide access
to all the important notetaker functions at any time,
– editing functions that allow using six- and
eight-dot Braille and navigating through the
text quickly.
5. Principles of communication between an Android-based device and a Braille notetaker
Our concept assumes that the Android-based
device and the Braille notetaker communicate via Bluetooth, which is common in smartphones, programmable via API (Application Interface Programming) using the android.bluetooth pack
-age and supported by many Braille notetakers. Notetakers not equipped with an integrated Blue -tooth adapter may be expanded with an external
Bluetooth adapter.
The Bluetooth protocol is a carrier layer for
commands (queries) sent from the notetaker and for answers sent from the smartphone. A layer for the developed communication application
pro-tocol is provided by the Bluetooth propro-tocol. The
XML format is used for application protocol writ-ing. The application used for communication be-tween the two devices consists of two modules: Commander, which operates in Windows CE
environment of the Braille notetaker, and Con -troller, which operates in Android environment. Commands generated by Commander and sent to Controller as well as answers generated by Con-troller, including any requested data and sent to Commander are based on the XML format. The
notetaker sends queries with identifiers of services
to be provided by the Controller module, which works as a service server.
Figure 2 shows an example of a request-related XML command for service execution “Create-ContactEntry”.
Figure 3 shows an example of an XML answer (transmitting service result.)
Figure 2. Example of a command with a request sent from the notetaker to the smartphone
Figure 3. Example of an answer with the data sent from the smartphone to the notetaker after service execution
The Controller module, developed in its pilot version as an Android service, can connect with
the notetaker via Bluetooth in two ways: by manu -ally choosing the notetaker from a list generated by Controller and sending a connection request to
Commander or by accepting a Bluetooth connec -tion request incoming from the notetaker’s
Com-mander module. Controller generates a list of
de-vices available via Bluetooth as a result of detect
-ing them by scann-ing, us-ing the BluetoothAdapter
class instance provided in the Android system. A link between the two devices is established upon
successfully confirming the connection request
sent to the notetaker or upon accepting the con-nection incoming from the notetaker. Then the Controller module waits for Commander com-mands and executes them.
The smartphone’s Controller module works in the following modes:
– sending Bluetooth connection requests to
the notetaker (if chosen from the list by the user),
– waiting for Bluetooth connection requests
incoming from the notetaker,
– accepting a Bluetooth connection request
incoming from the notetaker,
– receiving and sending text and binary data (executing commands incoming from the notetaker).
For the purposes of providing services request-ed by the notetaker, the Controller module works in server mode in relation to the Commander mod-ule. For running in server mode the Controller module keeps an open server socket (an instance
of the Android BluetoothServerSocket class),
which waits for incoming queries. The connection is accepted only if the notetaker has sent a query
with a specific UUID value of ControllerService.
The Commander module performs the following functions for the purposes of smartphone access:
– detecting Android-based devices (with the Controller module running) present within
the range of Bluetooth communication,
– receiving text data from the paired Android-based device,
– sending the input text commands to the Android-based device.
The essential function of the Commander mod-ule is to send commands and data to the Android-based device with a request of service execution. The transmitted commands they begin execution of services that allow e.g. employing the wire-less keyboard functionality when the user inputs
Version for pr
eview
are then sent as corresponding commands to the
smartphone’s Controller module via Bluetooth. Besides transmitting commands the Commander
module receives answers as a result of services. All text data contained in answers is sent to the
speech synthesiser and/or a Braille line.
The following types of commands have been implemented in the Commander module for the purposes of sending:
– character – a command of this type simulates inputting a given character by the user; A command corresponding to inputting the let-ter “a” by the user will have the following form: <command type=”character”>a</command>
– navigating – a command of this type simulates pressing a certain navigation key (routing);
A command corresponding to pressing the left arrow will have the following form:
<command type=”navigating”>left</command>, – function – a command of this type simulates
pressing a certain function key.
Such command will have the following form: <command type=”function”>function_key</ command>
Figure 4 shows an overview of the proposed communication concept.
Figure 4. Communication layers between an Android-based device and a Braille note taker
6. Methodology
In order to test the concept we have developed communication protocol as a sequence of XML
query-response messages; the Bluetooth protocol
serves as the transport layer.
We have developed also two interacting mod-ule. One module runs on an Android-based
smart-phone and one on a notetaker (Kajetek SD - Pol
-ish product) operated under Windows CE 5. Both modules support wireless Bluetooth connectivity.
The module on Android system has been
im-plemented in Java, using Android API level 17,
and tested on a Samsung Galaxy S III smartphone. Feedback is realised using text-to-speech, with the
TalkBack screen reader.
The notetaker software is a C++ library that ex-tends Kajetek application with support for a proto-col used to communicate with the Android module.
During the research the basic concept assump -tions have been tested by managing contacts us-ing the remote interface of the notetaker. The pilot software allows the user to operate the smartphone
using Kajetek as a physical Braille keyboard; this
way he or she can quickly navigate through smart-phone interface items using notetaker keystrokes. The software also provides easy wireless access to contact management functions of the smartphone. The following contact management services have been implemented in the Kajetek notetaker:
– browsing contact lists (using specific Braille
keystrokes),
– reviewing details of selected contacts,
– adding new contacts (using the editing functions integrated in Kajetek software), – deleting contacts.
Several methods of inputting text into a smart-phone have also been tested during the research. The test application provides three ways of com-posing messages to selected recipients: using a
virtual QWERTY keyboard, a virtual Braille key -board or the editing functions of the notetaker.
Practical tests of text input speed have been con -ducted on a virtual QWERTY keyboard and on the physical keyboard of the Kajetek notetaker.
The tests have been conducted on a group of 5 users with different experience in the use of smart-phones: from a few weeks to a couple months. The
test was carried out on the iPhone with VoiceOver
screen reader and on the smartphones
Android-based (version 4.02 or later) equipped with the TalkBack screen reader. All the testers knew Braille well and they were ones with experience in the practical use of Braille notetakers.
7. Results
Practical tests of pilot application have shown
that the use of a notetaker to operate the inter-face of a smartphone is not only a solution that
improves a blind user’s work efficiency and con -venience, but also a practical alternative for those for whom a touch interface is manually
inacces-sible. Feedback from the test group confirms that
the existing accessibility mechanisms of the
An-droid system (such as TalkBack and BrailleBack),
especially the ones related to navigation based on the available gestures (e.g. implemented through
TalkBack) and text input using a virtual QWERTY keyboard, are subject to significant limitations.
The tests have proven that managing contacts with the interface of the Kajetek notetaker and its
phys-ical Braille keyboard is more efficient than doing
so using the standard smartphone application for
managing contacts combined with the TalkBack
screen reader, gestures and the virtual QWERTY keyboard. Selecting desired contacts from the list
using Kajetek is more efficient primarily because
of lesser erroneousness of browsing actions; op-erating a touch interface requires greater manual precision. One of the testers suggested extending the list item selection functionality of Kajetek e.g.
with a quick search function looking for the first
few entered letters of a contact.
The received feedback also confirms that the
existing smartphone accessibility mechanisms
re-lated to text input are insufficient. Message com
-posing tests indicated a significant advantage of the physical Braille keyboard of a notetaker over a virtual QWERTY keyboard. The use of a Braille
notetaker with its editing functions is much more
efficient than using a virtual QWERTY keyboard:
the slowest and the fastest tester reached a text input speed of 11 and 38 characters per minute
(CPM) in the case of the latter and respectively 87 and 160 CPM in the case of the former.
8. Conclusions
The results of preliminary research on pilot im-plementation into the Android system, concerning functions like managing contacts and composing
messages to selected recipients, confirm that the ad -opted concept of using interaction mechanisms for
touchscreen-based mobile devices and Braille no -tetakers is a promising direction for further research and give grounds for continuing works. The research suggests new ways of solving the essential problem, which is the inaccessibility of new, advanced tech-nologies to the blind. The main cause of this prob-lem is that hardware manufacturers and software developers usually marginalise this group of users by providing (mostly behindhand) only accessibility tools (such as screen readers) that rather adapt meth-ods of operating graphical touch interfaces than provide other, suitable ones. The proposed solution for improving the accessibility of touch interfaces is quite unusual compared to those currently provided
in smartphones: it is based on using a Braille inter -face, smart notetaker mechanisms and a dedicated communication protocol for convenient operation of advanced mobile device functions and services. The feedback from blind users is very positive.
Currently we develop that conception and its application in national research project
concern-ing the PlatMat platform [11] which aims is in
-creasing efficiency of real-time math-related com -munication between teacher and blind students using mobile devices.
References
1. d’Andrea FM. Preferences and Practices Among Stu-dents Who Read Braille and Use Assistive Technol-ogy. Journal of Visual Impairment & Blindness, Year 2012, October-November 2012; 106: 585-596. 2. Southern C, Clawson J, Frey B, Abowd GD, Romero
M. An Evaluation of BrailleTouch: Mobile Touch-screen Text Entry for the Visually Impaired. Proc. MobileHCI’12, ACM 2012; 317-326.
3. Information about Kajetek SD on the manufac-turer’s website: http://www.ece.com.pl/index. php?option=com_virtuemart&page=shop. browse&category_id=5&Itemid=2
4. Kane SK, Wobbrock JO, Ladner RE. Usable gestures for blind people: Understanding preference and per-formance. Proc. CHI’11, ACM, 2011; 413-422.
Version for pr
eview
5. Oliveira J, Guerreiro T, Nicolau H, Jorge J, Gon-çalves D. Blind people and mobile touch-based
text-entry: acknowledging the need for different flavors.
Proc. ASSETS ‘11, ACM 2011; 179-186.
6. Azenkot S, Wobbrock JO, Prasain S, Ladner RE.
Input finger detection for nonvisual touch screen text
entry in Perkinput. Proc. Graphics Interface, Cana-dian Information Processing Society 2012; 121-129. 7. Findlater L, Lee BQ, Wobbrock JO. Beyond
QWERTY: Augmenting Touch-Screen Keyboards with Multi-Touch Gestures for Non-Alphanumeric Input. Proc. CHI’21, ACM 2012; 2679-2682. 8. Costagliola G, Fuccella V, Di Capua M. Text Entry
with KeyScretch. Proc. IUI, ACM 2011; 277-286. 9. Ruamviboonsuk V, Azenkot S, Ladner RE. Tapulator:
A Non-Visual Calculator using Natural Prefix-Free
Codes. Proc. ASSETS’12, ACM 2012; 221-222. 10. Kocieliński D, Brzostek-Pawłowska J. Linear In
-terface for Graphical In-terface of Touch-Screen: a Pilot Study on Improving Accessibility of the Android-Based Mobile Devices, Proc. Mobile HCI 2013, ACM 2013; 546-551.
11. Brzostek-Pawłowska J, Mikułowski D. A Concept of
Mobile Technology for Remotely Supporting Math-ematical Education of the blind, Proc. REV 2014, © 2014 IEEE, 2014; 54-60.
Corresponding Author
Jolanta Brzostek-Pawłowska,
Institute of Mathematical Machines, Warsaw,
Poland,
