The inkML (language)
and other W3C techs
applied to a
multi-channel document
processing
system.
José Antonio Magaña Mesa (
[email protected]
)
R+D Software Engineer. Member of inkML WG.
HP – Barcelona Division
“In the next 100 years, half of
the 6000 languages spoken
in the world will disappear.”
American Language Society
(Chicago)
What is inkML?
http://www.w3.org/2002/mmi/ink
•
InkML is an XML data format for representing digital ink data that is input
with an electronic pen or stylus as part of a multimodal system.
•
In the context of the
W3C Multimodal Interaction Framework
, the markup
provides a format for:
1.
transferring digital ink data between devices and software components
2.
storing hand-input traces for:
•
Handwriting recognition (including text, mathematics, chemistry)
•
Signature verification
•
Gesture interpretation
•
The specification is based on the following documents:
1.
Ink Requirements Note
, published by the Working Group on 22 January 2003,
2.
InkXML
(
W3C Members only
), a specification contributed by IBM, Intel, the
International Unipen Foundation and Motorola.
More on inkML
The aim of the standard is to
remain
application agnostic
,
and provide the basis for
building based on it any XML
based language that
manipulates digital ink or serve
as transport layer for ink.
inkML
dppML
Note taking
HPLI-Annotated data
Companies participating in the working group:
IBM, Fraunhofer Gesellschaft, Apple, Corel, (Motorola), HP
Last draft(3rd draft): 28 Sept 2004
Statement of work of inkML
• Overview
• As more electronic devices with pen interfaces have and continue to become available for entering and manipulating information,
applications need to be more effective at leveraging this method of input. Handwriting is an input modality that is very familiar for most users since everyone learns to write in school. Hence, users will tend to use this as a mode of input and control when available.
• A pen-based interface is enabled by a transducer device and a pen that allow movements of the pen to be captured as digital ink. Digital ink
can be passed on to recognition software that will convert the pen input into appropriate computer actions. Alternatively, the handwritten input can be organized into ink documents, notes or messages that can be stored for later retrieval or exchanged through
telecommunications means. Such ink documents are appealing because they capture information as the user composed it, including text in any mix of languages and drawings such as equations and graphs.
• Hardware and software vendors have typically stored and represented digital ink using proprietary or restrictive formats. The lack of a public
and comprehensive digital ink format has severely limited the capture, transmission, processing, and presentation of digital ink across heterogeneous devices developed by multiple vendors. In response to this need, the Ink Markup Language (InkML) provides a simple and platform-neutral data format to promote the interchange of digital ink between software applications.
• InkML supports a complete and accurate representation of digital ink. For instance, in addition to the pen position over time, InkML allows
recording of information about transducer device characteristics and detailed dynamic behavior to support applications such as handwriting recognition and authentication. For example, there is support for recording additional channels such as pen tilt, or pen tip force (often referred to as pressure in manufacturers' documentation).
• InkML provides means for extension. By virtue of being an XML-based language, users may easily add application-specific information to ink
files to suit the needs of the application at hand.
• 1.1 Uses of InkML
• With the establishment of a non-proprietary ink standard, a number of applications, old and new, are expanded where the pen can be used
as a very convenient and natural form of input. Here are a few examples.
− Ink MessagingTwo-way transmission of digital ink, possibly wireless, offers mobile-device users a compelling new way to communicate. Users can draw or write with a pen on the device's screen to compose a note in their own handwriting. Such an ink note can then be addressed and delivered to other mobile users, desktop users, or fax machines. The recipient views the message as the sender composed it, including text in any mix of languages and drawings.
− Ink and SMILA photo taken with a digital camera can be annotated with a pen; the digital ink can be coordinated with a spoken commentary. The
ink annotation could be used for indexing the photo (for example, one could assign different handwritten glyphs to different categories of pictures).
− Ink Archiving and RetrievalA software application may allow users to archive handwritten notes and retrieve them using either the time of creation of the handwritten notes or the tags associated with keywords. The tags are typically text strings created using a handwriting recognition system.
− Electronic Form-FillingIn support of natural and robust data entry for electronic forms on a wide spectrum of keyboardless devices, a handwriting recognition engine developer may define an API that takes InkML as input.
− Pen Input and Multimodal SystemsRobust and flexible user interfaces can be created that integrate the pen with other input modalities such as speech. Higher robustness is achievable because cross-modal redundancy can be used to compensate for imperfect recognition on each individual mode. Higher flexibility is possible because users can choose the most appropriate from among various modes for achieving a task or issuing commands. This choice might be based on user preferences, suitability for the task, or external conditions. For instance, when noise in the environment or privacy is a concern, the pen modality is preferred over voice.
Application:
HP Forms Automation
System
HP Forms Automation System
Fill out the printed form using
the HP Digital Pen 200
HP Forms Automation System
Processed information as it
appears in an Excel spreadsheet.
Download captured
information
HP Forms Automation System
Form Processing Workflow
Digital Form PDF
Form Processing Application
Service Controller
Digital Paper Run-time (SDK) Acrobat Reader Program Digital Paper Printer Driver Digital Pen Dow nload Softw are HP LaserJet Printer HP Digital Pen Printed digital form Print Request Req./resp. for unique dot pattern
Print form command
Output digital paper form, w ith unique dot pattern
Req./resp. for form processing service
Pen data and related information
Req. to upload pen data, via USB cradle
Customer’s IT Infrastructure
Form processing results
User w rites on digital form and pen captures stroke data DLD (parser) Plug-in General comm.
dppML
HTTP/
HTTPS
XHTML
CSS
XSLT
SOAP
SVG
HTTP/
HTTPS
SOAP
CSS
PDF/XFDF
XFORM
S
WSDL
HP Forms Automation System
Form Processing Workflow
1.
User opens digital form using Acrobat
Reader.
2.
User issues print command, using Digital
Paper Printer driver, from Acrobat
Reader.
3.
Digital Paper Printer driver sends
requests, along with form id, to Service
Controller for unique dot pattern. Service
Controller responds with dot pattern
information.
4.
Digital Paper Printer sends print
command to printer.
5.
HP LaserJet prints paper form with unique
dot pattern.
6.
User writes on digital paper form and pen
simultaneously captures written pen
stroke data. User marks Send box when
done filling form.
7.
User docks digital pen to cradle to
being data upload to appropriate form
processing application server.
8.
Digital Pen software contacts Service
Controller for upload request. Service
controller checks for appropriate
application server and responds with
contact information. (Secured comm.)
9.
Digital Pen software contacts
application server to begin upload.
(Optional secured comm.)
10.
Forms processing application performs
business logic to process form data;
e.g. interfaces with IT applications, etc.
dppML – XML language for HP FAS
dppML is the key point in XML inter-application connectivity:
•Internal XML language
•Basis for XML transformations.
•Storage format.
dppML
Form Info
Ink Info
(inkML)
Logical Info
Print Info
dppML description
XML Schema driven (xsd)
inkML compliant
Different levels: Form definition, logical, ink Interoperability and XML Connector
Output and Input format
FAS - dppML structure flexibility
Contains information about:
•
Document structure ( forms )
•
Information filled by the user (digital ink)
•
Process results at different stages: ICR, data
correction, data validation
•
System information
•
User information
This information is separated in different blocks so
that only the required layers are transmitted.
FAS: dppML samples
Device definition
FAS - dppML samples.
FAS - dppML samples
Ink representation
FAS - dppML samples
Text Fields
FAS - dppML samples
Boolean fields
FAS -
Multi-channel environment
Device independence: CC/PP
Forms
Automation
System
Form data
repository
External
Form
Processing
Apps
dppML
WSDL
dppML
dppML
dppML
dppML
dppML
dppML
XM
L
Form
Document
RD
F
Advantages of use of W3C standards
•
Benefit from knowledge and experience of other
people in fast evolving technologies
•
Higher interoperability possibilities
•
Reduce development costs by use of
implementations of these standards:
−
Open source: Xerces (XML family), Axis (SOAP), ...
−
Java technologies: J2SE(DOM), J2EE (HTTP), JAI
(PNG), ..
Forms Automation System workflow
Current forms processes
mail form Deliver form Receive form Scan form Interpret form Validate form Data output fax form Fill form
Customer benefits:
- Reduced administration
- Less resources needed
- Forms Compliance
- ”Image” Digital Quality
Lowest level of automation
Interpret form Validate form Data output Fill form