Assistive Application Programming Guide For OS X 2.0

(1)

Assistive Application

 

Programming Guide

 

For OS X

2.0 For the PFAssistive and PFEventTaps Frameworks

(2)

(3)

Introduction to the

 

Assistive Application

 

Programming Guide

1 The PFiddlesoft™ PFAssistive and PFEventTaps Frameworks

1

2 Licensing Terms

3 Free for Personal Use and for Distribution and Use With Free Products

3 One-Time Licensing Fee for Distribution and Use With Paid Products

3 DISCLAIMERS

3 How to Use the

 

PFAssistive Framework

5 Discovering UI Elements

6 1. Reading the Screen

6 2. Browsing the Accessibility Hierarchy

8 3. Observing Notifications

9 Reading a UI Element's Attributes

11 1. Reading Simple Attributes

11 2. Reading Unknown Attributes Using Key-Value Coding

12 3. Reading Parameterized Attributes

13 4. Navigating the Accessibility Hierarchy

16 Controlling an Application

17 1. Setting a UI Element's Attributes

18 2. Performing Actions on a UI Element

20 3. Sending Keystrokes to an Application

21 Subclassing

22 1. The Class Registry

22 2. Subclass Requirements

23 Miscellaneous

23 How to Use the

 

PFEventTaps Framework

24 Monitoring Events Using an Event Tap

25 Filtering, Modifying, Blocking, and Responding to Events

29 Posting Synthetic Events

31 Reading the State of an Event Source

33 How to Grant Access

 

(4)

How to Build an Application Using the Frameworks

39 Historical Background (Mountain Lion and Older)

39 Modern Requirements (Mavericks and Newer)

40 Building an Application When the Frameworks are Installed as Shared Frameworks

41 Building an Application to Run As a Trusted Accessibility Process Before Mavericks

42

(5)

(6)

Introduction to the

 

Assistive Application

 

Programming Guide

This Programming Guide explains how to write an assistive application for Macintosh computer users with disabilities using the PFiddlesoft PFAssistive and PFEventTaps Frameworks. The PFiddlesoft Frameworks are written in Objective-C and designed for use with Cocoa applications written in Objective-C or Swift. They support and enhance Apple’s Accessibility and Quartz Event Taps APIs, enabling Cocoa developers to use familiar programming techniques to create assistive applications without having to master the technicalities of Apple’s procedural C Accessibility, Core Foundation, and Core Graphics APIs.

Apple's Accessibility technology grew out of Section 508 of the Workforce Investment Act of 1998 and its requirements regarding access to electronic and information technology for persons with disabilities. Compliance with Section 508 is a prerequisite for sale of computer and other products to the federal government and to many state agencies and

educational institutions. The Accessibility API is designed for use both by developers incorporating its features into their own accessible applications and by developers of assistive devices and applications for users with disabilities. The Event Taps API is also a Section 508 enabling technology.

Because Accessibility is built into every standard OS X User Interface element, whether written using the Cocoa or Carbon frameworks, it is capable of much broader uses. Software testing tools, network administration tools,

troubleshooting tools, plug-ins for applications that don't have a plug-in architecture, and remote control applications are only some of the possibilities.

The PFiddlesoft™ PFAssistive and PFEventTaps Frameworks

The PFAssistive Framework was created in 2003 as the engine driving PFiddlesoft’s highly regarded UI Browser utility for developers and for users of Apple’s AppleScript GUI Scripting technology. The PFEventTaps Framework was added in 2007 as the engine underlying PFiddlesoft’s Event Taps Testbench utility for developers. Both frameworks have been revised and updated over a period of years, and they have demonstrated their power and reliability in PFiddlesoft’s commercial and free developer utilities. PreForm Assistive Technologies, LLC is making the PFiddlesoft Frameworks available to all Macintosh developers. Together, the PFiddlesoft Frameworks bring to Cocoa developers the full range of Accessibility capabilities needed to write assistive applications and other software that explores, manipulates, and monitors the User Interface elements and user inputs of most OS X applications.

The current versions of the PFiddlesoft Frameworks require OS X v10.7 Lion or newer, and they support all Accessibility and Event Taps features introduced by Apple in subsequent releases of OS X through OS X v10.11 El Capitan. They support clients that run natively on Intel processors using 64-bit architectures with reference counted memory

management or ARC. Support for garbage-collected client applications is not available. They may be installed either as shared frameworks in the local /Library/Frameworks folder or as embedded frameworks in an assistive application’s bundle, for use by any assistive application in OS X v10.9 Mavericks or newer and for use by assistive applications using

(7)

global access for assistive devices in OS X v10.8 or v10.7. Use as a shared instead of an embedded framework is required only in the special case of an assistive application that is to be treated as a “trusted” application in OS X v10.8 or v10.7. See How to Grant Access to Assistive Applications for more information. Assistive applications that use the Accessibility and Quartz Event Taps APIs are ineligible for the Mac App Store whether or not they use the PFiddlesoft Frameworks.

The two frameworks are independent of one another. Both together enable you to write a full-featured assistive application, but you may need only one or the other to accomplish more limited purposes.

Download the PFiddlesoft Frameworks at pfiddlesoft.com. PFiddlesoft’s free developer utilities and free 30-day trial versions of its commercial products are also available for download there.

This version of the Assistive Application Programming Guide is for use with PFAssistive Framework 3.5.0 and PFEventTaps Framework 1.5.0.

Licensing Terms

1

The PFiddlesoft Frameworks are copyrighted software.

Free for Personal Use and for Distribution and Use With Free Products

The PFiddlesoft Frameworks may be licensed free of charge for personal use, including use during development of any client application or other software. They may also be licensed free of charge for distribution and use with any client application or other software that you distribute to the public free of charge (including freeware as well as free beta or trial versions of a product for which you intend to request or require payment in the future). You are required only to give notice to PreForm Assistive Technologies, LLC, to provide attribution to PreForm Assistive Technologies, LLC in your client application or other software, and to include the copyright notice and license in your client application or other software.

One-Time Licensing Fee for Distribution and Use With Paid Products

If you distribute the PFiddlesoft Frameworks with or in a client application or other software product for which you request or require payment, or if you distribute a client application or other software product that includes or uses the PFiddlesoft Frameworks for which you request or require payment, such as donationware, shareware, and commercial applications, or for internal use within a for-profit organization, you must within thirty days of initial distribution of your product pay PreForm Assistive Technologies, LLC a flat one-time license fee of $250 U.S. for each framework that you distribute or use, regardless of the number of units of your product you distribute or use. This fee covers all present and future versions of your product, but any separate and distinct product requires you to pay PreForm Assistive

Technologies, LLC an additional licensing fee of $250 U.S. for each framework that you distribute or use, as described above.

An executed license is required both for free distribution or use, and for distribution or use subject to a flat one-time license fee with a product for which you request or require payment. Download the PFAssistive Framework distribution license or the PFEventTaps Framework distribution license or both of them, depending on which of the PFiddlesoft Frameworks you distribute or use. Then print the licenses in duplicate, fill in the blanks, sign them, and mail them to:

PreForm Assistive Technologies, LLC 

P.O. Box 326 

Quechee, VT 05059-0326

DIFFERENT TERMS APPLY TO LARGE OR ESTABLISHED COMMERCIAL SOFTWARE DEVELOPERS. The source code is available for an additional fee. Contact us at [email protected] for details.

DISCLAIMERS

The PFiddlesoft Frameworks are provided on an "AS IS" basis. The following disclaimers apply to each of the frameworks:

This is a summary of the PreForm Assistive Technology, LLC licenses. For the legally binding terms consult the licenses

1

themselves. The licenses are embedded in the frameworks’ bundles, and they are also available for download at pfiddle-soft.com.

(9)

PREFORM ASSISTIVE TECHNOLOGIES, LLC MAKES NO WARRANTIES, EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, REGARDING THE FRAMEWORK OR ITS USE AND OPERATION ALONE OR IN COMBINATION WITH OTHER PRODUCTS. THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE EXPRESSLY DISCLAIMED.

IN NO EVENT SHALL PREFORM ASSISTIVE TECHNOLOGIES, LLC BE LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) ARISING IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF THE FRAMEWORK, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF PREFORM ASSISTIVE TECHNOLOGIES, LLC HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. YOUR REMEDY FOR ANY DEFECT OR FAULT IN THE FRAMEWORK IS LIMITED TO REFUND OF THE LICENSE FEE YOU PAID. 

(10)

How to Use the

 

PFAssistive Framework

Apple's Accessibility API implements the concept of a UI element, an object that represents a user interface element on the screen in any running application, such as a menu, a window, or a button, or the application itself. The Accessibility API also implements the concept of an observer, an object that registers to observe a UI element that issues Accessibility notifications when changes occur.

The PFAssistive Framework implements these same concepts in its PFUIElement, PFApplicationUIElement, and PFObserver classes, each of which instantiates and encapsulates an associated Accessibility API object and makes its capabilities available to an assistive application using standard Objective-C, Swift and Cocoa techniques. For example, an assistive application using the PFAssistive Framework can implement optional delegate methods declared in the framework’s PFUIElementDelegate and PFObserverDelegate formal protocols to respond to Accessibility notifications. In this chapter, you learn how to discover UI elements, how to read UI element attributes, and how to control UI elements and, through them, the target application, all in the interest of supporting assistive applications. Assistive applications enable a user with disabilities to use the computer to perform the same tasks that any user can perform with the graphical user interface. An assistive application typically does this by performing these tasks:

• An assistive application discovers individual UI elements in the target application in one of three ways: by locating an element on the screen, for example, the element currently under the mouse or the element that is the target application’s frontmost window; by receiving a notification from an element that something about it has just changed; or by navigating the element hierarchy from a known starting point. Read the Discovering UI Elements

section for details.

• When an assistive application discovers a UI element of potential interest, it ascertains the element’s identity and nature by reading its attributes, such as its role, its title, and its position and size on the screen. Read the Reading a UI Element’s Attributes section for details.

• Once it identifies and understands a UI element, an assistive application manipulates or controls the element and, through it, the target application, at the user's direction. There are three ways to do this, depending on the nature of the element: by setting the element's value or other attribute; by performing an action on the element; or by sending a keystroke to the target application while the element has keyboard focus (or by sending a key combination that the application recognizes as a keyboard shortcut). Read the Controlling an Application section for details.

By performing these tasks—discovering, reading, and controlling a UI element—an assistive application enables a user with disabilities to do everything that a user without disabilities can do with the target application. This parity of treatment defines and delimits the Accessibility API. Apple rigorously enforces the notion that the Accessibility API should enable a user with disabilities to do everything that any user can do with a target application, and no more. What the API cannot do is as important as what it can do. For example, the Accessibility API cannot read or control UI elements in a window that is offscreen, even if the window still exists in memory and the Cocoa frameworks can see it.

(11)

This design principle explains a key feature of the Accessibility API, namely, the hierarchy of UI elements that an assistive application navigates to find any element's parent and children. In principle, the Accessibility hierarchy includes only those elements that a user can read and control in the graphical user interface of the target application. Although the Cocoa view hierarchy includes many invisible views that contain other views for purposes of programmatic control, the Accessibility hierarchy ignores them because they play no direct role in a user's work with the target application. In the first section of this chapter, you learn how to discover a UI element, including the application UI element and the special system-wide UI element. This involves using the PFUIElement class and its PFApplicationUIElement subclass to read the screen or to browse an application's Accessibility hierarchy, or using the PFObserver class to register for and observe notifications from an application or its individual UI elements. In the second section, you learn how to read a UI element's attributes. In the last section, you learn how to control an application by setting those attributes of its UI element that are settable, by performing actions that are recognized by some of its UI elements, and by sending keystrokes to it.

Discovering UI Elements

There are three primary ways in which an assistive application discovers UI elements:

1. A screen reader is the most common kind of assistive application. It ascertains the identify of the UI element currently under the mouse or at a specified position on the screen, either in the active application or in an application that is running but is not currently the frontmost application. Examples are Apple's VoiceOver application, its Accessibility Inspector, and the Screen Reader in PFiddlesoft's UI Browser. Read the Reading the Screen subsection for details.

2. A browser is another kind of assistive application. The user specifies a target application and then navigates that application's Accessibility hierarchy until it finds a UI element of interest. Examples are Apple's System Events application, which supports GUI Scripting for AppleScript using the Accessibility API, and the main browser view in UI Browser. Read the Browsing the Accessibility Hierarchy subsection for details.

3. An observer is a third kind of assistive application. It registers to monitor specified kinds of activity in a target application or a particular UI element, and it responds when it receives a notification of interest. UI Browser is an example. Read the Observing Notifications subsection for details.

Some assistive applications, such as UI Browser, implement all three modes of operation.

1. Reading the Screen

A screen reader discovers UI elements by determining what UI element is located at a specified point on the screen or screens attached to the computer, typically the point where the mouse pointer is located. To implement this feature in your assistive application, use the +elementAtPoint:withDelegate:error: class method of the PFUIElement class or the -elementAtPoint: method of the PFApplicationUIElement class, both declared in PFUIElement.h. Use the PFUIElement class method when an assistive application needs to know what UI element is visible at the specified location without regard to which running application owns it. Use the PFApplicationUIElement instance method when it needs to know what UI element belonging to a specific application is at the specified location, even if the element is currently obscured by a UI element belonging to another application.

Both of these methods take an NSPoint structure specifying a point on the screen as an argument. The Accessibility API uses the Quartz 2D device space coordinate system in which the origin is at the top-left corner of the primary screen, the

(12)

screen on which the menu bar appears. Positive movement flows down and to the right; negative movement, up and to 2

the left. From the Cocoa perspective, device space is flipped.

A screen reader typically focuses on the current mouse pointer. In Cocoa, the location of the mouse pointer can be obtained at any time in bottom-left relative screen coordinates, regardless of the current event or pending events, using NSEvent's +mouseLocation class method. You must convert it to the Accessibility API's top-left relative screen coordinates before passing it to either of these methods. To avoid having to convert the coordinates, use Carbon's HIGetMousePosition() function, which returns top-left relative screen coordinates.

An assistive application may need to read the screen continuously as the user moves the mouse, or it may only need to read the screen at user direction, for example, in response to a keyboard shortcut, a global hot key, or an AppleScript command. To read the screen continuously, an application would typically use a repeating timer with a resolution that is tight enough to capture everything the mouse moves over, perhaps 2 or 3 times per second. To read the screen at user direction, simply call +elementAtPoint:withDelegate:error: or 

-elementAtPoint: in an action method or other method that is called on command.

Here is a simplified version of the -updateScreenReader method in UI Browser’s screen reader window controller: - (void)updateScreenReader {

HIPoint location;

HIGetMousePosition(kHICoordSpaceScreenPixel, NULL, &location); NSPoint point = NSMakePoint(location.x, location.y);

PFUIElement *element =

[PFUIElement elementAtPoint:point withDelegate:nil error:NULL]]; // ...

}

UI Browser’s -updateScreenReader method is called from a repeating timer, which is set up in the window controller’s -awakeFromNib method like this:

[NSTimer scheduledTimerWithTimeInterval:0.4 target:self

selector:@selector(updateScreenReader) userInfo:nil repeats:YES];

The +elementAtPoint:delegate:error: method takes two other arguments, delegate and error. They may be nil or NULL, respectively, as shown here. Pass an object in the delegate argument and implement either or both of the PFUIElementDelegate formal protocol methods -PFUIElementWasDestroyed: and 

–PFUIElementReportError: if you wish to take advantage of those capabilities. The error argument is an indirect reference to a standard NSError object. See the PFUIElement Class Reference for more information.

Both +elementAtPoint:withDelegate:error: and -elementAtPoint: create and return an autoreleased PFUIElement object representing the UI element at the specified location. An assistive application can read its attributes,

The primary screen may differ from the screen returned by NSScreen’s -mainScreen method, which returns the

2

(13)

navigate the Accessibility hierarchy from it using its parent and children attributes, register to observe changes to it, and control it using the other features of the PFAssistive Framework.

2. Browsing the Accessibility Hierarchy

A browser discovers UI elements by navigating the Accessibility hierarchy from a known starting point, typically the root application element of a specified application or the system-wide element that is aware of the frontmost running application. To do this in your assistive application, start by creating an application or system-wide UI element using one of the initializers declared in the PFApplicationUIElement class, such as -initWithPath:delegate:, 

-initWithPid:delegate:, and -initSystemWideWithDelegate:, in PFUIElement.h.

The easiest way to create an application UI element is to use the path or file URL identifying the application package or file, or the application package’s bundle identifier. To access TextEdit, for example, you could use any available Cocoa technique to get the path to TextEdit’s application file, such as spelling it out like this:

NSString *appPath = @"/Applications/TextEdit.app";

Then create and initialize the application UI element like this: PFApplicationUIElement *appElement =

[[PFApplicationUIElement alloc] initWithPath:appPath delegate:nil];

The designated initializer for the PFApplicationUIElement class is -initWithPid:delegate:. Use the designated initializer when an application has easy access to the BSD Unix process identifier number (PID) of the target application. Every PFUIElement object makes its application PID available in its -pid method, so the application can use the designated initializer to create a PFApplicationUIElement object for the target application whenever a PFUIElement object is available, for example, after reading the screen. Convenience initializers taking the application’s full path, URL or bundle identifier are also provided. Alternatively, simply call the PFUIElement object’s -applicationElement method. The Accessibility API recognizes a special UI element known as the system-wide element. Use it when an application needs to get a PFApplicationUIElement object representing the application that is currently frontmost, using

PFUIElement’s AXFocusedApplication property.

The delegate argument to the three PFApplicationUIElement initializers plays the same role that it plays in

+elementAtPoint:withDelegate:error:, discussed above. Set it to nil if the PFUIElement delegate methods are not needed.

The first two initializers create and return an autoreleased PFApplicationUIElement object representing a running application. The third creates and returns an autoreleased PFApplicationUIElement object representing the system, from which the active or frontmost running application can be determined by reading its AXFocusedApplication attribute. An assistive application can read a target application element’s attributes, register to observe changes to it, and control it using the other features of the PFAssistive Framework. Most importantly for purposes of browsing, it can navigate the Accessibility hierarchy by using the AXParent and AXChildren attributes to get every element in the hierarchy. The root application UI element’s AXChildren property is the starting point. For more about reading Accessibility attributes to navigate the hierarchy using PFUIElement properties, consult the Navigating the Accessibility Hierarchy subsection of the Reading a UI Element’s Attributes section.

(14)

3. Observing Notifications

An observer discovers UI elements by registering to be notified when a target application’s UI elements are changed by user activity or by other means such as an AppleScript script or another assistive application. When the user of a target application types into a text field, adjusts a slider, moves or resizes a window, or selects a menu item, for example, an observer that has registered to observe such activity receives a notification. The notification includes a reference to the affected UI element, and the observer can then use it to respond.

To observe Accessibility notifications in your assistive application, create a PFObserver object using one of the factory convenience methods declared in PFObserver.h, or create a PFObserver object and initialize it with one of the declared initializers. Once a PFObserver has been created, register it to observe a particular notification using the 

-registerForNotification:fromElement:contextInfo: method. 3

A PFObserver has a choice of two techniques to watch for and respond to Accessibility notifications: a delegate method and a callback method. The delegate method is easier to use because it provides full observer management, and it suffices for most purposes. The callback method allows for more complex behavior when circumstances require it, but the assistive application must take responsibility for managing the observer. The choice is made when the PFObserver is created.

To create a PFObserver that uses a delegate method, call a factory method, such as +observerWithPath:, passing in the full path of the target application. Then, when the assistive application is ready to begin observing the target application, take two additional steps: set the delegate, and register to observe a specified notification from a specified UI element. The delegate must declare that it conforms to the PFObserverDelegate formal protocol, which is declared in PFObserver.h. Set the delegate by sending PFObserver’s -setDelegate: message.

To create a PFObserver that uses a callback method instead of a delegate method, call the corresponding

+observerWithPath:notificationDelegate:callback: factory method, passing in the full path of the target application. Also pass in an object to serve as the notification delegate and the selector for the callback method that the notification delegate implements.

As an alternative to using the target application’s full path, call factory methods that use the target application’s file URL, bundle identifier, or BSD Unix process identification number (PID) to create a PFObserver.

All of the factory methods have corresponding initializers for use when factory methods are inappropriate. The designated initializers are -initWithPid: and -initWithPid:notificationDelegate:callbackSelector:.

Registration using the -registerForNotification:fromElement:contextInfo: method requires two arguments, notification, a string identifying a valid Accessibility API notification, and element, the PFUIElement that is to be observed. As a convenience when the application as a whole is to be observed instead of one of its UI elements, pass nil as the element argument; the framework creates a temporary application element using the observer’s UNIX process identifier and saves you the trouble of creating one yourself. The contextInfo argument is optional; it is provided in case an assistive application needs to pass information to the observer for use when a notification is received. The element to be observed can be a PFApplicationUIElement. This is often desirable, because an observed application element issues notifications whenever the specified notification is issued by any UI element in the

When using the PFAssistive Framework under Mac OS X v10.5 Leopard and earlier, registering for either the AXApplica

3

-tionActivated or AXApplicationDeactivated notification also registered for the other. The same was true of the AXApplica-tionHidden and AXApplicationShown notifications. The framework did this in order to work around a longstanding Acces-sibility API bug that Apple fixed in Mac OS X v10.6 Snow Leopard.

(15)

target application. For example, if the target application is observed for kAXWindowMovedNotification, it issues a notification when any window in the application is moved, and the notification identifies the affected window. If a specific window is observed, it issues a notification only when that window is moved. In that case, the observed element and the affected element are one and the same element.

One PFObserver can be registered to observe many notifications, or an assistive application can create multiple PFObservers and register each of them to observe different notifications. For example, UI Browser creates a single PFObserver object and registers it to observe many notifications selected by the user from a table in UI Browser’s Notifications drawer. It uses a callback method to respond to notifications. This code, from the repeat block governing a single row in the table, calls a number of internal UI Browser methods that are not explained here, but the pattern should be clear:

// Lazily create observer if not yet created. if (![self notificationObserver]) {

[self setNotificationObserver:

[PFObserver observerWithPid:[[self currentElement] pid] notificationDelegate:self callbackSelector:

@selector(observer:notification:element:contextInfo:)]]; }

// Register observer for notifications and element. NSIndexSet *indexSet = [table selectedRowIndexes]; NSUInteger indexBuffer[[indexSet count]];

NSUInteger idx;

for (idx = 0; idx < [indexSet getIndexes:indexBuffer

maxCount:[indexSet count] inIndexRange:NULL]; idx++) { [[self notificationObserver] registerForNotification:

[(NSDictionary *)[combinedNotifications objectAtIndex:indexBuffer[idx]] objectForKey:@"notification"] fromElement:[self currentElement]

contextInfo:(void *)[[self currentElement] elementInfo]]; }

The payoff comes when notifications are issued in response to user-initiated changes in the target application’s graphical user interface. In the case of a delegate-based observer, the delegate method declared in the PFObserverDelegate protocol, -applicationWithIdentifier:atPath:didPostAccessibilityNotification:

fromObservedUIElement:forAffectedUIElement:, is called. In the case of a callback-based observer, the callback method declared in the notification delegate is called. It must have this signature:

- (void)observer:(PFObserver *)observer notification:(NSString *)notification element:(PFUIElement *)element contextInfo:(void *)contextInfo

The delegate method is provided with several useful items of information: the bundle identifier and path of the target application, the notification string, the observed PFUIElement and the affected PFUIElement. The callback method is provided with somewhat different information, but all needed items can be determined from the PFObserver object that is

(16)

passed in. The key item in both cases is the affected UI element. An assistive application can read its attributes, navigate the Accessibility hierarchy from it using its parent and children attributes, register to observe other changes to it, and control it using the other features of the PFAssistive Framework.

The PFObserver class also implements methods to create observers that, in OS X v10.9 Mavericks and newer, respond to notifications that include information dictionaries. See the PFAssistive Framework Reference for more information. In addition to PFObserver delegate and callback methods, an assistive application can implement optional delegate methods declared in the PFUIElement class or register to receive corresponding notifications whenever a UI element is destroyed in the user interface, or to learn of Accessibility errors. See the UIElementDelegate formal protocol declared in PFUIElement.h.

Reading a UI Element's Attributes

Some assistive applications are passive. They discover the target application’s UI elements and read the values of their attributes, but they do not enable the user to control the target application. Even active assistive applications typically must read the attributes of the target application’s elements.

There are four important kinds of operations involved in reading attributes of a UI element:

1. Reading any of dozens of simple attributes by name to obtain information about the current state of a UI element. Read the Reading Simple Attributes subsection for details.

2. Reading any of the simple attributes by using Key-Value Coding (KVC) when their names are not known until runtime. This operation is very powerful, because in many situations an assistive application can get a list of relevant attribute names and iterate over them to get their values. In addition, this operation allows an assistive application to read new attributes introduced in future versions of OS X and custom attributes used by third-party applications. Read the Reading Unknown Attributes Using Key-Value Coding subsection for details. 3. Reading several so-called parameterized attributes to obtain the value of some part of a complex UI element

given a variable parameter such as the position of the mouse on the screen. For example, an assistive application can get a character or line of text under the mouse in a text area or a cell at the intersection of a given column and row in a cell-based table. Read the Reading Parameterized Attributes subsection for details. 4. Reading the AXParent and AXChildren attributes of any UI element to navigate the accessibility hierarchy. Read

the Navigating the Accessibility Hierarchy subsection for details.

1. Reading Simple Attributes

An assistive application normally reads the values of a UI element’s Accessibility attributes using the properties declared in the Attributes category on the PFUIElement class. The Attributes category declares several dozen properties, each corresponding to an Accessibility API attribute. The value of an Attributes property is always a Cocoa object. Each Attributes property is named for the NSString used to identify the attribute in the Accessibility API, such as AXRole for the attribute named “AXRole” and AXTitle for the attribute named “AXTitle.”

Depending on the attribute, the value of an Attributes property of the receiving UI element may be another PFUIElement object or an NSString, NSNumber, NSValue, or other object. For example, the value of the AXParent property is a PFUIElement object representing the receiving PFUIElement’s parent element in the Accessibility hierarchy. Some properties return an NSArray object containing multiple PFUIElements, NSStrings, or other objects. For example, the AXChildren property returns an array of PFUIElement objects representing the receiving PFUIElement’s children

(17)

elements in the hierarchy. For NSNumber and NSValue attribute values, an assistive application must use standard Cocoa methods to extract the value, such as -boolValue, -intValue, -pointValue, and -rangeValue. For example, the AXSize property returns an NSValue object representing the height and width of the receiving

PFUIElement on the screen; use NSValue’s -sizeValue method to extract its NSSize structure. Consult the PFAssistive

Framework Reference for information about each property, including its type, the type of its contents if it is an NSArray, and the method to use to extract its value if it is an NSNumber or NSValue.

The PFUIElement class includes several utility methods that help an assistive application to read UI element attributes. The -exists method returns YES if the receiving UI element still exists in the target application’s graphical user interface; for example, if a window has not been closed by the user. It is prudent to test whether an element still exists before using it, because objects representing destroyed elements may be nil or may have been recycled to refer to other elements, leading to unexpected behavior. The -existsAttribute: method returns YES if the receiving element exists and the name of the attribute passed in the attribute argument identifies an attribute that is supported by the receiving PFUIElement. The -existsValueForAttribute: method returns YES if the receiving element exits, it supports the attribute, and the attribute returns a value. This is important because Cocoa applications often indicate that they support an attribute but do not in fact return a value for it. The -typeForAttribute: method returns an NSString identifying the type of an attribute’s value, such as “array,” “Boolean,” “range,” “rect,” or “UIElement,” suitable for use as the identifier of a table column, for determining which NSNumber or NSValue method to use to extract a structure, and in other situations where an attribute’s value must be processed differently depending on its type. This example from UI Browser reports either the plain English description or the technical name (beginning with “AX”) of each attribute listed in UI Browser’s Attributes drawer, depending on a user preference setting.

NSString *description;

if ([[NSUserDefaults standardUserDefaults]

integerForKey:TERMINOLOGY_STYLE_DEFAULTS_KEY] == 0) {

description = [element AXRoleDescription]; } else {

description = [element AXRole]; }

Many methods in the PFUIElement class return PFUIElement or PFApplicationUIElement objects. Think of these methods as factories that can be called upon to churn out these objects as often as needed, whenever needed, even if they are used only for a moment. In addition, an assistive application can create a temporary PFUIElement object at any time, for example, an object representing a known UI element in order to test it for equality with a cached element. The application can allocate as many UI elements as desired in this way, as often as desired, and initialize each of them with 

-initWithElementRef:delegate:. Typically, in this situation, it gets the CGElementRef argument from an existing or newly-created element by sending it an -elementRef message. Separate PFUIElement or PFApplicationUIElement objects representing the same UI element in the running application are interchangeable. Whether they represent the same element can be tested using -isEqual: or -isEqualToElement:.

2. Reading Unknown Attributes Using Key-Value Coding

There are many circumstances in which an assistive application does not know the names of the attributes it will read until it is running. This frequently happens, for example, when reading all of the attributes of a UI element currently under

(18)

the mouse and calling PFUIElement’s -attributes method to get an array of the element’s attribute names. It may also happen when targeting an application that implements custom UI elements with custom attributes.

The PFAssistive Framework includes two methods based on Key-Value Coding (KVC) for these situations, 

-valueForAttribute: and -valuesForAttributes:. They take advantage of the fact that the name of every simple Attributes property is the same as the NSString used in the Accessibility API to identify it. The 

-valueForAttribute: method takes as its argument an NSString containing the name of an attribute, and the   -valuesForAttributes: method takes an array of NSStrings containing the names of several attributes. The attributes passed to these two methods, for example, the attributes returned by the -attributes method, may be unknown to your assistive application or to the Accessibility API. Nevertheless, these KVC-based methods return their values. Use the -exists, -existsAttribute:, -existsValueForAttribute:, and -typeForAttribute: methods to help understand and process the values they return.

An assistive application based on the PFAssistive Framework continues to work even under new versions of OS X that introduce new attributes and with new target applications that use custom UI elements with custom attributes. This example from UI Browser is the action method for the Find in Browser button in UI Browser’s Attributes drawer: - (IBAction)elementFindButtonAction:(id)sender {

if ([self isTargetAccessible]) {

if ([[self currentElement] exists]) {

NSTableView *table = [self attributeTable];

NSString *selectedAttribute = [(NSDictionary *)[[self cachedAttributeArray]

objectAtIndex:[table selectedRow]] objectForKey:@"attribute"];

PFUIElement *selectedElement = [[self currentElement]

valueForAttribute:selectedAttribute]; [self displayElementInCurrentApplication:selectedElement]; } else { NSBeep(); } } }

3. Reading Parameterized Attributes

The Accessibility API recognizes several attributes of a special kind known as parameterized attributes. These are attributes that have a different value depending on the value of an argument passed to them. For example, parameterized attributes recognized by a text area UI element return the character located at a given point in the element’s coordinate system, the NSString constituting a line of text in the view given the line number, and the bounds of a text passage in the view given the range of characters. Similarly, a parameterized attribute recognized by a cell-based table in Apple’s Numbers application returns the cell located at a given row and column intersection, and parameterized attributes recognized by a layout area in Numbers convert between locations in screen coordinates and locations in scaled layout area coordinates.

The Attributes category of the PFUIElement class declares methods for all of these parameterized attributes, such as 

(19)

application can use them in a variety of ways. For example, a screen reader can get the location of the mouse pointer on the screen, and then use the -AXRangeForPosition: parameterized attribute method to obtain the range of the Unicode glyph at that location. The location field of the NSRange structure returned by sending the -rangeValue message to the returned NSValue object is the index of the first character in the glyph in the Cocoa text storage object under the mouse, and the length field is the distance to the character at the beginning of the next glyph, bearing in mind that Unicode glyphs may be composed of several Unicode characters. These values apply even if the text storage object spans several text containers such as multiple columns or pages. An assistive application uses the character’s range as a springboard into the other parameterized attribute properties, such as -AXLineForIndex:, -AXRangeForLine:, and -AXAttributedStringForRange:.

To help an assistive application distinguish between simple and parameterized attributes, the PFUIElement class supplements its -attributes method with two similar methods that return lists of attributes, 

-nonParameterizedAttributes and -parameterizedAttributes. In addition, the kind of a particular attribute can be determined with the -isParameterizedAttribute: method.

(20)

- (void)recordParametersForParameterizedElement:(PFUIElement *)element atLocation:(NSPoint)location {

// Get the string, range and bounds of the line of text under the mouse. NSValue *mousePosition = [NSValue valueWithPoint:location];

// Get the primary parameterized value: (1) Get the index of the first character composing the glyph under the mouse.

if ([element isRole:NSAccessibilityTextAreaRole]

|| [element isRole:NSAccessibilityTextFieldRole]) {

NSValue *characterRangeForPosition =

[element AXRangeForPosition:mousePosition];

NSNumber *characterIndex =

[NSNumber numberWithInt:[characterRangeForPosition rangeValue].location];

// Get the secondary parameterized values: (2) use the character index to get the range of characters composing the glyph under the mouse (this should be the same as characterRangeForPosition); (3) use the character index to obtain the line index of the line under the mouse; (4) use the line index to obtain the range of characters composing the line under the mouse; and (5) use the character index to obtain the range of characters composing the style run under the mouse.

NSValue *characterRangeForGlyph = [element AXRangeForIndex:characterIndex];

NSNumber *lineIndex = [element AXLineForIndex:characterIndex];

NSValue *characterRangeForLine = [element AXRangeForLine:lineIndex];

NSValue *characterRangeForStyleRun =

[element AXStyleRangeForIndex:characterIndex];

// Get the tertiary parameterized values: (6) Get the string for the line; (7) Get the RTF data for the line; (8) Get the bounds of the line. UI Browser's use of the line located under the mouse as the base for calculating tertiary values is arbitrary; any other range could have been used as the base (such as characterRangeForStyleRun or characterRangeForGlyph).

NSString *stringForLine = [element AXStringForRange:characterRangeForLine];

NSData *RTFDataForLine = [element AXRTFForRange:characterRangeForLine];

NSAttributedString *attributedStringForLine =

[element AXAttributedStringForRange:characterRangeForLine];

NSValue *boundsForLine = [element AXBoundsForRange:characterRangeForLine];

// Store all parameter values in a dictionary. // ....

(21)

4. Navigating the Accessibility Hierarchy

Many Accessibility attributes return UI elements. These attributes are yet another way in which an assistive application can discover UI elements in a target application, in addition to the three described in the Discovering UI Elements

section. For example, the AXCloseButton property returns the PFUIElement object representing the close button in the title bar of the receiving window UI element, and the AXSelectedRows property returns an array of PFUIElement objects representing the rows in a table or outline.

An assistive application navigates or browses the target application’s Accessibility hierarchy by using certain of these attributes after discovering the starting point by reading the screen, getting the application element, or receiving a notification. Specifically, the AXParent property enables an assistive application to navigate “up” the several levels of the hierarchy toward the root application element at the top, and the AXChildren property enables it to navigate “down” the hierarchy toward a leaf element visible on the screen. The root application element has no parent (its AXParent property is nil), and a leaf element has no children (its AXChildren property is an empty array).

There are several Accessibility attributes that allow an assistive application to skip levels when navigating to an important container in the target application’s Accessibility hierarchy, such as the window, drawer, or sheet containing the UI element under the mouse. The AXWindow property returns the PFUIElement representing the receiving PFUIElement’s containing window, ignoring any subcontainer levels that might lie between the window and the leaf element under the mouse. The AXTopLevelUIElement property is similar, but the PFUIElement it returns can represent something other than a window, such as a drawer or a sheet. The top-level container of an application dock item is the Dock application’s list of dock items. To ascertain what kind of UI element the container is, get its AXRole property. To determine whether it is a given kind of UI element, use PFUIElement’s -isRole: method.

The example on the next page is part of UI Browser’s screen reader window controller’s code to detect and report mismatches in a target application’s accessibility hierarchy. An important requirement of the Accessibility API is that the hierarchy when looking “up” from a leaf element on the screen should be a mirror image of the hierarchy when looking “down” from the root application element. Otherwise, a screen reader will be unable to navigate to an element that a browser can reach, or vice versa. A mismatch should always be considered a bug in the target application. It exists wherever a given UI element’s AXParent attribute omits the given UI element from its AXChildren attribute, and wherever a given UI element’s AXChildren attribute contains one or more elements whose AXParent attribute is not the given

(22)

if (![element isRole:NSAccessibilityApplicationRole]) { // root element has no parent

PFUIElement *browserParent = [[[self elementArray] objectAtIndex:(column - 1)]

objectAtIndex:[[self elementBrowser] selectedRowInColumn:(column - 1)]];

if (![element AXParent]) {

[cell setStringValue:[NSLocalizedString(@"[MISMATCH-no parent] ",

@"Warning string for no parent mismatch in target application's UI element hierarchy for browser") stringByAppendingString:

[self descriptionWithTitleAndIndexOfElement:element atColumn:column]]];

NSLog(@"Mismatch browsing the children tree from the root UI element: the %@ is

a child of %@ <%p>, but it has no parent.", [element AXRoleDescription], [browserParent AXRoleDescription],browserParent);

} else if (![[element AXParent] isEqualToElement:browserParent]) {

[cell setStringValue:[NSLocalizedString (@"[MISMATCH-different parent] ", @"Warning string for different parent mismatch in target application's UI element hierarchy for browser") stringByAppendingString:

[self descriptionWithTitleAndIndexOfElement:element atColumn:column]]];

NSLog(@"Mismatch browsing the children tree from the root UI element: the %@ is

a child of %@ <%p>, but it has a different parent, %@ <%p>.", [element AXRoleDescription], [browserParent AXRoleDescription], browserParent, [[element AXParent] AXRoleDescription],

[element AXParent]); } else {

[cell setStringValue:[self descriptionWithTitleAndIndexOfElement:element atColumn:column]];

}

[cell setLeaf:[element childrenCount] == 0]; }

The PFUIElement and PFApplicationUIElement classes declare several other methods relating to attributes. Read the

PFAssistive Framework Reference for comprehensive information about them.

Controlling an Application

There are three ways in which an active assistive application can control a target application using the Accessibility API: 1. An assistive application can determine whether a UI element of interest has an Accessibility attribute that can be

set. If it does, the assistive application sets the attribute to a new value, and the effect takes place immediately in the target application and on the screen. Read the Setting a UI Element’s Attributes subsection for details. 2. An assistive application can determine whether a UI element of interest responds to any of several Accessibility

actions. If it does, the assistive application directs the target application to perform the action, and the effect takes place immediately in the target application and on the screen. Read the Performing Actions on a UI Element subsection for details.

3. An assistive application can send a keystroke, optionally combined with modifier keys, to the target application. If the target application recognizes the key combination as a keyboard shortcut, it performs the shortcut

(23)

immediately. Otherwise, if the UI element in the target application that currently has keyboard focus can respond to the keystroke, it responds immediately, for example, by typing a character into the focused text field or text area. Read the Sending Keystrokes to an Application subsection for details.

Whenever an assistive application controls a target application using any of these techniques, the target application responds exactly as it would if a user had done the same thing using the graphical user interface. The target application’s internal data store and its user interface are updated appropriately, its documents are marked dirty as appropriate, and its undo mechanism takes note so that the user can undo whatever changes were made.

There is a fourth mechanism by which an assistive application can control a target application, namely, by manipulating or creating user input events using the PFiddlesoft PFEventTaps Framework. The PFEventTaps Framework is discussed in the next chapter, How to Use the PFEventTaps Framework.

1. Setting a UI Element's Attributes

The values of many of a UI element’s attributes can be set by an assistive application. In the parlance of Objective-C, their properties in the Attributes category of the PFUIElement class are declared readwrite, not readonly. Setting a settable attribute is a common means used by active assistive applications to control a target application.

Before attempting to set an attribute, an assistive application should test whether it is marked as settable in the Accessibility API. To do this, send it an -isSettableAttribute: message, passing in the name of the attribute of interest. Beware that some attributes are marked as settable but their values cannot in fact be set. Code defensively to guard against this possibility.

This statement is used in UI Browser as the first step in marking an attribute as settable in the Attributes drawer: [tempDictionary setObject:[NSNumber numberWithBool:

[[self currentElement] isSettableAttribute:attribute]] forKey:@"settable"];

To set the value of a settable attribute by name, use a standard setter method. Some attributes are always settable and need not be tested first, like this:

[myWindowElement setAXMinimized:[NSNumber numberWithBool:YES]];

Just as an assistive application can read an unknown attribute using Key-Value Coding (KVC), it can also set the value of an unknown attribute using KVC. As when reading unknown attributes, this works only with non-parameterized

attributes. To do this, call PFUIElement’s -setValue:forAttribute: method. The example on the next page is part of UI Browser’s code to set the target application’s attributes when the user edits the existing values in the Attributes drawer.

(24)

- (void)commitEdits {

PFBrowserController *controller = [self browserController];

NSTableView *table = [controller attributeTable];

if ([[self currentElement] isSettableAttribute:[self selectedAttribute]]) {

if ([[[self currentElement] typeForAttribute:[self selectedAttribute]]

isEqualToString:@"string"]) {

// Set attribute table to new string.

[[self currentElement] setValue:[[self bottomTextView] string]

forAttribute:[self selectedAttribute]];

[(NSMutableDictionary *)[[controller cachedAttributeArray]

objectAtIndex:[table selectedRow]] setObject:

[[self bottomTextView] string] forKey:@"value"];

[(NSMutableDictionary *)[[controller cachedAttributeArray]

objectAtIndex:[table selectedRow]] setObject:

[controller descriptionOfAttributeValue:[[self bottomTextView] string] ofType:@"string" element:[self currentElement]]

forKey:@"valueDescription"];

[table reloadData];

// Set string textfield to new string.

NSString *selectedAttributeValue =

[[self currentElement] valueForAttribute:[self selectedAttribute]];

[[controller settingStringTextField] setStringValue:selectedAttributeValue]; [[controller settingStringTextField] selectText:table];

} } }

The -setValue:forAttribute: method returns YES if the attribute’s value was successfully set, or NO if not or if the given attribute is a parameterized attribute. However, some applications mark attributes as settable when they are not, and this method may return YES even though the value was not changed. Code defensively to guard against this possibility.

There are some cases where you might think that a UI element’s value attribute should be settable, but it isn’t. In those cases, the UI element can be modified by performing an Accessibility action on it, instead. A common example that often trips up users of GUI Scripting is a checkbox. Apple advises that a checkbox should be modified only by sending it an “AXPress” action simulating a click in the graphical user interface, not by setting its AXValue attribute. An assistive application can nevertheless read its AXValue attribute. In many cases, however, the target application can be

controlled either by setting an attribute or by sending an action. For example, a window can be minimized to the Dock by setting the window element’s AXMinimized property, as shown previously, or by sending an “AXPress” action to its minimize button element, as shown in the next subsection.

(25)

2. Performing Actions on a UI Element

Many UI elements in a target application can be controlled by instructing them to perform one of a handful of Accessibility actions. The Accessibility API recognizes these actions in OS X v10.9 Mavericks and newer, each of which is a string:

• “AXCancel” - a cancel action, such as pressing the Cancel button. • “AXConfirm” - a confirm action, such as pressing Return in a text field.

• “AXDecrement” - a decrement action, such as pressing a stepper's down arrow. • “AXDelete” - a delete action.

• “AXIncrement” - an increment action, such as pressing a stepper's up arrow. • “AXPress” - a press action, such as clicking a button or a menu.

• “AXRaise” - a raise action, such as bringing a window to the front within the application.

• “AXShowMenu” - a show menu user action, such as opening a pop-up button's menu or a contextual menu in a text view or text field.

• “AXShowDefaultUI” - a show default UI action, such as changing a transient UI element to its default appearance when the mouse leaves the element (introduced in OS X v10.9 Mavericks).

• “AXShowAlternateUI” - a show alternate UI action, such as changing a transient UI element to its alternate appearance when the mouse hovers over it (introduced in OS X v10.9 Mavericks).

Applications may respond to custom actions, as well, such as the “AXOpen” action in the Finder.

To get a list of the actions that a particular UI element recognizes, send it the -actions message. It returns an array of NSStrings. To determine whether an element responds to a given action, send it the -existsAction: method, passing in the name of the action. To cause the target application to perform the action on the element, send the element the -performAction: method, passing in the name of the action.

This example is the UI Browser action method that is sent to the target application when the user double-clicks any UI element in UI Browser’s main browser view. The action method brings the target application and the window containing the UI element to the front and highlights the UI element with a yellow overlay. It starts by activating the target application using a PFUIElement utility method, -activateApplication. Then, if the UI element that was double-clicked is a window element, it performs the “AXRaise” action on it to make it the target application’s frontmost window. If it is any other element, it gets the element’s AXWindow property and then performs the “AXRaise” action on that.

(26)

- (IBAction)highlightDoubleClickAction:(id)sender {

if ([self isTargetAccessible]) {

[[self currentElement] activateApplication];

if ([[self currentElement] isRole:NSAccessibilityWindowRole]) {

[[self currentElement] performAction:NSAccessibilityRaiseAction];

} else {

[[[self currentElement] AXWindow]

performAction:NSAccessibilityRaiseAction];

}

[[self highlightButton] setState:NSOnState]; [self highlightAction:sender];

} }

3. Sending Keystrokes to an Application

The final way to control a target application using the PFAssistive Framework is to send the application a keystroke, optionally combined with modifier keys. To send a keystroke to a specified application, use PFApplicationUIElement’s4  

-typeCharacters:keyCode:withModifierFlags: method. It posts keyboard events to the application

represented by the receiving PFApplicationUIElement even if it is not currently the active application. To send a keystroke to the active application without first determining which application is active, send the

+typeCharactersSystemWide:keyCode:withModifierFlags: class method. It creates a temporary system-wide element and posts keyboard events to the frontmost application.

Both methods are sent directly to the application, not to individual UI elements. They differ from the other techniques for controlling a target application in this respect. Usually, the target application types the character into whatever UI element currently has keyboard focus, just as typing on the keyboard would do. Therefore, an assistive application should set focus to a specific text field or text area first, using the AXFocused property, by sending a -setAXFocused: message to the element.

Both methods send a single keystroke with optional modifier keys. Apple’s corresponding GUI Scripting commands for AppleScript—the ‘keystroke’ command and the ‘keycode’ command—are different in this regard; they send multi-character strings by calling the Accessibility API function underlying these PFAssistive Framework methods multiple times. An assistive application can accomplish the same result that GUI Scripting does by similarly sending either of these methods multiple times in succession. However, it is often easier and more efficient to set the AXValue attribute of a UI element if it is a text field or a text area. An assistive application can set the value of a text field or a text area by passing an NSString of arbitrary length in a -setAXValue: or -setValue:forAttribute: method.

If the -typeCharacters:keyCode:withModifierFlags: or +typeCharactersSystemWide:keyCode: withModifierFlags: method specifies that the Command key is down, and if the target application recognizes the key combination as a keyboard shortcut, it executes the shortcut. It is important to pass the command character as a

These methods were deprecated in OS X v10.9 Mavericks because Apple deprecated the underlying Accessibility API

4

functions. Developers are advised to use Quartz Event Taps through the CGEventCreateKeyboardEvent() function in CGEvent.h or its Cocoa counterpart, +[NSEvent keyEventWithType:location:modifierFlags: time-stamp:windowNumber:context:characters:charactersIgnoringModifiers: isARepeat:keyCode:.

(27)

lowercase letter, because an uppercase character is interpreted as a keyboard shortcut with the Shift key down. In some applications this performs a different keyboard shortcut, but in most applications it either does nothing or it types the uppercase character into the active UI element.

The characters and flags arguments are equivalent to those used in NSEvent's 

-charactersIgnoringModifiers and -modifierFlags methods. The virtualKey argument is the virtual key code provided by NSEvent's -keyCode method. It is a hardware-independent integer value provided by system resources for every known keyboard, mapped from the hardware-dependent raw key code using the current keyboard layout resource.

On Roman systems, the characters argument is optional and should be passed as nil or an empty string unless you are knowledgeable regarding the difficult and arcane subject of keyboard layouts. It is not optional on some other systems where it is used as a hint to supplement the virtual key code during key translation.

Subclassing

A client may subclass PFApplicationUIElement. The designated initializer, -initWithPid:delegate:, automatically registers PFApplicationUIElement's base class or subclass so that PFUIElement methods which allocate and initialize a new root application UI Element will know what class of object to instantiate based on the target application's PID. The designated initializer also registers the default PFUIElement base class for the target application. A client that subclasses

PFApplicationUIElement may optionally register a subclass of PFUIElement by calling a registration class method immediately after creating and initializing the root application UI element. Both the root application and the UI Element class can be retrieved by calling class methods.

1. The Class Registry

PFApplicationUIElement declares in its implementation file a private static variable, classRegistry, of type NSMutableDictionary. It acts as a class variable, created only once and available directly only within the PFApplicationUIElement base class object itself. All other classes and applications can access it through the +

[PFApplicationUIElement applicationUIElementClassForPid:] and +[PFApplicationUIElement UIElementClassForPid:] class methods.

PFApplicationUIElement's designated initializer, -initWithPid:delegate:, creates an empty registry dictionary the first time any PFApplicationUIElement or subclass object is created and initialized. Once the registry is confirmed to exist, the designated initializer adds its own Class (PFApplicationUIElement or a subclass) along with the default PFUIElement class to a subdictionary within the classRegistry dictionary keyed to the target application's PID in the form of an NSNumber object. No matter how many different subclasses of PFApplicationUIElement a client creates, and no matter how many instances of each it creates, the designated initializer populates the registry with a unique Class/PID pair for each application. Because each running application has a unique PID, the PID can be used to look up its

PFApplicationUIElement class or subclass whenever a PFUIElement object is asked to create and return its application element. To facilitate this, PFApplicationUIElement implements a class method, +applicationUIElementClassForPid:, which takes a pid_t (int) parameter and returns the associated Class object. The -[PFUIElement applicationElement] method and other methods use this Class value to create and return a PFApplicationUIElement or subclass object of the correct class.

In addition, the +[PFApplicationUIElement registerUIElementClass:forPid:] class method can be called, immediately after creating and initializing a PFApplicationUIElement subclass, to register a subclass of PFUIElement for the same PID, replacing the default PFUIElement class registered in the designated initializer. It can be accessed through the + [PFApplicationUIElement UIElementClassForPid:] class method. Subclassing PFUIElement is optional.

(28)

The target application's PID as an NSNumber object is always available from the -[PFApplicationUIElement pidNumber] method, and the PID is always available as a pid_t (int) value from the -[PFUIElement pid] method.

2. Subclass Requirements

Clients that do not subclass PFApplicationUIElement or PFUIElement need not pay any attention to the registry mechanism.

Clients that do subclass PFApplicationUIElement or both of them need to observe a few simple requirements: (1) They must ensure that their PFApplicationUIElement subclass initializers ultimately call the PFApplicationUIElement designated initializer, -initWithPid:delegate:, which updates the registry. Subclasses must always call through to the base class designated initializer anyway, so this should never be an issue.

(2) If they subclass PFUIElement as well, they must call the +[PFApplicationUIElement registerUIElementClass:forPid:] class method immediately after creating and initializing an instance of the PFApplicationUIElement subclass, to register the subclass in place of the default PFUIElement class.

(3) If they implement any PFUIElement methods that return the UI element's root application UI element, those methods must call the +[PFApplicationUIElement applicationUIElementClassForPid:] class method to create, initialize and return a PFApplicationUIElement class or subclass instance of the correct class for the target application.

(4) If a screen reader or similar application uses a subclass of PFApplicationUIElement, it must create and initialize a root application UI element early in the execution path, in order to register the subclass before calling a method like -applicationElement on the UI element object. If it subclasses PFUIElement, it must also call @link

registerUIElementClass:forPid: +registerUIElementClass:forPid: @/link at the same time. A screen reader application might not normally create and initialize a root application UI element directly but instead use a method like +[PFUIElement elementAtPoint] to get a specific UI element. If it then navigates the UI element hierarchy using a method like -AXParent, which might encounter the root application UI element, it must first have registered the PFApplicationUIElement subclass by creating and initializing a dummy subclass instance; and, of course, it must first have registered any PFUIElement subclass.

In general, any application that subclasses PFApplicationUIElement or both UI element classes should create and initialize a root application UI element at the outset, and also register any PFUIElement subclass immedidately afterward.

Miscellaneous

There are a few methods in the PFAssistive Framework that have not been covered in this Programming Guide. They are covered in detail in the PFAssistive Framework Reference.

Assistive Application Programming Guide For OS X 2.0

Assistive Application

Programming Guide

For OS X

2.0

For the PFAssistive and PFEventTaps Frameworks

Introduction to the

Assistive Application

Programming Guide

1

The PFiddlesoft™ PFAssistive and PFEventTaps Frameworks

1

Related Documentation

2

Licensing Terms

3

Free for Personal Use and for Distribution and Use With Free Products

3

One-Time Licensing Fee for Distribution and Use With Paid Products

3

DISCLAIMERS

3

How to Use the

PFAssistive Framework

5

Discovering UI Elements

6

1. Reading the Screen

6

2. Browsing the Accessibility Hierarchy

8

3. Observing Notifications

9

Reading a UI Element's Attributes

11

1. Reading Simple Attributes

11

2. Reading Unknown Attributes Using Key-Value Coding

12

3. Reading Parameterized Attributes

13

4. Navigating the Accessibility Hierarchy

16

Controlling an Application

17

1. Setting a UI Element's Attributes

18

2. Performing Actions on a UI Element

20

3. Sending Keystrokes to an Application

21

Subclassing

22

1. The Class Registry

22

2. Subclass Requirements

23

Miscellaneous

23

How to Use the

PFEventTaps Framework

24

Monitoring Events Using an Event Tap

25

Filtering, Modifying, Blocking, and Responding to Events

29

Posting Synthetic Events

31

Reading the State of an Event Source

33

How to Grant Access

How to Build an Application Using the Frameworks

39

Historical Background (Mountain Lion and Older)

39

Modern Requirements (Mavericks and Newer)

40

Building an Application When the Frameworks are Installed as Shared Frameworks

41

Building an Application to Run As a Trusted Accessibility Process Before Mavericks