-
Notifications
You must be signed in to change notification settings - Fork 0
Internals
The Sprites add-on for Google Chrome developed by researchers from the Make4All group at the University of Washington is a research product based on exploration of interaction techniques to make table navigation more efficient. Simply put, the Sprites mode is a table exploration mode, when enabled, maps the rows of a table to the top rows of a user's keyboard (keys 1-equals) and maps the column corresponding to a row to the keys on the left-hand side of a keyboard (`, tab, capslock, shift and control). This page contains details of the implementation of the add-on and should help you get started working on the codebase.
Activating Sprites mode: NVDA+SHIFT+T
Note that once Sprites mode is activated, all existing gestures from NVDA will be disabled until the user quits SPRITEs mode with Escape, and only the keys defined below can be activated:
- Table exploration:
- For switching columns: keys 1 to 0 and -, =
- For switching rows: keys `, tab, capslock, left shift, and left control
- For announcing row number: r
- For announcing column number: c
- For announcing both: b
- search
- Activating search mode: F
- Jumping between search results (if any): up arrow and down arrow
- Exiting search (when in search mode): Escape
- misc
- Exiting Sprites mode (when not in search mode): Escape
- Interrupt speech: right shift, right control
The add-on is structured into five separate files plus one test file in the chrome folder, and a script specifying the setup upon installation. The purpose of the code in each of these files is outlined below.
This file contains the code to perform setup upon installation of the add-on. This setup includes creating a data log file in %appdata%/nvda/sprites directory, and stores information relevant to logging in config.conf, which is NVDA's internal dictionary to store states.
This file contains the AppModule class and is the main file interfacing with NVDA. It also defines the Sprites gestures and all the announcements. For each table, it creates a new SpritesTable object, stored in spritesTable.py.
This file contains the definition of the SpritesTable class. When Sprites mode is activated, a new SpritesTable is created to store the current state of the table, including the number of rows and columns, the rows and columns currently mapped to the keyboard, the location where the user is exploring. It also stores the search configurations including the current search term, a list of locations for any search results, the index of the result the user is currently on, and whether row or column filters are applied.
This file contains definitions for the Sprites search and search result dialogs. It uses the wx library and some gui helper functions defined in the NVDA source code to create the pop up windows. It also defines the Sprites settings panel, which shows up in the NVDA preferences menu. The settings panel contains an option to open the directory containing the data log as well as a link to the GitHub issue template for submitting feedback and bug reports.
This file contains some helper functions for logging usage information as defined in the data logging spec. It also has the logic that checks if the log has persisted for more than 30 days, if so it will clear the log file.
This file contains a variable storing the message shown on first-time launch in HTML format. This is imported from __init__.py and is displayed to the user when they launch Chrome for the first time after installing the add-on.
This file contains some simple unit tests that verifies the sprites table object is behaving correctly. Please note that it is not technically possible to automatically test the entire functionality of Sprites at this point so please perform manual testing before creating pull requests.
The add-on is a Chrome AppModule, meaning it will only run if the user focus is on the Chrome browser. When the user starts up Chrome, the gesture (NVDA+SHIFT+T) to activate Sprites mode is added. A set of gestures only available to Sprites is also defined, but not yet bound to NVDA. We also inject a hook into the handleCaretMove function in the review module so that we can detect when the user cursor reaches a table and remind them that Sprites mode is available.
When the user reaches a table and uses the gesture to activate Sprites mode, here is what happens.
- We first verify that the cursor is indeed on a table (more details in specific workarounds section).
- We initialize a
SpritesTableobject with the object with the tableID (assigned by NVDA), its row and column dimension, as well as the number of row keys and column keys available to map to rows and columns in the table. The tableID is one necessary for retrieving specific cells from NVDA later on, and the row and column dimensions are crucial for the navigation to work properly. - We bind all the gestures we defined upon start-up to NVDA, and disable other gestures by replacing the
captureFuncininput.manager(more details in specific workarounds section). - We announce the initial mapping of the keys.
Upon activating Sprites mode on a table, the user always starts at (1, 1) (row and column indices use 1-based indexing), meaning the graav and 1 key, when pressed, will always announce the content of the cell at (1, 1). Pressing on any of the keys that switch rows will modify the row index. For example, if the user presses Tab, the cell announced next will be at (2, 1), and if they then press 3, the cell announced next will be at (2, 3). If there are more keys than rows or columns available, double pressing the first and last keys (graav and left control for row and 1 and = for column) can scroll to the previous or next set of rows and columns. For example, if the user double presses left control after activating Sprites, the graav and 1 key will now map to (6,1) instead.
The current cursor position (currRow and currColumn) and scroll state of the table (rowOffset and columnOffset) is maintained in the SpritesTable object.
The getCellAt function in SpritesTable handles the retrieval of a TextInfo object containing the content of the cell at the given row and column. It uses the _getTableCellAt() function of the treeInterceptor of the current document, which is built into NVDA. The returned TextInfo is spoken in the speakCell function in __init__.py with the speakTextInfo() function from the speech module, and the configurations for the proper formatting of the announcement is inspired by the EasyTableNavigator add-on.
Sprites enables a keyword search within a table, with extra configurations like case-sensitivity and filters for rows and columns. When the user presses F, a SearchDialog (defined in dialogs.py) pops up, allowing the user to enter a keyword and set specific configurations. After the user presses “Search”, a background thread (see onSearch in __init__.py) is initialized. This thread scans through the whole table for the keyword and stores the results in a list. When the search is complete (see onSearchComplete in __init__.py), another window will pop up notifying the user how many instances of the keyword have been found and also provides an option to jump to the first search result.
It is expected that when there is a huge table with a lot of data, the search can take more time. The user could continue to explore the table with Sprites gestures When the background thread is searching, but they won’t be able to initiate a different search until the current search has been completed.
Once we have populated the search results, the user can jump between them using the up/down arrow keys. Jumping this way remaps the `, 1 keys to the specific search result and aligns the upper left corner of the mapped table region to that search result. If a row and/or column filter is applied, or the user decides to only explore the rows and/or columns containing the search result, jumping to the first search result is mandatory.
As the user navigates through a table with search mode on, NVDA also announces the number of occurrences found in the same row or column. If the user is currently on a search result, the calculation will exclude the current instance. For example, if the two search results are at (1, 1) and (1, 2) and the user is currently on (1, 1), NVDA will announce “1 more occurrences found in this row”. If the user is currently on (1, 3), NVDA will announce “2 more occurrences found in this row”.
Whether the Sprites search mode is on or not depends on whether the search result is an empty list. Upon exiting the search, we will clear the current search keyword and the list of the search results in the SpritesTable.
Upon exiting, we will clear all Sprites navigation gestures, add back the Sprites activation gesture and restore the captureFunc used by input.manager. The cursor position will be the last cell explored by the user. This way the user can continue exploring using NVDA as usual.
Since Sprites defines a set of controls different from the traditional table navigation scheme for screen readers, some workarounds were implemented to achieve the desired behavior. We detail these workarounds below.
When using NVDA in browse mode, the cursor follows the navigator instead of the focus. In NVDA, the only event that is associated with certain cursor movements is event_onFocusChange, but that doesn’t always capture all cursor movement events, especially when navigating through a plain text document. To detect if the current navigator object is on a table, we inject a hook into the review module of NVDA (see injectHooks and post_handleCaretMove in __init__.py for implementation). Essentially, we replaced the original handleCaretMove in the review module with our own version, which first calls the original version, then checks whether the navigator object is currently on the table. Since the handleCaretMove function is called every time a cursor movement occurs, it achieves our purpose of detecting when the cursor has reached a table, so we can remind the users to turn on Sprites mode.
Note that it is necessary to do the detection after we run the original handleCaretMove instead of before, otherwise the api.getNavigatorObject() call to check for tables will trigger a series of exceptions when Chrome is first started, causing NVDA to freeze.
We also have a hasAnnouncedTable flag that would be set to True if the cursor is on a table and the “Sprites mode available” message has been announced, which is necessary because we don’t want the announcement to keep repeating if the user doesn't want to use Sprites mode.
Checking if the navigator is on a table is a bit more complicated than simply checking the role of the current navigator object, since it is usually true that the navigator object is pointing at a child of the table object instead of the table itself. The check utilizes the focus.treeInterceptor._getTableCellCoords() function, which when passed in the current selection will raise an AttributeError or LookupError if the cursor is not on a table (see getTableInfo in __init__.py for implementation, and this approach is inspired by the EasyTableNavigator add-on). If no errors are raised, it means the cursor is on a table. Then we can access the TextInfo object associated with the current selection with focus.treeInterceptor.selection and obtain relevant attributes like the tableID and the number of rows and columns at the table from that object, which is inspired by the _getTableCellCoords implementation from the NVDA source code.
While the official NVDA add-on development guide refers to one way of binding gestures by using the script decorator, it doesn’t fit the use case for Sprites as the Sprites gestures should only be accessible when Sprites mode is activated. To support this functionality, we use the bindGesture function that takes in the gesture string and a function name with the script_ prefix which can bind gestures dynamically when the add-on is running (see bindSpritesGesture in __init__.py). To remove the gestures, we use clearGestureBindings and then bind back the gesture to activate Sprites mode (see clearTableGesture in __init__.py).
NVDA has many built-in gestures or shortcut keys for browse mode and it looks up gesture bindings in a specific order (defined in the developer guide), with a couple other gesture maps taking precedence over the ones defined in appModules. Considering the significant number of new gestures introduced by Sprites, it is almost inevitable that there will be conflicts, and users could accidentally trigger existing gestures and leave the table when Sprites mode is on. To mitigate the issue, we replaced the _captureFunc of the manager instance in the inputCore module so we can intercept the gestures that are not defined as part of the Sprites gestures. We also define a set of special keys, which includes the set of modifier keys that by default cannot perform actions when pressed alone. When those keys are pressed, they are intercepted just like other undefined gestures, but they also trigger a callLater in our version of the captureFunc that calls the functions for changing rows. We are able to attach actions to modifier key presses (see captureFunc in __init__.py for implementation) with this workaround.
Though NVDA has a built-in getLastScriptRepeatCount() function for detecting double-press gesture, it doesn’t work for the actions bound to modifier keys because the function objects initiated by callLater don’t share the same address. So we keep track of the time between presses instead, and only register double presses if the interval between presses is between 0.1 seconds and 0.5 seconds (both bounds exclusive). The bounds are chosen after doing some timing experiments, but it might be beneficial in the future to add a setting on double press sensitivity so that users can make adjustments.