Accessability with Semantic Marker® - Advanced Optical Visual Markers

Click on Titles below to learn about unique Accessability Features.

The 3x3 matrix is the main collection of SMART Buttons that are user accessable (clickable). There are a couple modes for what a "click" means, but for voice control to work, the user says the 1 through 9. For consistency with accessability, especially with sight challenged users, is that the matrix values are ALWAYS 1 through 9. That isn't always the same with other displays, as the numbers are dynamically assigned by Apple.

One way to invoke these values is for another app to "speak" those numbers, and this app "listens". One client using Eye Tracking can work this way if this app isn't on their eye tracking supported display.

With the spoken feature of this app, the matrix itself will speak out the title of each SMART button. So for example, it might say Item 1 (Row 1, Col 1) shows the Eiffle Tower and the Music from 1972 of the Grateful Dead in Paris. Then, using the voice control, you could say "1" and that music will start playing with this apps SMART Music feature. (The user can also get a map of all the SMART Matrix items).

Each of these SMART Buttons have a full meta-data backing, including what messages to send, and most import is the global address to get to that information. SemanticMarker.org is an ".org" since the markers only work if anyone can access them. But within that framework is the ability to require security on which button an be invoked.

Accessability through eye tracking is amazing, and really provides much needed accessibility for some of our clients. Eye Tracking is a unique new tool that the iPhone/iPad provides. While powerful (I've tried it) there are other products that have extended eye tracking (eg. think about swiping vs touching). One of our clients uses Tobii eye tracker which has been taylored through 20 years of use.

Using this app, the SMART Matrix is the easiest approach to "clicking" or "touching" via eye tracking (similar to the Voice Control). But other parts of the app can use Eye Tracking to configure the IoT devices, talk Bluetooth to the devices, and lead or follow remote education using the Semantic Marker® features.

The Semantic Marker® framework supports a full IoT messaging capability. These messages can be packaged up and send via other tools. For example, the "feed dog" message is available in our app via one of those Matrix SMART Buttons. But that same message can be sent from other tools as they are "web" calls. So the eye-tracker can use their own matrix - but then send the appropriate message to feed their dog.

In addition, the Semantic Marker® app can also accept messages and as if the user touched the button. Thus a message to cell "1" will click that button. Even more advanced is the ability to run any random SMART Button using the addressability language we designed. That same information can be retreived and mapped to an physical IoT button (see those below).

The following image describes Kevin O'Brien using his Eye Tracker to feed his dog Abby. This was almost 2 years ago and was in his words "Ground Breaking".

Our Braille concept incorporates a 3d printed holder (the blue one in this image). This helps the sight imparired user find the correct mini-matrix of SMART Buttons. Once there, they can use the Semantic Marker® app's camera to scan the QR codes. Some of those will speak because of the Semantic Marker® (a feature we call TRON). This might say "these turn on the xmas tree lights" or "these feed the dog".

So the sight imparied users can relate the Blue colored plastic to the "dog feed" feature. But even cooler, is that they can say to sighted guests, that they can grab the Blue colored plastic to find the QR codes that will feed the dog. The "Blue" would be the Braille to help the user know which one is which. Talk about Agency!!

Augmented Reality is a powerful feature of the Semantic Marker® app. In one mode, you get feedback with graphics to know the marker is the one you want (eg. a graphic with "FEED DOG" shows up versus "PLAY MUSIC".

A really powerful feature of our Augmented Reality is that the SMART Buttons are interactive. This means some can be showing the results of the environment (such as the amount of food left for the dog, or the state of a sensor). This bi-directionaly interaction is only available because the MQTT messaging of the full Semantic Marker® framework. You can see more at Github

This image shows a user scanning a Semantic Marker® located on that round device - which is our 3d printed Internet of Things (IoT) dog feeder. The user feels via "haptics" that they are scanning that QR code, but they are also seeing the results through the dog image in the iPhone display. This is Scooby Doo and defined what "Augmented Reality" means. You see an image representing the object you are scanning. Oh, the app is also sending an Internet of Things message via bluetooth (or MQTT) to that device as well - and it will feed the dog. Again "Agency!!"

A lot of the accessability features are described in the white paper linked here. Written in 2024 for the #Ability conference in LA, this shows a lot of the features especially tailored with accessability in mind

2024 Paper on Accessability with the full Semantic Marker® suite of capabilities

The Semantic Marker® SMART Architecture Framework is extensive. It provides a unique messaging approach to Internet of Things (IoT) processing. This framework is described starting at Github Konacurrents. But the entire addressability features are a necessary foundation for all the ability concepts describe on this web page.

Users with an account can access the SMART User Control (needs user account)

Full IoT (Internet of Things) capabilities, including easy to touch physical SMART Buttons (that are programmable), motion sensors, and others. This image shows some of those buttons in various configurations, and accessable by disabled users.


All these features and more are available through this app.