Kona Currents, LLC - Architecting in the Internet of Things (IoT) Era

The SMART Button is the key to accessability.
The trick is how to "press" that button, and what does the button do. The sections below are all about defining, pushing, and running those buttons.

- Also: if you don't like this button, we have many more to choose from.

The 3x3 matrix is the main collection of SMART Buttons that are user accessable (clickable). There are a couple modes for what a "click" means, but for voice control to work, the user says the 1 through 9. For consistency with accessability, especially with sight challenged users, is that the matrix values are ALWAYS 1 through 9. That isn't always the same with other displays, as the numbers are dynamically assigned by Apple.

One way to invoke these values is for another app to "speak" those numbers, and this app "listens". One client using Eye Tracking can work this way if this app isn't on their eye tracking supported display.

With the spoken feature of this app, the matrix itself will speak out the title of each SMART button. So for example, it might say Item 1 (Row 1, Col 1) shows the Eiffle Tower and the Music from 1972 of the Grateful Dead in Paris. Then, using the voice control, you could say "1" and that music will start playing with this apps SMART Music feature. (The user can also get a map of all the SMART Matrix items).

Each of these SMART Buttons have a full meta-data backing, including what messages to send, and most import is the global address to get to that information. SemanticMarker.org is an ".org" since the markers only work if anyone can access them. But within that framework is the ability to require security on which button an be invoked.

Accessability through eye tracking is amazing, and really provides much needed accessibility for some of our clients. Eye Tracking is a unique new tool that the iPhone/iPad provides. While powerful (I've tried it) there are other products that have extended eye tracking (eg. think about swiping vs touching). One of our clients uses Tobii eye tracker which has been taylored through 20 years of use.

Using this app, the SMART Matrix is the easiest approach to "clicking" or "touching" via eye tracking (similar to the Voice Control). But other parts of the app can use Eye Tracking to configure the IoT devices, talk Bluetooth to the devices, and lead or follow remote education using the Semantic Marker® features.

The Semantic Marker® framework supports a full IoT messaging capability. These messages can be packaged up and send via other tools. For example, the "feed dog" message is available in our app via one of those Matrix SMART Buttons. But that same message can be sent from other tools as they are "web" calls. So the eye-tracker can use their own matrix - but then send the appropriate message to feed their dog.

In addition, the Semantic Marker® app can also accept messages and as if the user touched the button. Thus a message to cell "1" will click that button. Even more advanced is the ability to run any random SMART Button using the addressability language we designed. That same information can be retreived and mapped to an physical IoT button (see those below).

The following image describes Kevin O'Brien using his Eye Tracker to feed his dog Abby. This was almost 2 years ago and was in his words "Ground Breaking".

NFC tags are another approach to run any of these SMART Buttons. Scanning with an phone's radio can access. This is different than the QR code, the optical marker, which uses a phots GPU based optial scanner (eg. the camera). Adk for a demo.

QR codes are a form of bar code that can define Adk for a demo.

To define and run a SMART button, an entire framework was designed and built. This the Semantic Marker(R) architecture. A unique addressabiilty approach defines buttons, and provides for a runtime framework to run those buttons. Everthing centers on finding ways to find, touch and run those buttons. Much like a programming language, the user creates these SMART Buttons, and the Semantic Marker(R) Agent Framework supports running the IoT framework as well as the Photo Bookmark to almost anything hypertext ready in the world wide web (the internet). This includes any sharable (linkable) artifact, photos, social media posts, workouts, etc. This implements the Grandmas Attic vision.

Our Braille concept incorporates a 3d printed holder (the blue one in this image). This helps the sight imparired user find the correct mini-matrix of SMART Buttons. Once there, they can use the Semantic Marker® app's camera to scan the QR codes. Some of those will speak because of the Semantic Marker® (a feature we call TRON). This might say "these turn on the xmas tree lights" or "these feed the dog".

So the sight imparied users can relate the Blue colored plastic to the "dog feed" feature. But even cooler, is that they can say to sighted guests, that they can grab the Blue colored plastic to find the QR codes that will feed the dog. The "Blue" would be the Braille to help the user know which one is which. Talk about Agency!!

Various spoken features are supported natively with the app. Some was mentioned above under Voice Control. But the Semantic Marker® Tron feature also speaks natively as someone scans or invokes the special QR code (with the Semantic Marker® SMART app).
The type of language and language speaker can also be changed through the app.
Try Tron out (click on blue button Speak Tron)

For user that can hold the iPhone, when scanning a QR code, there is definite feedback to the user through haptics, vibrating the device. This helps the user know they have found matches for what they are searching for. For sight imparired users, this lets them know they are over target.

For sighted users, the Semantic Marker® app provides an Augmented Reality graphic overlay when a Semantic Marker® (a QR Code) is found. Scanning can also be sending messages to the light switch to turn on - so the feedback can be (1) haptics, (2) results (light turn on/off), (3) Augmented Reality Overlay, and (4) spoken results (like "feeding the dog").

Augmented Reality is a powerful feature of the Semantic Marker® app. In one mode, you get feedback with graphics to know the marker is the one you want (eg. a graphic with "FEED DOG" shows up versus "PLAY MUSIC".

A really powerful feature of our Augmented Reality is that the SMART Buttons are interactive. This means some can be showing the results of the environment (such as the amount of food left for the dog, or the state of a sensor). This bi-directionaly interaction is only available because the MQTT messaging of the full Semantic Marker® framework. You can see more at Github

This image shows a user scanning a Semantic Marker® located on that round device - which is our 3d printed Internet of Things (IoT) dog feeder. The user feels via "haptics" that they are scanning that QR code, but they are also seeing the results through the dog image in the iPhone display. This is Scooby Doo and defined what "Augmented Reality" means. You see an image representing the object you are scanning. Oh, the app is also sending an Internet of Things message via bluetooth (or MQTT) to that device as well - and it will feed the dog. Again "Agency!!"

Another feature of the Semantic Marker® is the ability to denote geospatial values, GPS values. The app will run a SMART Button if the location is within some tolerance of the specified GPS value.

A lot of the accessability features are described in the white paper linked here. Written in 2024 for the #Ability conference in LA, this shows a lot of the features especially tailored with accessability in mind

2024 Paper on Accessability with the full Semantic Marker® suite of capabilities

This section will be updated soon, but the remote education is a unique feature of Semantic Marker®. In particular, you can send a Semantic Marker message in what we call the DOCFOLLOW remote education control feature. Instead of sending images, documents or a zoom replicate of your screen, a specific message is sent that denotes the specific optical marker. The users (or followers) then do things with that information. One valuable use is to change their display to the page described by that marker's address.

The Semantic Marker® SMART Architecture Framework is extensive. It provides a unique messaging approach to Internet of Things (IoT) processing. This framework is described starting at Github Konacurrents. But the entire addressability features are a necessary foundation for all the ability concepts describe on this web page.

Users with an account can access the SMART User Control (needs user account)

Full IoT (Internet of Things) capabilities, including easy to touch physical SMART Buttons (that are programmable), motion sensors, and others. This image shows some of those buttons in various configurations, and accessable by disabled users.

All these features and more are available through this app.

Accessability with Semantic Marker® - Advanced Optical Visual Markers