SenseAR Specification

Home / SenseAR / SenseAR Specification

SenseAR

Navigation. Awareness. Freedom.

Overview

SenseAR is an open-source, AI-powered smart glasses and environmental intelligence platform designed to help people understand, interpret, and navigate the world around them. Inspired by the advanced perception technologies used in autonomous driving vehicles, SenseAR combines artificial intelligence, computer vision, spatial awareness, depth perception, and real-time sensor fusion to create a continuous understanding of the user’s surroundings.

Designed particularly for people who are blind or visually impaired, SenseAR aims to function as an intelligent visual companion, delivering real-time play-by-play descriptions of the surrounding environment through natural voice guidance. Rather than simply identifying objects, SenseAR interprets what is happening, understands spatial relationships, monitors movement, anticipates potential hazards, and communicates relevant information to help users make informed decisions.

The platform provides intelligent navigation, obstacle detection, environmental interpretation, object recognition, contextual awareness, and conversational assistance. Its modular architecture allows capabilities to operate independently while sharing environmental information through a unified intelligence system. SenseAR is designed to support different wearable hardware configurations, local AI processing, privacy controls, and community-developed extensions.

Core Objectives

Provide real-time environmental awareness through intelligent visual perception.
Empower blind and visually impaired individuals with greater independence and mobility.
Deliver continuous, natural-language descriptions of the surrounding world.
Apply autonomous-driving-inspired perception and prediction capabilities to pedestrian accessibility.
Identify objects, people, vehicles, obstacles, landmarks, and environmental features.
Provide intelligent indoor and outdoor navigation.
Detect potential hazards and communicate timely warnings.
Understand spatial relationships, distances, directions, and movement.
Support hands-free voice interaction and conversational environmental assistance.
Prioritize privacy, local processing, accessibility, and user control.
Provide a modular architecture that supports interchangeable hardware and independently developed capabilities.
Establish an extensible open-source platform for assistive technology innovation.

Core Modules

Environmental Perception and Scene Understanding Module

Provides continuous environmental observation and contextual interpretation of the user’s surroundings.

Real-time recognition of objects, people, animals, vehicles, buildings, and environmental features.
Indoor and outdoor scene classification.
Recognition of foreground and background objects.
Identification of stationary and moving objects.
Spatial relationship analysis between detected objects.
Recognition of environmental changes.
Detection of objects entering or leaving the user’s field of view.
Recognition of crowded, open, confined, and obstructed spaces.
Contextual scene descriptions.
Identification of relevant environmental activity.
Prioritization of important observations.
Continuous environmental model updates.
Confidence scoring for environmental observations.
Configurable environmental observation frequency.

Autonomous-Driving-Inspired Perception Module

Applies perception and environmental analysis techniques inspired by autonomous driving systems to pedestrian awareness and navigation.

Multi-camera environmental perception.
Object detection and classification.
Multi-object tracking.
Object position estimation.
Relative velocity estimation.
Object trajectory prediction.
Pedestrian detection and tracking.
Vehicle detection and tracking.
Free-space detection.
Walkable area identification.
Road boundary and curb detection.
Crosswalk and lane recognition.
Traffic movement analysis.
Collision trajectory assessment.
Environmental movement prediction.
Dynamic obstacle recognition.
Sensor fusion for improved perception.
Continuous scene interpretation.

Spatial Awareness and Depth Perception Module

Provides three-dimensional awareness of nearby objects, surfaces, and environmental structures.

Stereo vision and depth estimation.
RGB-D camera integration.
Optional LiDAR and Time-of-Flight sensor support.
Distance estimation to nearby objects.
Relative object positioning.
Three-dimensional environmental reconstruction.
Detection of overhead and ground-level obstacles.
Recognition of walls, doors, furniture, and structures.
Detection of elevation changes.
Identification of stairs, ramps, slopes, and curbs.
Detection of drop-offs and potentially dangerous edges.
Recognition of narrow passages.
Identification of low-hanging obstacles.
Spatial orientation and head-position tracking.
Continuous spatial awareness updates.

Real-Time AI Voice Guidance Module

Translates environmental perception and contextual intelligence into natural, conversational audio guidance.

Continuous play-by-play descriptions of the surrounding environment.
Natural-language environmental explanations.
Context-aware spoken notifications.
Directional instructions including left, right, ahead, behind, above, and below.
Distance-based object announcements.
Movement-aware audio guidance.
Priority-based notifications.
Adjustable speech frequency and verbosity.
Customizable voice selection and speech speed.
Interruptible voice responses.
Intelligent suppression of repetitive notifications.
Emergency hazard announcements.
On-demand environmental descriptions.
Conversational questions about nearby objects and surroundings.
Adaptive audio guidance based on environmental complexity.

Intelligent Navigation Module

Provides indoor and outdoor navigation through environmental understanding, localization, and obstacle-aware route planning.

GPS and visual localization.
Indoor and outdoor navigation.
Real-time pedestrian route guidance.
Visual and spatial localization.
Simultaneous Localization and Mapping.
Landmark recognition.
Destination identification.
Route planning and recalculation.
Obstacle-aware route recommendations.
Walkable path identification.
Sidewalk and pedestrian pathway recognition.
Crosswalk and intersection identification.
Doorway and entrance recognition.
Elevator and staircase identification.
Indoor room-to-room navigation.
Building and facility navigation.
Campus navigation.
Turn-by-turn spoken directions.
Voice-controlled destination selection.
Destination arrival confirmation.
Personalized navigation preferences.

Predictive Hazard Detection and Safety Module

Continuously evaluates the surrounding environment to identify potential hazards and communicate relevant warnings.

Real-time hazard identification.
Approaching vehicle detection.
Recognition of vehicles turning across pedestrian paths.
Moving bicycle and scooter detection.
Moving pedestrian detection.
Collision trajectory prediction.
Identification of obstacles within the projected walking path.
Detection of uneven sidewalks and pavement.
Recognition of potholes and ground-level hazards.
Detection of curbs, drop-offs, and platform edges.
Identification of low-hanging objects.
Recognition of construction zones and temporary obstructions.
Identification of potentially dangerous intersections.
Risk prioritization and severity classification.
Early warning notifications.
Dynamic hazard monitoring.
Continuous reassessment of potential threats.
Explicit communication of detection uncertainty.

Object and Text Recognition Module

Identifies everyday objects and interprets written information within the user’s environment.

Everyday object identification.
Household object recognition.
Furniture and appliance recognition.
Product and packaging identification.
Clothing and accessory recognition.
Personal belonging identification.
Currency recognition.
Barcode and QR code recognition.
Optical character recognition.
Reading printed documents.
Reading signs, labels, and menus.
Street sign recognition.
Building number and room number recognition.
Reading digital displays.
Recognition of public transportation information.
Document interpretation and summarization.
On-demand object descriptions.

People and Social Awareness Module

Provides contextual awareness of people and social activity without requiring the user to continuously ask for environmental descriptions.

Detection of nearby people.
Estimation of the number of people in an area.
Recognition of approaching individuals.
Tracking of people moving through the environment.
Conversational proximity awareness.
Crowd detection and analysis.
Recognition of waving and pointing gestures.
Detection of available seating.
Recognition of people approaching from behind.
Personal-space obstruction awareness.
Optional recognition of previously authorized contacts with explicit enrollment and privacy controls.
Privacy-conscious people detection without default facial identification.

Traffic and Transportation Awareness Module

Provides awareness of traffic conditions, pedestrian crossings, and public transportation environments.

Traffic light recognition.
Pedestrian crossing signal recognition.
Crosswalk identification.
Traffic sign recognition.
Vehicle direction and movement analysis.
Turning vehicle detection.
Traffic flow awareness.
Bus stop identification.
Bus and train recognition.
Public transportation information recognition.
Platform and boarding area awareness.
Train and bus arrival information integration.
Parking lot navigation.
Vehicle entrance and exit identification.
Transit terminal navigation.

Indoor Environmental Intelligence Module

Interprets indoor environments and provides navigation assistance within buildings and facilities.

Room and hallway recognition.
Door and doorway detection.
Room number identification.
Office and building navigation.
Elevator and staircase recognition.
Restroom identification.
Exit and emergency exit recognition.
Furniture and fixture detection.
Indoor obstacle avoidance.
Indoor positioning and localization.
Room layout mapping.
Building directory recognition.
Store and business navigation.
Indoor point-of-interest identification.
Environmental change detection.

Outdoor Environmental Intelligence Module

Provides contextual awareness of outdoor environments and pedestrian surroundings.

Street and sidewalk recognition.
Building and landmark identification.
Street sign reading.
Address identification.
Park and recreational area recognition.
Terrain and elevation awareness.
Sidewalk condition analysis.
Pedestrian route identification.
Construction and roadwork awareness.
Weather-related environmental observations.
Outdoor spatial mapping.
Recognition of entrances, exits, and access points.

Spatial Mapping and Environmental Memory Module

Maintains a dynamic representation of the user’s surroundings to support localization, navigation, and environmental recall.

Visual and visual-inertial mapping.
Depth-based environmental mapping.
Three-dimensional scene reconstruction.
Occupancy mapping.
Semantic mapping.
Dynamic obstacle mapping.
Landmark-based localization.
Local environmental maps.
Route history mapping.
Environmental change detection.
Map persistence and updating.
Temporary spatial memory.
Recognition of previously encountered landmarks.
Familiar environment recognition.
User-defined location bookmarks.
Saved navigation preferences.
User-controlled memory retention.
Optional encrypted personal environmental maps.

Contextual AI Intelligence Module

Combines information from perception, navigation, spatial awareness, and sensor systems to interpret environmental situations.

Environmental context interpretation.
Prioritization of important events.
Prediction of likely environmental changes.
Understanding of user movement and direction.
Context-aware recommendations.
Intelligent event filtering.
Integration of visual, spatial, and sensor information.
Explanation of detected situations.
Conversational environmental reasoning.
Confidence-based notifications.
Distinction between observed facts and AI-generated interpretations.
Adaptive assistance based on environmental complexity.
Uncertainty-aware responses.
Contextual decision support.

Voice Command and Conversational Interaction Module

Enables hands-free interaction with SenseAR through natural speech.

Voice-activated commands.
Natural-language interaction.
Conversational environmental questions.
On-demand scene descriptions.
Voice-controlled navigation.
Voice-controlled settings.
Multi-turn conversations.
Speech recognition.
Text-to-speech.
Context-aware conversational responses.
Offline voice command support.
Customizable command vocabulary.
Optional external AI model integration.
Adjustable interaction preferences.
User-controlled AI verbosity.

Audio and Spatial Sound Module

Provides directional and contextual audio feedback to help users interpret their surroundings.

Directional audio guidance.
Spatial audio positioning.
Left and right audio cues.
Distance-based audio notifications.
Object-specific audio alerts.
Priority-based sound notifications.
Bone-conduction headphone support.
Open-ear audio support.
Adjustable audio levels.
Environmental sound awareness.
Background noise detection.
Audio focus management.
Emergency audio prioritization.

Accessibility Personalization Module

Adapts SenseAR’s behavior to individual accessibility needs and preferences.

Customizable spoken guidance.
Adjustable speech speed.
Adjustable notification frequency.
Adjustable hazard sensitivity.
High-priority hazard announcements.
Quiet and detailed guidance modes.
Directional audio preferences.
Personalized environmental descriptions.
Configurable assistance profiles.
Support for different levels of visual impairment.
Support for different mobility requirements.
Multilingual voice guidance.
Accessible configuration and setup.
User-defined notification preferences.

Emergency Assistance Module

Provides user-controlled emergency support and configurable assistance features.

Emergency voice commands.
Manual emergency activation.
User-defined emergency contacts.
Optional location sharing.
Emergency location reporting.
Hazard escalation notifications.
Potential fall detection using motion sensors.
User-confirmed emergency alerts.
Optional emergency service integrations where supported.
Low-battery emergency notifications.
Hardware failure warnings.
Emergency assistance mode.

Sensor Fusion and Hardware Integration Module

Combines multiple sensor inputs to improve environmental understanding and spatial accuracy.

Multiple camera integration.
Stereo camera support.
RGB-D camera support.
LiDAR integration.
Time-of-Flight sensor integration.
IMU integration.
GPS and GNSS integration.
Compass and orientation sensors.
Microphone integration.
Ambient light sensors.
Optional ultrasonic sensors.
Sensor synchronization.
Sensor calibration.
Multi-sensor data fusion.
Sensor failure detection.
Hardware abstraction interfaces.
Configurable sensor combinations.

Hardware and Wearable Interface Module

Provides support for modular smart glasses and external wearable processing configurations.

Modular smart glasses hardware.
Camera mounting options.
Lightweight wearable frame support.
Integrated microphone support.
Open-ear speaker support.
Bone-conduction audio support.
Replaceable battery options.
External processing unit support.
Wireless hardware communication.
Bluetooth, Wi-Fi, and USB connectivity.
Power management.
Thermal management.
Hardware diagnostics.
Battery health monitoring.
Low-power operating modes.
Optional visual augmented reality display.
Hardware configuration and compatibility management.

Privacy and Security Module

Protects user information and provides transparent control over environmental data.

Local-first processing.
On-device computer vision.
Encrypted data storage.
Encrypted communications.
User-controlled camera processing.
Configurable recording policies.
No mandatory cloud dependency.
Optional cloud AI processing.
User-controlled data retention.
Secure authentication.
Permission-based integrations.
Camera and microphone status indicators.
Configurable data deletion.
Protection against unauthorized access.
Privacy-aware handling of bystander information.
No facial identification by default.

Developer API and Extensibility Module

Provides interfaces for extending SenseAR with additional functionality, hardware, and community-developed capabilities.

Modular plugin architecture.
Public developer APIs.
Hardware abstraction interfaces.
AI model integration interfaces.
Navigation provider integrations.
Sensor integration interfaces.
Event-driven architecture.
Custom perception modules.
Custom voice assistants.
Custom navigation providers.
Third-party accessibility integrations.
Plugin lifecycle management.
Module configuration.
Developer documentation.
Testing and simulation interfaces.
Community-developed extensions.

Optional Plugin Modules

Optional plugins extend SenseAR’s functionality without requiring changes to the core modules. Each plugin should operate independently where practical and integrate with the shared environmental intelligence system through defined interfaces.

Advanced Object Intelligence Plugin

Expanded object classification.
Detailed product and object descriptions.
Object condition and state recognition.
Advanced visual attribute recognition.
Contextual object comparisons.
Object search and identification assistance.

Retail Assistant Plugin

Store and shopping center navigation.
Product identification.
Price and product label reading.
Shelf and aisle recognition.
Shopping list assistance.
Product comparison.
Barcode and QR code scanning.

Restaurant Assistant Plugin

Restaurant and dining area recognition.
Menu reading.
Menu descriptions and summarization.
Table identification.
Seating availability awareness.
Restaurant entrance and restroom navigation.
Food label recognition.

Travel Assistant Plugin

Airport navigation.
Hotel navigation.
Train station and terminal navigation.
Boarding gate identification.
Travel document reading.
Baggage claim identification.
Travel information interpretation.
Landmark and destination assistance.

Public Transportation Assistant Plugin

Bus, train, subway, and transit navigation.
Transit stop identification.
Route and schedule information.
Platform identification.
Boarding and exit assistance.
Transit arrival notifications.
Transfer guidance.

Medical Information Reader Plugin

Medication packaging recognition.
Prescription label reading.
Healthcare document reading.
Medical instruction interpretation.
Healthcare appointment document assistance.
Medical information summarization.

Document Intelligence Plugin

Advanced document reading.
Printed document interpretation.
Document summarization.
Form recognition.
Table and diagram descriptions.
Multi-page document navigation.
Text extraction and organization.

Currency and Financial Recognition Plugin

Banknote recognition.
Coin recognition.
Currency identification.
Financial document reading.
Receipt recognition.
Invoice and bill interpretation.

Home Assistant Plugin

Smart home integration.
Household device identification.
Appliance awareness.
Room navigation.
Smart lighting integration.
Voice-controlled home automation.
Household environmental awareness.

Navigation Mapping Plugin

External mapping provider integration.
Alternative route recommendations.
Community accessibility maps.
Accessible point-of-interest information.
Custom map provider support.
User-defined navigation destinations.

Weather Awareness Plugin

Current weather information.
Weather forecast integration.
Rain and storm notifications.
Temperature and environmental condition alerts.
Weather-aware navigation recommendations.

Gesture Interaction Plugin

Hand gesture recognition.
Gesture-based commands.
Hands-free interaction.
Custom gesture configuration.
Gesture-driven navigation controls.

Sign Language Recognition Plugin

Experimental sign language recognition.
Gesture interpretation.
Sign language translation assistance.
Configurable communication support.
Visual communication assistance.

Advanced Spatial Audio Plugin

Advanced directional sound positioning.
Three-dimensional audio cues.
Object-specific spatial audio.
Environmental sound localization.
Configurable audio environments.

Cloud AI Intelligence Plugin

Optional remote AI model integration.
Advanced multimodal reasoning.
Expanded scene interpretation.
Cloud-based language processing.
Optional remote image analysis.
Configurable cloud processing policies.

Offline AI Intelligence Plugin

Expanded on-device AI capabilities.
Offline environmental interpretation.
Local conversational assistance.
Offline object recognition.
Offline text recognition.
Disconnected navigation support where maps and positioning data are available.

Community Environment Maps Plugin

Community-contributed accessibility maps.
Shared accessible route information.
Public point-of-interest data.
Accessible entrance and exit information.
User-submitted environmental observations.
Map updates and verification.
Privacy-controlled community participation.

Assistive Communication Plugin

Communication assistance.
Text-to-speech communication.
Speech-to-text transcription.
Conversational accessibility tools.
Multilingual communication support.
Optional communication board integration.

Sports and Recreation Plugin

Recreational environment awareness.
Outdoor activity assistance.
Trail and pathway recognition.
Sports facility navigation.
Activity-specific environmental descriptions.
Configurable activity safety notifications.

Specification Branding License (SBL)

Standard

Fully AGPL-3.0+ compliant system.
Copyleft enforced for network deployments.
Required attribution:
- Roxanne Ardary
- https://www.roxanneardary.com/

Optional

Specification Branding License (SBL)
- Attribution-free commercial deployment.
- Pricing based on scale, usage, and deployment scope.
- https://roxanneardary.com/sensear/

License & Notice Requirements

SenseAR is released under the GNU Affero General Public License v3.0 or later (AGPL-3.0+).

By contributing to any Open Arsenal project, you agree that your contributions will also be released under this license.

Please note the following:

All contributions must comply with the AGPL-3.0+ terms.
Under Section 7 of the license, all redistributions, forks, and derivative works must preserve attribution to: Roxanne Ardary and roxanneardary.com.
SenseAR specifications are free to use with attribution. A Specification Branding License can be negotiated upon request.
The project’s notice.md file tracks attribution requirements and contributor acknowledgments. Any update that adds new contributors or modifies attribution should also update notice.md.
When submitting a pull request, ensure that any new files maintain the attribution headers where applicable.
Network-deployed versions of this software must also remain fully AGPL-3.0+ compliant, including exposure of source code modifications when applicable under the license.

For full legal details, please refer to the AGPL-3.0+ license and the project’s notice.md file.

Notice – SenseAR

Attribution Requirement: Under Section 7 of the AGPL 3.0+ license, all redistributions, forks, and derivative works, including network-deployed versions of this project, must provide attribution to Roxanne Ardary and roxanneardary.com.

Contributors

This file tracks contributors and their specific contributions to the project.

Roxanne Ardary, roxanneardary.com – November 2, 2026
Created the repository for SenseAR. Developed the project concept and specification for an open-source AI-powered smart glasses platform featuring real-time environmental perception, intelligent navigation, spatial awareness, predictive hazard detection, and conversational audio guidance.
Add other contributors here – [Date]
[Describe contribution in one sentence]

License – SenseAR

This repository is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0+).

Key Points:

You are free to use, modify, and distribute the code.
All redistributions, forks, and derivative works or network-deployed versions must also be licensed under AGPL-3.0+ and provide attribution to Roxanne Ardary and roxanneardary.com as required under Section 7 of the license.
The software is provided “as is,” without warranty of any kind.

For the full license text, see GNU AGPL-3.0 License.