🪅 Visual Genius: Communication Assistant for ASD

Visual Genius is an application that helps facilitate communication between parents and children with autism spectrum disorders (ASD) through visual cards and structured conversations.

Important

This application was developed for proof-of-concept purposes. Please make sure to test and verify its features before using it.

• Teach mode: Uses visual cards inspired by Applied Behavior Analysis (ABA) and addresses problems of existing solutions. Examples of visual aid products on the market, These products can be quite costly: Bing Search Results, Amazon Search Result.
• Communication mode: Uses cards inspired by this paper: AACessTalk.
• Letterboard: Inspired by the book The Reason I Jump.

🎡 Screens

Teach Teaching words with visual cards	Parent Start conversation session	Child Interact with conversation cards
Letterboard Spell-based communication	Auth Sign in & sign up	Settings API key & session management

🌟 Features

1. Parent Interface - Conversation Facilitation

Start and manage conversations with your child using:

• 8 Predefined Topics: Daily routines, emotions, food, activities, school, bedtime, family, safety
• Custom Prompts: Create conversation starters for any situation
• Real-time State Management: Start, pause, resume, or stop conversations
• Quick Response Tracking: Yes/No/"I don't know" buttons
• Conversation Timeline: View complete interaction history

2. Child Interface - Visual Response Board

Simple, touch-friendly interface where children can:

• View available visual cards
• Select cards to communicate
• See immediate feedback
• All selections are logged for parent review

3. Teach Mode - Card Creation & Sequencing

Create educational visual sequences:

• Generate step-by-step cards using AI
• Drag-and-drop to reorder teaching sequences
• Save card arrangements for future use
• Build reusable teaching libraries

4. Letter Board - Alternative Communication Method

Spell-based communication inspired by The Reason I Jump:

• Full alphabet grid for letter-by-letter communication
• Text-to-speech for constructed messages
• Ideal for children who can spell but struggle with verbal expression
• Lower cognitive load than card selection for some users

🏗️ Technology Stack

• Framework: Next.js 14+ (App Router)
• UI: Tailwind CSS
• State: Zustand
• Database: PostgreSQL (or Azure Database for PostgreSQL)
• ORM: Drizzle ORM
• AI Services:
  • Azure OpenAI (card generation)
  • Unsplash API (free image search for visual suggestions)

🚀 Getting Started

1. Clone the Repository

git clone <repository-url>
cd visual-genius

2. Install Dependencies

npm install

3. Configure Environment Variables

Option A: Web UI (Recommended)

1. Start the development server: npm run dev
2. Navigate to /settings in your browser
3. Configure all settings through the web interface
4. Settings are stored in the database

Option B: Environment File

Create a .env.local file in the root directory:

# PostgreSQL Connection
POSTGRES_URL=postgres://user:password@hostname:5432/database

# Azure OpenAI
AZURE_OPENAI_ENDPOINT=https://<your-openai-name>.openai.azure.com/
AZURE_OPENAI_DEPLOYMENT_NAME=<gpt-deployment-name>

# Unsplash Image Search (Free)
# Get your free API key at: https://unsplash.com/developers
UNSPLASH_ACCESS_KEY=<your-unsplash-access-key>

Note: Settings in the database override .env.local values.

4. Initialize Database

The application will automatically create required tables on first run:

• conversation_session
• visual_card
• utterance

5. Start Development Server

npm run dev

The application will be available at http://localhost:3001

📱 Usage

Communication Workflow

1. Start a Conversation (/parent)

   • Select a predefined topic or enter a custom prompt
   • Click "Start Conversation"
   • AI generates relevant visual cards with images
   • Cards appear on the main board and child screen
   • Click "->" button to hand over the conversation to a child

2. Response to a conversation facilitated by a parent

   • View available cards from the active session
   • Tap cards to select
   • Click "->" button to hand over the conversation to a parent

3. Manage Conversation

   • Pause: Temporarily halt the conversation
   • Resume: Continue from where you paused
   • Stop: End conversation and clear session
   • Quick response buttons log child's answers

Teaching Workflow

1. Navigate to /teach
2. Enter a teaching prompt (e.g., "How to brush teeth")
3. Review generated cards
4. Drag-and-drop to reorder steps
5. Save arrangement for future lessons

🚢 Deployment

Docker Deployment

The application is fully containerized and can be deployed to any cloud service or on-premises infrastructure.

Quick Start with Docker Compose:

# 1. Copy environment template
cp .env.example .env

# 2. Edit .env with your credentials
nano .env

# 3. Build and start services
# docker compose up --build -d → Only builds/starts the app, not postgres
docker compose --profile local-db up --build -d # → Builds/starts both postgres and app

# 4. Access the application
open http://localhost:3001

Azure Native Deployment

Azure Resources Needed:

1. Azure App Service (Linux, Node.js 18+) or Container Instances
2. Azure OpenAI Service
3. Azure Database for PostgreSQL Flexible Server
4. Application Insights (optional, for monitoring)

External Services:

• Unsplash API (free image search)

Deployment Steps

1. Provision Azure resources
2. Configure App Service environment variables
3. Enable Managed Identity for OpenAI access
4. Configure PostgreSQL firewall rules
5. Deploy code via GitHub Actions or Azure CLI

🔒 Security

• API keys stored server-side only (environment variables)
• DefaultAzureCredential for Azure service authentication
• PostgreSQL connections over SSL
• Input validation on all API endpoints
• No sensitive data exposed to frontend

📄 License

MIT

🙏 Acknowledgments

• Inspired by ABA (Applied Behavior Analysis) principles
• Research on visual supports for autism communication
• Azure AI services for accessibility

---

Note: The .old/ directory contains the original proof-of-concept implementation and is kept for reference only. All active development occurs in the src/ directory.

For detailed architecture documentation, see ARCHITECTURE.md