DeathNote

DeathNote is a multimodal LLM-powered React Native mobile app for taking notes using text, images, and audio. The app summarizes your entries with witty commentary from your chosen Shinigami, and supports searching, organizing, and reviewing your notes.

---

Features

• Text, Image, and Audio Notes: Add rich content to your notes.
• AI Summaries: Get amusing summaries from your Shinigami.
• Multimodal Uploads: Notes are sent to a backend that processes text, images, and audio.
• Search & Organize: Quickly find notes using semantic search.
• Customizable Settings: Choose your language and Shinigami.
• Chat with Shinigami: Interact for fun or advice.

---

Frontend

The frontend is built with React Native and Expo. It provides a seamless mobile experience for note-taking and interacting with the AI.

Main Components

• Note Creation: index.jsx
  Users can create notes with text, images (camera/gallery), and audio recordings. Each note is stored locally and can be uploaded to the backend for AI processing.
• Note List & Detail: NodeList.jsx, NoteDetail.jsx
  Displays all notes, their summaries, and details. Summaries are fetched from the backend and stored locally.
• Settings: SettingsForm.jsx
  Lets users set their name, gender, preferred language, and Shinigami persona.
• Chat: AiChat.jsx
  Enables users to chat with their Shinigami, sending text, images, and audio messages.
• Note Upload: sendNoteToServer.js
  Handles sending notes and media to the backend using FormData.

Data Flow

1. Notes are created and stored locally.
2. When a note is ready, it is uploaded to the backend with user settings and previous summaries.
3. The backend returns a summary and title, which are saved locally and displayed in the app.
4. Semantic search and chat features interact with backend endpoints for advanced AI functionality.

---

Backend

The backend is built with Django and Django REST Framework. It powers all AI features, note management, and multimodal processing.

Main Responsibilities

• Note Storage & Processing:
  Receives notes (text, images, audio) via API, processes media (transcribes audio, describes images), and stores them.
• AI Summarization:
  Uses a multimodal LLM (via OpenAI or Ollama) to generate summaries and witty commentary in the style of Ryük, referencing previous entries for context. See utils.py for prompt engineering and LLM calls.
• Semantic Search:
  Embeds notes using AI models and allows users to search by meaning, not just keywords. See views.py.
• Chat with Shinigami:
  Provides a conversational endpoint where the AI responds as Ryük, drawing from past summaries and user settings.
• User Settings:
  Applies user preferences (name, gender, language, Shinigami persona) to personalize AI responses.

Key Files

• views.py:
  Main API endpoints for note upload, summarization, search, and chat.
• utils.py:
  Handles LLM prompts, media processing (audio/image), and chat logic.
• models.py:
  Database models for notes and user data.
• serializers.py:
  Serializes notes and responses for API communication.
• urls.py:
  API routing.

API Endpoints

• POST /api/notes/summarize/ — Upload a note and receive a summary.
• GET /api/notes/search/ — Semantic search for notes.
• POST /api/chat/ — Chat with your Shinigami.

Summarization Flow

1. The frontend sends note data, user settings, and previous summaries.
2. The backend processes media (transcribes audio, describes images).
3. All content is sent to the LLM with a custom prompt (see utils.py).
4. The LLM returns a title and summary, which are sent back to the frontend.

Chat Flow

1. The frontend sends chat messages, working memory (recent summaries), and user settings.
2. The backend uses a specialized prompt to generate Ryük-style responses, referencing past entries and maintaining persona.

Semantic Search

• Notes are embedded using AI models.
• Search queries are embedded and compared for similarity.
• Results are ranked and returned to the frontend.

---

Getting Started

Prerequisites

• Node.js
• Expo CLI
• Python 3 (for backend)
• Django & Django REST Framework

Setup

1. Install frontend dependencies:

   cd frontend
   npm install

2. Start the frontend app:

   npx expo start

3. Install backend dependencies:

   cd server/DeathNote
   pip install -r requirements.txt

4. Run Django server:

   python manage.py runserver

---

Project Structure

frontend/
  app/
    (tabs)/
      index/
      components/
      constants/
      hooks/
      assets/
server/
  DeathNote/
    notes/
      views.py
      utils.py
      urls.py

---

Backend Endpoints

• /api/notes/summarize/ — Summarize notes
• /api/notes/search/ — Semantic search
• /api/chat — Chat with Shinigami

---

*Contributions and feedback