Team

IT guys:

• Mykhailo Lapko: Motivated DevOps, developed a boatload of algorithms for the app. Pure fun and pleasure to work with.
• Illia Martynov: Pentester with passion(and Frontender in past), always knows how to support and speedrunned the frontend tasks.
• Arsen Rudnytskyi: Machine Learning guy thanks to whom you can read our code. Excellent at refactoring and organizing information.
• Sviatoslav Pylyp: Greatest Backender I've worked with. Need a backend, DevOps solution? Describe a task in couple of words, he will deliver perfectly. Did it for the startup using clean and maintainable APIs in Java, microservice infrastructure, and hybrid infrastructure leveraging AWS.
• Volodymyr Lapkin: Software Developer. Quickly adapts to new environments. Played a key role in developing algorithms for furniture placement, 3D model integration, and improving image quality. His problem-solving skills and rapid learning make him an invaluable asset to any team.
• Kyrylo Turchyn: Machine Learning engineer. Worked outsourced, so you dont see his commits. Helped a lot though with 3D estimation. He is driven to learn AI to train ginormous revolutionary models. Just give him oportunity.
• Oleksandr Milko: Guess we'll find out some day. Cheers!

Business guys:

• Dawid Tambor: The man thanks to whom I started doing this project. He's been a good mentor for me.
• Agnieszka Solarz: Managed all the work with Designing the app and more. Always brought good energy to our meetings even on the hardest days.
• Marcin Mika: Helped significantly with accounting projections and bureaucracy. Realist, makes conclusions based on facts and statistics.

Documentation

An application for virtual staging of rooms(empty room photos).

• 74% accuracy from the last test(on simple bedroom layouts)! 😎
• Takes only 2 minutes(per picture) to stage a room.
• Uses real 3D models. Not an ordinary GenAI.
  

We made the app use real-world 3D furniture models because it is the right step toward what was desired by many interior designers.

This application consists of several components that work together to create a staged room image based on an empty room input and style preferences. Below is an overview of the key parts and installation steps.

Interesting links:

0. link to Miro Management Board(since Summer 2024)
1. link to Trello Management Board(until Summer 2024)
2. Vistager GUI designs
3. link to Miro flowchart
4. link to System Design(incomplete)
5. Last tests from November - December of 2024 for Bedrooms
6. Some of the good results we got with postprocessing
7. links to Backend DockerHub and Github(Made by Sviatoslav): auth-service, email-service, api-gateway, ai-queue, Backend Github
8. links to Frontend(Made by Illia): illiamartynov/startup
9. link for testing scripts

Application Components

1. codename25 Repo
   The core of the application, serving as the API for image creation.
   Input:

   Base64 encoded empty room image, style and budget string in the format:
   "{Style}, {Budget}"
   Room type string in the format:
   "{bedroom/living room/kitchen}"

   Output: Base64 encoded staged room image.

2. Blender 3D furniture models

3. Stable Diffusion
   Handles preprocessing (segmentation) and postprocessing (image enhancement). You can use either WebUI or ComfyUI for this task.

   • WebUI: Superior for postprocessing. Arguably easier to install.
   • ComfyUI: Manages GPU memory more efficiently but is less effective at postprocessing.

4. Conda Environment
   Contains all the necessary packages for the application to function correctly.

Installing each component

Conda Environment and CUDA

1. Install CUDA: Follow the CUDA installation guide.
2. Install Conda: Download and install Miniconda from here.
3. Create a new Conda environment: conda create -n app python=3.11.9 anaconda
4. Install PyTorch for AI(Dont forget to activate conda env with conda activate app beforehand): conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
5. pip install -r requirements.txt

Blender models

Unfortunately, we cannot provide pre-existing 3D models due to licensing restrictions. Follow the steps below.

1. Find or create your own 3d models in USDC format.
2. Test mode1s in Blender to confirm they work as intended, including textures, scaling, and positioning.
3. Prepare the Folder Structure
   Navigate to the Projects/codename25/visuals/3Ds/ directory. Identify the appropriate category folder for your model (e.g., living_room for living room assets). If the category does not exist, create a new folder with a relevant name (e.g., bedroom, living_room).
4. Place the .usdc file in the appropriate category folder.
   

APP itself (codename25)

1. git clone https://github.com/AlexandrMilko/codename25.git
2. Put depth_pro.pt model into the folder codename25/ml_depth_pro/src/depth_pro/cli/checkpoints/
3. Create folder codename25\ml_depth_pro\output
4. If you have macbook: pip install git+https://github.com/rwightman/pytorch-image-models.git

Stable Diffusion

1. Install WebUI
2. Install Controlnet through extensions
3. Download and put the controlnet models(control_v11f1p_sd15_depth.pth, control_v11p_sd15_seg.pth) in the directory: extensions/sd-webui-controlnet/models
4. Download and put stable diffusion model in models/Stable-diffusion
5. Replace webui.py with the webui.py from codename25other

Running the App

Web UI (First run the WEBUI! The app will check connection to it on the start)

IMPORTANT: change the ui parameter in constants.py to 'webui'

Run the following command:

webui.bat --nowebui --server-name=0.0.0.0

Or if you run UNIX:

./webui.sh --nowebui --server-name=0.0.0.0

App

Next, activate the conda environment and run the application:

conda activate app
cd codename25
python run.py

Test

To test the API, activate the conda environment and run the test script:

conda activate app
python test_api.py  # (WARNING: Put the right image path in test_api.py)

How to work with Docker on the Server

1. Create your container

sudo docker run -it --name vistagerVova --network host --gpus all oleksandrmilko/vistager:demo1309

2. (Just in case):

sudo docker restart vistagerVova

3. Activate github token for the git in the container if it is expired
4. Enter a session in container

sudo docker exec -it vistagerVova bash

5. Copy files(and images for example) from docker:

sudo docker cp <docker_container>:<path_in_docker> <path_in_host_system>

Special thanks

• ml_depth_pro: for depth estimation to calculate 3D structure of rooms
• segment_anything: for masking neighbouring rooms