Install instructions which worked for me

[philippwulff]

Using NVCC version 11.8.0.

This is what worked for me:

git clone https://github.com/luciddreamer-cvlab/LucidDreamer.git
cd LucidDreamer

cd submodules/depth-diff-gaussian-rasterization-min/third_party
git clone https://github.com/g-truc/glm.git glm
cd ../../..

conda create -n lucid python=3.9 -y
conda activate lucid
which nvcc
nvcc --version

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d gradio omegaconf
pip install timm==0.6.7
pip install plyfile==0.8.1

cd submodules/depth-diff-gaussian-rasterization-min/
python setup.py install

cd ../simple-knn
python setup.py install
cd ../..

pip install --upgrade \
    torch==2.1.0+cu118 \
    torchvision==0.16.0+cu118 \
    torchaudio==2.1.0 \
    --index-url https://download.pytorch.org/whl/cu118

According to ChatGPT this should work:

git clone https://github.com/luciddreamer-cvlab/LucidDreamer.git
cd LucidDreamer

# vendor GLM for the CUDA extension:
cd submodules/depth-diff-gaussian-rasterization-min/third_party
git clone https://github.com/g-truc/glm.git glm
cd ../../../

conda create -n lucid python=3.9 -y
conda activate lucid

# (optional) verify that nvcc is coming from your CUDA 11.8 install
which nvcc
nvcc --version

# 1) core numeric libs
pip install --upgrade "numpy<2.0.0"

# 2) torch 2.1.0+cu118
pip install --upgrade \
    torch==2.1.0+cu118 \
    torchvision==0.16.0+cu118 \
    torchaudio==2.1.0 \
    --index-url https://download.pytorch.org/whl/cu118

# 3) everything else
pip install peft diffusers scipy imageio[ffmpeg] opencv-python Pillow open3d gradio omegaconf
pip install timm==0.6.7 plyfile==0.8.1

# 4) build the two C++ extensions
cd submodules/depth-diff-gaussian-rasterization-min
python setup.py install

cd ../simple-knn
python setup.py install

[philippwulff]

I also needed to fix a type error.. so the dependency versions I installed are not optimal:

  File "/storage/user/wph/LucidDreamer/./ZoeDepth/zoedepth/models/base_models/midas.py", line 173, in __call__
    return nn.functional.interpolate(x, (height, width), mode='bilinear', align_corners=True)

I added this monkey patch to run.py

import torch.nn.functional as _F

_orig_interpolate = _F.interpolate
def _safe_interpolate(input, size=None, *args, **kwargs):
    # Coerce numpy ints (or other scalar types) to Python int
    if size is not None:
        try:
            size = tuple(int(s) for s in size)
        except Exception:
            # if size is a single int, just cast it
            size = int(size)
    return _orig_interpolate(input, size=size, *args, **kwargs)

_F.interpolate = _safe_interpolate

[philippwulff]

You get this error if the input image is too large or the aspect ratio is too skewed from 1:1 I think.

Reading Training Transforms
Traceback (most recent call last):
  File "/storage/user/wph/LucidDreamer/run.py", line 82, in <module>
    ld.create(rgb_cond, txt_cond, neg_txt_cond, args.campath_gen, args.seed, args.diff_steps, model_name=args.model_name)
  File "/storage/user/wph/LucidDreamer/luciddreamer.py", line 188, in create
    self.scene = Scene(self.traindata, self.gaussians, self.opt)
  File "/storage/user/wph/LucidDreamer/scene/__init__.py", line 22, in __init__
    info = readDataInfo(traindata, opt.white_background)
  File "/storage/user/wph/LucidDreamer/scene/dataset_readers.py", line 411, in readDataInfo
    preset_minicams = loadCameraPreset(traindata, presetdata=get_camerapaths())
  File "/storage/user/wph/LucidDreamer/scene/dataset_readers.py", line 376, in loadCameraPreset
    W, H = traindata["frames"][0]["image"].size
IndexError: list index out of range

512x512 works for me.

[philippwulff]

If the input image is not a 3-channel image you get:

Traceback (most recent call last):
  File "/storage/user/wph/LucidDreamer/run.py", line 82, in <module>
    ld.create(rgb_cond, txt_cond, neg_txt_cond, args.campath_gen, args.seed, args.diff_steps, model_name=args.model_name)
  File "/storage/user/wph/LucidDreamer/luciddreamer.py", line 187, in create
    self.traindata = self.generate_pcd(rgb_cond, txt_cond, neg_txt_cond, pcdgenpath, seed, diff_steps)
  File "/storage/user/wph/LucidDreamer/luciddreamer.py", line 356, in generate_pcd
    depth_curr = self.d(image_curr)
  File "/storage/user/wph/LucidDreamer/luciddreamer.py", line 157, in d
    return self.d_model.infer_pil(im)
  File "/home/stud/wph/miniconda3/envs/lucid/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/storage/user/wph/LucidDreamer/./ZoeDepth/zoedepth/models/depth_model.py", line 141, in infer_pil
    out_tensor = self.infer(x, pad_input=pad_input, with_flip_aug=with_flip_aug, **kwargs)
  File "/storage/user/wph/LucidDreamer/./ZoeDepth/zoedepth/models/depth_model.py", line 126, in infer
    return self.infer_with_flip_aug(x, pad_input=pad_input, **kwargs)
  File "/storage/user/wph/LucidDreamer/./ZoeDepth/zoedepth/models/depth_model.py", line 110, in infer_with_flip_aug
    out = self._infer_with_pad_aug(x, pad_input=pad_input, **kwargs)
  File "/storage/user/wph/LucidDreamer/./ZoeDepth/zoedepth/models/depth_model.py", line 77, in _infer_with_pad_aug
    assert x.shape[1] == 3, "x must have 3 channels, got {}".format(x.shape[1])
AssertionError: x must have 3 channels, got 4

Run convert input.png -alpha off output.png

[philippwulff]

I ran into this issue when looking at the output files (noisy depth video, mostly white .ply, black llff.mp4).

This answer solves it for me:

UPDATE: It appears there is an issue with the rasterizer method in the depth_diff_gaussian_rasterization_min package. The issue is resolved when you set the convert_SHs_python parameter to True in arguments.py.

[philippwulff]

If you want to use other camera trajectories during optimization, you need to edit run.py and utils/trajectory.py