phoebe routine error in PHonons of Phoebe-QE-7.3

[mrnbd]

I've build phoebe and patched qe-7.3 as docker image with following Dockerfile.

ARG CUDA_VERSION=12.6

# Build environment
FROM nvcr.io/nvidia/nvhpc:25.1-devel-cuda${CUDA_VERSION}-ubuntu24.04 AS build

RUN apt-get update -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
cmake \
gcc \
doxygen \
graphviz \
libomp-dev \
libopenmpi3 \
libhdf5-openmpi-dev

## Build Phoebe
WORKDIR /qsim
RUN git clone --recurse-submodules https://github.com/phoebe-team/phoebe.git

ARG GPU_ARCH
ARG CUDA_CC

WORKDIR /qsim/phoebe/build
RUN cmake .. \
-DKokkos_ENABLE_CUDA=ON \
-DKokkos_ARCH_${GPU_ARCH}${CUDA_CC}=ON \
-DOMP_AVAIL=ON \
-DCMAKE_EXE_LINKER_FLAGS="-lgfortran" \
-DCMAKE_INSTALL_PREFIX=/qsim/install

RUN make -j$(nproc)
RUN make install

## Build Phoebe QE
WORKDIR /qsim
RUN git clone https://github.com/phoebe-team/phoebe-quantum-espresso.git phoebe-qe

ARG CUDA_VERSION

WORKDIR /qsim/phoebe-qe
RUN git checkout phoebe-qe-7.3

RUN ./configure \
--with-cuda=${CUDA_HOME} \
--with-cuda-cc=${CUDA_CC} \
--with-cuda-runtime=${CUDA_VERSION} \
--with-cuda-mpi=yes \
--enable-openmp \
--prefix=/qsim/install

### Apply NVHPC patch for QE-7.3.1 PH.x
# COPY files/0001-fix-for-nvhpc-23.11-in-phq_setup.patch .
# RUN git apply 0001-fix-for-nvhpc-23.11-in-phq_setup.patch

### Build and install Phoebe QE
RUN make -j$(nproc) pw
RUN make -j$(nproc) pp
RUN make -j$(nproc) ph
RUN make -j$(nproc) w90
RUN make install

# Release environment
FROM nvcr.io/nvidia/nvhpc:25.1-runtime-cuda${CUDA_VERSION}-ubuntu24.04 AS release

RUN apt-get update -y && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
ssh \
make \
libsz2 \
python3-pip \
ca-certificates \
wget \
nano \
libomp-dev \
libopenmpi3 \
libhdf5-openmpi-dev

WORKDIR /home/ubuntu
COPY --chown=ubuntu:ubuntu --from=build /qsim qsim

RUN echo ubuntu:ubuntu | chpasswd

RUN echo "\nif [ -d \$HOME/qsim/install/bin ]; then\n\tPATH=\"\$HOME/qsim/install/bin:${PATH}\";\nfi\n" >> .profile
RUN echo "\nLD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/25.1/cuda/12.6/targets/x86_64-linux/lib:${LD_LIBRARY_PATH}" >> .profile
RUN ln -s ../../phoebe/build/phoebe qsim/install/bin

COPY files/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

EXPOSE 22
ENTRYPOINT [ "/entrypoint.sh" ]

Then i tried to calculate electrical conductivity of the researched structure. A strange error occurs during the calculation of the second q point in PHonon code. To verify build, I performed a calculation using an example of silicon-EPA, and the same issue occurred.

Increasing of ecutwfc, ecutrho does not fix it.
Why can this happen?

[jcoulter12]

Hi @mrnbd,

Thanks for reporting this, indeed that's unexpected behavior.
I can think of a couple reasons this could happen, but it would be much easier for me to test this locally.

Would you mind uploading the full input and output files that produced this error? (if sensitive research-wise, you are free to email them to me, my email is on my github profile).

Best,
Jenny

[mrnbd]

This reproduces on silicon-epa from example folder. My calculation results with doubled ecutwfc attached.
Kokkos arch: ADA89
qe-elph.zip

[jcoulter12]

Great -- I'll take a look and hopefully let you know what's going on in the next day or so!

[jcoulter12]

Hi @mrnbd,

I downloaded the zip files and tested them with a build of phoebe-qe-7.3 on my local machine -- everything seems to work ok on my end.

I suspect something has gone wrong with the docker set up. I can try to help with that, and ideally I will again get to try your docker image specifically in the next day or so -- but also -- do you mind saying what motivated you to build with docker initially? This is just in case I can fix the cause of using the docker more simply than fixing the docker issue.

### Apply NVHPC patch for QE-7.3.1 PH.x
# COPY files/0001-fix-for-nvhpc-23.11-in-phq_setup.patch .
# RUN git apply 0001-fix-for-nvhpc-23.11-in-phq_setup.patch

In particular, I'm unsure what the effect of these lines combined with the Phoebe patch might be. I am sure you want this patch, I just wonder if it's causing something in the Phoebe patch to behave unexpectedly. Is there somewhere I can see what these modify?

Best,
Jenny

[mrnbd]

Thank you for test. And sorry for late reply.

1. Why Docker? For computing, I use a home PC with Win 11 as the main system and WSL for UNIX-based development. Therefore WSL distro has too dirty environment. Instead of duplicating WSL distros I use solution providing controllable environment - Docker.
2. Why GPU (CUDA)? Reason in first half of previous point.
3. Patch. In qe-7.3 found issue for new NVHPC and fixed by commit QEF/q-e@28944c2. You can see these lines commented out, because it not occurs in 'epa' calculations (other subroutine is called I think).

I've question. Are you run test with GPU or CPU build?

[jcoulter12]

Ok, great -- that patch is indeed super small, ideally not an issue. Also indeed, Windows is not a good option for running these computations -- so I understand the choice of Docker, though you might find that in general, phonon and electron-phonon transport calculations are too expensive to sensibly run on a home PC.

I ran the test on CPU. I have basically never run QE on GPUs, because up to very recently, my understanding was ph.x was not GPU optimized. I suppose this indicates it now might be.

Is it possible for you to run a CPU test in your container?