Flow Rate Calculation Across Processors in Turbulent Channel Flow with Non-Uniform Grid

[IMSRG]

Hello everyone,

I'm currently modifying the turbulent channel flow case for a custom problem in Xcompact3D. My want to to compute flow rate vs. time at a cross-section perpendicular to the x-axis. I am calculating average velocity inside the code for the flow rate.
Since the mesh is non-uniform in channel height direction, I'm using the area weighted average rule to compute the average velocity using the velocity component ux1. The integration is performed locally on each MPI process, and the results are combined using MPI_ALLREDUCE.

The issue arises due to domain decomposition. For instance, if the y-direction grid spans 1–100 and is split across 2 MPI processes (1–50 and 51–100), the area weighted part of the cell between the interface points 50 and 51 is missed in the calculation. This leads to an under-prediction of the average velocity or integrated area. I want to avoid gathering the entire field on one process due to memory and performance constraints.

Below is a simplified version of my implementation in case_custom.f90:

subroutine flowrate_channel_tpg(ux1, uy1, uz1)
implicit none

real(mytype), intent(in), dimension(xsize(1), xsize(2), xsize(3)) :: ux1, uy1, uz1
real(mytype) :: local_area, local_avg_vel, area, avg_vel, vel_cellcenter, dydz
integer :: i, j, k, code

i = xend(1)-5                                                  ! slice location
local_area = 0.0d0
local_avg_vel = 0.0d0
do j = xstart(2), xend(2)-1
  do k = xstart(3), xend(3)-1
     dydz = (yp(j+1)-yp(j))*(zp(k+1)-zp(k))
     vel_cellcenter = 0.250d0*(ux1(i, j, k) + ux1(i, j+1, k) + ux1(i, j+1, k+1) + ux1(i, j, k+1))
     local_area = local_area+ dydz
     local_avg_vel = local_avg_vel + vel_cellcenter*dydz
  end do
end do

area = local_area
avg_vel = local_avg_vel
call MPI_ALLREDUCE(MPI_IN_PLACE,area,1,real_type,MPI_SUM,MPI_COMM_WORLD,code)
call MPI_ALLREDUCE(MPI_IN_PLACE,avg_vel,1,real_type,MPI_SUM,MPI_COMM_WORLD,code)
avg_vel = avg_vel/area

end subroutine flowrate_channel_tpg

My question:

Is there a recommended approach for computing cross-sectional integrals on non-uniform grids in a parallel-decomposed domain correctly inside the code?

Any guidance or suggestions would be greatly appreciated!

Thanks and best regards,
Ravi

[mathrack]

Hello,

I think you should use ppy on non-uniform grids. See https://github.com/xcompact3d/Incompact3d/blob/master/src/Case-Channel.f90#L220