Skip to content

Latest HyperDX can't scan data from ClickHouse #1283

@simadimonyan

Description

@simadimonyan

I have deployed clickstack with Docker.

Set containers timedate with host machine timedate

Image
...

clickhouse:
    image: clickhouse/clickhouse-server:25.9
    container_name: clickhouse
    ulimits:
      nofile:
        soft: 262144
        hard: 262144
    environment:
      - CLICKHOUSE_USER=${CLICKHOUSE_USER}
      - CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD}
    ports:
      - "8123:8123"    # http
      - "9090:9000"    # native
    volumes:
      - ./volumes/clickhouse/data:/var/lib/clickhouse
      - ./volumes/clickhouse/logs:/var/log/clickhouse-server
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: 1
    networks:
      - schedule_net

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    container_name: open-telemetry
    restart: always
    depends_on:
      - clickhouse
    ports:
      - "4317:4317"
      - "4318:4318"
      - "8888:8888"
    volumes:
      - ./configs/opentelemetry/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: 1
    networks:
      - schedule_net
    
  hyperdx:
    image: docker.hyperdx.io/hyperdx/hyperdx-all-in-one
    container_name: hyperdx-ui
    depends_on:
      - clickhouse
      - otel-collector
    environment:
      - HYPERDX_API_URL=${HYPERDX_API_URL}
      - HYPERDX_API_PORT=${HYPERDX_API_PORT}
      - HYPERDX_APP_URL=${HYPERDX_APP_URL}
      - HYPERDX_APP_PORT=${HYPERDX_APP_PORT}
      #- FRONTEND_URL=${FRONTEND_URL} - для production
    ports:
      - "8081:8080"
      - "8000:8000"
    volumes:
      - ./volumes/hyperdx/data:/data/db
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: 1
    networks:
      - schedule_net

...

HyperDX-UI shows empty list when requesting data from the ClickHouse. Datasource is correct, auth is correct but the list is empty. There is a txt file of console logs from HyperDX-UI which sending requests with sql query:

SELECT
  Timestamp,
  ServiceName,
  SeverityText,
  Body
FROM
  default.otel_logs
WHERE
  (
    TimestampTime >= fromUnixTimestamp64Milli (1761036102302)
    AND TimestampTime <= fromUnixTimestamp64Milli (1761039702311)
  )
ORDER BY
  TimestampTime DESC
Image

HyperDX Logs.txt

ClickHouse query:

879405f1eb9b :) select * from default.otel_logs limit 10

SELECT *
FROM default.otel_logs
LIMIT 10

Query id: 401bb22b-c306-4c61-b731-2dac7dcd3e6f

Row 1:
──────
Timestamp:          2025-10-21 12:03:41.702000000
TimestampTime:      2025-10-21 12:03:41
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Starting App using Java 21.0.8 with PID 109 (/service/schedule-parse-service/build/classes/java/main started by root in /service/schedule-parse-service)
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          app.App
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {'thread.name':'main'}

Row 2:
──────
Timestamp:          2025-10-21 12:03:41.702000000
TimestampTime:      2025-10-21 12:03:41
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Starting App using Java 21.0.8 with PID 109 (/service/schedule-parse-service/build/classes/java/main started by root in /service/schedule-parse-service)
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          app.App
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {}

Row 3:
──────
Timestamp:          2025-10-21 12:03:41.721000000
TimestampTime:      2025-10-21 12:03:41
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               No active profile set, falling back to 1 default profile: "default"
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          app.App
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {'thread.name':'main'}

Row 4:
──────
Timestamp:          2025-10-21 12:03:41.721000000
TimestampTime:      2025-10-21 12:03:41
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               No active profile set, falling back to 1 default profile: "default"
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          app.App
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {}

Row 5:
──────
Timestamp:          2025-10-21 12:03:43.121000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Multiple Spring Data modules found, entering strict repository configuration mode
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {'thread.name':'main'}

Row 6:
──────
Timestamp:          2025-10-21 12:03:43.121000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Multiple Spring Data modules found, entering strict repository configuration mode
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {}

Row 7:
──────
Timestamp:          2025-10-21 12:03:43.122000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Bootstrapping Spring Data JPA repositories in DEFAULT mode.
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {'thread.name':'main'}

Row 8:
──────
Timestamp:          2025-10-21 12:03:43.122000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Bootstrapping Spring Data JPA repositories in DEFAULT mode.
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {}

Row 9:
───────
Timestamp:          2025-10-21 12:03:43.276000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Finished Spring Data repository scanning in 144 ms. Found 4 JPA repository interfaces.
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {'thread.name':'main'}

Row 10:
───────
Timestamp:          2025-10-21 12:03:43.276000000
TimestampTime:      2025-10-21 12:03:43
TraceId:            
SpanId:             
TraceFlags:         0
SeverityText:       INFO
SeverityNumber:     9
ServiceName:        schedule-parse-service
Body:               Finished Spring Data repository scanning in 144 ms. Found 4 JPA repository interfaces.
ResourceSchemaUrl:  https://opentelemetry.io/schemas/1.24.0
ResourceAttributes: {'container.id':'2f058d58d39d488f93e4b5601d14d95ccfc8a006f72a8c1b28d0db23688e1daa','host.arch':'amd64','host.name':'2f058d58d39d','os.description':'Linux 6.8.0-85-generic','os.type':'linux','process.command_line':'/opt/java/openjdk/bin/java -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant app.App','process.executable.path':'/opt/java/openjdk/bin/java','process.pid':'109','process.runtime.description':'Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.8+9-LTS','process.runtime.name':'OpenJDK Runtime Environment','process.runtime.version':'21.0.8+9-LTS','service.instance.id':'2899312e-62bc-4781-93c2-634a6334799b','service.name':'schedule-parse-service','telemetry.distro.name':'opentelemetry-spring-boot-starter','telemetry.distro.version':'2.14.0','telemetry.sdk.language':'java','telemetry.sdk.name':'opentelemetry','telemetry.sdk.version':'1.49.0'}
ScopeSchemaUrl:     
ScopeName:          org.springframework.data.repository.config.RepositoryConfigurationDelegate
ScopeVersion:       
ScopeAttributes:    {}
LogAttributes:      {}

10 rows in set. Elapsed: 0.008 sec. 

Some logs showing after 2 hours of waiting. No network issues.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions