Skip to content

java exception during startup of opentsdb init container #61

@kgreczka9

Description

@kgreczka9

ISSUE
The main pod opentsdb crashes afer some time/some amount of data processed. In our case it is regular - after about two weeks of usage.
The opentsdb logs:

opentsdb-create-hbase-tables Checking if given compression is supported...
opentsdb-create-hbase-tables 2022-08-16 07:59:33,333 INFO  [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesIm
opentsdb-create-hbase-tables 2022-08-16 07:59:33,350 INFO  [main] hfile.CacheConfig: Created cacheConfig: CacheConfig:disabled
opentsdb-create-hbase-tables 2022-08-16 07:59:33,523 INFO  [main] zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
opentsdb-create-hbase-tables 2022-08-16 07:59:33,538 INFO  [main] compress.CodecPool: Got brand-new compressor [.gz]
opentsdb-create-hbase-tables 2022-08-16 07:59:33,542 INFO  [main] compress.CodecPool: Got brand-new compressor [.gz]
opentsdb-create-hbase-tables 2022-08-16 07:59:33,727 INFO  [main] hfile.CacheConfig: Created cacheConfig: CacheConfig:disabled
opentsdb-create-hbase-tables 2022-08-16 07:59:33,741 INFO  [main] compress.CodecPool: Got brand-new decompressor [.gz]
opentsdb-create-hbase-tables SUCCESS
opentsdb-create-hbase-tables HBase script:
opentsdb-create-hbase-tables ----
opentsdb-create-hbase-tables create 'tsdb-uid',
opentsdb-create-hbase-tables {NAME => 'id', COMPRESSION => 'GZ', BLOOMFILTER => 'ROW'},
opentsdb-create-hbase-tables {NAME => 'name', COMPRESSION => 'GZ', BLOOMFILTER => 'ROW'}
opentsdb-create-hbase-tables create 'tsdb',
opentsdb-create-hbase-tables {NAME => 't', VERSIONS => 1, COMPRESSION => 'GZ', BLOOMFILTER => 'ROW'}
opentsdb-create-hbase-tables create 'tsdb-tree',
opentsdb-create-hbase-tables {NAME => 't', VERSIONS => 1, COMPRESSION => 'GZ', BLOOMFILTER => 'ROW'}
opentsdb-create-hbase-tables create 'tsdb-meta',
opentsdb-create-hbase-tables {NAME => 'name', COMPRESSION => 'GZ', BLOOMFILTER => 'ROW'}
opentsdb-create-hbase-tables ----
opentsdb-create-hbase-tables Checking if opentsdb tsdb-uid hbase table exists
opentsdb-create-hbase-tables java exception

PRECONDITIONS
chart version - 0.1.7
custom values:

config: {"tsd.http.request.enable_chunked":true,"tsd.http.request.max_chunk":65535}
daemons: 1
hbase:
  enabled: true
  hbase:
    master:
      replicas:3
    regionServer:
      replicas: 3
      resources:
        requests:
          memory: 1Gi
          cpu: 100m
  hdfs:
    enabled: true
    dataNode:
      replicas: 3
    persistence:
      nameNode:
        enabled: true
        accessMode: ReadWriteOnce
        size: 5Gi
      dataNode:
        enabled: true
        accessMode: ReadWriteOnce
        size: 10Gi
  zookeeper:
    enabled: true
    replicaCount: 1
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: "kubernetes.io/hostname"
            labelSelector:
              matchLabels:
                release: zookeeper
    persistence:
      enabled: true
      accessMode: ReadWriteOnce
      size: 1Gi
resources:
  requests:
    memory: 1Gi
    cpu: 100m

STEPS TO REPRODUCE
Install chart with custom values, use it as normal for a while (about a week or two depending on configuration)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions