How to set master-slave replication for standalone deployment?

[utilbox]

Hi team,

We're planning to deploy Milus as standalone with Docker compose. The document says

Milvus Standalone supports high availability through master-slave replication.

We are considering enabling master-slave replication(on two EC2 nodes) for HA, but unable to find any further guidance from the official document or the Github Discussions.

Could anyone provide some details about how to set it up? Thanks a lot ❤️

Best Regards
James

[yhmo]

A possible solution: milvus standalone + pulsar/kafka + milvus-cdc
milvus-cdc is a change data capture tool for Milvus. It can capture the changes of upstream Milvus collections and sink them to downstream Milvus.

1. deploy a milvus standalone A with pulsar as message queue: https://milvus.io/docs/deploy_pulsar.md
2. deploy another milvus standalone B (pulsar, kafka, or local message queue)
3. deploy milvus-cdc https://milvus.io/docs/deploy-cdc-server.md, in the configuration file cdc.yaml, set the standalone A as source milvus, set the pulsar address of A
4. create a cdc task to sync data of source A to target B: https://milvus.io/docs/manage-cdc-tasks.md

[libaokang]

Thank you @yhmo for the solution. Let me try it

[utilbox]

Hi @yhmo, thank you again for your solution 🙏 . We followed the design above and deployed the services as shown below. The source Milvus works as expected. But the CDC server seems not working well.

We configured the CDC and the source/target Milvus Standalone instances according to the documents listed below:

• Milvus-CDC Usage
• Deploy CDC Server

For the first time we starts the CDC server, it only synchronized the collection creation operation. The data insertion was missing.

CDC logs

[WARN] [reader/target_client.go:99] ["fail to get collection info"] [error="can't find collection[database=default][collection=collection_demo]"] [errorVerbose="can't find collection[database=default][collection=collection_demo]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20241009111120-8008f14f2c16/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).DescribeCollection\n  | \t/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20241009111120-8008f14f2c16/client/collection.go:274\n  | github.com/zilliztech/milvus-cdc/core/reader.(*TargetClient).GetCollectionInfo.func1\n  | \t/app/core/reader/target_client.go:87\n  | github.com/zilliztech/milvus-cdc/core/reader.(*TargetClient).milvusOp\n  | \t/app/core/reader/target_client.go:68\n  | github.com/zilliztech/milvus-cdc/core/reader.(*TargetClient).GetCollectionInfo\n  | \t/app/core/reader/target_client.go:86\n  | github.com/zilliztech/milvus-cdc/core/reader.(*replicateChannelManager).startReadCollectionForMilvus.func1\n  | \t/app/core/reader/replicate_channel_manager.go:220\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg@v0.0.0-20241129095919-97f9205bd805/util/retry/retry.go:44\n  | github.com/zilliztech/milvus-cdc/core/reader.(*replicateChannelManager).startReadCollectionForMilvus\n  | \t/app/core/reader/replicate_channel_manager.go:219\n  | github.com/zilliztech/milvus-cdc/core/reader.(*replicateChannelManager).StartReadCollection\n  | \t/app/core/reader/replicate_channel_manager.go:315\n  | github.com/zilliztech/milvus-cdc/core/reader.(*CollectionReader).StartRead.func1.1\n  | \t/app/core/reader/collection_reader.go:122\n  | github.com/zilliztech/milvus-cdc/core/reader.(*EtcdOp).WatchCollection.func1.1.2.2\n  | \t/app/core/reader/etcd_op.go:277\n  | github.com/zilliztech/milvus-cdc/core/util.(*Map[...]).Range.func1\n  | \t/app/core/util/atomic.go:102\n  | sync.(*Map).Range\n  | \t/usr/local/go/src/sync/map.go:476\n  | github.com/zilliztech/milvus-cdc/core/util.(*Map[...]).Range\n  | \t/app/core/util/atomic.go:101\n  | github.com/zilliztech/milvus-cdc/core/reader.(*EtcdOp).WatchCollection.func1.1.2\n  | \t/app/core/reader/etcd_op.go:276\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/go/pkg/mod/github.com/!sim!f!g/milvus/pkg@v0.0.0-20241129095919-97f9205bd805/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.2/worker.go:67\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) can't find collection[database=default][collection=collection_demo]\nError types: (1) *withstack.withStack (2) *errutil.leafError"]

Then we cleaned everything for the source/target Milvus instances and CDC server(including the data), and re-deployed everything from scratch. The source MIlvus instance worked well. However, the CDC server even doesn't synchronize the collection creation operation this time.

CDC logs for the second time

maxNameLength: 256
logLevel: info
detectDeadLock: false
[2025/04/17 12:59:17.946 +00:00] [INFO] [paramtable/component_param.go:4433] ["DeployModeEnv is not set, use default"] [default=0.5]
[2025/04/17 12:59:17.949 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2025/04/17 12:59:17.949 +00:00] [INFO] [tag/tag.go:34] ["base info"] [BuildTime="Mon Dec  9 02:56:00 UTC 2024"] [GitCommit=369b60f] [GoVersion="go version go1.21.13 linux/amd64"]
[2025/04/17 12:59:17.990 +00:00] [INFO] [msgstream/mq_msgstream.go:121] ["Msg Stream state"] [can_produce=true]
[2025/04/17 12:59:18.142 +00:00] [INFO] [msgstream/mq_msgstream.go:211] ["Successfully create consumer"] [channel=milvus-replicate-msg] [subname=milvus-replicate-msg1747797053080405302]
[2025/04/17 12:59:18.142 +00:00] [INFO] [msgstream/mq_msgstream.go:221] ["start to close mq msg stream"] ["producer num"=0] ["consumer num"=1]
[2025/04/17 12:59:18.162 +00:00] [INFO] [server/server.go:56] ["start server..."]

The WebUIs of both Milvus instances show that their components are healthy. It seems that the behavior of the CDC is not as stable as expected. Below are our docker compose files and configs. Could you please help take a look? Thank you

Source Milvus docker-compose.yaml

services:
  # ZooKeeper Service
  zookeeper:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      - metadataStoreUrl=zk:zookeeper:2181
      - cluster-name=cluster-a
      - managedLedgerDefaultEnsembleSize=1
      - managedLedgerDefaultWriteQuorum=1
      - managedLedgerDefaultAckQuorum=1
    volumes:
      - /data/volumes/zookeeper:/pulsar/data/zookeeper
    command: >
      bash -c "bin/apply-config-from-env.py conf/zookeeper.conf &&
               bin/generate-zookeeper-config.sh conf/zookeeper.conf &&
               exec bin/pulsar zookeeper"
    healthcheck:
      test:
        ["CMD", "sh", "-c", "echo ruok | nc -w 2 127.0.0.1 2181 | grep imok"]
      interval: 5s
      timeout: 30s
      retries: 5
      start_period: 20s

  # One-shot Metadata Initialization Container
  initialize-pulsar-cluster-metadata:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: initialize-pulsar-cluster-metadata
    depends_on:
      zookeeper:
        condition: service_healthy
    command:
      [
        "bin/pulsar",
        "initialize-cluster-metadata",
        "--cluster",
        "cluster-a",
        "--zookeeper",
        "zookeeper:2181",
        "--configuration-store",
        "zookeeper:2181",
        "--web-service-url",
        "http://broker:8080",
        "--broker-service-url",
        "pulsar://broker:6650",
      ]
    restart: "no"
    # This container runs once and then exits.

  # Bookie Service
  bookie:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: bookie
    hostname: bookie
    depends_on:
      zookeeper:
        condition: service_healthy
      initialize-pulsar-cluster-metadata:
        condition: service_completed_successfully
    environment:
      - clusterName=cluster-a
      - zkServers=zookeeper:2181
      - metadataServiceUri=metadata-store:zk:zookeeper:2181
    volumes:
      - /data/volumes/bookkeeper:/pulsar/data/bookkeeper
    command: >
      bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf &&
               exec bin/pulsar bookie"
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f 'bin/pulsar bookie' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3

  # Broker Service
  broker:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: broker
    hostname: broker
    depends_on:
      zookeeper:
        condition: service_healthy
      initialize-pulsar-cluster-metadata:
        condition: service_completed_successfully
      bookie:
        condition: service_healthy
    ports:
      - "6650:6650" # Pulsar binary protocol
      - "8080:8080" # Pulsar Admin/API
    environment:
      - metadataStoreUrl=zk:zookeeper:2181
      - zookeeperServers=zookeeper:2181
      - clusterName=cluster-a
      - managedLedgerDefaultEnsembleSize=1
      - managedLedgerDefaultWriteQuorum=1
      - managedLedgerDefaultAckQuorum=1
    command: >
      bash -c "bin/apply-config-from-env.py conf/broker.conf &&
               exec bin/pulsar broker"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/admin/v2/clusters"]
      interval: 10s
      timeout: 5s
      retries: 5

  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    ports:
      - "2379:2379"
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - /data/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.5.8
    command: ["milvus", "run", "standalone"]
    security_opt:
      - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
    volumes:
      - /data/volumes/milvus-config/milvus.yaml:/milvus/configs/milvus.yaml
      - /data/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      broker:
        condition: service_healthy
      etcd:
        condition: service_healthy

networks:
  default:
    name: milvus

Source Milvus config milvus.yaml

etcd:
  endpoints: etcd:2379
  rootPath: by-dev
  metaSubPath: meta

  log:
    level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
    path: stdout
  ssl:
    enabled: false # Whether to support ETCD secure connection mode
    tlsCert: /path/to/etcd-client.pem # path to your cert file
    tlsKey: /path/to/etcd-client-key.pem # path to your key file
    tlsCACert: /path/to/ca.pem # path to your CACert file
    tlsMinVersion: 1.3
  requestTimeout: 10000 # Etcd operation timeout in milliseconds
  use:
    embed: false # Whether to enable embedded Etcd (an in-process EtcdServer).
  data:
    dir: default.etcd # Embedded Etcd only. please adjust in embedded Milvus: /tmp/milvus/etcdData/
  auth:
    enabled: false # Whether to enable authentication
    userName: # username for etcd authentication
    password: # password for etcd authentication

metastore:
  type: etcd # Default value: etcd, Valid values: [etcd, tikv]
  snapshot:
    ttl: 86400 # snapshot ttl in seconds
    reserveTime: 3600 # snapshot reserve time in seconds

localStorage:
  path: /var/lib/milvus/data/

minio:
  address: s3.us-east-2.amazonaws.com
  port: 443 # Port of MinIO or S3 service.
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: true # Switch value to control if to access the MinIO or S3 service through SSL.
  ssl:
    tlsCACert: /path/to/public.crt # path to your CACert file
  bucketName: milvus-data-staging
  rootPath: milvus-primary
  useIAM: true
  cloudProvider: aws
  gcpCredentialJSON:
  iamEndpoint:
  logLevel: fatal # Log level for aws sdk log. Supported level:  off, fatal, error, warn, info, debug, trace
  region: us-east-2 # Specify minio storage system location region
  useVirtualHost: false # Whether use virtual host mode for bucket
  requestTimeoutMs: 10000 # minio timeout for request time in milliseconds
  listObjectsMaxKeys: 0

mq:
  type: pulsar
  enablePursuitMode: true # Default value: "true"
  pursuitLag: 10 # time tick lag threshold to enter pursuit mode, in seconds
  pursuitBufferSize: 8388608 # pursuit mode buffer size in bytes
  pursuitBufferTime: 60 # pursuit mode buffer time in seconds
  mqBufSize: 16 # MQ client consumer buffer length
  dispatcher:
    mergeCheckInterval: 1 # the interval time(in seconds) for dispatcher to check whether to merge
    targetBufSize: 16 # the lenth of channel buffer for targe
    maxTolerantLag: 3 # Default value: "3", the timeout(in seconds) that target sends msgPack

pulsar:
  address: broker
  port: 6650 # Port of Pulsar service.
  webport: 8080 # Web port of of Pulsar service. If you connect direcly without proxy, should use 8080.
  maxMessageSize: 5242880
  tenant: public
  namespace: default # A Pulsar namespace is the administrative unit nomenclature within a tenant.
  requestTimeout: 60 # pulsar client global request timeout in seconds
  enableClientMetrics: false # Whether to register pulsar client metrics into milvus metrics path.


...


msgChannel:
  chanNamePrefix:
    cluster: by-dev
    rootCoordTimeTick: rootcoord-timetick
    rootCoordStatistics: rootcoord-statistics
    rootCoordDml: rootcoord-dml
    replicateMsg: replicate-msg
    queryTimeTick: queryTimeTick
    dataCoordTimeTick: datacoord-timetick-channel
    dataCoordSegmentInfo: segment-info-channel
  subNamePrefix:
    dataCoordSubNamePrefix: dataCoord
    dataNodeSubNamePrefix: dataNode

Target Milvus docker-compose.yaml

services:
  # ZooKeeper Service
  zookeeper:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      - metadataStoreUrl=zk:zookeeper:2181
      - cluster-name=cluster-a
      - managedLedgerDefaultEnsembleSize=1
      - managedLedgerDefaultWriteQuorum=1
      - managedLedgerDefaultAckQuorum=1
    volumes:
      - /data/volumes/zookeeper:/pulsar/data/zookeeper
    command: >
      bash -c "bin/apply-config-from-env.py conf/zookeeper.conf &&
               bin/generate-zookeeper-config.sh conf/zookeeper.conf &&
               exec bin/pulsar zookeeper"
    healthcheck:
      test:
        ["CMD", "sh", "-c", "echo ruok | nc -w 2 127.0.0.1 2181 | grep imok"]
      interval: 5s
      timeout: 30s
      retries: 5
      start_period: 20s

  # One-shot Metadata Initialization Container
  initialize-pulsar-cluster-metadata:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: initialize-pulsar-cluster-metadata
    depends_on:
      zookeeper:
        condition: service_healthy
    command:
      [
        "bin/pulsar",
        "initialize-cluster-metadata",
        "--cluster",
        "cluster-a",
        "--zookeeper",
        "zookeeper:2181",
        "--configuration-store",
        "zookeeper:2181",
        "--web-service-url",
        "http://broker:8080",
        "--broker-service-url",
        "pulsar://broker:6650",
      ]
    restart: "no"
    # This container runs once and then exits.

  # Bookie Service
  bookie:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: bookie
    hostname: bookie
    depends_on:
      zookeeper:
        condition: service_healthy
      initialize-pulsar-cluster-metadata:
        condition: service_completed_successfully
    environment:
      - clusterName=cluster-a
      - zkServers=zookeeper:2181
      - metadataServiceUri=metadata-store:zk:zookeeper:2181
    volumes:
      - /data/volumes/bookkeeper:/pulsar/data/bookkeeper
    command: >
      bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf &&
               exec bin/pulsar bookie"
    healthcheck:
      test: ["CMD-SHELL", "pgrep -f 'bin/pulsar bookie' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3

  # Broker Service
  broker:
    image: apachepulsar/pulsar-all:2.8.2
    container_name: broker
    hostname: broker
    depends_on:
      zookeeper:
        condition: service_healthy
      initialize-pulsar-cluster-metadata:
        condition: service_completed_successfully
      bookie:
        condition: service_healthy
    ports:
      - "6650:6650" # Pulsar binary protocol
      - "8080:8080" # Pulsar Admin/API
    environment:
      - metadataStoreUrl=zk:zookeeper:2181
      - zookeeperServers=zookeeper:2181
      - clusterName=cluster-a
      - managedLedgerDefaultEnsembleSize=1
      - managedLedgerDefaultWriteQuorum=1
      - managedLedgerDefaultAckQuorum=1
    command: >
      bash -c "bin/apply-config-from-env.py conf/broker.conf &&
               exec bin/pulsar broker"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/admin/v2/clusters"]
      interval: 10s
      timeout: 5s
      retries: 5

  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.18
    ports:
      - "2379:2379"
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - /data/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://etcd:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.5.8
    command: ["milvus", "run", "standalone"]
    security_opt:
      - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
    volumes:
      - /data/volumes/milvus-config/milvus.yaml:/milvus/configs/milvus.yaml
      - /data/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      broker:
        condition: service_healthy
      etcd:
        condition: service_healthy

networks:
  default:
    name: milvus

Target Milvus config milvus.yaml

etcd:
  endpoints: etcd:2379
  rootPath: by-dev
  metaSubPath: meta
  kvSubPath: kv
  log:
    level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
    path: stdout
  ssl:
    enabled: false # Whether to support ETCD secure connection mode
    tlsCert: /path/to/etcd-client.pem # path to your cert file
    tlsKey: /path/to/etcd-client-key.pem # path to your key file
    tlsCACert: /path/to/ca.pem # path to your CACert file
    tlsMinVersion: 1.3
  requestTimeout: 10000 # Etcd operation timeout in milliseconds
  use:
    embed: false # Whether to enable embedded Etcd (an in-process EtcdServer).
  data:
    dir: default.etcd # Embedded Etcd only. please adjust in embedded Milvus: /tmp/milvus/etcdData/
  auth:
    enabled: false # Whether to enable authentication
    userName: # username for etcd authentication
    password: # password for etcd authentication

metastore:
  type: etcd # Default value: etcd, Valid values: [etcd, tikv]
  snapshot:
    ttl: 86400 # snapshot ttl in seconds
    reserveTime: 3600 # snapshot reserve time in seconds

localStorage:
  path: /var/lib/milvus/data/

minio:
  address: s3.us-east-2.amazonaws.com
  port: 443 # Port of MinIO or S3 service.
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: true # Switch value to control if to access the MinIO or S3 service through SSL.
  ssl:
    tlsCACert: /path/to/public.crt # path to your CACert file
  bucketName: milvus-data-staging
  rootPath: milvus-replica
  useIAM: true
  cloudProvider: aws
  gcpCredentialJSON:
  iamEndpoint:
  logLevel: fatal # Log level for aws sdk log. Supported level:  off, fatal, error, warn, info, debug, trace
  region: us-east-2 # Specify minio storage system location region
  useVirtualHost: false # Whether use virtual host mode for bucket
  requestTimeoutMs: 10000 # minio timeout for request time in milliseconds
  listObjectsMaxKeys: 0

mq:
  type: pulsar
  enablePursuitMode: true # Default value: "true"
  pursuitLag: 10 # time tick lag threshold to enter pursuit mode, in seconds
  pursuitBufferSize: 8388608 # pursuit mode buffer size in bytes
  pursuitBufferTime: 60 # pursuit mode buffer time in seconds
  mqBufSize: 16 # MQ client consumer buffer length
  dispatcher:
    mergeCheckInterval: 1 # the interval time(in seconds) for dispatcher to check whether to merge
    targetBufSize: 16 # the lenth of channel buffer for targe
    maxTolerantLag: 3 # Default value: "3", the timeout(in seconds) that target sends msgPack

pulsar:
  address: broker
  port: 6650 # Port of Pulsar service.
  webport: 8080 # Web port of of Pulsar service. If you connect direcly without proxy, should use 8080.
  maxMessageSize: 5242880
  tenant: public
  namespace: default # A Pulsar namespace is the administrative unit nomenclature within a tenant.
  requestTimeout: 60 # pulsar client global request timeout in seconds
  enableClientMetrics: false # Whether to register pulsar client metrics into milvus metrics path.

...
Common:
  ttMsgEnabled: false
...

msgChannel:
  chanNamePrefix:
    cluster: by-dev
    rootCoordTimeTick: rootcoord-timetick
    rootCoordStatistics: rootcoord-statistics
    rootCoordDml: rootcoord-dml
    replicateMsg: replicate-msg
    queryTimeTick: queryTimeTick
    dataCoordTimeTick: datacoord-timetick-channel
    dataCoordSegmentInfo: segment-info-channel
  subNamePrefix:
    dataCoordSubNamePrefix: dataCoord
    dataNodeSubNamePrefix: dataNode

CDC docker-compose.yaml

services:
  mysql:
    image: mysql:9.3
    container_name: milvus-cdc-mysql
    environment:
      MYSQL_ROOT_PASSWORD: xxxx
      MYSQL_DATABASE: milvuscdc
    ports:
      - "3306:3306"
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
    healthcheck:
      test:
        ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-proot"]
      interval: 5s
      timeout: 5s
      retries: 5
    volumes:
      - /data/volumes/mysql:/var/lib/mysql

  milvus-cdc:
    container_name: milvus-cdc
    image: milvusdb/milvus-cdc:latest
    volumes:
      - /data/volumes/cdc-config/cdc.yaml:/app/configs/cdc.yaml
    ports:
      - "8086:8444"
    depends_on:
      mysql:
        condition: service_healthy

networks:
  default:
    name: milvus
    external: true

CDC config cdc.yaml

address: 0.0.0.0:8444
maxTaskNum: 100
metaStoreConfig:
  storeType: mysql
  mysqlSourceUrl: xxxx:xxxx@tcp(mysql:3306)/milvuscdc?charset=utf8
  rootPath: milvus
sourceConfig:
  etcd:
    address:
      - http://milvus-1.staging-private.internal.io:2379
    rootPath: milvus
    metaSubPath: meta
    enableAuth: false
    enableTLS: false
  readChanLen: 4
  defaultPartitionName: _default
  replicateChan: by-dev-replicate-msg
  pulsar:
    address: pulsar://milvus-1.staging-private.internal.io:6650
    webAddress: milvus-1.staging-private.internal.io:8080
    maxMessageSize: 5242880
    tenant: public
    namespace: default
maxNameLength: 256
logLevel: info
detectDeadLock: false

[xiaofan-luan]

yes, we recommended to use k8s for high availability.

The easiest way is to buy our managed cloud service, see https://zilliz.com/.

[utilbox]

Ah.. I see. We'll take the purchase into consideration once we have enough budget. When our data size reach some level, a purchase for managed service will be a natural thing.

May I know what kind of background knowledge is needed to play the CDC? I'm ready to learn anything necessary. I've read all the documents available in the community and the Milvus official site and followed the guide. But it seems not working. So I'm asking the community for help.

[yhmo]

[WARN] [reader/target_client.go:99] ["fail to get collection info"] [error="can't find collection[database=default][collection=collection_demo]"]

This error indicates the collection "collection_demo" doesn't exist in the source milvus.

[utilbox]

Thanks @yhmo. That's one reason why we thought it's wired. We saw the collection was synchronized to the replica on the WebUI of the replica. But the following inserted data was not. And when we redeployed everything again with the same Ansbile playbook and executed the same test code in python, no error logs were printed any more and no synchronization happened. 🤔

[SimFG]

It seems that there should be an error here. Generally speaking, if the collection can be synchronized normally, but the message is not synchronized, it is usually because there is a problem with the mq configuration. You can check whether the mq parameter in the source config of cdc.yaml is correct.