Redis Sentinel

[modzilla99]

Hey there,

I'm in the process of making Nebox highly available, but I am not able to get Redis Sentinel to work. It always complains about No master found for 'mymaster'. I think I don't exactly understand whats going on and how redis handles replication, but I think I've tried everything... Also shouldn't there be 2 databases for tasks and caching?

I really hope that someone's got it working and could help me. This is my configuration. I deploy Netbox on Kubernetes:

REDIS = {
    'tasks': {
        'SENTINELS': [
            ('netbox-redis-node-0.netbox-redis-headless', 26379),
            ('netbox-redis-node-1.netbox-redis-headless', 26379),
            ('netbox-redis-node-2.netbox-redis-headless', 26379)
        ],
        'SENTINEL_SERVICE': 'mymaster',
        'SENTINEL_TIMEOUT': 300,
        'DATABASE': 0,
        'SSL': False,
        'INSECURE_SKIP_TLS_VERIFY': False
    },
    'caching': {
        'SENTINELS': [
            ('netbox-redis-node-0.netbox-redis-headless', 26379),
            ('netbox-redis-node-1.netbox-redis-headless', 26379),
            ('netbox-redis-node-2.netbox-redis-headless', 26379)
        ],
        'SENTINEL_SERVICE': 'mymaster',
        'SENTINEL_TIMEOUT': 300,
        'DATABASE': 1,
        'SSL': False,
        'INSECURE_SKIP_TLS_VERIFY': False
    }
}

Redis nodes hostnames + fqdn:

netbox-redis-node-0 netbox-redis-node-0.netbox-redis-headless.netbox.svc.cluster.local
netbox-redis-node-1 netbox-redis-node-1.netbox-redis-headless.netbox.svc.cluster.local
netbox-redis-node-2 netbox-redis-node-2.netbox-redis-headless.netbox.svc.cluster.local

Redis-cli output on redis-node-1:

modzilla@jumphost:/opt/deployments/netbox$ kubectl exec -it netbox-redis-node-1 -- redis-cli --askpass -p 26379
Defaulted container "redis" out of: redis, sentinel, metrics
Please input password: **********
127.0.0.1:26379> sentinel masters
1)  1) "name"
    2) "mymaster"
    3) "ip"
    4) "netbox-redis-node-2.netbox-redis-headless.netbox.svc.cluster.local"
    5) "port"
    6) "6379"
    7) "runid"
    8) "3d9fdaacd02d57c5c01ccffbd52df760e14731d0"
    9) "flags"
   10) "master"
   11) "link-pending-commands"
   12) "0"
   13) "link-refcount"
   14) "1"
   15) "last-ping-sent"
   16) "0"
   17) "last-ok-ping-reply"
   18) "977"
   19) "last-ping-reply"
   20) "977"
   21) "down-after-milliseconds"
   22) "60000"
   23) "info-refresh"
   24) "7030"
   25) "role-reported"
   26) "master"
   27) "role-reported-time"
   28) "2968700"
   29) "config-epoch"
   30) "1"
   31) "num-slaves"
   32) "2"
   33) "num-other-sentinels"
   34) "2"
   35) "quorum"
   36) "2"
   37) "failover-timeout"
   38) "18000"
   39) "parallel-syncs"
   40) "1"
127.0.0.1:26379>
modzilla99@jumphost:/opt/deployments/netbox$ kubectl exec -it netbox-redis-node-1 -- redis-cli --askpass -p 6379
Defaulted container "redis" out of: redis, sentinel, metrics
Please input password: **********
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:netbox-redis-node-2.netbox-redis-headless.netbox.svc.cluster.local
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:33640910
slave_repl_offset:33640910
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:cf39dc9fd74c923c3d64c24734cc39d51c09adc2
master_replid2:234ad4d74caeee0158a41725c71d739b330bb4f1
master_repl_offset:33640910
second_repl_offset:32500281
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:32592335
repl_backlog_histlen:1048576
127.0.0.1:6379> info kespace
127.0.0.1:6379> info keyspace
# Keyspace
db0:keys=3,expires=0,avg_ttl=0
127.0.0.1:6379> client list
id=5 addr=10.244.121.26:6379 laddr=10.244.111.96:47956 fd=11 name= age=3182 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=(superuser) redir=-1
id=68 addr=10.244.121.26:48274 laddr=10.244.111.96:6379 fd=10 name=sentinel-554a1fb1-cmd age=3054 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
id=69 addr=10.244.121.26:48276 laddr=10.244.111.96:6379 fd=12 name=sentinel-554a1fb1-pubsub age=3054 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=70 addr=10.244.82.52:35248 laddr=10.244.111.96:6379 fd=13 name=sentinel-b60aa088-cmd age=3053 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
id=71 addr=10.244.82.52:35250 laddr=10.244.111.96:6379 fd=14 name=sentinel-b60aa088-pubsub age=3053 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=73 addr=10.244.111.96:57830 laddr=10.244.111.96:6379 fd=9 name=sentinel-25e38adc-cmd age=3053 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=publish user=default redir=-1
id=74 addr=10.244.111.96:57832 laddr=10.244.111.96:6379 fd=15 name=sentinel-25e38adc-pubsub age=3053 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=1282 addr=127.0.0.1:35096 laddr=127.0.0.1:6379 fd=16 name= age=45 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=40928 argv-mem=10 obl=0 oll=0 omem=0 tot-mem=61466 events=r cmd=client user=default redir=-1

[modzilla99]

No1? :/

[candlerb]

You asked the same question in July and there was no answer, so I think you can take it as read that nobody else is using Redis clustering with Netbox.

[modzilla99]

That wasn't me, but yeah seems like it

[smutel]

I am using redis sentinel with Netbox and it works correctly except for #7770.
I didn't see the PASSWORD field in your configuration but I see in the redis-cli command that you use a password ...

Please see my redis config below:

REDIS = {
    'tasks': {
        'SENTINELS': [
            ('xx.xx.xx.xx', yyyy),
            ('xx.xx.xx.xx', yyyy),
            ('xx.xx.xx.xx', yyyy),
        ],
        'SENTINEL_SERVICE': 'master0',
        'SENTINEL_TIMEOUT': 1800,
        'PASSWORD': 'xxxxxx',
        'DATABASE': 0,
        'SSL': False,
    },
    'caching': {
        'SENTINELS': [
            ('xx.xx.xx.xx', yyyy),
            ('xx.xx.xx.xx', yyyy),
            ('xx.xx.xx.xx', yyyy),
        ],
        'SENTINEL_SERVICE': 'master0',
        'SENTINEL_TIMEOUT': 1800,
        'PASSWORD': 'xxxxxx',
        'DATABASE': 1,
        'SSL': False,
    }
}