Deploiement kubernetes worker-subdefcreation en erreur #4576
Description
Bonjour,
Nous avons un problème de déploiement sur un cluster kube auto-hénergé.
Phraseanet est déployé en gitops (via ArgoCD) à partir des fichiers Helm de votre dépot.
L'application se déploie bien, mais le worker de création des sous-définitions ne fonctionne pas.
Il semble avoir un problème avec RabittMq.
Voici un extrait du values.yaml pour les workers déployés.
worker:
.....
launch:
monoworker: false
records: true
subdefcreation: true
writemetadatas: true
exportemail: false
exportftp: false
ps: false
noscalable: false
cmd: false
scheduler: false
Le worker phraseanet-worker-subdefcreation donne les logs suivants:
2025-01-14 14:10:01 - chown app:app on cache/ repository
2025-01-14 14:10:01 - Newrelic extension deactivation.
Multiples Phraseanet workers will be launched with bin/console worker:execute
Worker subdefCreation defined with parallelism 1
1 workers defined
tail: cannot open '/var/alchemy/Phraseanet/logs/worker_service.log' for reading: No such file or directory
[PhpAmqpLib\Exception\AMQPConnectionClosedException]
INTERNAL_ERROR(0, 0)
worker:execute [-p|--preserve-payload] [--queue-name QUEUE-NAME] [-m|--max-processes MAX-PROCESSES] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [--] <command>
2025-01-14 14:10:02 - 0 running workers
One or more worker:execute is not running, exiting...
Voici un log de RabittMq (désolé c'est un peu long)
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ** Generic server <0.28695.0> terminating
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ** Last message in was {'$gen_cast',
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {queue_event,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {deliver,<<"e806b9dd-2fcb-4ec5-9025-be9c7e52a285">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> true,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{{resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <0.974.0>,35,true,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {mc,mc_amqpl,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {content,60,none,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<176,0,16,97,112,112,108,105,99,97,116,105,111,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 110,47,106,115,111,110,0,0,0,196,7,120,45,100,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 101,97,116,104,65,0,0,0,137,70,0,0,0,132,5,99,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 111,117,110,116,73,0,0,0,2,8,101,120,99,104,97,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 110,103,101,83,0,0,0,16,97,108,99,104,101,109,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 121,45,101,120,99,104,97,110,103,101,5,113,117,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 101,117,101,83,0,0,0,14,115,117,98,100,101,102,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 67,114,101,97,116,105,111,110,12,114,111,117,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 116,105,110,103,45,107,101,121,115,83,0,0,0,14,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 115,117,98,100,101,102,67,114,101,97,116,105,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 111,110,6,114,101,97,115,111,110,83,0,0,0,8,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 114,101,106,101,99,116,101,100,4,116,105,109,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 101,84,0,0,0,0,103,134,63,19,14,119,111,114,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 107,101,114,45,109,101,115,115,97,103,101,83,0,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 0,0,26,83,117,98,100,101,102,32,103,101,110,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 101,114,97,116,105,111,110,32,102,97,105,108,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 101,100,32,33,2>>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> rabbit_framing_amqp_0_9_1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [<<"{\"message_type\":\"subdefCreation\",\"payload\":{\"recordId\":5,\"databoxId\":1,\"subdefName\":\"thumbnail\",\"status\":\"\",\"workerJobId\":\"84\",\"published\":1736851219}}">>]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{id =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<22,131,147,17,122,156,83,194,190,155,212,12,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 200,138,178,158>>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ttl => undefined,x => <<"alchemy-exchange">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> rk => [<<"subdefCreation">>],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> deaths =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{{<<"subdefCreation_retry">>,expired},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {death,<<"retry-alchemy-exchange">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [<<"subdefCreation_retry">>],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> 1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{first_time => 1736851229692,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> last_time => 1736851229692}}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {{<<"subdefCreation">>,rejected},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {death,<<"alchemy-exchange">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"subdefCreation">>,2,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{first_time => 1736851219000,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> last_time => 1736851219000}}}],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> rts => 1736851219691,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"x-last-death-exchange">> =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"retry-alchemy-exchange">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"x-last-death-queue">> =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"subdefCreation_retry">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"x-last-death-reason">> =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"expired">>}}}]}}}
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ** When Server state == {ch,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {conf,running,rabbit_framing_amqp_0_9_1,1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <0.28678.0>,<0.28693.0>,<0.28678.0>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"10.42.171.197:46580 -> 10.42.171.229:5672">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> undefined,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {user,<<"rabbit">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [administrator],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{rabbit_auth_backend_internal,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #Fun<rabbit_auth_backend_internal.3.26105038>}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"/">>,<<"subdefCreation_retry">>,<0.28682.0>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{<<"authentication_failure_close">>,bool,true},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"publisher_confirms">>,bool,true},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"consumer_cancel_notify">>,bool,true},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"exchange_exchange_bindings">>,bool,true},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"basic.nack">>,bool,true},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"connection.blocked">>,bool,true}],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> none,1,134217728,1800000,#{},infinity,1000000000,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> false},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {lstate,<0.28694.0>,false},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> none,1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {0,[],[]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {state,#{},erlang},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{<<"e806b9dd-2fcb-4ec5-9025-be9c7e52a285">> =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {{amqqueue,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> true,false,none,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{<<"x-dead-letter-exchange">>,longstr,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"retry-alchemy-exchange">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"x-dead-letter-routing-key">>,longstr,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <<"subdefCreation_retry">>}],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> <0.974.0>,[],[],[],undefined,undefined,[],[],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> live,0,[],<<"/">>,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{user => <<"rabbit">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> rabbit_classic_queue,#{}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {false,1,false,[]}}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{{resource,<<"/">>,queue,<<"subdefCreation">>} =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {<<"e806b9dd-2fcb-4ec5-9025-be9c7e52a285">>,nil,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> nil}}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {state,fine,5000,#Ref<0.4142020336.1529610241.56622>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> false,1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {rabbit_confirms,undefined,#{}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [],[],none,flow,[],
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {rabbit_queue_type,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{{resource,<<"/">>,queue,<<"subdefCreation">>} =>
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {ctx,rabbit_classic_queue,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {rabbit_classic_queue,<0.974.0>,#{},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #{<0.974.0> => ok}}}}},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> #Ref<0.4142020336.1529610241.56618>,false}
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ** Reason for termination ==
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> ** {{bad_generator,<<"subdefCreation">>},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{mc_amqpl,'-death_table/7-lc$^0/1-0-',1,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{file,"mc_amqpl.erl"},{line,814}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {mc_amqpl,death_table,7,[{file,"mc_amqpl.erl"},{line,814}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {lists,map_1,2,[{file,"lists.erl"},{line,1564}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {lists,map,2,[{file,"lists.erl"},{line,1559}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {mc_amqpl,deaths_to_headers,2,[{file,"mc_amqpl.erl"},{line,770}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {mc_amqpl,protocol_state,2,[{file,"mc_amqpl.erl"},{line,407}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {rabbit_channel,handle_deliver0,4,
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> [{file,"rabbit_channel.erl"},{line,2697}]},
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0> {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}]}
2025-01-14 14:10:01.676073+00:00 [error] <0.28695.0>
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> crasher:
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> initial call: rabbit_channel:init/1
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> pid: <0.28695.0>
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> registered_name: []
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> exception exit: {{bad_generator,<<"subdefCreation">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{mc_amqpl,'-death_table/7-lc$^0/1-0-',1,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{file,"mc_amqpl.erl"},{line,814}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {mc_amqpl,death_table,7,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{file,"mc_amqpl.erl"},{line,814}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {lists,map_1,2,[{file,"lists.erl"},{line,1564}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {lists,map,2,[{file,"lists.erl"},{line,1559}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {mc_amqpl,deaths_to_headers,2,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{file,"mc_amqpl.erl"},{line,770}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {mc_amqpl,protocol_state,2,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{file,"mc_amqpl.erl"},{line,407}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {rabbit_channel,handle_deliver0,4,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{file,"rabbit_channel.erl"},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {line,2697}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {lists,foldl,3,[{file,"lists.erl"},{line,1594}]}]}
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> in function gen_server2:terminate/3 (gen_server2.erl, line 1172)
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> ancestors: [<0.28692.0>,<0.28689.0>,<0.28677.0>,<0.28676.0>,<0.834.0>,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> <0.833.0>,<0.832.0>,<0.830.0>,<0.829.0>,rabbit_sup,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> <0.254.0>]
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> message_queue_len: 0
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> messages: []
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> links: [<0.28692.0>]
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> dictionary: [{permission_cache,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [{{resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},read},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,exchange,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> <<"retry-alchemy-exchange">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},read},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,queue,<<"subdefCreation_retry">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},write},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,queue,<<"subdefCreation_retry">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},configure},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,exchange,<<"alchemy-exchange">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},read},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},write},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,queue,<<"subdefCreation">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},configure},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,exchange,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> <<"retry-alchemy-exchange">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},configure},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{resource,<<"/">>,exchange,<<"alchemy-exchange">>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> #{},configure}]},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {permission_cache_can_expire,false},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {channel_operation_timeout,15000},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {{xtype_to_module,direct},rabbit_exchange_type_direct},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {rand_seed,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {#{max => 288230376151711743,type => exsplus,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> next => #Fun<rand.5.65977474>,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> jump => #Fun<rand.3.65977474>},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> [218724849696446143|285489977274797483]}},
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {process_name,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {rabbit_channel,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> {<<"10.42.171.197:46580 -> 10.42.171.229:5672">>,
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> 1}}}]
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> trap_exit: true
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> status: running
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> heap_size: 46422
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> stack_size: 28
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> reductions: 95072
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0> neighbours:
2025-01-14 14:10:01.678473+00:00 [error] <0.28695.0>
2025-01-14 14:10:01.680162+00:00 [error] <0.28678.0> Error on AMQP connection <0.28678.0> (10.42.171.197:46580 -> 10.42.171.229:5672, vhost: '/', user: 'rabbit', state: running), channel 1:
A noter que la version avec TaskManager ne fonctionne pas non plus, une erreur SQL.
Phraseanet version 4.1.9
Bien cordialement