about AskTimeoutException

[fix-all]

hi,

The Gateway service calls the Things service and reports an error "org.eclipse.ditto.base.model.exceptions.AskException: There was an internal timeout." The specific error is as follows. I am using Ditto version 3.1.2. How should I troubleshoot this problem?

2024-05-09 10:48:45,775 WARN  [55ee31e8-45c2-4bcf-99dc-18c5df464416] o.e.d.i.u.c.AskWithRetry  - Got AskTimeout during ask for message <ModifyThing> - retrying.. : <Ask timed out on [Actor[akka://ditto-cluster/system/sharding/thingProxy#-833199036]] after [3000 ms]. Message of type [org.eclipse.ditto.things.model.signals.commands.modify.ModifyThing]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.>
2024-05-09 10:48:49,035 WARN  [55ee31e8-45c2-4bcf-99dc-18c5df464416] o.e.d.i.u.c.AskWithRetry  - Got AskTimeout during ask for message <ModifyThing> - retrying.. : <Ask timed out on [Actor[akka://ditto-cluster/system/sharding/thingProxy#-833199036]] after [3000 ms]. Message of type [org.eclipse.ditto.things.model.signals.commands.modify.ModifyThing]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.>
2024-05-09 10:48:51,495 ERROR [55ee31e8-45c2-4bcf-99dc-18c5df464416] o.e.d.e.s.d.AskWithRetryCommandForwarder  - Encountered timeout in edge forwarding
org.eclipse.ditto.base.model.exceptions.AskException: There was an internal timeout.
	at org.eclipse.ditto.base.model.exceptions.AskException$Builder.doBuild(AskException.java:120)
	at org.eclipse.ditto.base.model.exceptions.AskException$Builder.doBuild(AskException.java:105)
	at org.eclipse.ditto.base.model.exceptions.DittoRuntimeExceptionBuilder.build(DittoRuntimeExceptionBuilder.java:176)
	at org.eclipse.ditto.base.model.exceptions.DittoRuntimeException.fromJson(DittoRuntimeException.java:245)
	at org.eclipse.ditto.base.model.exceptions.AskException.fromJson(AskException.java:88)
	at jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.ditto.base.model.signals.AnnotationBasedJsonParsable.parse(AnnotationBasedJsonParsable.java:94)
2024-05-09 10:48:51,495 ERROR [55ee31e8-45c2-4bcf-99dc-18c5df464416] o.e.d.e.s.d.AskWithRetryCommandForwarder  - Encountered timeout in edge forwarding
org.eclipse.ditto.base.model.exceptions.AskException: There was an internal timeout.
	at org.eclipse.ditto.base.model.exceptions.AskException$Builder.doBuild(AskException.java:120)
	at org.eclipse.ditto.base.model.exceptions.AskException$Builder.doBuild(AskException.java:105)
	at org.eclipse.ditto.base.model.exceptions.DittoRuntimeExceptionBuilder.build(DittoRuntimeExceptionBuilder.java:176)
	at org.eclipse.ditto.base.model.exceptions.DittoRuntimeException.fromJson(DittoRuntimeException.java:245)
	at org.eclipse.ditto.base.model.exceptions.AskException.fromJson(AskException.java:88)
	at jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.ditto.base.model.signals.AnnotationBasedJsonParsable.parse(AnnotationBasedJsonParsable.java:94)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.fromJson(AbstractJsonParsableRegistry.java:97)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.lambda$parse$0(AbstractJsonParsableRegistry.java:79)
	at org.eclipse.ditto.base.model.exceptions.DittoJsonException.wrapJsonRuntimeException(DittoJsonException.java:158)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.parse(AbstractJsonParsableRegistry.java:78)
	at org.eclipse.ditto.base.model.signals.GlobalErrorRegistry.parse(GlobalErrorRegistry.java:91)
	at org.eclipse.ditto.base.model.signals.GlobalErrorRegistry.parse(GlobalErrorRegistry.java:36)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.deserializeJson(AbstractJsonifiableWithDittoHeadersSerializer.java:423)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.createJsonifiableFrom(AbstractJsonifiableWithDittoHeadersSerializer.java:325)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.tryToCreateKnownJsonifiableFrom(AbstractJsonifiableWithDittoHeadersSerializer.java:296)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.fromBinary(AbstractJsonifiableWithDittoHeadersSerializer.java:273)
	at akka.serialization.Serialization.deserializeByteBuffer(Serialization.scala:266)
	at akka.remote.MessageSerializer$.deserializeForArtery(MessageSerializer.scala:101)
	at akka.remote.artery.Deserializer$$anon$3.onPush(Codecs.scala:668)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:542)
	at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:496)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:650)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:521)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.fromJson(AbstractJsonParsableRegistry.java:97)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.lambda$parse$0(AbstractJsonParsableRegistry.java:79)
	at org.eclipse.ditto.base.model.exceptions.DittoJsonException.wrapJsonRuntimeException(DittoJsonException.java:158)
	at org.eclipse.ditto.base.model.signals.AbstractJsonParsableRegistry.parse(AbstractJsonParsableRegistry.java:78)
	at org.eclipse.ditto.base.model.signals.GlobalErrorRegistry.parse(GlobalErrorRegistry.java:91)
	at org.eclipse.ditto.base.model.signals.GlobalErrorRegistry.parse(GlobalErrorRegistry.java:36)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.deserializeJson(AbstractJsonifiableWithDittoHeadersSerializer.java:423)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.createJsonifiableFrom(AbstractJsonifiableWithDittoHeadersSerializer.java:325)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.tryToCreateKnownJsonifiableFrom(AbstractJsonifiableWithDittoHeadersSerializer.java:296)
	at org.eclipse.ditto.internal.utils.cluster.AbstractJsonifiableWithDittoHeadersSerializer.fromBinary(AbstractJsonifiableWithDittoHeadersSerializer.java:273)
	at akka.serialization.Serialization.deserializeByteBuffer(Serialization.scala:266)
	at akka.remote.MessageSerializer$.deserializeForArtery(MessageSerializer.scala:101)
	at akka.remote.artery.Deserializer$$anon$3.onPush(Codecs.scala:668)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:542)
	at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:496)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:650)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:521)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
	at akka.actor.ActorCell.invoke(ActorCell.scala:547)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
2024-05-09 10:48:51,497 INFO  [55ee31e8-45c2-4bcf-99dc-18c5df464416] o.e.d.e.s.a.AcknowledgementAggregatorActor akka://ditto-cluster/user/gatewayRoot/streaming/0-ede7ff16-5095-436b-baed-d679312f4cb1/ackr966-55ee31e8-45c2-4bcf-99dc-18c5df464416 - Stopped waiting for acknowledgements because of ditto runtime exception <EdgeServiceTimeoutException [message='The service you wanted to reach is currently not responding.', errorCode=service.timeout, httpStatus=HttpStatus [code=504, category=SERVER_ERROR], description='Please contact the service team if this happens again.', href=null, dittoHeaders=ImmutableDittoHeaders [{ditto-origin=ede7ff16-5095-436b-baed-d679312f4cb1, ditto-auth-context={"type":"pre-authenticated-http","subjects":["nginx:ditto"]}, declared-acks=[], correlation-id=55ee31e8-45c2-4bcf-99dc-18c5df464416, version=2, response-required=true, if-match=*, content-type=application/json, ditto-entity-id=thing:openRoad:openRoad214873094821888, requested-acks=["twin-persisted"], ditto-ackgregator-address=akka://ditto-cluster@10.42.35.63:2551/user/gatewayRoot/streaming/0-ede7ff16-5095-436b-baed-d679312f4cb1/ackr966-55ee31e8-45c2-4bcf-99dc-18c5df464416#-689386565, ditto-originator=nginx:ditto}]]>.
2024-05-09 10:48:54,659 INFO  [5fd962a3-550f-4673-879c-016797d8235d] o.e.d.g.s.s.a.p.PreAuthenticatedAuthenticationProvider  - Pre-authentication has been applied resulting in AuthorizationContext <ImmutableAuthorizationContext [type=pre-authenticated-http, authorizationSubjects=[nginx:ditto]]>.
```txt

[thjaeckle]

I would suggest to update to a supported version of Ditto and check if all the services are running or if some eg crashed, eg due to too less resources.