-
|
About "is being rate limited" Feb 24 20:23:18 test-server vector[25183]: 2023-02-24T12:23:18.555798Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=25}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being rate limited. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 6 replies
-
|
Hi @mlonV ,
In your case, the |
Beta Was this translation helpful? Give feedback.
-
|
Ah, yes, we recommend having Loki accept out-of-order writes. It should be the default behavior on newer versions of Loki. |
Beta Was this translation helpful? Give feedback.
-
|
Although out-of-order writes have been set, the restriction is still received 2023-02-27T10:48:14.799450Z INFO source{component_kind="source" component_id=victory_phpapp component_type=file component_name=victory_phpapp}:file_server: vector::internal_events::file::source: Found new file to watch. file=/data/logs/app_log-app-2023-02-27-00.log |
Beta Was this translation helpful? Give feedback.
-
|
Hi bro |
Beta Was this translation helpful? Give feedback.
-
|
It surprises me that Loki wouldn't log any errors when it returns a 400. I'd double check if there are additional logging parameters on the Loki side that could show more detail. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much. I think I also need to find out whether it is loki's problem |
Beta Was this translation helpful? Give feedback.
-
|
Maybe it's my loki's limitation or rpc error, but I don't know what's wrong level=warn ts=2023-02-27T02:37:57.942100344Z caller=grpc_logging.go:43 method=/logproto.Pusher/Push duration=1.148304ms err="rpc error: code = Code(400) desc = entry with timestamp 2023-02-26 16:30:04 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:04 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:04 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:04 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\nentry with timestamp 2023-02-26 16:30:05 +0000 UTC ignored, reason: 'entry out of order' for stream: {hostname="test-server", level="DEBUG", service="victory.phpapp_dev"},\ntotal ignored: 789 out of 789" msg=gRPC |
Beta Was this translation helpful? Give feedback.
Hi @mlonV ,
is being rate limitedis just talking about Vector's internal logs that are being written to stderr. Vector is not rate limiting the stream of events flowing through it.In your case, the
lokisink is failing to send at least some events to Loki though. You can see the HTTP 400 responses there. I would check your Loki logs to see if indicates why the requests are failing.