Skip to content

azure-eventhub consumer stops processing events after Received message 257:None is not WSMsgType.BINARY #43073

@erewok

Description

@erewok

Describe the bug
An Async Eventhub consumer completely stopped processing events after it hit this exception:

"EventProcessor instance '27933907-a7f5-4a14-8405-4d298e948c46' of eventhub 'bankupdates_v2' partition '0' consumer group '$default'. An error occurred while receiving. The exception is ConnectError(\"Can not read frame due to exception: Websocket disconnected: WSMessageTypeError('Received message 257:None is not WSMsgType.BINARY')\\nError condition: ErrorCondition.SocketError\\n Error Description: Can not read frame due to exception: Websocket disconnected: WSMessageTypeError('Received message 257:None is not WSMsgType.BINARY')\")."

We saw one more log message after this from our on_partition_error handler:

An exception: 0 occurred during receiving from Partition: Can not read frame due to exception: Websocket disconnected: WSMessageTypeError('Received message 257:None is not WSMsgType.BINARY')\nError condition: ErrorCondition.SocketError\n Error Description: Can not read frame due to exception: Websocket disconnected: WSMessageTypeError('Received message 257:None is not WSMsgType.BINARY').

And we saw this one from our on_partition_close handler:

"<CloseReason.SHUTDOWN: 0>"

To Reproduce
Steps to reproduce the behavior:

  1. good question!

Expected behavior
(no bugs forever ;) )

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
We found this related (seemingly similar?) issue for azure-servicebus, fixed earlier this year: #36280

For further context (some of which may be red herrings), I can offer the following:

  • This application uses Avro serialization for its events.
  • The Eventhub is a Standard SKU (with 1mb event size max)
  • The events themselves are large (we don't see this in our other apps with very small events). The publisher of these events sometimes sees rejections for sending events that are too large. I don't know if this is involved in this issue, but figured it was worth sharing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ClientThis issue points to a problem in the data-plane of the library.Event HubsService AttentionWorkflow: This issue is responsible by Azure service team.customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions