[python] [Feature] Add flexibility to 'peer' and 'op' label when using sw_rabbitmq and sw_requests

[dcryans]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I would like to add options on how to name 'op' and 'peer' when using some of the plugins.

I have already written some code to address this feature and would like to make them part of the product for anyone to use.

Default would still be exactly the same as it is right now, but it would be possible to change the reporting behavior of the plugins for specific needs/services.

Right now, sw_rabbitmq uses:

• peer => host:port
• op => RabbitMQ/Topic/' + exchange + '/Queue/' + routing_key

I would like to add 2 env. variables to enable this change ( 1 bool + 1 label (Default RabbitMQ)):

• peer => {env.var.label}: {exchange}: {routing_key}

Same goes for sw_requests:

• peer => netloc
• op => path or "/"

With 3 env. var (1 peer bool, 1 peer label (Default Web Request), 1 op bool):

• peer => {label}
• op => netloc

Use case

RabbitMQ:
container 1 -> queue 1 -> container 2 -> queue 2 -> container 3

instead of:
service 1 <-> RabbitMQ
service 2 <-> RabbitMQ
service 3 <- RabbitMQ

Requests:
One of our containers calls different urls on the web to get additional information on the data being processed.
This creates a crazy topology pointing to each and every webpage being read (tons of them)

We'd rather see:

• service 1 -> Web Request

Instead of a storm like this:

• service 1 -> url1, url2, url3, url4, url5..., url1000

Plus, in traces, we prefer seeing which websites are slow (netloc) than their precise path.

Related issues

There could be more of this coming in later as we start using Skywalking for all our services.

Just a question, I know this could be a big endeavor / change:

• Did you plan on having some control over this type of labeling problem from the OAP Server instead of changing every plugin 1 by 1?
• I've used AppDynamics and Dynatrace in the past and I really appreciated the possibility to change labels even after the applications went up. it gave APM people much faster results when they didn't have access to the service's configurations or restarts had to be approved.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

I am transferring this to the discussion, the feature request is about a solid solution.

[wu-sheng]

I am a little confused about your case. First of all, the peer is a very important concept in the SkyWalking agent.
Read the paper if you want to get the design, https://wu-sheng.github.io/STAM/

peer => {env.var.label}: {exchange}: {routing_key}

Peer like this is super wrong, peer is a network identifier to define a server or a cluster.

[wu-sheng]

What you proposed seems hard to follow in the SkyWalking context.

I've used AppDynamics and Dynatrace in the past and I really appreciated the possibility to change labels even after the applications went up. it gave APM people much faster results when they didn't have access to the service's configurations or restarts had to be approved.

I think I mentioned, don't mention vendors in a tech discussion. What any vendor did is belonging to their design, it is their intellectual property. We have no intent and definitely should not copy. Let's focus on what is the issue, what you faced is Python agent could be found and resolved by other agents. Focus please.

[dcryans]

Sorry about that

[dcryans]

I will read through the provided link and come back with more 'skywalking-centric' feature.

[dcryans]

@wu-sheng
Ok, focusing on rabbitmq only:

I went through some Java agent plugins + the STAM paper
I understand that for many synchronized requests, using host:port, then matching it to service can work wonders to avoid reprocessing the same data.

In the case of async queues, added to the fact that RabbitMQ itself isn't instrumented with Skywalking, it creates a problem.
The producer doesn't know who the consumer is in advance.
There can be multiple different services that produce messages to that single consumer service.

Using standard Skywalking STAM implementation, the result is that all containers are linked to the RabbitMQ server, regardless of which Queue is being used.
This leaves us without any way to link the containers that communicate together through MQ.

I hope this explains more clearly the issue I'm trying to resolve.

[wu-sheng]

The producer doesn't know who the consumer is in advance.

It does from the trace. What do you mean it doesn't?

There can be multiple different services that produce messages to that single consumer service.

Yes, this is a common case.

Using standard Skywalking STAM implementation, the result is that all containers are linked to the RabbitMQ server, regardless of which Queue is being used.

The endpoint dependency is designed for showing URI to URI level dependency. Service topology doesn't care the case. You could think one RPC service could be called by different services, and you can't tell which URI(s) are being called by which service. Same case in the MQ, nothing special.

[dcryans]

Thank you for your time, this is appreciated.

Currently, traces & endpoint dependencies only link 1 exitspan with 1 entryspan, then it cuts.

I have looked at traces and endpoint dependencies + debugged the agent a little
It seems the python code loses it's spans and returns an empty context at some point, which restarts links between services.
I will continue investigating until I understand better and come back with more information.

[dcryans]

@wu-sheng
Thank you again for pointing to all of this information.
I dived into the code + documentation <
The tracing issue was simple after all.
MQ's instrumentation is like this

Entry (deliver):

• span linked to previous service span
• this span is linked with the exitSpan from previous service

Run code on message (outside of an Entry span)

Exit (publish):

• this is a new span
• this span will be linked with entry span of next service

[dcryans]

@wu-sheng
I managed to get traces working with async, traces now show service 1 through service x, no problem

But when looking at Service 3 Endpoint Dependency:

• it doesn't go farther than 'Service 2 Exit -> Service 3 Entry' (1 depth)

Do you agree I can create a 'bug' issue for the problem I will describe below?

• See attached image showing my findings

There is a small bug in server-oap for endpoint "getEndpointDependencies":

• Since we hide RabbitMQ, the depth#2 oap returns to the UI is 'Service 2: Q3 Producer'
• that is an Exit, not Entry
• it is actually an Entry for RabbitMQ

Idea#1 for a solution:

• The right entry point would be Service 2: Q2 Consumer
• Pro: Would get complete dependencies
• Con: Q3: Producer would not be shown on screen (not so bad)

Idea#2

• not to hiding Rabbit MQ from endpoint Dependency
• Pro: But both consumers and producers would be shown on screen.
• Con: it would limit the maximum depth

[wu-sheng]

First, I want to br clear, endpoint dependency is not exit span to entry span. Endpoint dependency describes the relationship of the entry spans in the distributed call chain.
So, in MQ case, only consumer will shoe up in this dependency. The exit and entry relationship is already showing in the trace, as well as service relationship, and there is MQ consuming latency metrics in the latest versions.

[dcryans]

@wu-sheng
Thank you, this explains what I was seeing

Do you agree that Endpoint Dependency should be fixed for a case like mine?

Currently, in my case, it shows Exit to Entry

[dcryans]

well, to be fair, it is an Entry that is showing up, but an Entry to RabbitMQ, un uninstrumented node

Since the node showing up is 'Service 2', we shoud be seeing it's Entry, right?

[wu-sheng]

Since we hide RabbitMQ, the depth#2 oap returns to the UI is 'Service 2: Q3 Producer'

I am not sure what makes this happen, but producer usually doesn't show up in the endpoint dependency, unless it is a only span in the segments. Such as if you are using a timer(w/o instrumentation) to produce messages, then it is expected to see this.

[dcryans]

Thank you!
So it's about pika's IOloop
I guess I'll have to do without endpoint dependency for my use case

[dcryans]

@wu-sheng
Now that RabbitMQ point is closed.

The second point was 'sw_requests'
We have a web scraper type container, it calls a lot of different netlocs that are external
We don't really want to see these in Topology, there is no value for us.

Is there a preferred way to do that?

I could disable the plugin, but then I would be missing the Exit call with it's 'op' (URI)
That information can be very interesting in traces if some websites answer slowly, which slows down the web scraper.

Thank you!

[wu-sheng]

Right now, I think all outgoing calls, no matter external or internal, are showing on that topology. Generally, because we don't know what is external from the system perspective.