[receiver/pprof] attempt to convert from pprof to pdata and back #40548
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package internal // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/pprofreceiver/internal" | ||
|
|
||
| import ( | ||
| "crypto/md5" //nolint:gosec | ||
| "time" | ||
|
|
||
| "github.com/google/pprof/profile" | ||
| "go.opentelemetry.io/collector/pdata/pcommon" | ||
| "go.opentelemetry.io/collector/pdata/pprofile" | ||
| ) | ||
|
|
||
| func pprofToPprofile(parsed *profile.Profile, originalPayload []byte) pprofile.Profiles { | ||
| result := pprofile.NewProfiles() | ||
| // create a new profile record | ||
| resultProfile := result.ResourceProfiles().AppendEmpty().ScopeProfiles().AppendEmpty().Profiles().AppendEmpty() | ||
|
|
||
| var hash [16]byte | ||
| // set the original payload on the record | ||
| // per https://github.com/open-telemetry/semantic-conventions/blob/main/model/profile/registry.yaml | ||
| resultProfile.SetOriginalPayloadFormat("go") | ||
|
Comment on lines
+22
to
+23
There was a problem hiding this comment. There is no direct correlation between the frame types that are reported with OTel Profiling (https://github.com/open-telemetry/semantic-conventions/blob/main/model/profile/registry.yaml) and the There was a problem hiding this comment. I didn't understand, sorry. Is this value invalid? There was a problem hiding this comment. Sorry, I was confused about this element. The best value, when converting Google/pprof to OTel Profiling should be There was a problem hiding this comment. that's not what the spec says though - pointing back at the schema you linked, the exhaustive list of enums has go: https://github.com/open-telemetry/semantic-conventions/blob/main/model/profile/registry.yaml#L71 There was a problem hiding this comment. The spec you are referring to is about profile.frame.type and The type of a single frame is different to the format of The field |
||
| resultProfile.OriginalPayload().FromRaw(originalPayload) | ||
| hash = md5.Sum(originalPayload) //nolint:gosec | ||
|
|
||
| // set period | ||
| resultProfile.SetPeriod(parsed.Period) | ||
|
There was a problem hiding this comment. Here is also a conflict with off CPU profiling as off CPU profiling does not have a regular period but is triggered by scheduler events. So its cadence can change depending on the workload and is not constant, like sampling based on CPU profiling. There was a problem hiding this comment. should I put something else here? Or not set it in some situations? There was a problem hiding this comment. It depends, I guess 😅 If the input originates from a sampling based approach, then this value should be set. Otherwise, it should be left out. There was a problem hiding this comment. I cannot really know that, can I? Whatever pprof provides can be passed in and pprof can make that assertion? |
||
| // TODO set period type? | ||
| // TODO set doc url? | ||
|
|
||
| // set timestamps - both start and duration | ||
| resultProfile.SetStartTime(pcommon.NewTimestampFromTime(time.Unix(0, parsed.TimeNanos))) | ||
| resultProfile.SetDuration(pcommon.NewTimestampFromTime(time.Unix(0, parsed.DurationNanos))) | ||
|
|
||
| // set the default simple type. Given this is a string, it gets cached in the string table and referred to via its index. | ||
| result.ProfilesDictionary().StringTable().Append(parsed.DefaultSampleType) | ||
| resultProfile.SetDefaultSampleTypeIndex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
|
|
||
| // set the profile ID. | ||
| resultProfile.SetProfileID(hash) | ||
|
|
||
| // add all mappings to the global dictionary | ||
| for _, m := range parsed.Mapping { | ||
| // create a mapping in the mapping table | ||
| resultM := result.ProfilesDictionary().MappingTable().AppendEmpty() | ||
| resultM.SetFileOffset(m.Offset) | ||
| resultM.SetMemoryStart(m.Start) | ||
| resultM.SetMemoryLimit(m.Limit) | ||
| resultM.SetHasFilenames(m.HasFilenames) | ||
| resultM.SetHasFunctions(m.HasFunctions) | ||
| resultM.SetHasInlineFrames(m.HasInlineFrames) | ||
| resultM.SetHasLineNumbers(m.HasLineNumbers) | ||
| result.ProfilesDictionary().StringTable().Append(m.File) | ||
| resultM.SetFilenameStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| } | ||
|
|
||
| // now iterate over each sample of the pprof data | ||
| for _, s := range parsed.Sample { | ||
| // for each sample create a corresponding sample record in our pprofile record | ||
| resultS := resultProfile.Sample().AppendEmpty() | ||
| // set the value of the sample. | ||
| resultS.Value().FromRaw(s.Value) | ||
| // now iterate over each location. | ||
|
|
||
| // set the start index at which we will start to read locations. | ||
| resultS.SetLocationsStartIndex(int32(result.ProfilesDictionary().LocationTable().Len())) | ||
|
|
||
| // read each location and map it over to pprofile | ||
| for _, l := range parsed.Location { | ||
| // add to the location table | ||
| resultL := result.ProfilesDictionary().LocationTable().AppendEmpty() | ||
| resultL.SetAddress(l.Address) | ||
| resultL.SetIsFolded(l.IsFolded) | ||
| for i, m := range result.ProfilesDictionary().MappingTable().All() { | ||
|
There was a problem hiding this comment. This might not scale well, if OTels There was a problem hiding this comment. we are not trying to scale, just make a roundtrip pprof -> otlp -> pprof. After that works we can fine tune. |
||
| if m.FileOffset() == l.Mapping.Offset && result.ProfilesDictionary().StringTable().At(int(m.FilenameStrindex())) == l.Mapping.File { | ||
| resultL.SetMappingIndex(int32(i)) | ||
| break | ||
| } | ||
| } | ||
| } | ||
| resultS.SetLocationsLength(int32(len(parsed.Location))) | ||
| } | ||
|
|
||
| for _, f := range parsed.Function { | ||
|
There was a problem hiding this comment. Similar to mappings, the resulting connection between OTel Sample.Location.Line.Function is missing. So I would also suggest to add the google/pprof Function to the OTel Function Dictionary while encountering it while iterating over google Samples. |
||
| resultF := result.ProfilesDictionary().FunctionTable().AppendEmpty() | ||
| result.ProfilesDictionary().StringTable().Append(f.Filename) | ||
| resultF.SetFilenameStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| result.ProfilesDictionary().StringTable().Append(f.Name) | ||
| resultF.SetNameStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| resultF.SetStartLine(f.StartLine) | ||
| result.ProfilesDictionary().StringTable().Append(f.SystemName) | ||
| resultF.SetSystemNameStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| } | ||
|
|
||
| for _, s := range parsed.SampleType { | ||
| resultSampleType := resultProfile.SampleType().AppendEmpty() | ||
| result.ProfilesDictionary().StringTable().Append(s.Unit) | ||
| resultSampleType.SetUnitStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| result.ProfilesDictionary().StringTable().Append(s.Type) | ||
| resultSampleType.SetTypeStrindex(int32(result.ProfilesDictionary().StringTable().Len() - 1)) | ||
| // TODO set aggregation temporality | ||
| } | ||
|
|
||
| commentsIndices := make([]int32, len(parsed.Comments)) | ||
|
There was a problem hiding this comment. As google/pprof mixes sample types and profiles in general, it might be hard to assign the google/Profile.comment to the corresponding OTel Profile.comment_strindices, as there could be multiple OTel Profiles per google/Profile. |
||
| start := result.ProfilesDictionary().StringTable().Len() - 1 | ||
| result.ProfilesDictionary().StringTable().Append(parsed.Comments...) | ||
| for i := range parsed.Comments { | ||
| commentsIndices[i] = int32(start + i) | ||
| } | ||
| resultProfile.CommentStrindices().FromRaw(commentsIndices) | ||
|
|
||
| return result | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| // Copyright The OpenTelemetry Authors | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| package internal | ||
|
|
||
| import ( | ||
| "bytes" | ||
| "runtime/pprof" | ||
| "testing" | ||
|
|
||
| "github.com/google/pprof/profile" | ||
| "github.com/stretchr/testify/require" | ||
| "go.opentelemetry.io/collector/pdata/pprofile" | ||
| ) | ||
|
|
||
| func TestMapPprofToPdata(t *testing.T) { | ||
| p := pprof.Lookup("goroutine") | ||
| pprofData := new(bytes.Buffer) | ||
| err := p.WriteTo(pprofData, 0) | ||
| require.NoError(t, err) | ||
| parsed, err := profile.Parse(pprofData) | ||
| require.NoError(t, err) | ||
| converted := pprofToPprofile(parsed, pprofData.Bytes()) | ||
| require.NotNil(t, converted) | ||
| pprofBack := pprofileToPprof(converted) | ||
| require.Equal(t, parsed, pprofBack[0]) | ||
| } | ||
|
|
||
| func pprofileToPprof(pdataProfiles pprofile.Profiles) []*profile.Profile { | ||
| result := []*profile.Profile{} | ||
| functions := make([]*profile.Function, pdataProfiles.ProfilesDictionary().FunctionTable().Len()) | ||
| for i, f := range pdataProfiles.ProfilesDictionary().FunctionTable().All() { | ||
| functions[i] = &profile.Function{ | ||
| ID: uint64(i + 1), // this is not preserved, so we just make it up. | ||
| Name: pdataProfiles.ProfilesDictionary().StringTable().At(int(f.NameStrindex())), | ||
| SystemName: pdataProfiles.ProfilesDictionary().StringTable().At(int(f.SystemNameStrindex())), | ||
| Filename: pdataProfiles.ProfilesDictionary().StringTable().At(int(f.FilenameStrindex())), | ||
| StartLine: f.StartLine(), | ||
| } | ||
| } | ||
|
|
||
| mappings := make([]*profile.Mapping, pdataProfiles.ProfilesDictionary().MappingTable().Len()) | ||
| for i, m := range pdataProfiles.ProfilesDictionary().MappingTable().All() { | ||
| mappings[i] = &profile.Mapping{ | ||
| ID: uint64(i + 1), // this is not preserved, so we just make it up. | ||
| Start: m.MemoryStart(), | ||
| Limit: m.MemoryLimit(), | ||
| Offset: m.FileOffset(), | ||
| File: pdataProfiles.ProfilesDictionary().StringTable().At(int(m.FilenameStrindex())), | ||
| BuildID: "", // TODO no mapping | ||
| HasFunctions: m.HasFunctions(), | ||
| HasFilenames: m.HasFilenames(), | ||
| HasLineNumbers: m.HasLineNumbers(), | ||
| HasInlineFrames: m.HasInlineFrames(), | ||
| KernelRelocationSymbol: "", // TODO no mapping | ||
| } | ||
| } | ||
| for _, rp := range pdataProfiles.ResourceProfiles().All() { | ||
| for _, sp := range rp.ScopeProfiles().All() { | ||
| for _, p := range sp.Profiles().All() { | ||
| var comments []string | ||
| if p.CommentStrindices().Len() > 0 { | ||
| comments = make([]string, p.CommentStrindices().Len()) | ||
| for i, index := range p.CommentStrindices().All() { | ||
| comments[i] = pdataProfiles.ProfilesDictionary().StringTable().At(int(index)) | ||
| } | ||
| } | ||
| sampleTypes := make([]*profile.ValueType, p.SampleType().Len()) | ||
| for i, st := range p.SampleType().All() { | ||
| sampleTypes[i] = &profile.ValueType{ | ||
| Type: pdataProfiles.ProfilesDictionary().StringTable().At(int(st.TypeStrindex())), | ||
| Unit: pdataProfiles.ProfilesDictionary().StringTable().At(int(st.UnitStrindex())), | ||
| } | ||
| } | ||
|
|
||
| locations := make([]*profile.Location, p.LocationIndices().Len()) | ||
| for i, l := range p.LocationIndices().All() { | ||
| pdataLocation := pdataProfiles.ProfilesDictionary().LocationTable().At(int(l)) | ||
| lines := make([]profile.Line, pdataLocation.Line().Len()) | ||
| for j, k := range pdataLocation.Line().All() { | ||
| lines[j] = profile.Line{ | ||
| Function: functions[k.FunctionIndex()], | ||
| Line: k.Line(), | ||
| Column: k.Column(), | ||
| } | ||
| } | ||
| locations[i] = &profile.Location{ | ||
| ID: uint64(l + 1), // TODO we don't map this over. | ||
| Mapping: mappings[pdataLocation.MappingIndex()], | ||
| Address: pdataLocation.Address(), | ||
| Line: lines, | ||
| IsFolded: pdataLocation.IsFolded(), | ||
| } | ||
| } | ||
|
|
||
| samples := make([]*profile.Sample, p.Sample().Len()) | ||
| for i, s := range p.Sample().All() { | ||
| sampleLocations := make([]*profile.Location, s.LocationsLength()) | ||
| for li := 0; li < int(s.LocationsLength()); li++ { | ||
| pdataLocation := pdataProfiles.ProfilesDictionary().LocationTable().At(i + int(s.LocationsStartIndex())) | ||
| lines := make([]profile.Line, pdataLocation.Line().Len()) | ||
| for j, k := range pdataLocation.Line().All() { | ||
| lines[j] = profile.Line{ | ||
| Function: functions[k.FunctionIndex()], | ||
| Line: k.Line(), | ||
| Column: k.Column(), | ||
| } | ||
| } | ||
| sampleLocations[li] = &profile.Location{ | ||
| ID: uint64(li + 1), // TODO we don't map this over. | ||
| Mapping: mappings[pdataLocation.MappingIndex()], | ||
| Address: pdataLocation.Address(), | ||
| Line: lines, | ||
| IsFolded: pdataLocation.IsFolded(), | ||
| } | ||
| } | ||
| samples[i] = &profile.Sample{ | ||
| Location: sampleLocations, | ||
| Value: s.Value().AsRaw(), | ||
| Label: nil, // TODO no mapping | ||
| NumLabel: nil, // TODO no mapping | ||
| NumUnit: nil, // TODO no mapping | ||
| } | ||
| } | ||
|
|
||
| result = append(result, &profile.Profile{ | ||
| SampleType: sampleTypes, | ||
| DefaultSampleType: pdataProfiles.ProfilesDictionary().StringTable().At(int(p.DefaultSampleTypeIndex())), | ||
| Sample: samples, | ||
| Mapping: mappings, | ||
| Location: locations, | ||
| Function: functions, | ||
| Comments: comments, | ||
| DocURL: "", // TODO no mapping | ||
| DropFrames: "", // TODO no mapping | ||
| KeepFrames: "", // TODO no mapping | ||
| TimeNanos: p.Time().AsTime().UnixNano(), | ||
| DurationNanos: p.Duration().AsTime().UnixNano(), | ||
| PeriodType: nil, | ||
| Period: p.Period(), | ||
| }) | ||
| } | ||
| } | ||
| } | ||
| return result | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A single Resource Profile in OTel Profiling can have multiple Scope Profiles. And OTel eBPF Profiler is already using multiple Scope Profiles to differentiate between on CPU and off CPU profiling. Mixing these profiles will cause issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that defined somewhere I can reuse?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To add more complexity 🙈 open-telemetry/opentelemetry-ebpf-profiler#517 will implement the recent changes in the OTel Profiling signal to generate a single
ResourceProfileper container and an additional ResourceProfile for all non-containerized processes. As, I think, google/pprof does not differentiate between containerized elements and others, I think it is hard to map this correctly. Google/pprof might be able to set attributes accordingly, but they might not be standartized.But I think, there should be one OTel/ScopeProfile per Google/pprof Profile.sample_type. This will then also be according to open-telemetry/opentelemetry-proto#649.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, can this be clarified in a spec? This is a bit of a departure from other signals where scope typically is to identify the instrumentation used.