[apm] Improve instructions for monitoring APM on premise

[colleenmcginnis]

Description

@lucabelluccini shared a few ways we could improve the instructions for monitoring APM on premise:

• Warn users that if they have already enrolled Elastic Agent, following these instructions will likely cause them to lose the state of the agent.
• Suggest that users who have already enrolled Elastic Agent, use Fleet UI to change the agent's HTTP monitoring endpoint.
• Remove username and password from the sample modules.d/beat-xpack.yml.

Documentation sets edited in this PR

Check all that apply.

• Stateful (docs/en/observability/*)
• Serverless (docs/en/serverless/*)
• Integrations Developer Guide (docs/en/integrations/*)
• None of the above

Related issue

Closes #4726

Checklist

• Product/Engineering Review
• Writer Review

Follow-up tasks

Select one.

• This PR does not need to be ported to another doc set because:
  • The concepts in this PR only apply to one doc set (serverless or stateful)
  • The PR contains edits to both doc sets (serverless and stateful)
• This PR needs to be ported to another doc set:
  • Port to stateful docs: <link to PR or tracking issue>
  • Port to serverless docs: <link to PR or tracking issue>

[github-actions]

A documentation preview will be available soon.

• 🔨 Buildkite builds
• 📚 HTML diff
• 📙 Preview page

Request a new doc build by commenting

• Rebuild this PR: run docs-build
• Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

[bmorelli25]

✨

[eedugon]

IMO the title Configure Elastic Agent to **send** monitoring data is incorrect and it's probably the reason for the original document to have issues like including a username and a password that shouldn't be there.

The title should be: Configure Elastic Agent to serve monitoring data (instead of serve we can use another word like provide), as that part of the configuration just enables an HTTP endpoint to provide monitoring data to any client, it doesn't send anything.

What do you think @lucabelluccini and @colleenmcginnis ?

@colleenmcginnis , I hope this comment doesn't feel off-topic (I tend to add too many off-topic comments :-D ), as I think it's related with the original issue.

[lucabelluccini]

Ideally we would have:

• For versions pre 8.15 we have to say:

*This procedure works only if you've _not_ yet enrolled {agent}.*
If you've already enrolled {agent}, you'll need to re-enroll {agent} and you'll lose the state of the agent.

• For version 8.15 and more recent

Use the Fleet UI to change the agent's HTTP monitoring endpoint using {fleet-guide}/agent-policy.html#advanced-agent-monitoring-settings[Advanced agent monitoring settings].

[lucabelluccini]

Also, I would invite a developer / engineer to test this flow if possible.

[colleenmcginnis]

@simitt is someone from your team available to take a look at this recommendation?

[simitt]

@raultorrecilla know better about the team's availability.

[lucabelluccini]

Hello, I've taken the time to test this and it works. I'm leaving a comment below for a catch...

[lucabelluccini]

I've left a comment.

[lucabelluccini]

This is actually not true...
The basepath has to match the component ID of the APM Integration.

For example, if ones creates an APM Integration policy and its ID is 7b12d130-018d-4fa8-908b-80a677e7cfce, then the basepath has to be set to basepath: "/processes/apm-7b12d130-018d-4fa8-908b-80a677e7cfce.

How to tell a user how to get the component ID? I do not know the fastest solution, but it might be:

• Go to Fleet UI
• Go to the Elastic Agent policy which contains the APM Package Integration policy
• Go in Actions / View Policy
• Search for the id of the integration apm
• Use the id, prefixing it with apm-

Or... from command line...

 elastic-agent status --output full | grep apm

Will show up:

   ├─ apm-7b12d130-018d-4fa8-908b-80a677e7cfce <------ this one
   │  ├─ apm-7b12d130-018d-4fa8-908b-80a677e7cfce
   │  └─ apm-7b12d130-018d-4fa8-908b-80a677e7cfce-1720f832-262b-43fa-a3fd-0629bcb3d530

The only situation the APM Integration has apm-server-default is when the integration is created via kibana.yml and the ID is provided by the user, which is quite rare.

[lucabelluccini]

I've reserved some time to work on this and try it out. I've left some comments....