Skip to content

Add Indices filter flag #574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 50 commits into from
Closed

Add Indices filter flag #574

wants to merge 50 commits into from

Conversation

chapost1
Copy link

As can be seen earlier, certain people asked for the option to filter the indices they are exporting.

Especially if you have a lot of them and you need to monitor just certain, the rest of it, is just a waste of resources.

ATM, fetching indices settings/stats/mappings queries all the existing ones using this kind of GET requests:
GET /_all/_settings, GET /_all/_stats and GET /_all/_mappings

As mentioned in the docs:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html

It's possible to just retrieve the ones which match 2 types of queries:

  1. Matching using wildcard(*) expression. i.e:
    GET /prefix_*/_stats

  2. Matching a list of indices, using comma:
    GET /first_name,second_name/_stats

It is possible to mix them both. ie:
GET /prefix_1_*,prefix_2_*/_stats

The new added flag:
es.indices_filter (default is '_all')

Note: The flag will be used only if any export indices flag is used.
To use it, pass an expression to match your use-case indices, default '_all' just like before.
By passing one of the options mentioned above as a value (statement with a wildcard, list with commas), it will be used in the http request to retrieve statistics.

This change would be very appreciated,
Thanks.

@ghost
Copy link

ghost commented Jul 31, 2023

Any update about this one?

@chapost1
Copy link
Author

chapost1 commented Aug 2, 2023

I've fixed the merge conflicts, but I'm failing to pass the DCO due to invalid signature.
Is it possible to work it out?

@sysadmind
Copy link
Contributor

The details button on the DCO should help you to resolve. My suggestion is to rebase onto the master branch, squashing all of your commits into a single commit. This makes it easy to know that there's only one commit to sign.

bobo333 and others added 26 commits August 17, 2023 19:51
* Add support for adding aliases to index and shard stats

Optionally will include aliases in a label to index and shard stats

Signed-off-by: Steven Cipriano <cipriano@squareup.com>

* move aliases to separate informational metrics

Signed-off-by: Steven Cipriano <cipriano@squareup.com>

* address review

- switch boolean flag
- fix linting errors

Signed-off-by: Steven Cipriano <cipriano@squareup.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
… queries

Signed-off-by: Shahar Tal <chapo12345@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/go-kit/log](https://github.com/go-kit/log) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/go-kit/log/releases)
- [Commits](go-kit/log@v0.2.0...v0.2.1)

---
updated-dependencies:
- dependency-name: github.com/go-kit/log
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Thibault Richard <thb.krkr@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.34.0 to 0.35.0.
- [Release notes](https://github.com/prometheus/common/releases)
- [Commits](prometheus/common@v0.34.0...v0.35.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/imdario/mergo](https://github.com/imdario/mergo) from 0.3.12 to 0.3.13.
- [Release notes](https://github.com/imdario/mergo/releases)
- [Commits](darccio/mergo@0.3.12...v0.3.13)

---
updated-dependencies:
- dependency-name: github.com/imdario/mergo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.1 to 1.12.2.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](prometheus/client_golang@v1.12.1...v1.12.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* Support for AWS Elasticsearch using AWS SDK v2 (#1)

Signed-off-by: Roshan Patil <patilroshan443@gmail.com>

* Fixes PR comments

Signed-off-by: Roshan Patil <patilroshan443@gmail.com>

* Resolved PR comments

Signed-off-by: Roshan Patil <patilroshan443@gmail.com>

* Fix PR comments

Signed-off-by: Roshan Patil <patilroshan443@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Mike Eves <michael.eves@autotrader.co.uk>
…set (#603)

Signed-off-by: akazs <5749172+akazs@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Use semver/v4 for better Go module compatibility.
* Update all Go modules.

Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* [FEATURE] Add metrics collection for data stream statistics #592
* [FEATURE] Support for AWS Elasticsearch using AWS SDK v2 #597
* [BUGFIX] Fix cluster settings collection when max_shards_per_node is manually set. #603

Signed-off-by: SuperQ <superq@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Roshan Patil <patilroshan443@gmail.com>

Signed-off-by: Roshan Patil <patilroshan443@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Simon Zengerling <simon.zengerling@camunda.com>

Signed-off-by: Simon Zengerling <simon.zengerling@camunda.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* Add https to metrics server

Signed-off-by: Yuriy Kis <yurakis9@gmail.com>

* Delete tls-config.yml

Signed-off-by: Yuriy Kis <54007065+yuriykis@users.noreply.github.com>

* Apply suggestions

Co-authored-by: Joe Adams <adams10301@gmail.com>
Signed-off-by: Yuriy Kis <54007065+yuriykis@users.noreply.github.com>

* Add webflag import

Signed-off-by: Yuriy Kis <yurakis9@gmail.com>

Signed-off-by: Yuriy Kis <yurakis9@gmail.com>
Signed-off-by: Yuriy Kis <54007065+yuriykis@users.noreply.github.com>
Co-authored-by: Joe Adams <adams10301@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>

Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.15.14 to 1.17.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.15.14...config/v1.17.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Abhilash Mandaliya <abhilashmandaliya@gmail.com>

Signed-off-by: Abhilash Mandaliya <abhilashmandaliya@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>

Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* number_of_replicas

Signed-off-by: Alexandr Marchenko <marchenko.alexandr@gmail.com>

* simplify metric name by removing redundant number_of_ prefix

Co-authored-by: Ben Kochie <superq@gmail.com>
Signed-off-by: Alexandr Marchenko <marchenko.alexandr@gmail.com>

* Update README.md

Thanks indeed

Co-authored-by: Joe Adams <adams10301@gmail.com>
Signed-off-by: Alexandr Marchenko <marchenko.alexandr@gmail.com>

Signed-off-by: Alexandr Marchenko <marchenko.alexandr@gmail.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
Co-authored-by: Joe Adams <adams10301@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.8 to 1.18.7.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.17.8...config/v1.18.7)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* feat(*): add ilm metrics

Signed-off-by: iishabakaev <iishabakaev@gmail.com>

* Refactor ilm collectors

Signed-off-by: Шабакаев Илья Исмаилович <ishabakaev@alfabank.ru>
Signed-off-by: Шабакаев Илья Исмаилович <iishabakaev@gmail.com>

* fix lint checks

Signed-off-by: Шабакаев Илья Исмаилович <iishabakaev@gmail.com>

Signed-off-by: iishabakaev <iishabakaev@gmail.com>
Signed-off-by: Шабакаев Илья Исмаилович <ishabakaev@alfabank.ru>
Signed-off-by: Шабакаев Илья Исмаилович <iishabakaev@gmail.com>
Co-authored-by: Шабакаев Илья Исмаилович <ishabakaev@alfabank.ru>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* Fixing field counting logic in countFieldsRecursive function for the indices_mappings feature

Signed-off-by: Alain Lefebvre <hartfordfive@gmail.com>

* Added test case to validate counting of fields in index mapping

Signed-off-by: Alain Lefebvre <hartfordfive@gmail.com>

* Updated conditional statement to exclude counting field when the type field is set to object as it will be counted in the recursion case

Signed-off-by: Alain Lefebvre <hartfordfive@gmail.com>

* Reverting version change

Signed-off-by: Alain Lefebvre <hartfordfive@gmail.com>

Signed-off-by: Alain Lefebvre <hartfordfive@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
prombot and others added 24 commits August 17, 2023 19:53
Signed-off-by: prombot <prometheus-team@googlegroups.com>

Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Co-authored-by: Christoph Burmeister <christoph.burmeister@idealo.de>
Signed-off-by: Steve Teuber <steve.teuber@idealo.de>

Signed-off-by: Steve Teuber <steve.teuber@idealo.de>
Co-authored-by: Christoph Burmeister <christoph.burmeister@idealo.de>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
…679)

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.13.7 to 1.13.10.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@service/mq/v1.13.7...service/mq/v1.13.10)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
…682)

Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.18.2 to 1.18.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@config/v1.18.2...config/v1.18.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [github.com/prometheus/common](https://github.com/prometheus/common) from 0.37.0 to 0.40.0.
- [Release notes](https://github.com/prometheus/common/releases)
- [Commits](prometheus/common@v0.37.0...v0.40.0)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](golang/net@v0.4.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
…1.0 (#689)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.0.0-20210616213533-5ff15b29337e to 0.1.0.
- [Release notes](https://github.com/golang/crypto/releases)
- [Commits](https://github.com/golang/crypto/commits/v0.1.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* feat: additional elasticcloud roles

---------

Signed-off-by: Christophe MORIO <cmorio@talend.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* Refactor cluster settings collector

Refactor cluster settings collector into new Collector interface. Update tests to use prometheus/testutil to validate the output instead of checking the parsed elasticsearch response.

Signed-off-by: Joe Adams <github@joeadams.io>

* Fix rebase

Signed-off-by: Joe Adams <github@joeadams.io>

---------

Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
* Update Go to 1.20.
* Update Makefile.common
* Update golangci-lint.
* Update exporter-toolkit.
* Cleanup useless `_` assignment for go-kit/log.
* Cleanup deprecated and unused code.

Signed-off-by: SuperQ <superq@gmail.com>
The metric labels in the prometheus description were set to all 3 status options, instead of the name of the status label. The code exports a metric for each of the statuses individually, not all 3 at the same time.

fixes #677

Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: innerpeacez <innerpeace.zhai@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: innerpeacez <innerpeace.zhai@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
In the Prometheus ecosystem, info metrics (those with the `_info`
suffix) usually have a value of 1. Although not enforced, this has
become a standard pattern across Prometheus and several exporters (see
`prometheus_build_info`).

Signed-off-by: Jorge Luis Betancourt Gonzalez <jorge-luis.betancourt@trivago.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Fix comment typo error in Retriever struct of the clusterinfo package

Signed-off-by: bagmeg <joonsoo181005@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Raz Amir <88726761+ramir-savvy@users.noreply.github.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Joe Adams <github@joeadams.io>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: prombot <prometheus-team@googlegroups.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Tomáš Kadaně <tomas.kadane@cdn77.com>
Co-authored-by: Tomáš Kadaně <tomas.kadane@cdn77.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
Signed-off-by: Shahar Tal <stalmail10@gmail.com>
@chapost1 chapost1 closed this by deleting the head repository Aug 17, 2023
@chapost1
Copy link
Author

chapost1 commented Aug 17, 2023

@sysadmind
@shani-e

The details button on the DCO should help you to resolve. My suggestion is to rebase onto the master branch, squashing all of your commits into a single commit. This makes it easy to know that there's only one commit to sign.

I've tried to use the DCO guidelines, but it went off so I've closed this PR and created a new fork.
New PR can be found in here:
#764

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.