How to use s3 upload to minio?

[DimentR]

gdg:0.4.5

My config:

context_name: devcontext
storage_engine:
    minio_hub:
        access_key_id: "xXx"
        bucket_name: "minio.example.com/backup"
        kind: "cloud"
        cloud_type: s3
        secret_key: "xXx"
        prefix: "grafana"
contexts:
    devcontext:
        storage: minio_hub
        url: https://grafana.example.com/
        token: xXx
        user_name: ""
        password: ""
        organization: ""
        watched:
            - Kafka
            - Logs
            - General
        datasources: {}
        filter_override: null
        output_path: dev
global:
    debug: true
    ignore_ssl_errors: true

I run:

dash download

Output:

WARN[0000] Unable to get Admin Auth.  Basic Auth credentials, continuing with token
WARN[0000] Error getting organizations
ERRO[0000] blob (key "grafana/dev/dashboards/General/backups.json") (code=Unknown): MissingRegion: could not find region configuration for backups
ERRO[0000] blob (key "grafana/dev/dashboards/General/blackbox-exporter.json") (code=Unknown): MissingRegion: could not find region configuration for blackbox-exporter
...
INFO[0001] Importing dashboards for context: 'devcontext'
┌──────┬──────────┐
│ TYPE │ FILENAME │
├──────┼──────────┤
└──────┴──────────┘

How to use s3 upload in minio?

[safaci2000]

Technically Minio/Ceph isn't supported yet. I am using Minio in a test case, but I had to write some custom code to get that working. I need to make that a bit more generic for general consumption.

You can see if there's anything useful in the lib GDG is using: ttps://github.com/google/go-cloud or the S3 specific driver which is invoked when you set the type to S3.

Otherwise, will have to wait till I have some cycles to add Minio support.

[safaci2000]

Here's a few more references: google/go-cloud#3240 I asked the question a while back.

https://software.es.net/gdg/docs/gdg/cloud_configuration/ more info.

Also, the access_key and secret_key are no longer supported. You should be configuring your bucket via AWS tools and GDG will pick up the auth information from there.

[safaci2000]

@DimentR next release will address this change. Waiting on the code review and that should get be the last ticket for 0.5.0 to be released with this change.

[safaci2000]

These are the instructions on how to set it up @DimentR https://software.es.net/gdg/docs/gdg/cloud_configuration/#custom

Update: https://software.es.net/gdg/docs/gdg/configuration/storage/ is probably the better URL since the last one is a 404 now.

[NikiVinn]

still need an instruction, can locally upload and download dashboards to grafana, and from grafana
but
when we talkin about s3 minio, it just cant
i cant even see its attempt to upload, 0 logs, 0 useful things is debug
would really appreciate for cool instruction how can i locally upload mine dashboards from grafana -> to -> minio s3

[safaci2000]

You could I suppose. You're defining a backup destination so if you configured the backup to go to s3, then it ignores local files and looks in S3 for backups. If you're uploading from S3 it won't find any and won't do anything. I would suggest using a local config to push things to Grafana and switch to S3 context and fetch the data to S3.

it's not intended for you to constantly switch between local and S3. You need to choose where you're backing up to and that is the source of 'truth'. Otherwise you could potentially just recreate the structure that GDG creates locally on AWS the easiest pattern is:

Local ->Grafana, update config to use S3, Grafana -> S3. You can also have two contexts defined that are fairly similar except one uses S3 and one is using local storage. Just switch context as needed.

[NikiVinn]

[NikiVinn]

@safaci2000

[safaci2000]

Let me know if you need more help, beyond the answer provided: here

[NikiVinn]

@safaci2000 maybe ima kinda stupid, but what i need is:

i press the the thing, or maybe even doing some automated job and what happened next is -> grafan dashboards -> going to mine s3 minio bucket, this is what i need, im testing rn it on localhost, so dont really care bout the credentials, so here they are:

context_name: testing 
 
storage_engine: 
  test: 
    self_hosted: true
    kind: cloud 
    cloud_type: s3 
    prefix: dummy 
    bucket_name: "grafana-backups" 
    access_id: "admin_user_root" 
    secret_key: "admin_user_admin" 
    init_bucket: "true" 
    endpoint: "http://minio:9000" 
    ssl_enabled: "false" 
 
contexts: 
  testing: 
    url: "http://grafana:3000" 
    user_name: "admin" 
    password: "admin" 
    dashboard_settings: 
      ignore_filters: false 
      nested_folders: true 
    watched: 
      - Test 
    output_path: grafana-backups 
 
global: 
  debug: true 
  api_debug: false 
  clear_output: false 
  retry_count: 3 
  retry_delay: 5s 
  ignore_ssl_errors: true

i have everything in docker, and all the things are in mine docker network monitoring, the thing is right now all the dashboards are going to local directory on mine machine, while i need it to save it right to the minio, yea i might need the functionality to backup from minio, if mine grafana is down, but thats another part which i dont really need right now.

so can you please give me some advice what im doing wrong with mine config, and why its keepin saving on mine machine, but not in the minio

sadly, i already did a systemd job, which is doing curl +X PUT to the minio from the local directory but thats kinda suck

most of the setting are vanilla, if you need some more information, surely let me know, id like to give u all the info if it will help

[safaci2000]

You haven't told to use the storage test, you just defined a type of staroge called test. You need to add "storage: test" to the testing context.

Reference: https://software.es.net/gdg/docs/gdg/configuration/contexts/#storage

You can see the full example here notice the storage label inside the context. If it's blank it assumes local storage always, otherwise it looks for a storage engine with the label you gave it.

[NikiVinn]

damn, i really apreciate it, thanks a lot, i get it now. dont u mind if i'd have some troubles i can reply u here?

[safaci2000]

Yeah that's fine. That's the purpose of these discussions to help people debug issues they run into.