Skip to content

Experimental: Add query type definition and schemas #897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 79 commits into from
Mar 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
8781f2f
add query type support
ryantxu Feb 12, 2024
0ec8629
move values
ryantxu Feb 13, 2024
556b816
no refs
ryantxu Feb 13, 2024
fa45a90
add k8s placeholder
ryantxu Feb 13, 2024
645173e
parallel k8s
ryantxu Feb 13, 2024
ff55c53
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Feb 13, 2024
f6bb323
now with examples
ryantxu Feb 13, 2024
e2341d6
ok with dataframe now
ryantxu Feb 13, 2024
992463d
ok with dataframe now
ryantxu Feb 13, 2024
58b32ee
no versions
ryantxu Feb 14, 2024
d791fc9
add apiVersion
ryantxu Feb 14, 2024
0d526d8
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Feb 16, 2024
4ae1f0c
add settings example
ryantxu Feb 16, 2024
b9382d8
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Feb 17, 2024
238f7c9
fix build
ryantxu Feb 17, 2024
7ed252c
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Feb 20, 2024
1fe487a
multiple discriminators
ryantxu Feb 20, 2024
f5a35b3
difference between save and post
ryantxu Feb 21, 2024
dc41e8c
rename to spec
ryantxu Feb 21, 2024
1202776
rename to spec
ryantxu Feb 21, 2024
10b145d
more reference stuff
ryantxu Feb 24, 2024
e854402
more reference stuff
ryantxu Feb 24, 2024
074b8f8
more reference stuff
ryantxu Feb 25, 2024
c673243
with definitions
ryantxu Feb 25, 2024
940bc8e
with definitions
ryantxu Feb 25, 2024
5c2de27
cleanup
ryantxu Feb 25, 2024
64c06c2
cleanup
ryantxu Feb 25, 2024
36b841c
use kube-openapi
ryantxu Feb 25, 2024
2c8c4ad
fix error state
ryantxu Feb 25, 2024
2b78392
refactor (again)
ryantxu Feb 25, 2024
e717941
refactor (again)
ryantxu Feb 25, 2024
d9f700e
now with panel schema
ryantxu Feb 25, 2024
ee12be2
now with panel schema
ryantxu Feb 25, 2024
77f2055
now with dataFrame type
ryantxu Feb 26, 2024
f8eb48f
now with dataFrame type
ryantxu Feb 26, 2024
8f647ba
lint
ryantxu Feb 26, 2024
9e14771
refactor
ryantxu Feb 26, 2024
29beda6
refactor
ryantxu Feb 26, 2024
a09ab12
another constructor
ryantxu Feb 26, 2024
81d51b3
merge main
ryantxu Feb 28, 2024
2151c09
update query parser
ryantxu Feb 29, 2024
5af1ec4
update query parser
ryantxu Feb 29, 2024
1e64757
merge main
ryantxu Feb 29, 2024
f18d1cb
use real type for JSONSchema object
ryantxu Feb 29, 2024
ff38e88
use real type for JSONSchema object
ryantxu Feb 29, 2024
34c635a
use real type for JSONSchema object
ryantxu Feb 29, 2024
0847e1d
add spec definition
ryantxu Feb 29, 2024
8482109
more specs
ryantxu Feb 29, 2024
8dbdba0
more specs
ryantxu Feb 29, 2024
bb0cd98
lint
ryantxu Feb 29, 2024
2b3cc45
now with unstructured
ryantxu Mar 1, 2024
0b0e1ea
less verbose
ryantxu Mar 1, 2024
a804614
now with unstructured
ryantxu Mar 1, 2024
424be9a
now with unstructured
ryantxu Mar 1, 2024
3a97c12
more copy methods
ryantxu Mar 1, 2024
f8336e6
fix lint remove parser
ryantxu Mar 2, 2024
2a7ba03
fix lint
ryantxu Mar 2, 2024
2ec1471
panel cleanup
ryantxu Mar 2, 2024
0237a40
panel cleanup
ryantxu Mar 2, 2024
d35e870
inline timerange
ryantxu Mar 3, 2024
a7986a7
more deepcopy
ryantxu Mar 3, 2024
71374e7
hopefully last big package rename
ryantxu Mar 4, 2024
d2c2e90
loading frame types
ryantxu Mar 4, 2024
0e568d6
loading frame types
ryantxu Mar 4, 2024
e7e1b68
fix test
ryantxu Mar 4, 2024
81d7990
remove unused settings configs
ryantxu Mar 4, 2024
0fe8c9c
move panel to schema builder
ryantxu Mar 4, 2024
bd58a3b
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Mar 4, 2024
e9b643e
use k8s style paths
ryantxu Mar 4, 2024
c52d81c
use codegen
ryantxu Mar 4, 2024
91df7c5
use codegen
ryantxu Mar 4, 2024
66438c4
lint
ryantxu Mar 4, 2024
557e208
rename to data
ryantxu Mar 5, 2024
283c5df
update readme
ryantxu Mar 6, 2024
782e1f5
update headers
ryantxu Mar 6, 2024
342bf4e
update headers
ryantxu Mar 6, 2024
0fd38ec
Merge remote-tracking branch 'origin/main' into query-type-handler
ryantxu Mar 6, 2024
86b43fa
move package
ryantxu Mar 6, 2024
95b4050
move package
ryantxu Mar 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions apis/data/v0alpha1/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package v0alpha1

import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"

"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data/utils/jsoniter"
)

type QueryDataClient interface {
QueryData(ctx context.Context, req QueryDataRequest, headers ...string) (int, *backend.QueryDataResponse, error)
}

type simpleHTTPClient struct {
url string
client *http.Client
headers []string
}

func NewQueryDataClient(url string, client *http.Client, headers ...string) QueryDataClient {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure I understand the purpose of this new client? Is it an example?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is what alert ruler would call (currently we say "call /ds/query" but there is no typed client).

It is also the abstraction used inside the query service that will execute query requests. We have a few implementations:

We don't necessarily need the implementation here (could be internal) but then not sure how to best have alert ruler use the client.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it may make sense in its own package (and perhaps with a generic for the query type) but given this is all in experimental/v0alpha I don't think it matters too much yet

if client == nil {
client = http.DefaultClient
}
return &simpleHTTPClient{
url: url,
client: client,
headers: headers,
}
}

func (c *simpleHTTPClient) QueryData(ctx context.Context, query QueryDataRequest, headers ...string) (int, *backend.QueryDataResponse, error) {
body, err := json.Marshal(query)
if err != nil {
return http.StatusBadRequest, nil, err
}

req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.url, bytes.NewBuffer(body))
if err != nil {
return http.StatusBadRequest, nil, err
}
headers = append(c.headers, headers...)
if (len(headers) % 2) != 0 {
return http.StatusBadRequest, nil, fmt.Errorf("headers must be in pairs of two")
}
for i := 0; i < len(headers); i += 2 {
req.Header.Set(headers[i], headers[i+1])
}
req.Header.Set("Content-Type", "application/json")

rsp, err := c.client.Do(req)
if err != nil {
return rsp.StatusCode, nil, err
}
defer rsp.Body.Close()

qdr := &backend.QueryDataResponse{}
iter, err := jsoniter.Parse(jsoniter.ConfigCompatibleWithStandardLibrary, rsp.Body, 1024*10)
if err == nil {
err = iter.ReadVal(qdr)
}
return rsp.StatusCode, qdr, err
}
52 changes: 52 additions & 0 deletions apis/data/v0alpha1/client_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
package v0alpha1_test

import (
"context"
"encoding/json"
"fmt"
"net/http"
"testing"

"github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
"github.com/stretchr/testify/require"
)

func TestQueryClient(t *testing.T) {
t.Skip()

client := v0alpha1.NewQueryDataClient("http://localhost:3000/api/ds/query", nil,
"Authorization", "Bearer YOURKEYHERE",
)
body := `{
"from": "",
"to": "",
"queries": [
{
"refId": "X",
"scenarioId": "csv_content",
"datasource": {
"type": "grafana-testdata-datasource",
"uid": "PD8C576611E62080A"
},
"csvContent": "a,b,c\n1,hello,true",
"hide": true
}
]
}`
qdr := v0alpha1.QueryDataRequest{}
err := json.Unmarshal([]byte(body), &qdr)
require.NoError(t, err)

code, rsp, err := client.QueryData(context.Background(), qdr)
require.NoError(t, err)
require.Equal(t, http.StatusOK, code)

r, ok := rsp.Responses["X"]
require.True(t, ok)

for _, frame := range r.Frames {
txt, err := frame.StringTable(20, 10)
require.NoError(t, err)
fmt.Printf("%s\n", txt)
}
}
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ var f embed.FS

func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
"github.com/grafana/grafana-plugin-sdk-go/backend.DataResponse": schemaDataResponse(ref),
"github.com/grafana/grafana-plugin-sdk-go/data.Frame": schemaDataFrame(ref),
"github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1.DataQuery": schemaDataQuery(ref),
"github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1.QueryTypeDefinitionSpec": schemaQueryTypeDefinitionSpec(ref),
"github.com/grafana/grafana-plugin-sdk-go/backend.DataResponse": schemaDataResponse(ref),
"github.com/grafana/grafana-plugin-sdk-go/data.Frame": schemaDataFrame(ref),
"github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1.DataQuery": schemaDataQuery(ref),
"github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1.QueryTypeDefinitionSpec": schemaQueryTypeDefinitionSpec(ref),
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func init() { //nolint:gochecknoinits
jsoniter.RegisterTypeDecoder("v0alpha1.DataQuery", &genericQueryCodec{})
}

type DataQueryRequest struct {
type QueryDataRequest struct {
// Time range applied to each query (when not included in the query body)
TimeRange `json:",inline"`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ func TestParseQueriesIntoQueryDataRequest(t *testing.T) {
"to": "1692646267389"
}`)

req := &DataQueryRequest{}
req := &QueryDataRequest{}
err := json.Unmarshal(request, req)
require.NoError(t, err)

Expand Down Expand Up @@ -74,7 +74,7 @@ func TestParseQueriesIntoQueryDataRequest(t *testing.T) {
})

t.Run("same results from either parser", func(t *testing.T) {
typed := &DataQueryRequest{}
typed := &QueryDataRequest{}
err = json.Unmarshal(request, typed)
require.NoError(t, err)

Expand Down
File renamed without changes.
File renamed without changes.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 10 additions & 10 deletions experimental/schemabuilder/example/query_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"reflect"
"testing"

sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
data "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
"github.com/grafana/grafana-plugin-sdk-go/experimental/schemabuilder"
"github.com/stretchr/testify/require"
)
Expand All @@ -23,30 +23,30 @@ func TestQueryTypeDefinitions(t *testing.T) {
})
require.NoError(t, err)
err = builder.AddQueries(schemabuilder.QueryTypeInfo{
Discriminators: sdkapi.NewDiscriminators("queryType", QueryTypeMath),
Discriminators: data.NewDiscriminators("queryType", QueryTypeMath),
GoType: reflect.TypeOf(&MathQuery{}),
Examples: []sdkapi.QueryExample{
Examples: []data.QueryExample{
{
Name: "constant addition",
SaveModel: sdkapi.AsUnstructured(MathQuery{
SaveModel: data.AsUnstructured(MathQuery{
Expression: "$A + 11",
}),
},
{
Name: "math with two queries",
SaveModel: sdkapi.AsUnstructured(MathQuery{
SaveModel: data.AsUnstructured(MathQuery{
Expression: "$A - $B",
}),
},
},
},
schemabuilder.QueryTypeInfo{
Discriminators: sdkapi.NewDiscriminators("queryType", QueryTypeReduce),
Discriminators: data.NewDiscriminators("queryType", QueryTypeReduce),
GoType: reflect.TypeOf(&ReduceQuery{}),
Examples: []sdkapi.QueryExample{
Examples: []data.QueryExample{
{
Name: "get max value",
SaveModel: sdkapi.AsUnstructured(ReduceQuery{
SaveModel: data.AsUnstructured(ReduceQuery{
Expression: "$A",
Reducer: ReducerMax,
Settings: ReduceSettings{
Expand All @@ -57,9 +57,9 @@ func TestQueryTypeDefinitions(t *testing.T) {
},
},
schemabuilder.QueryTypeInfo{
Discriminators: sdkapi.NewDiscriminators("queryType", QueryTypeResample),
Discriminators: data.NewDiscriminators("queryType", QueryTypeResample),
GoType: reflect.TypeOf(&ResampleQuery{}),
Examples: []sdkapi.QueryExample{},
Examples: []data.QueryExample{},
})
require.NoError(t, err)

Expand Down
18 changes: 9 additions & 9 deletions experimental/schemabuilder/examples.go
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
package schemabuilder

import (
sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
data "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
)

func exampleRequest(defs sdkapi.QueryTypeDefinitionList) sdkapi.DataQueryRequest {
rsp := sdkapi.DataQueryRequest{
TimeRange: sdkapi.TimeRange{
func exampleRequest(defs data.QueryTypeDefinitionList) data.QueryDataRequest {
rsp := data.QueryDataRequest{
TimeRange: data.TimeRange{
From: "now-1h",
To: "now",
},
Queries: []sdkapi.DataQuery{},
Queries: []data.DataQuery{},
}

for _, def := range defs.Items {
for _, sample := range def.Spec.Examples {
if sample.SaveModel.Object != nil {
q := sdkapi.NewDataQuery(sample.SaveModel.Object)
q := data.NewDataQuery(sample.SaveModel.Object)
q.RefID = string(rune('A' + len(rsp.Queries)))
for _, dis := range def.Spec.Discriminators {
_ = q.Set(dis.Field, dis.Value)
Expand All @@ -36,13 +36,13 @@ func exampleRequest(defs sdkapi.QueryTypeDefinitionList) sdkapi.DataQueryRequest
return rsp
}

func examplePanelTargets(ds *sdkapi.DataSourceRef, defs sdkapi.QueryTypeDefinitionList) []sdkapi.DataQuery {
targets := []sdkapi.DataQuery{}
func examplePanelTargets(ds *data.DataSourceRef, defs data.QueryTypeDefinitionList) []data.DataQuery {
targets := []data.DataQuery{}

for _, def := range defs.Items {
for _, sample := range def.Spec.Examples {
if sample.SaveModel.Object != nil {
q := sdkapi.NewDataQuery(sample.SaveModel.Object)
q := data.NewDataQuery(sample.SaveModel.Object)
q.Datasource = ds
q.RefID = string(rune('A' + len(targets)))
for _, dis := range def.Spec.Discriminators {
Expand Down
2 changes: 1 addition & 1 deletion experimental/schemabuilder/panel.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
package schemabuilder

import sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
import sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"

// This is only used to write out a sample panel query
// It is not public and not intended to represent a real panel
Expand Down
2 changes: 1 addition & 1 deletion experimental/schemabuilder/reflector.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (

"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/invopop/jsonschema"
"github.com/stretchr/testify/assert"
Expand Down
12 changes: 6 additions & 6 deletions experimental/schemabuilder/reflector_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import (
"reflect"
"testing"

sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/invopop/jsonschema"
"github.com/stretchr/testify/require"
Expand All @@ -16,8 +16,8 @@ func TestWriteQuerySchema(t *testing.T) {
PluginID: []string{"dummy"},
ScanCode: []CodePaths{
{
BasePackage: "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi",
CodePath: "../../apis/sdkapi/v0alpha1",
BasePackage: "github.com/grafana/grafana-plugin-sdk-go/apis/data",
CodePath: "../../apis/data/v0alpha1",
},
{
BasePackage: "github.com/grafana/grafana-plugin-sdk-go/data",
Expand All @@ -40,7 +40,7 @@ func TestWriteQuerySchema(t *testing.T) {
// // Hide this old property
query.Properties.Delete("datasourceId")

outfile := "../../apis/sdkapi/v0alpha1/query.schema.json"
outfile := "../../apis/data/v0alpha1/query.schema.json"
old, _ := os.ReadFile(outfile)
maybeUpdateFile(t, outfile, query, old)

Expand All @@ -54,10 +54,10 @@ func TestWriteQuerySchema(t *testing.T) {
updateEnumDescriptions(query)
query.ID = ""
query.Version = draft04 // used by kube-openapi
outfile = "../../apis/sdkapi/v0alpha1/query.definition.schema.json"
outfile = "../../apis/data/v0alpha1/query.definition.schema.json"
old, _ = os.ReadFile(outfile)
maybeUpdateFile(t, outfile, query, old)

def := sdkapi.GetOpenAPIDefinitions(nil)["github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1.QueryTypeDefinitionSpec"]
def := sdkapi.GetOpenAPIDefinitions(nil)["github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1.QueryTypeDefinitionSpec"]
require.Equal(t, query.Properties.Len(), len(def.Schema.Properties))
}
2 changes: 1 addition & 1 deletion experimental/schemabuilder/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"encoding/json"
"fmt"

sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/sdkapi/v0alpha1"
sdkapi "github.com/grafana/grafana-plugin-sdk-go/apis/data/v0alpha1"
"k8s.io/kube-openapi/pkg/validation/spec"
)

Expand Down