Skip to content

Commit ce10db8

Browse files
psainicspsainics
committed
DEBUG E2E
1 parent 601f62b commit ce10db8

File tree

17 files changed

+17
-178
lines changed

17 files changed

+17
-178
lines changed

‎src/e2e-test/features/bigquery/multitablesink/BigQueryMultiTableAdditional.feature‎

Lines changed: 0 additions & 160 deletions
Original file line numberDiff line numberDiff line change
@@ -56,163 +56,3 @@ Feature: BigQueryMultiTable sink -Verification of BigQuery to BigQueryMultiTable
5656
Then Open and capture logs
5757
Then Verify the pipeline status is "Succeeded"
5858
Then Validate data transferred from BigQuery To BigQueryMultiTable is equal
59-
60-
@BQ_SINGLE_SOURCE_BQMT_TEST @BQ_SINK_BQMT_TEST
61-
Scenario: Verify data successfully transferred from BigQuery To BigQueryMultiTable in one table
62-
Given Open Datafusion Project to configure pipeline
63-
When Expand Plugin group in the LHS plugins list: "Source"
64-
When Select plugin: "BigQuery" from the plugins list as: "Source"
65-
Then Navigate to the properties page of plugin: "BigQuery"
66-
Then Replace input plugin property: "project" with value: "projectId"
67-
Then Replace input plugin property: "dataset" with value: "dataset"
68-
Then Replace input plugin property: "table" with value: "bqSourceTable"
69-
Then Click on the Get Schema button
70-
Then Validate "BigQuery" plugin properties
71-
Then Close the Plugin Properties page
72-
When Expand Plugin group in the LHS plugins list: "Sink"
73-
When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink"
74-
Then Connect plugins: "BigQuery" and "BigQuery Multi Table" to establish connection
75-
Then Navigate to the properties page of plugin: "BigQuery Multi Table"
76-
And Enter input plugin property: "referenceName" with value: "Reference"
77-
And Replace input plugin property: "project" with value: "projectId"
78-
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
79-
And Enter input plugin property: "dataset" with value: "dataset"
80-
Then Override Service account details if set in environment variables
81-
Then Click plugin property: "flexibleSchema"
82-
Then Validate "BigQuery Multi Table" plugin properties
83-
And Close the Plugin Properties page
84-
Then Save the pipeline
85-
Then Deploy the pipeline
86-
Then Run the Pipeline in Runtime
87-
Then Wait till pipeline is in running state
88-
Then Open and capture logs
89-
Then Verify the pipeline status is "Succeeded"
90-
Then Validate data transferred from BigQuery To BigQueryMultiTable in one table is equal
91-
92-
@BQ_TWO_SOURCE_BQMT_TEST @BQ_EXISTING_TARGET_TEST @BQ_DELETE_TABLES_TEST
93-
Scenario: Verify data successfully transferred from BigQuery To BigQueryMultiTable in two existing tables
94-
Given Open Datafusion Project to configure pipeline
95-
When Expand Plugin group in the LHS plugins list: "Source"
96-
When Select plugin: "BigQuery" from the plugins list as: "Source"
97-
Then Navigate to the properties page of plugin: "BigQuery"
98-
Then Replace input plugin property: "project" with value: "projectId"
99-
Then Replace input plugin property: "dataset" with value: "dataset"
100-
Then Replace input plugin property: "table" with value: "bqSourceTable"
101-
Then Click on the Get Schema button
102-
Then Validate "BigQuery" plugin properties
103-
Then Close the Plugin Properties page
104-
When Expand Plugin group in the LHS plugins list: "Source"
105-
When Select plugin: "BigQuery" from the plugins list as: "Source"
106-
Then Navigate to the properties page of plugin: "BigQuery2"
107-
Then Replace input plugin property: "project" with value: "projectId"
108-
Then Replace input plugin property: "dataset" with value: "dataset"
109-
Then Replace input plugin property: "table" with value: "bqSourceTable2"
110-
Then Click on the Get Schema button
111-
Then Validate "BigQuery2" plugin properties
112-
Then Close the Plugin Properties page
113-
When Expand Plugin group in the LHS plugins list: "Sink"
114-
When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink"
115-
Then Connect plugins: "BigQuery" and "BigQuery Multi Table" to establish connection
116-
Then Connect plugins: "BigQuery2" and "BigQuery Multi Table" to establish connection
117-
Then Navigate to the properties page of plugin: "BigQuery Multi Table"
118-
And Enter input plugin property: "referenceName" with value: "Reference"
119-
And Replace input plugin property: "project" with value: "projectId"
120-
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
121-
And Enter input plugin property: "dataset" with value: "dataset"
122-
Then Override Service account details if set in environment variables
123-
Then Click plugin property: "flexibleSchema"
124-
Then Validate "BigQuery Multi Table" plugin properties
125-
And Close the Plugin Properties page
126-
Then Save the pipeline
127-
Then Deploy the pipeline
128-
Then Run the Pipeline in Runtime
129-
Then Wait till pipeline is in running state
130-
Then Open and capture logs
131-
Then Verify the pipeline status is "Succeeded"
132-
Then Validate data transferred from BigQuery To BigQueryMultiTable is equal
133-
134-
@BQ_TWO_SOURCE_BQMT_TEST @BQ_EXISTING_TARGET_TEST @BQ_DELETE_TABLES_TEST
135-
Scenario: Verify data successfully transferred from BigQuery To BigQueryMultiTable in two existing tables using truncate
136-
Given Open Datafusion Project to configure pipeline
137-
When Expand Plugin group in the LHS plugins list: "Source"
138-
When Select plugin: "BigQuery" from the plugins list as: "Source"
139-
Then Navigate to the properties page of plugin: "BigQuery"
140-
Then Replace input plugin property: "project" with value: "projectId"
141-
Then Replace input plugin property: "dataset" with value: "dataset"
142-
Then Replace input plugin property: "table" with value: "bqSourceTable"
143-
Then Click on the Get Schema button
144-
Then Validate "BigQuery" plugin properties
145-
Then Close the Plugin Properties page
146-
When Expand Plugin group in the LHS plugins list: "Source"
147-
When Select plugin: "BigQuery" from the plugins list as: "Source"
148-
Then Navigate to the properties page of plugin: "BigQuery2"
149-
Then Replace input plugin property: "project" with value: "projectId"
150-
Then Replace input plugin property: "dataset" with value: "dataset"
151-
Then Replace input plugin property: "table" with value: "bqSourceTable2"
152-
Then Click on the Get Schema button
153-
Then Validate "BigQuery2" plugin properties
154-
Then Close the Plugin Properties page
155-
When Expand Plugin group in the LHS plugins list: "Sink"
156-
When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink"
157-
Then Connect plugins: "BigQuery" and "BigQuery Multi Table" to establish connection
158-
Then Connect plugins: "BigQuery2" and "BigQuery Multi Table" to establish connection
159-
Then Navigate to the properties page of plugin: "BigQuery Multi Table"
160-
And Enter input plugin property: "referenceName" with value: "Reference"
161-
And Replace input plugin property: "project" with value: "projectId"
162-
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
163-
And Enter input plugin property: "dataset" with value: "dataset"
164-
Then Override Service account details if set in environment variables
165-
Then Click plugin property: "flexibleSchema"
166-
Then Toggle BigQuery sink property truncateTable to true
167-
Then Validate "BigQuery Multi Table" plugin properties
168-
And Close the Plugin Properties page
169-
Then Save the pipeline
170-
Then Deploy the pipeline
171-
Then Run the Pipeline in Runtime
172-
Then Wait till pipeline is in running state
173-
Then Open and capture logs
174-
Then Verify the pipeline status is "Succeeded"
175-
Then Validate data transferred from BigQuery To BigQueryMultiTable is equal
176-
177-
@BQ_SOURCE_UPDATE_TEST @BQ_EXISTING_TARGET_TEST @BQ_DELETE_TABLES_TEST
178-
Scenario: Verify data successfully transferred from BigQuery To BigQueryMultiTable in two existing tables after updating schema
179-
Given Open Datafusion Project to configure pipeline
180-
When Expand Plugin group in the LHS plugins list: "Source"
181-
When Select plugin: "BigQuery" from the plugins list as: "Source"
182-
Then Navigate to the properties page of plugin: "BigQuery"
183-
Then Replace input plugin property: "project" with value: "projectId"
184-
Then Replace input plugin property: "dataset" with value: "dataset"
185-
Then Replace input plugin property: "table" with value: "bqSourceTable"
186-
Then Click on the Get Schema button
187-
Then Validate "BigQuery" plugin properties
188-
Then Close the Plugin Properties page
189-
When Expand Plugin group in the LHS plugins list: "Source"
190-
When Select plugin: "BigQuery" from the plugins list as: "Source"
191-
Then Navigate to the properties page of plugin: "BigQuery2"
192-
Then Replace input plugin property: "project" with value: "projectId"
193-
Then Replace input plugin property: "dataset" with value: "dataset"
194-
Then Replace input plugin property: "table" with value: "bqSourceTable2"
195-
Then Click on the Get Schema button
196-
Then Validate "BigQuery2" plugin properties
197-
Then Close the Plugin Properties page
198-
When Expand Plugin group in the LHS plugins list: "Sink"
199-
When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink"
200-
Then Connect plugins: "BigQuery" and "BigQuery Multi Table" to establish connection
201-
Then Connect plugins: "BigQuery2" and "BigQuery Multi Table" to establish connection
202-
Then Navigate to the properties page of plugin: "BigQuery Multi Table"
203-
And Enter input plugin property: "referenceName" with value: "Reference"
204-
And Replace input plugin property: "project" with value: "projectId"
205-
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
206-
And Enter input plugin property: "dataset" with value: "dataset"
207-
Then Override Service account details if set in environment variables
208-
Then Click plugin property: "flexibleSchema"
209-
Then Select radio button plugin property: "updateSchema" with value: "true"
210-
Then Validate "BigQuery Multi Table" plugin properties
211-
And Close the Plugin Properties page
212-
Then Save the pipeline
213-
Then Deploy the pipeline
214-
Then Run the Pipeline in Runtime
215-
Then Wait till pipeline is in running state
216-
Then Open and capture logs
217-
Then Verify the pipeline status is "Succeeded"
218-
Then Validate data transferred from BigQuery To BigQueryMultiTable is equal

‎src/e2e-test/java/io/cdap/plugin/bigquery/runners/sinkrunner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
features = {"src/e2e-test/features"},
2828
glue = {"io.cdap.plugin.bigquery.stepsdesign", "io.cdap.plugin.gcs.stepsdesign",
2929
"stepsdesign", "io.cdap.plugin.common.stepsdesign"},
30-
tags = {"@BigQuery_Sink and not @CDAP-20830"},
30+
tags = {"@BigQuery_Sinks and not @CDAP-20830"},
3131
//TODO: Enable test once issue is fixed https://cdap.atlassian.net/browse/CDAP-20830
3232
monochrome = true,
3333
plugin = {"pretty", "html:target/cucumber-html-report/bigquery-sink",

‎src/e2e-test/java/io/cdap/plugin/bigquery/runners/sourcerunner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
features = {"src/e2e-test/features"},
2828
glue = {"io.cdap.plugin.bigquery.stepsdesign", "io.cdap.plugin.gcs.stepsdesign",
2929
"stepsdesign", "io.cdap.plugin.common.stepsdesign", "io.cdap.plugin.pubsub.stepsdesign"},
30-
tags = {"@BigQuery_Source"},
30+
tags = {"@BigQuery_Sources"},
3131
monochrome = true,
3232
plugin = {"pretty", "html:target/cucumber-html-report/bigquery-source",
3333
"json:target/cucumber-reports/cucumber-bigquery-source.json",

‎src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQueryMultiTable.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,7 @@ public void validateDataTransferredFromBigQueryToBigQueryMultiTableIsEqual()
3535
List<String> sourceTables = Arrays.asList(PluginPropertyUtils.pluginProp("bqSourceTable"),
3636
PluginPropertyUtils.pluginProp("bqSourceTable2"));
3737

38-
List<String> targetTables = Arrays.asList(PluginPropertyUtils.pluginProp("bqTargetTable"),
39-
PluginPropertyUtils.pluginProp("bqTargetTable2"));
38+
List<String> targetTables = Arrays.asList("tabA", "tabB");
4039
boolean recordsMatched = BigQueryMultiTableValidation.validateBQToBigQueryMultiTable(sourceTables, targetTables);
4140
Assert.assertTrue("Value of records transferred to the BQ sink should be equal to the value " +
4241
"of the records in the source table", recordsMatched);
@@ -47,7 +46,7 @@ public void validateDataTransferredFromBigQueryToBigQueryMultiTableInOneTableIsE
4746
throws IOException, InterruptedException {
4847
boolean recordsMatched = BigQueryMultiTableValidation.
4948
validateBQToBigQueryMultiTable(Collections.singletonList(PluginPropertyUtils.pluginProp("bqSourceTable")),
50-
Collections.singletonList(PluginPropertyUtils.pluginProp("bqTargetTable")));
49+
Collections.singletonList("tabA"));
5150
Assert.assertTrue("Value of records transferred to the BQ sink should be equal to the value " +
5251
"of the records in the source table", recordsMatched);
5352
}

‎src/e2e-test/java/io/cdap/plugin/bigqueryexecute/runner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
features = {"src/e2e-test/features"},
2828
glue = {"io.cdap.plugin.bigqueryexecute.stepsdesign", "io.cdap.plugin.common.stepsdesign", "stepsdesign",
2929
"io.cdap.plugin.bigquery.stepsdesign"},
30-
tags = {"@BQExecute"},
30+
tags = {"@BQExecutes"},
3131
monochrome = true,
3232
plugin = {"pretty", "html:target/cucumber-html-report/bqExecute",
3333
"json:target/cucumber-reports/cucumber-bqExecute.json",

‎src/e2e-test/java/io/cdap/plugin/bigtable/runners/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
@CucumberOptions(
2828
features = {"src/e2e-test/features"},
2929
glue = {"io.cdap.plugin.bigtable.stepsdesign", "io.cdap.plugin.common.stepsdesign", "stepsdesign"},
30-
tags = {"@BigTable"},
30+
tags = {"@BigTables"},
3131
monochrome = true,
3232
plugin = {"pretty", "html:target/cucumber-html-report/bigtable",
3333
"json:target/cucumber-reports/cucumber-bigtable.json",

‎src/e2e-test/java/io/cdap/plugin/common/runners/cmekrunner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
glue = {"io.cdap.plugin.gcs.stepsdesign", "io.cdap.plugin.bigquery.stepsdesign",
2929
"stepsdesign", "io.cdap.plugin.common.stepsdesign", "io.cdap.plugin.pubsub.stepsdesign",
3030
"io.cdap.plugin.gcsmove.stepsdesign"},
31-
tags = {"@CMEK"},
31+
tags = {"@CMEKS"},
3232
plugin = {"pretty", "html:target/cucumber-html-report/cmek", "json:target/cucumber-reports/cucumber-cmek.json",
3333
"junit:target/cucumber-reports/cucumber-cmek.xml", "io.cdap.e2e.utils.PropModifier:cmek-config.properties"},
3434
monochrome = true

‎src/e2e-test/java/io/cdap/plugin/datastore/runner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
features = {"src/e2e-test/features"},
2828
glue = {"io.cdap.plugin.datastore.stepsdesign", "io.cdap.plugin.common.stepsdesign",
2929
"stepsdesign"},
30-
tags = {"@DataStore"},
30+
tags = {"@DataStores"},
3131
monochrome = true,
3232
plugin = {"pretty", "html:target/cucumber-html-report/datastore",
3333
"json:target/cucumber-reports/cucumber-datastore.json",

‎src/e2e-test/java/io/cdap/plugin/gcs/runners/sinkrunner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
features = {"src/e2e-test/features"},
2828
glue = {"io.cdap.plugin.gcs.stepsdesign", "io.cdap.plugin.bigquery.stepsdesign",
2929
"stepsdesign", "io.cdap.plugin.common.stepsdesign"},
30-
tags = {"@GCS_Sink"},
30+
tags = {"@GCS_Sinks"},
3131
monochrome = true,
3232
plugin = {"pretty", "html:target/cucumber-html-report/gcs-sink",
3333
"json:target/cucumber-reports/cucumber-gcs-sink.json",

‎src/e2e-test/java/io/cdap/plugin/gcs/runners/sourcerunner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
features = {"src/e2e-test/features"},
2929
glue = {"io.cdap.plugin.gcs.stepsdesign", "io.cdap.plugin.bigquery.stepsdesign",
3030
"stepsdesign", "io.cdap.plugin.common.stepsdesign"},
31-
tags = {"@GCS_Source and not @PLUGIN-823 and not @PLUGIN-1113 and not @PLUGIN-825"},
31+
tags = {"@GCS_Sources and not @PLUGIN-823 and not @PLUGIN-1113 and not @PLUGIN-825"},
3232
/* TODO :Enable tests once issues fixed https://cdap.atlassian.net/browse/PLUGIN-823,
3333
https://cdap.atlassian.net/browse/PLUGIN-1113, https://cdap.atlassian.net/browse/PLUGIN-825 */
3434
monochrome = true,

0 commit comments

Comments
 (0)