Merge pull request #9208 from ariesdevil/stream-load-doc

docs: Update streaming load docs and tests

Author: BohuTANG

docs/doc/11-integrations/00-api/03-streaming-load.md

@@ -14,20 +14,20 @@ The Streaming Load API is used to read data from your local files and load it in
 To create a request with the Streaming Load API, follow the format below:
 
 ```bash
-curl -H "<parameter>:<value>"  [-H "<parameter>:<value>"...] -F "upload=@<file_location>" [-F "upload=@<file_location>"] -XPUT http://<user_name>:[password]@<http_handler_host>:<http_handler_port>/v1/streaming_load
+curl -H "insert_sql:<value>" -F "upload=@<file_location>" [-F "upload=@<file_location>"] -XPUT http://<user_name>:[password]@<http_handler_host>:<http_handler_port>/v1/streaming_load
 ```
 ## Explaining Argument `-H`
 
-The request usually includes many occurrences of the argument `-H` and each is followed by one of the following parameters to tell Databend how to handle the file you're loading data from. Please note that `insert_sql` is required, and other parameters are optional.
+The request usually includes many occurrences of the argument `-H` and each is followed by one of the following parameters to tell Databend how to handle the file you're loading data from. Please note that `insert_sql` is required.
 
-| Parameter               | Values                                                                                                                                                                                                                                                                                    | Supported Formats         | Examples                                                                                                                              |
-|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
-| insert_sql              | [INSERT_statement] + format [file_format]                                                                                                                                                                                                                                                 | All                       | -H "insert_sql: insert into ontime format CSV"                                                                                        |
-| format_skip_header      | Tells Databend how many lines at the beginning of the file to skip for header.<br /> 0 (default): No lines to skip;<br /> 1: Skip the first line;<br /> N: Skip the first N lines.                                                                                                        | CSV / TSV / NDJSON | -H "format_skip_header: 1"                                                                                                            |
-| format_compression      | Tells Databend the compression format of the file.<br /> NONE (default): Do NOT decompress the file;<br /> AUTO: Automatically decompress the file by suffix;<br />  You can also use one of these values to explicitly specify the compression format: GZIP \                            | BZ2 \| BROTLI \                                                                                                                              | ZSTD \|  DEFALTE \| RAW_DEFLATE. | CSV / TSV / NDJSON | -H "format_compression:auto"                   |
-| format_field_delimiter  | Tells Databend the characters used in the file to separate fields.<br /> Default for CSV files: `,`.<br /> Default for TSV files: `\t`.<br /> Hive output files using [SOH control character (\x01)]( https://en.wikipedia.org/wiki/C0_and_C1_control_codes#SOH) as the field delimiter.  | CSV / TSV                 | -H "format_field_delimiter:,". |
-| format_record_delimiter | Tells Databend the new line characters used in the file to separate records.<br />  Default: `\n`.                                                                                                                                                                                        | CSV / TSV                 | -H "format_recorder_delimiter:\n"                                                                                                     |
-| format_quote           | Tells Databend the quote characters for strings in CSV file.<br /> Default: ""(Double quotes).                                                                                                                                                                                            | CSV                       |                                                                                                                                       |
+| Parameter               | Values                              | Supported Formats         | Examples                                                                                                                              |
+|-------------------------|-------------------------------------|---------------------------|---------------------------------------------------------------------------------------------------------------------------------------|
+| insert_sql              | [INSERT_statement] +  [FILE_FORMAT] | All                       | -H "insert_sql: insert into ontime file_format = (type = 'CSV' skip_header = 1 compression = 'bz2')"                                                                                        |                                                                                                                                                                                          | CSV                       |                                                                                                                                       |
+
+
+> FILE_FORMAT = ( TYPE = { CSV | TSV | NDJSON | PARQUET | XML} [ formatTypeOptions ] )
+> 
+> The `formatTypeOptions` contains the same options as the one for the [COPY_INTO](../../14-sql-commands/10-dml/dml-copy-into-table.md) command.
 
 ## Alternatives to Streaming Load API
 

docs/doc/12-load-data/02-local.md

@@ -43,7 +43,7 @@ CREATE TABLE books
 Create and send the API request with the following scripts:
 
 ```bash
-curl -XPUT 'http://root:@127.0.0.1:8000/v1/streaming_load' -H 'insert_sql: insert into book_db.books format CSV' -H 'format_skip_header: 0' -H 'format_field_delimiter: ,' -H 'format_record_delimiter: \n' -F 'upload=@"./books.csv"'
+curl -XPUT 'http://root:@127.0.0.1:8000/v1/streaming_load' -H 'insert_sql: insert into book_db.books file_format = (type = "CSV" skip_header = 0 field_delimiter = "," record_delimiter = "\n")' -F 'upload=@"./books.csv"'
 ```
 
 Response Example:
@@ -101,7 +101,7 @@ CREATE TABLE bookcomments
 Create and send the API request with the following scripts:
 
 ```bash
-curl -XPUT 'http://root:@127.0.0.1:8000/v1/streaming_load' -H 'insert_sql: insert into book_db.bookcomments(title,author,date)format CSV' -H 'format_skip_header: 0' -H 'format_field_delimiter: ,' -H 'format_record_delimiter: \n' -F 'upload=@"./books.csv"'
+curl -XPUT 'http://root:@127.0.0.1:8000/v1/streaming_load' -H 'insert_sql: insert into book_db.bookcomments(title,author,date) file_format = (type = "CSV" skip_header = 0 field_delimiter = "," record_delimiter = "\n")' -F 'upload=@"./books.csv"'
 ```
 
 Notice that the `insert_sql` part above specifies the columns (title, author, and date) to match the loaded data.

docs/doc/21-use-cases/01-analyze-ontime-with-databend-on-ec2-and-s3.md

@@ -48,7 +48,7 @@ unzip t_ontime.csv.zip
 ```
 
 ```shell title='Load TSV files into Databend'
-curl -H "insert_sql:insert into ontime format TSV" -H "format_skip_header:0" -F "upload=@t_ontime.csv"  -XPUT http://root:@127.0.0.1:8000/v1/streaming_load
+curl -H "insert_sql:insert into ontime file_format = (type = 'TSV' skip_header = 0)" -F "upload=@t_ontime.csv"  -XPUT http://root:@127.0.0.1:8000/v1/streaming_load
 ```
 
 :::tip

docs/doc/21-use-cases/05-analyze-hits-dataset-with-databend.md

@@ -47,7 +47,7 @@ gzip -d hits_1m.csv.gz
 ```
 
 ```shell title='Load CSV files into Databend'
-curl -H "insert_sql:insert into hits format TSV"  -F  "upload=@./hits_1m.tsv"  -XPUT http://user1:abc123@127.0.0.1:8000/v1/streaming_load
+curl -H "insert_sql:insert into hits file_format = (type = 'TSV')"  -F  "upload=@./hits_1m.tsv"  -XPUT http://user1:abc123@127.0.0.1:8000/v1/streaming_load
 ```
 
 ## Step 3. Queries

tests/suites/0_stateless/13_tpch/13_0000_prepare.sh

@@ -111,5 +111,6 @@ tar -zxf ${CURDIR}/data/tpch.tar.gz -C ${CURDIR}/data
 for t in customer lineitem nation orders partsupp part region supplier
 do
     echo "$t"
-    curl -s -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" -H 'insert_sql: insert into '$t' format CSV' -H 'format_skip_header: 0' -H 'format_field_delimiter:|' -H 'format_record_delimiter: \n' -F 'upload=@"'${CURDIR}'/data/tests/suites/0_stateless/13_tpch/data/'$t'.tbl"' > /dev/null 2>&1
+    insert_sql="insert into $t file_format = (type = 'CSV' skip_header = 0 field_delimiter = '|' record_delimiter = '\n')"
+    curl -s -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" -H "insert_sql: ${insert_sql}" -F 'upload=@"'${CURDIR}'/data/tests/suites/0_stateless/13_tpch/data/'$t'.tbl"' > /dev/null 2>&1
 done

tests/suites/1_stateful/01_load_v2/01_0000_streaming_load.result

@@ -10,8 +10,6 @@
 199	2020.0	769
 --ndjson
 198	2020.0	767
---csv using file_format
-199	2020.0	769
 --parquet less
 199	2020.0	769
 --parquet mismatch schema

tests/suites/1_stateful/01_load_v2/01_0000_streaming_load.sh

@@ -38,43 +38,37 @@ fi
 
 # load csv
 echo "--csv"
-curl -H "insert_sql:insert into ontime_streaming_load format Csv" -H "format_skip_header:1" -F "upload=@/tmp/ontime_200.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'CSV' skip_header = 1)" -F "upload=@/tmp/ontime_200.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
 echo "--csv.gz"
 # load csv gz
-curl -H "insert_sql:insert into ontime_streaming_load format Csv" -H "format_skip_header:1" -H "format_compression:gzip" -F "upload=@/tmp/ontime_200.csv.gz" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'CSV' skip_header = 1 compression = 'gzip')" -F "upload=@/tmp/ontime_200.csv.gz" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
 # load csv zstd
 echo "--csv.zstd"
-curl -H "insert_sql:insert into ontime_streaming_load format Csv" -H "format_skip_header:1" -H "format_compression:zstd" -F  "upload=@/tmp/ontime_200.csv.zst" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'CSV' skip_header = 1 compression = 'zstd')" -F  "upload=@/tmp/ontime_200.csv.zst" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
 # load csv bz2
 echo "--csv.bz2"
-curl -H "insert_sql:insert into ontime_streaming_load format Csv" -H "format_skip_header:1" -H "format_compression:bz2" -F  "upload=@/tmp/ontime_200.csv.bz2" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'CSV' skip_header = 1 compression = 'bz2')" -F  "upload=@/tmp/ontime_200.csv.bz2" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
 # load parquet
 echo "--parquet"
-curl -H "insert_sql:insert into ontime_streaming_load format Parquet" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'Parquet')" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
 # load ndjson
 echo "--ndjson"
-curl -H "insert_sql:insert into ontime_streaming_load format NdJson" -H "format_skip_header:1" -F "upload=@/tmp/ontime_200.ndjson" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
-echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
-echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
-
-# load csv using file_format syntax
-echo "--csv using file_format"
-curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'CSV' skip_header = 1)" -F "upload=@/tmp/ontime_200.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into ontime_streaming_load file_format = (type = 'NdJson' skip_header = 1)" -F "upload=@/tmp/ontime_200.ndjson" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT
 echo "truncate table ontime_streaming_load" | $MYSQL_CLIENT_CONNECT
 
@@ -90,13 +84,13 @@ echo 'CREATE TABLE ontime_less
 
 
 echo "--parquet less"
-curl -s -H "insert_sql:insert into ontime_less format Parquet" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load"  > /dev/null 2>&1
+curl -s -H "insert_sql:insert into ontime_less file_format = (type = 'Parquet')" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load"  > /dev/null 2>&1
 echo "select count(1), avg(Year), sum(DayOfWeek)  from ontime_less;" | $MYSQL_CLIENT_CONNECT
 
 # load parquet with mismatch schema
 echo "--parquet mismatch schema"
 cat $CURDIR/../ddl/ontime.sql | sed 's/ontime/ontime_test_mismatch/g' | sed 's/DATE/VARCHAR/g' | $MYSQL_CLIENT_CONNECT
-curl -s -H "insert_sql:insert into ontime_test_mismatch format Parquet" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" | grep -c 'parquet schema mismatch'
+curl -s -H "insert_sql:insert into ontime_test_mismatch file_format = (type = 'Parquet')" -F "upload=@/tmp/ontime_200.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" | grep -c 'parquet schema mismatch'
 
 
 echo "drop table ontime_streaming_load;" | $MYSQL_CLIENT_CONNECT

tests/suites/1_stateful/01_load_v2/01_0000_streaming_load_books.sh

@@ -14,11 +14,11 @@ echo "CREATE TABLE books
 );" | $MYSQL_CLIENT_CONNECT
 
 # load csv
-curl -H "insert_sql:insert into books format CSV" -F "upload=@${CURDIR}/books.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into books file_format = (type = 'CSV')" -F "upload=@${CURDIR}/books.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(), count_if(title is null), count_if(author is null), count_if(date is null), count_if(publish_time is null) from books " |  $MYSQL_CLIENT_CONNECT
 
 # load tsv
-curl -H "insert_sql:insert into books format TSV" -F "upload=@${CURDIR}/books.tsv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into books file_format = (type = 'TSV')" -F "upload=@${CURDIR}/books.tsv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select count(), count_if(title is null), count_if(author is null), count_if(date is null), count_if(publish_time is null) from books " |  $MYSQL_CLIENT_CONNECT
 
 

tests/suites/1_stateful/01_load_v2/01_0004_streaming_parquet_int96.sh

@@ -19,7 +19,7 @@ if [ $? -ne 0 ]; then
 fi
 
 # load parquet
-curl -H "insert_sql:insert into mytime format Parquet" -F "upload=@/tmp/mytime.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into mytime file_format = (type = 'Parquet')" -F "upload=@/tmp/mytime.parquet" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select * from mytime" | $MYSQL_CLIENT_CONNECT
 echo "drop table mytime;" | $MYSQL_CLIENT_CONNECT
 

tests/suites/1_stateful/01_load_v2/01_0004_streaming_variant_load.sh

@@ -32,12 +32,13 @@ if [ $? -ne 0 ]; then
 fi
 
 # load csv
+# todo(ariesdevil): change to new syntax when format_quote landing
 curl -H "insert_sql:insert into variant_test format Csv" -H "format_skip_header:0" -H 'format_field_delimiter: ,' -H 'format_record_delimiter: \n' -H "format_quote: \'" -F "upload=@/tmp/json_sample1.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 curl -H "insert_sql:insert into variant_test format Csv" -H "format_skip_header:0" -H 'format_field_delimiter: |' -H 'format_record_delimiter: \n' -H "format_quote: \'" -F "upload=@/tmp/json_sample2.csv" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select * from variant_test order by Id asc;" | $MYSQL_CLIENT_CONNECT
 
 # load ndjson
-curl -H "insert_sql:insert into variant_test2 format NdJson" -H "format_skip_header:0" -F "upload=@/tmp/json_sample.ndjson" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
+curl -H "insert_sql:insert into variant_test2 file_format = (type = 'NdJson' skip_header = 0)" -F "upload=@/tmp/json_sample.ndjson" -u root: -XPUT "http://localhost:${QUERY_HTTP_HANDLER_PORT}/v1/streaming_load" > /dev/null 2>&1
 echo "select * from variant_test2 order by b asc;" | $MYSQL_CLIENT_CONNECT
 
 echo "drop table variant_test;" | $MYSQL_CLIENT_CONNECT