Skip to content

Commit 2dbd70b

Browse files
yiguoleimorningmanDoris-Extras
authored
1.1.1 rc03 (#11161)
* [fix](fe_plugins) update fe plugins' dependency to fe 1.0-SNAPSHOT (#11141) * 1.1.1 rc03 Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com> Co-authored-by: yiguolei <yiguolei@gmail.com>
1 parent a894905 commit 2dbd70b

File tree

7 files changed

+223
-5
lines changed

7 files changed

+223
-5
lines changed

‎docs/en/docs/ecosystem/audit-plugin.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
{
3+
"title": "Audit log plugin",
4+
"language": "en"
5+
}
6+
---
7+
8+
<!-
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
  http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
->
26+
27+
# Audit log plugin
28+
29+
Doris's audit log plugin was developed based on FE's plugin framework. Is an optional plugin. Users can install or uninstall this plugin at runtime.
30+
31+
This plugin can periodically import the FE audit log into the specified Doris cluster, so that users can easily view and analyze the audit log through SQL.
32+
33+
## Compile, Configure and Deploy
34+
35+
### FE Configuration
36+
37+
FE's plugin framework is an experimental feature, which is closed by default. In the FE configuration file, add `plugin_enable = true` to enable the plugin framework.
38+
39+
### AuditLoader Configuration
40+
41+
The configuration of the auditloader plugin is located in `$ {DORIS}/fe_plugins/auditloader/src/main/assembly/`.
42+
43+
Open `plugin.conf` for configuration. See the comments of the configuration items.
44+
45+
### Compile
46+
47+
After executing `sh build_plugin.sh` in the Doris code directory, you will get the `auditloader.zip` file in the `fe_plugins/output` directory.
48+
49+
### Deployment
50+
51+
You can place this file on an http download server or copy(or unzip) it to the specified directory of all FEs. Here we use the latter.
52+
53+
### Installation
54+
55+
After deployment is complete, and before installing the plugin, you need to create the audit database and tables previously specified in `plugin.conf`. The table creation statement is as follows:
56+
57+
```
58+
create table doris_audit_tbl__
59+
(
60+
query_id varchar(48) comment "Unique query id",
61+
`time` datetime not null comment "Query start time",
62+
client_ip varchar(32) comment "Client IP",
63+
user varchar(64) comment "User name",
64+
db varchar(96) comment "Database of this query",
65+
state varchar(8) comment "Query result state. EOF, ERR, OK",
66+
query_time bigint comment "Query execution time in millisecond",
67+
scan_bytes bigint comment "Total scan bytes of this query",
68+
scan_rows bigint comment "Total scan rows of this query",
69+
return_rows bigint comment "Returned rows of this query",
70+
stmt_id int comment "An incremental id of statement",
71+
is_query tinyint comment "Is this statemt a query. 1 or 0",
72+
frontend_ip varchar(32) comment "Frontend ip of executing this statement",
73+
cpu_time_ms bigint comment "Total scan cpu time in millisecond of this query",
74+
sql_hash varchar(50) comment "Hash value for this query",
75+
sql_digest varchar(48) comment "Sql digest for this query",
76+
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
77+
stmt string comment "The original statement, trimed if longer than 2G"
78+
) engine=OLAP
79+
duplicate key(query_id, `time`, client_ip)
80+
partition by range(`time`) ()
81+
distributed by hash(query_id) buckets 1
82+
properties(
83+
"dynamic_partition.time_unit" = "DAY",
84+
"dynamic_partition.start" = "-30",
85+
"dynamic_partition.end" = "3",
86+
"dynamic_partition.prefix" = "p",
87+
"dynamic_partition.buckets" = "1",
88+
"dynamic_partition.enable" = "true",
89+
"replication_num" = "3"
90+
);
91+
```
92+
93+
>**Notice**
94+
>
95+
> In the above table structure: stmt string, this can only be used in 0.15 and later versions, in previous versions, the field type used varchar
96+
97+
The `dynamic_partition` attribute selects the number of days to keep the audit log based on your needs.
98+
99+
After that, connect to Doris and use the `INSTALL PLUGIN` command to complete the installation. After successful installation, you can see the installed plug-ins through `SHOW PLUGINS`, and the status is `INSTALLED`.
100+
101+
Upon completion, the plug-in will continuously import audit date into this table at specified intervals.

‎docs/zh-CN/docs/ecosystem/audit-plugin.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
{
3+
"title": "审计日志插件",
4+
"language": "zh-CN"
5+
}
6+
---
7+
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
# 审计日志插件
28+
29+
Doris 的审计日志插件是在 FE 的插件框架基础上开发的。是一个可选插件。用户可以在运行时安装或卸载这个插件。
30+
31+
该插件可以将 FE 的审计日志定期的导入到指定 Doris 集群中，以方便用户通过 SQL 对审计日志进行查看和分析。
32+
33+
## 编译、配置和部署
34+
35+
### FE 配置
36+
37+
FE的插件框架当前是实验性功能，Doris中默认关闭，在FE的配置文件中，增加`plugin_enable = true`启用plugin框架
38+
39+
### AuditLoader 配置
40+
41+
auditloader plugin的配置位于`${DORIS}/fe_plugins/auditloader/src/main/assembly/`.
42+
43+
打开 `plugin.conf` 进行配置。配置项说明参见注释。
44+
45+
### 编译
46+
47+
在 Doris 代码目录下执行 `sh build_plugin.sh` 后，会在 `fe_plugins/output` 目录下得到 `auditloader.zip` 文件。
48+
49+
### 部署
50+
51+
您可以将这个文件放置在一个 http 服务器上，或者拷贝`auditloader.zip`(或者解压`auditloader.zip`)到所有 FE 的指定目录下。这里我们使用后者。
52+
53+
### 安装
54+
55+
部署完成后，安装插件前，需要创建之前在 `plugin.conf` 中指定的审计数据库和表。其中建表语句如下：
56+
57+
```
58+
create table doris_audit_tbl__
59+
(
60+
query_id varchar(48) comment "Unique query id",
61+
`time` datetime not null comment "Query start time",
62+
client_ip varchar(32) comment "Client IP",
63+
user varchar(64) comment "User name",
64+
db varchar(96) comment "Database of this query",
65+
state varchar(8) comment "Query result state. EOF, ERR, OK",
66+
query_time bigint comment "Query execution time in millisecond",
67+
scan_bytes bigint comment "Total scan bytes of this query",
68+
scan_rows bigint comment "Total scan rows of this query",
69+
return_rows bigint comment "Returned rows of this query",
70+
stmt_id int comment "An incremental id of statement",
71+
is_query tinyint comment "Is this statemt a query. 1 or 0",
72+
frontend_ip varchar(32) comment "Frontend ip of executing this statement",
73+
cpu_time_ms bigint comment "Total scan cpu time in millisecond of this query",
74+
sql_hash varchar(48) comment "Hash value for this query",
75+
sql_digest varchar(48) comment "Sql digest for this query",
76+
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
77+
stmt string comment "The original statement, trimed if longer than 2G "
78+
) engine=OLAP
79+
duplicate key(query_id, `time`, client_ip)
80+
partition by range(`time`) ()
81+
distributed by hash(query_id) buckets 1
82+
properties(
83+
"dynamic_partition.time_unit" = "DAY",
84+
"dynamic_partition.start" = "-30",
85+
"dynamic_partition.end" = "3",
86+
"dynamic_partition.prefix" = "p",
87+
"dynamic_partition.buckets" = "1",
88+
"dynamic_partition.enable" = "true",
89+
"replication_num" = "3"
90+
);
91+
```
92+
93+
>**注意**
94+
>
95+
> 上面表结构中：stmt string ，这个只能在0.15及之后版本中使用，之前版本，字段类型使用varchar
96+
97+
其中 `dynamic_partition` 属性根据自己的需要，选择审计日志保留的天数。
98+
99+
之后，连接到 Doris 后使用 `INSTALL PLUGIN` 命令完成安装。安装成功后，可以通过 `SHOW PLUGINS` 看到已经安装的插件，并且状态为 `INSTALLED`。
100+
101+
完成后，插件会不断的以指定的时间间隔将审计日志插入到这个表中。

‎fe/fe-core/src/main/java/org/apache/doris/common/util/DigitalVersion.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,8 @@ public static DigitalVersion fromString(String version) throws IllegalArgumentEx
8181
int revision = Integer.parseInt(list[2].trim());
8282

8383
if (major >= 100 || minor >= 100 || revision >= 100) {
84-
throw new IllegalArgumentException("Illegal version format: " + version);
84+
throw new IllegalArgumentException(
85+
"Illegal version format: " + version + ". Expected: major.minor.revision");
8586
}
8687

8788
return new DigitalVersion((byte) major, (byte) minor, (byte) revision);

‎fe_plugins/auditloader/src/main/assembly/plugin.properties

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,6 @@
1818
name=AuditLoader
1919
type=AUDIT
2020
description=load audit log to olap load, and user can view the statistic of queries
21-
version=0.13.1
21+
version=1.0.0
2222
java.version=1.8.0
2323
classname=org.apache.doris.plugin.audit.AuditLoaderPlugin

‎fe_plugins/auditloader/src/main/java/org/apache/doris/plugin/audit/DorisStreamLoader.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ private HttpURLConnection getConnection(String urlStr, String label) throws IOEx
6969

7070
conn.addRequestProperty("label", label);
7171
conn.addRequestProperty("max_filter_ratio", "1.0");
72-
conn.addRequestProperty("columns", "query_id, time, client_ip, user, db, state, query_time, scan_bytes," +
72+
conn.addRequestProperty("columns", "query_id, `time`, client_ip, user, db, state, query_time, scan_bytes," +
7373
" scan_rows, return_rows, stmt_id, is_query, frontend_ip, cpu_time_ms, sql_hash, peak_memory_bytes, stmt");
7474

7575
conn.setDoOutput(true);

‎fe_plugins/pom.xml

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ under the License.
6868
</modules>
6969
<properties>
7070
<log4j2.version>2.17.1</log4j2.version>
71-
<doris.version>0.15-SNAPSHOT</doris.version>
71+
<doris.version>1.0-SNAPSHOT</doris.version>
7272
<project.scm.id>github</project.scm.id>
7373
</properties>
7474
<profiles>
@@ -93,6 +93,21 @@ under the License.
9393
</pluginRepository>
9494
</pluginRepositories>
9595
</profile>
96+
<profile>
97+
<id>general-env</id>
98+
<activation>
99+
<property>
100+
<name>!env.CUSTOM_MAVEN_REPO</name>
101+
</property>
102+
</activation>
103+
<repositories>
104+
<!-- for java-cup -->
105+
<repository>
106+
<id>cloudera-public</id>
107+
<url>https://repository.cloudera.com/artifactory/public/</url>
108+
</repository>
109+
</repositories>
110+
</profile>
96111
</profiles>
97112
<dependencyManagement>
98113
<dependencies>

‎gensrc/script/gen_build_version.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
# contains the build version based on the git hash or svn revision.
2626
##############################################################
2727

28-
build_version="1.1.1-rc02"
28+
build_version="1.1.1-rc03"
2929

3030
unset LANG
3131
unset LC_CTYPE

0 commit comments

Comments
 (0)