[ Award ] Stability Problem Collection #7071

luzhijing · 2021-11-10T04:58:51Z

luzhijing
Nov 10, 2021

Hi the community,

In order to further improve the stability of Apache Doris , we will collect the stability issues for the whole community from now on.
If you meet some problems such as Frontend & Backend Crash , Backend Abnormal OOM etc.
Please Contact Us , We will help you solve the problems , and if it is confirmed as a BUG , we will fix it in the subsequent versions as soon as possible .

You can also provide the following information so that we can locate the fault causes faster:

Doris vision: e.g. 0.14.0 Release
Error message:
What operation cause the problem: Query , Load , ect.
Performance:
OS: e.g. Centos , Ubantu or else
Machine configuration: CPU Core , Memory , Storage
Architecture: e.g. x86 or ARM
Whether it can be Repetition or not and Repetition Step:
Relevant Issuees: #XXXXX

——————————

为了进一步提升 Apache Doris 的稳定性，从2021年Q4起我们将会持续向社区征集稳定性问题。

如果您遇到了诸如 FE&BE挂掉 或 BE异常OOM 之类的问题，请及时与我们取得联系，我们将会帮您解决问题。如果确认是Bug的话，我们将会尽快修复并在后续版本中完成上线，并且有Apache Doris周边礼品送出～

为了便于可以更好定位问题，可以提供以下信息：

系统版本：例如 0.14.0 Release
报错信息：
执行了什么操作导致的报错：比如执行的SQL、导入任务的数据量及频率等
具体表现：
操作系统：
机器配置：CPU核数、内存、磁盘
平台：x86 或 ARM平台
是否能复现以及复现步骤
相关Issue：

lollipopcs · 2021-11-11T04:05:42Z

lollipopcs
Nov 11, 2021

系统版本：0.15.0-rc03
报错信息：
建表语句中使用了String类型字段。
1.在运行flink程序开窗后，通过stream load实时写入数据到doris；
2.使用insert into select 批量导入数据到doris。
doris的各个日志没有明确报错信息，be节点会无故崩溃挂掉。
具体表现：
fe节点正常，但是be节点会无故崩溃挂掉。
操作系统：
centOS 7
机器配置：
测试环境：3台be节点，8核CPU，16G内存，500G HDD。其中一台 BE 混部一个 FE 节点，无Observer。
生产环境：3台be节点，8核CPU，32G内存，200G SSD。1台fe节点，8核CPU，16G内存，100G SSD。其中一台 BE 混部一个 Observer FE。
平台：x86
是否能复现以及复现步骤：
通过以上操作可以复现。
相关Issue：
同步业务库到doris，做宽表提供报表查询使用。
业务库数据量不大，测试环境每张表只有500W条或更少数据。

问题1：
String类型的字段用在where过滤条件，执行相应sql，导致be节点崩溃挂掉。

问题2：
重建表，并避免所有会涉及到where过滤条件的字段使用String类型，改用varchar。只在类似于memo这种字段使用String，做明细查询使用。
测试环境doris集群，增量数据不多，一开始运行正常，但有时be会偶尔挂掉，起初通过 dmesg -T 查看到系统有kill掉be，一开始会以为是测试环境配置较低，内存不够导致。
随后flink端有大量增量数据存入doris，或者使用insert into select批量导入数据到doris，be节点会无故崩溃挂掉。
查看doris的各个日志没有明确报错信息，通过 dmesg -T查看，内存够用，系统也没有kill掉be。
be.INFO:

be.out日志会有相关问题信息：

随后安装使用 Prometheus 和 Grafana 进项监控，发现be men 内存使用到3G左右，be就会挂掉（不一定是3G）：

为了确定不是资源问题，使用生产环境机器做以上操作，生产环境的be同样会奔溃挂掉。
从而确定不是资源问题导致的。

最后猜测可能还是String类型导致的问题，随后重新建表，并去除所有String类型，全部改用varchar。然后持续导入数据进行压测，doris集群稳定运行。

从而确定是String类型导致doris集群不稳定与be节点的崩溃。

1 reply

yangzhg Nov 23, 2021
Collaborator

0.15.1-rc09 may fix it, you can try it

guonc · 2021-11-12T06:02:36Z

guonc
Nov 12, 2021

系统版本：PALO分支的0.14.13/0.14.13.1/0.15.1-rc08

报错信息：
经单机、集群、不同硬件、是否自己编译等多次测试。结论：fe无报错信息，be报错信息如下
start time: Wed Oct 20 15:44:31 CST 2021
*** Aborted at 1634723518 (unix time) try "date -d @1634723518" if you are using GNU date ***
PC: @ 0x113941a doris::SlotRef::get_string_val()
*** SIGSEGV (@0x0) received by PID 26168 (TID 0x7fc03b949700) from PID 0; stack trace: ***
@ 0x1fd12e2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fc0f8d6a630 (unknown)
@ 0x113941a doris::SlotRef::get_string_val()
@ 0x1122df8 doris::ExprContext::get_value()
@ 0x11336a7 doris::ScalarFnCall::evaluate_children()
@ 0x11376ae doris::ScalarFnCall::interpret_eval<>()
@ 0x119c0c3 doris::EqDoubleValPred::get_boolean_val()
@ 0x18d7d82 doris::ExecNode::eval_conjuncts()
@ 0x1a3d191 doris::BaseScanner::fill_dest_tuple()
@ 0x1a2b84c doris::JsonScanner::get_next()
@ 0x19f1681 doris::BrokerScanNode::scanner_scan()
@ 0x19f21cc doris::BrokerScanNode::scanner_worker()
@ 0x2ff66e0 execute_native_thread_routine
@ 0x7fc0f8d62ea5 start_thread
@ 0x7fc0f87899fd __clone

具体表现：
Doris创建表，使用Routine Load 前置过滤 PRECEDING FILTER n_region_id = 1 and s_alias = 'XXXX' 从Kafka接收数据。
发现使用Routine Load 功能会发生不定期be节点crash的问题（多到一天3次以上，可以做保活，但是这样解决不了根本问题）。
而后偶然发现使用 WHERE n_region_id = 1 and s_alias = 'XXXX' 就没有此问题。

操作系统：
centOS 7

机器配置：
3台be节点，8核CPU，32G内存，4T SSD。1台fe节点，8核CPU，16G内存，1T SSD。同时使用be三个节点做fe高可用（2Fl 1Ob）。

平台：x86

是否能复现以及复现步骤：
建表语句：

CREATE TABLE IF NOT EXISTS gc_test (
    n_roleid BIGINT(20) NOT NULL COMMENT '',
    d_create DATETIME NOT NULL COMMENT '',
    xx BIGINT(20) NOT NULL COMMENT '',
    xx VARCHAR(32) NOT NULL COMMENT '',
    xx INT(11) NOT NULL COMMENT '',
    n_region_id INT(11) NOT NULL COMMENT '',
    s_alias VARCHAR(80) NOT NULL COMMENT ''
    )
    UNIQUE KEY(n_roleid,d_create)
    PARTITION BY RANGE(d_create) ()
    DISTRIBUTED BY HASH(n_roleid) BUCKETS 8
    PROPERTIES (
                   "dynamic_partition.create_history_partition" ="true" ,
                   "dynamic_partition.enable" = "true",
                   "dynamic_partition.time_unit" = "MONTH",
                   "dynamic_partition.start" = "-3",
                   "dynamic_partition.end" = "2",
                   "dynamic_partition.prefix" = "p",
                   "dynamic_partition.buckets" = "8",
                   "dynamic_partition.start_day_of_month" = "1",

                   "replication_num" = "3"
               );

创建 Routine Load：

CREATE ROUTINE LOAD test_db.gc_test_routine_load ON gc_test
COLUMNS(
n_roleid,
d_create,
xx,
xx,
xx,
n_region_id,
s_alias
),
PRECEDING FILTER n_region_id = 1 and s_alias = 'XXXX'
PROPERTIES
(
    "desired_concurrent_number"="1",
    "max_batch_interval" = "20",
    "max_batch_rows" = "300000",
    "max_batch_size" = "209715200",
    "strict_mode" = "false",
    "format" = "json"
)
FROM KAFKA
(
    "kafka_broker_list" = "xx",
    "kafka_topic" = "xx",
    "property.group.id" = "xx",
    "property.client.id" = "xx",
    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
);

Kafka 中消息：
每5S最多2000条，有重复
{"n_region_id":"1","s_alias":"XXXX",....}

最后解决办法：
PRECEDING FILTER 换成 WHERE，就可以规避此问题。

1 reply

morningman Nov 23, 2021
Collaborator

Fixed in #7193, will be release in next version, or you can patch it and compile by yourself.

fvhan · 2021-11-25T10:02:27Z

fvhan
Nov 25, 2021

Doris vision: 0.14.13.1 & 0.15.1-rc09
Performance:
OS: Centos 7
Machine configuration: CPU 40 Core , Memory 64G
Architecture: x86
Error message:
###############################################################################################################
be crash时 be.out文件的日志:
*** Aborted at 1637564200 (unix time) try "date -d @1637564200" if you are using GNU date ***
PC: @ 0x13580d6 doris::RowDescriptor::tuple_is_nullable()
*** SIGSEGV (@0x0) received by PID 3190926 (TID 0x7fff665de700) from PID 0; stack trace: ***
@ 0x1ec6742 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7ffff77c25d0 (unknown)
@ 0x13580d6 doris::RowDescriptor::tuple_is_nullable()
@ 0x1122027 doris::TupleIsNullPredicate::prepare()
@ 0x10e93b2 doris::Expr::prepare()
@ 0x10f3a33 doris::ExprContext::prepare()
@ 0x10ea2de doris::Expr::prepare()
@ 0x1a9a1e3 doris::ResultSink::prepare_exprs()
@ 0x1a9a9b7 doris::ResultSink::prepare()
@ 0x142912b doris::PlanFragmentExecutor::prepare()
@ 0x13a2b4e doris::FragmentExecState::prepare()
@ 0x13a6a27 doris::FragmentMgr::exec_plan_fragment()
@ 0x13a866d doris::FragmentMgr::exec_plan_fragment()
@ 0x146db24 doris::PInternalServiceImpl<>::_exec_plan_fragment()
@ 0x146dbdf doris::PInternalServiceImpl<>::exec_plan_fragment()
@ 0x2039477 brpc::policy::ProcessHttpRequest()
@ 0x2006897 brpc::ProcessInputMessage()
@ 0x200775e brpc::InputMessenger::OnNewMessages()
@ 0x20afd0e brpc::Socket::ProcessEvent()
@ 0x215864f bthread::TaskGroup::task_runner()
@ 0x2149a51 bthread_make_fcontext

################################################################################################################
be crash时 core dump文件用gdb分析的结果：
(gdb) bt
#0 0x00000000013580d6 in operator bool (this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_bvector.h:916
#1 operator* (this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_bvector.h:348
#2 operator[] (__n=, this=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_bvector.h:918
#3 doris::RowDescriptor::tuple_is_nullable (this=0x64371b1f8, tuple_idx=-1398457043) at ../src/runtime/descriptors.cpp:357
#4 0x0000000001122027 in doris::TupleIsNullPredicate::prepare(doris::RuntimeState*, doris::RowDescriptor const&, doris::ExprContext*) ()
at ../src/exprs/tuple_is_null_predicate.cpp:39
#5 0x00000000010e93b2 in doris::Expr::prepare(doris::RuntimeState*, doris::RowDescriptor const&, doris::ExprContext*) ()
at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1043
#6 0x00000000010f3a33 in doris::ExprContext::prepare(doris::RuntimeState*, doris::RowDescriptor const&, std::shared_ptrdoris::MemTracker const&) ()
at ../src/exprs/expr_context.cpp:61
#7 0x00000000010ea2de in doris::Expr::prepare(std::vector<doris::ExprContext*, std::allocatordoris::ExprContext* > const&, doris::RuntimeState*, doris::RowDescriptor const&, std::shared_ptrdoris::MemTracker const&) () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_vector.h:1061
#8 0x0000000001a9a1e3 in doris::ResultSink::prepare_exprs(doris::RuntimeState*) () at ../src/runtime/result_sink.cpp:57
#9 0x0000000001a9a9b7 in doris::ResultSink::prepare(doris::RuntimeState*) () at ../src/runtime/result_sink.cpp:69
#10 0x000000000142912b in doris::PlanFragmentExecutor::prepare(doris::TExecPlanFragmentParams const&, doris::QueryFragmentsCtx const*) ()
at /var/local/thirdparty/installed/include/boost/smart_ptr/scoped_ptr.hpp:109
#11 0x00000000013a2b4e in doris::FragmentExecState::prepare (this=this@entry=0x159272500, params=...) at ../src/runtime/fragment_mgr.cpp:229
#12 0x00000000013a6a27 in doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>)
() at ../src/runtime/fragment_mgr.cpp:609
#13 0x00000000013a866d in doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&) ()
at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:87
#14 0x000000000146db24 in doris::PInternalServiceImpldoris::PBackendService::_exec_plan_fragment (this=this@entry=0xa87bb80, ser_request=...)
at ../src/runtime/exec_env.h:117
---Type to continue, or q to quit---
#15 0x000000000146dbdf in doris::PInternalServiceImpldoris::PBackendService::exec_plan_fragment (this=0xa87bb80, cntl_base=,
request=, response=0xe9e1bdc0, done=0x1721923c0) at /var/local/thirdparty/installed/include/google/protobuf/arenastring.h:231
#16 0x0000000002039477 in brpc::policy::ProcessHttpRequest(brpc::InputMessageBase*) () at ../src/brpc/policy/http_rpc_protocol.cpp:1484
#17 0x0000000002006897 in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x12f5ac6d0) at ../src/brpc/input_messenger.cpp:135
#18 0x000000000200775e in operator() (last_msg=0x12f5ac6d0, this=) at ../src/brpc/input_messenger.cpp:141
#19 ~unique_ptr (this=, __in_chrg=) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unique_ptr.h:361
#20 brpc::InputMessenger::OnNewMessages(brpc::Socket*) () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unique_ptr.h:355
#21 0x00000000020afd0e in brpc::Socket::ProcessEvent(void*) () at ../src/brpc/socket.cpp:1017
#22 0x000000000215864f in bthread::TaskGroup::task_runner(long) () at ../src/bthread/task_group.cpp:296
#23 0x0000000002149a51 in bthread_make_fcontext () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/stl_iterator.h:953
Cannot access memory at address 0x7fff3cb75000
(gdb) f 10
#10 0x000000000142912b in doris::PlanFragmentExecutor::prepare(doris::TExecPlanFragmentParams const&, doris::QueryFragmentsCtx const*) ()
at /var/local/thirdparty/installed/include/boost/smart_ptr/scoped_ptr.hpp:109
109 /var/local/thirdparty/installed/include/boost/smart_ptr/scoped_ptr.hpp: No such file or directory.
(gdb) p _query_id
$1 = {_vptr.TUniqueId = 0x2f9e558 <vtable for doris::TUniqueId+16>, hi = -2936670627391388654, lo = -6006327264025392940}
(gdb) p /x -2936670627391388654
$2 = 0xd73ed99fa0034412
##########################################################################################################
select * from ( SELECT * FROM dwd_telemarketing_cust_market_register_dtl_v) user_defined_view limit 0 #be节点会崩溃
这个视图查询语句会导致be节点崩溃,fe节点正常，未在be节点的日志中发现明显异常
我们验证的结果是：
不加limit 0 就不会导致be节点崩溃：
SELECT * FROM dwd_telemarketing_cust_market_register_dtl_vlimit 0 ;#会崩溃
SELECT * FROM dwd_telemarketing_cust_market_register_dtl_v limit 1;#不会导致be崩溃
SELECT * FROM dwd_telemarketing_cust_market_register_dtl_vlimit 100000;#不会导致be崩溃
SELECT * FROM dwd_telemarketing_cust_market_register_dtl_v;#不会导致be崩溃
相关建表语句和视图语句分别为：

###################################################建表语句###############################################

CREATE TABLE `model_smt_mdm_user` (
  `id` int(11) NOT NULL ,
  `email` varchar(64) NULL ,
  `icc_account` varchar(64) NULL ,
  `create_time` datetime NULL ,
  `name` varchar(32) NULL ,
  `mobile` varchar(16) NULL ,
  `empno` varchar(32) NULL ,
  `emp_password` varchar(64) NULL ,
  `extension_no` int(11) NULL ,
  `extension_password` varchar(64) NULL ,
  `org_id` int(11) NULL ,
  `org_name` varchar(256) NULL ,
  `position_id` int(11) NULL ,
  `position_name` varchar(64) NULL ,
  `assign_limit_day` int(11) NULL ,
  `assign_limit_once` int(11) NULL ,
  `status` tinyint(4) NULL ,
  `create_userid` int(11) NULL ,
  `create_user_name` varchar(64) NULL ,
  `update_time` datetime NULL ,
  `update_userid` int(11) NULL ,
  `update_user_name` varchar(64) NULL ,
  `is_delete` tinyint(4) NULL ,
  `is_examine` tinyint(4) NULL DEFAULT "0" ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(32) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `email`, `icc_account`, `create_time`)
DISTRIBUTED BY HASH(`id`, `email`, `icc_account`, `create_time`) BUCKETS 1
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_smt_zib_call_result` (
  `id` int(11) NOT NULL ,
  `call_recordid` int(11) NULL ,
  `create_time` datetime NULL ,
  `market_order_id` bigint(20) NULL ,
  `activity_customerid` int(11) NULL ,
  `activity_id` varchar(32) NULL ,
  `activity_name` varchar(128) NULL DEFAULT "" ,
  `batch_no` varchar(255) NULL DEFAULT "" ,
  `customer_id` varchar(64) NULL ,
  `mobile` varchar(16) NULL ,
  `customer_name` varchar(32) NULL ,
  `action_first_code` varchar(32) NULL ,
  `action_second_code` varchar(32) NULL ,
  `status` tinyint(4) NULL ,
  `order_status` tinyint(4) NULL DEFAULT "0" ,
  `is_invite` tinyint(4) NULL ,
  `remark` varchar(512) NULL ,
  `user_id` int(11) NULL ,
  `user_name` varchar(32) NULL ,
  `user_empno` varchar(32) NULL ,
  `vgroup_id` int(11) NULL ,
  `vgroup_name` varchar(32) NULL ,
  `org_id` int(11) NULL ,
  `org_name` varchar(32) NULL ,
  `create_userid` int(11) NULL ,
  `update_time` datetime NULL ,
  `update_userid` int(11) NULL ,
  `is_delete` tinyint(4) NULL ,
  `type` tinyint(4) NULL ,
  `ignored` tinyint(4) NULL DEFAULT "0" ,
  `level` tinyint(4) NULL ,
  `first_dial_time` datetime NULL ,
  `last_dial_time` datetime NULL ,
  `success_count` int(11) NULL ,
  `today_dial_count` int(11) NULL DEFAULT "0" ,
  `cruise_call_result` varchar(256) NULL DEFAULT "" ,
  `mobile_enc` varchar(128) NULL ,
  `next_visit_time` varchar(10) NULL DEFAULT "" ,
  `valid_score` int(11) NULL DEFAULT "0" ,
  `customer_name_enc` varchar(128) NULL ,
  `book_time` varchar(20) NULL DEFAULT "" ,
  `last_visit_time` datetime NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(32) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `call_recordid`, `create_time`)
DISTRIBUTED BY HASH(`id`, `call_recordid`, `create_time`) BUCKETS 4
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_resu_info` (
  `id` bigint(20) NOT NULL ,
  `time_inst` datetime NOT NULL ,
  `ident` varchar(20) NULL ,
  `mobile` varchar(64) NOT NULL ,
  `unique_id` varchar(32) NULL ,
  `cust_id` bigint(20) NULL ,
  `name` varchar(60) NULL ,
  `name_enc` varchar(240) NULL ,
  `ident_enc` varchar(80) NULL ,
  `mobile_enc` varchar(80) NULL ,
  `password` varchar(50) NOT NULL ,
  `reg_time` datetime NOT NULL ,
  `modify_time` varchar(30) NOT NULL ,
  `reg_source` varchar(20) NOT NULL ,
  `reg_app_channel` varchar(20) NULL ,
  `reg_device_info` varchar(500) NULL ,
  `status` varchar(20) NULL ,
  `is_mobile_authed` char(1) NOT NULL DEFAULT "0" ,
  `complate_cust_info` varchar(3) NULL ,
  `time_upd` datetime NOT NULL ,
  `has_checked` varchar(3) NULL DEFAULT "0" ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(64) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `time_inst`)
PARTITION BY RANGE(`time_inst`)
(PARTITION p1 VALUES [('0000-01-01 00:00:00'), ('2016-01-01 00:00:00')),
PARTITION p201601 VALUES [('2016-01-01 00:00:00'), ('2016-02-01 00:00:00')),
PARTITION p201602 VALUES [('2016-02-01 00:00:00'), ('2016-03-01 00:00:00')),
PARTITION p201603 VALUES [('2016-03-01 00:00:00'), ('2016-04-01 00:00:00')),
PARTITION p201604 VALUES [('2016-04-01 00:00:00'), ('2016-05-01 00:00:00')),
PARTITION p201605 VALUES [('2016-05-01 00:00:00'), ('2016-06-01 00:00:00')),
PARTITION p201606 VALUES [('2016-06-01 00:00:00'), ('2016-07-01 00:00:00')),
PARTITION p201607 VALUES [('2016-07-01 00:00:00'), ('2016-08-01 00:00:00')),
PARTITION p201608 VALUES [('2016-08-01 00:00:00'), ('2016-09-01 00:00:00')),
PARTITION p201609 VALUES [('2016-09-01 00:00:00'), ('2016-10-01 00:00:00')),
PARTITION p201610 VALUES [('2016-10-01 00:00:00'), ('2016-11-01 00:00:00')),
PARTITION p201611 VALUES [('2016-11-01 00:00:00'), ('2016-12-01 00:00:00')),
PARTITION p201612 VALUES [('2016-12-01 00:00:00'), ('2017-01-01 00:00:00')),
PARTITION p201701 VALUES [('2017-01-01 00:00:00'), ('2017-02-01 00:00:00')),
PARTITION p201702 VALUES [('2017-02-01 00:00:00'), ('2017-03-01 00:00:00')),
PARTITION p201703 VALUES [('2017-03-01 00:00:00'), ('2017-04-01 00:00:00')),
PARTITION p201704 VALUES [('2017-04-01 00:00:00'), ('2017-05-01 00:00:00')),
PARTITION p201705 VALUES [('2017-05-01 00:00:00'), ('2017-06-01 00:00:00')),
PARTITION p201706 VALUES [('2017-06-01 00:00:00'), ('2017-07-01 00:00:00')),
PARTITION p201707 VALUES [('2017-07-01 00:00:00'), ('2017-08-01 00:00:00')),
PARTITION p201708 VALUES [('2017-08-01 00:00:00'), ('2017-09-01 00:00:00')),
PARTITION p201709 VALUES [('2017-09-01 00:00:00'), ('2017-10-01 00:00:00')),
PARTITION p201710 VALUES [('2017-10-01 00:00:00'), ('2017-11-01 00:00:00')),
PARTITION p201711 VALUES [('2017-11-01 00:00:00'), ('2017-12-01 00:00:00')),
PARTITION p201712 VALUES [('2017-12-01 00:00:00'), ('2018-01-01 00:00:00')),
PARTITION p201801 VALUES [('2018-01-01 00:00:00'), ('2018-02-01 00:00:00')),
PARTITION p201802 VALUES [('2018-02-01 00:00:00'), ('2018-03-01 00:00:00')),
PARTITION p201803 VALUES [('2018-03-01 00:00:00'), ('2018-04-01 00:00:00')),
PARTITION p201804 VALUES [('2018-04-01 00:00:00'), ('2018-05-01 00:00:00')),
PARTITION p201805 VALUES [('2018-05-01 00:00:00'), ('2018-06-01 00:00:00')),
PARTITION p201806 VALUES [('2018-06-01 00:00:00'), ('2018-07-01 00:00:00')),
PARTITION p201807 VALUES [('2018-07-01 00:00:00'), ('2018-08-01 00:00:00')),
PARTITION p201808 VALUES [('2018-08-01 00:00:00'), ('2018-09-01 00:00:00')),
PARTITION p201809 VALUES [('2018-09-01 00:00:00'), ('2018-10-01 00:00:00')),
PARTITION p201810 VALUES [('2018-10-01 00:00:00'), ('2018-11-01 00:00:00')),
PARTITION p201811 VALUES [('2018-11-01 00:00:00'), ('2018-12-01 00:00:00')),
PARTITION p201812 VALUES [('2018-12-01 00:00:00'), ('2019-01-01 00:00:00')),
PARTITION p201901 VALUES [('2019-01-01 00:00:00'), ('2019-02-01 00:00:00')),
PARTITION p201902 VALUES [('2019-02-01 00:00:00'), ('2019-03-01 00:00:00')),
PARTITION p201903 VALUES [('2019-03-01 00:00:00'), ('2019-04-01 00:00:00')),
PARTITION p201904 VALUES [('2019-04-01 00:00:00'), ('2019-05-01 00:00:00')),
PARTITION p201905 VALUES [('2019-05-01 00:00:00'), ('2019-06-01 00:00:00')),
PARTITION p201906 VALUES [('2019-06-01 00:00:00'), ('2019-07-01 00:00:00')),
PARTITION p201907 VALUES [('2019-07-01 00:00:00'), ('2019-08-01 00:00:00')),
PARTITION p201908 VALUES [('2019-08-01 00:00:00'), ('2019-09-01 00:00:00')),
PARTITION p201909 VALUES [('2019-09-01 00:00:00'), ('2019-10-01 00:00:00')),
PARTITION p201910 VALUES [('2019-10-01 00:00:00'), ('2019-11-01 00:00:00')),
PARTITION p201911 VALUES [('2019-11-01 00:00:00'), ('2019-12-01 00:00:00')),
PARTITION p201912 VALUES [('2019-12-01 00:00:00'), ('2020-01-01 00:00:00')),
PARTITION p202001 VALUES [('2020-01-01 00:00:00'), ('2020-02-01 00:00:00')),
PARTITION p202002 VALUES [('2020-02-01 00:00:00'), ('2020-03-01 00:00:00')),
PARTITION p202003 VALUES [('2020-03-01 00:00:00'), ('2020-04-01 00:00:00')),
PARTITION p202004 VALUES [('2020-04-01 00:00:00'), ('2020-05-01 00:00:00')),
PARTITION p202005 VALUES [('2020-05-01 00:00:00'), ('2020-06-01 00:00:00')),
PARTITION p202006 VALUES [('2020-06-01 00:00:00'), ('2020-07-01 00:00:00')),
PARTITION p202007 VALUES [('2020-07-01 00:00:00'), ('2020-08-01 00:00:00')),
PARTITION p202008 VALUES [('2020-08-01 00:00:00'), ('2020-09-01 00:00:00')),
PARTITION p202009 VALUES [('2020-09-01 00:00:00'), ('2020-10-01 00:00:00')),
PARTITION p202010 VALUES [('2020-10-01 00:00:00'), ('2020-11-01 00:00:00')),
PARTITION p202011 VALUES [('2020-11-01 00:00:00'), ('2020-12-01 00:00:00')),
PARTITION p202012 VALUES [('2020-12-01 00:00:00'), ('2021-01-01 00:00:00')),
PARTITION p202101 VALUES [('2021-01-01 00:00:00'), ('2021-02-01 00:00:00')),
PARTITION p202102 VALUES [('2021-02-01 00:00:00'), ('2021-03-01 00:00:00')),
PARTITION p202103 VALUES [('2021-03-01 00:00:00'), ('2021-04-01 00:00:00')),
PARTITION p202104 VALUES [('2021-04-01 00:00:00'), ('2021-05-01 00:00:00')),
PARTITION p202105 VALUES [('2021-05-01 00:00:00'), ('2021-06-01 00:00:00')),
PARTITION p202106 VALUES [('2021-06-01 00:00:00'), ('2021-07-01 00:00:00')),
PARTITION p202107 VALUES [('2021-07-01 00:00:00'), ('2021-08-01 00:00:00')),
PARTITION p202108 VALUES [('2021-08-01 00:00:00'), ('2021-09-01 00:00:00')),
PARTITION p202109 VALUES [('2021-09-01 00:00:00'), ('2021-10-01 00:00:00')),
PARTITION p202110 VALUES [('2021-10-01 00:00:00'), ('2021-11-01 00:00:00')),
PARTITION p202111 VALUES [('2021-11-01 00:00:00'), ('2021-12-01 00:00:00')),
PARTITION p202112 VALUES [('2021-12-01 00:00:00'), ('2022-01-01 00:00:00')))
DISTRIBUTED BY HASH(`time_inst`) BUCKETS 1
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "false",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "1",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_resu_info_back` (
  `id` bigint(20) NOT NULL ,
  `time_inst` datetime NOT NULL ,
  `unique_id` varchar(32) NULL ,
  `ident` varchar(20) NULL ,
  `cust_id` bigint(20) NULL ,
  `name` varchar(60) NULL ,
  `name_enc` varchar(240) NULL ,
  `ident_enc` varchar(80) NULL ,
  `mobile` varchar(64) NOT NULL ,
  `mobile_enc` varchar(80) NULL ,
  `password` varchar(50) NOT NULL ,
  `reg_time` datetime NOT NULL ,
  `modify_time` varchar(30) NOT NULL ,
  `reg_source` varchar(20) NOT NULL ,
  `reg_app_channel` varchar(20) NULL ,
  `reg_device_info` varchar(500) NULL ,
  `status` varchar(20) NULL ,
  `is_mobile_authed` char(1) NOT NULL DEFAULT "0" ,
  `complate_cust_info` varchar(3) NULL ,
  `time_upd` datetime NOT NULL ,
  `has_checked` varchar(3) NULL DEFAULT "0" ,
  `oper` varchar(50) NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(64) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `time_inst`)
DISTRIBUTED BY HASH(`time_inst`) BUCKETS 6
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_yllpa_main` (
  `apply_no` varchar(32) NOT NULL ,
  `unique_id` varchar(32) NOT NULL ,
  `apply_time` datetime NOT NULL ,
  `apply_serial_no` varchar(64) NULL ,
  `product_code` varchar(4) NULL ,
  `cooperator_id` varchar(8) NULL ,
  `cooperator_name` varchar(45) NULL ,
  `asset_type` varchar(16) NULL ,
  `pre_apply_no` varchar(32) NULL ,
  `status` varchar(32) NULL ,
  `rtf_state` varchar(32) NULL ,
  `refuse_code` varchar(200) NULL ,
  `refuse_desc` varchar(500) NULL ,
  `contract_no` varchar(32) NULL ,
  `sub_terminal_type` varchar(50) NULL ,
  `apply_type` varchar(10) NULL ,
  `time_inst` datetime NULL ,
  `time_upd` datetime NULL ,
  `comp_time` datetime NULL ,
  `acc_lmt` decimal(19, 4) NULL ,
  `order_id` varchar(32) NULL ,
  `workflow_flag` varchar(10) NULL ,
  `apply_lmt` decimal(19, 4) NULL ,
  `app_loan_term` int(11) NULL ,
  `loan_term` int(11) NULL ,
  `sug_lmt` decimal(19, 4) NULL ,
  `loan_purpose` varchar(10) NULL ,
  `join_life_insurance` char(1) NULL ,
  `loan_fixed_amt` decimal(19, 4) NULL ,
  `due_bill_no` varchar(32) NULL ,
  `apply_from_type` varchar(10) NULL ,
  `memo` varchar(500) NULL ,
  `app_remark` varchar(500) NULL ,
  `confirm_date` datetime NULL ,
  `sub_product_code` varchar(20) NULL ,
  `loan_rate` decimal(11, 8) NULL ,
  `execute_rate` decimal(10, 8) NULL ,
  `fee_rate` decimal(11, 8) NULL ,
  `fee_amt` decimal(19, 4) NULL ,
  `penalty_rate` decimal(10, 8) NULL ,
  `life_insu_fee_rate` decimal(10, 8) NULL ,
  `life_insu_amt` decimal(19, 4) NULL ,
  `installment_fee_rate` decimal(10, 8) NULL ,
  `installment_fee_amt` decimal(19, 4) NULL ,
  `cash_limit_rate` decimal(10, 8) NULL ,
  `pocket_money_limit_rate` decimal(10, 8) NULL ,
  `cash_handling_fee_rate` decimal(10, 8) NULL ,
  `cash_handling_fee_amount` decimal(19, 4) NULL ,
  `ins_rate` decimal(10, 8) NULL ,
  `ins_amt` decimal(19, 4) NULL ,
  `agent_fee_rate` decimal(10, 8) NULL ,
  `agent_fee_amt` decimal(19, 4) NULL ,
  `guaranty_id` varchar(32) NULL ,
  `central_bank_credit_auth_flag` varchar(1) NULL ,
  `social_auth_flag` varchar(1) NULL ,
  `providers_auth_flag` varchar(1) NULL ,
  `sesame_auth_flag` varchar(1) NULL ,
  `auth_flag_ext` varchar(100) NULL ,
  `credit_auth_serial_no` varchar(50) NULL ,
  `patch_type` varchar(50) NULL ,
  `patch_type_complete` varchar(50) NULL ,
  `patch_num` int(11) NULL ,
  `patch_cause` varchar(1024) NULL ,
  `data_complete_type` varchar(256) NULL ,
  `prepay_pkg_ind` varchar(10) NULL ,
  `is_old_customer` varchar(10) NULL ,
  `creadit_report_id` varchar(64) NULL ,
  `market_product_id` varchar(32) NULL ,
  `canvass_approve_state` varchar(32) NULL ,
  `premium_ind` varchar(1) NULL ,
  `premium_Amt` decimal(19, 2) NULL ,
  `client_contract_sign_time` datetime NULL ,
  `fail_reason` varchar(400) NULL ,
  `contract_exp_time` datetime NULL ,
  `apply_channel` varchar(10) NULL ,
  `prepay_pkg_ind_amt` decimal(19, 4) NULL ,
  `prepay_pkg_ind_rate` decimal(10, 8) NULL ,
  `join_screen_insu_ind` char(1) NULL ,
  `screen_insu_fee_rate` decimal(10, 8) NULL ,
  `screen_insu_fee_amt` decimal(19, 4) NULL ,
  `acct_type` varchar(16) NULL ,
  `credit_business_flag` varchar(32) NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(32) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`apply_no`, `unique_id`, `apply_time`)
PARTITION BY RANGE(`apply_time`)
(PARTITION p201511 VALUES [('0000-01-01 00:00:00'), ('2015-12-01 00:00:00')),
PARTITION p201512 VALUES [('2015-12-01 00:00:00'), ('2016-01-01 00:00:00')),
PARTITION p201601 VALUES [('2016-01-01 00:00:00'), ('2016-02-01 00:00:00')),
PARTITION p201602 VALUES [('2016-02-01 00:00:00'), ('2016-03-01 00:00:00')),
PARTITION p201603 VALUES [('2016-03-01 00:00:00'), ('2016-04-01 00:00:00')),
PARTITION p201604 VALUES [('2016-04-01 00:00:00'), ('2016-05-01 00:00:00')),
PARTITION p201605 VALUES [('2016-05-01 00:00:00'), ('2016-06-01 00:00:00')),
PARTITION p201606 VALUES [('2016-06-01 00:00:00'), ('2016-07-01 00:00:00')),
PARTITION p201607 VALUES [('2016-07-01 00:00:00'), ('2016-08-01 00:00:00')),
PARTITION p201608 VALUES [('2016-08-01 00:00:00'), ('2016-09-01 00:00:00')),
PARTITION p201609 VALUES [('2016-09-01 00:00:00'), ('2016-10-01 00:00:00')),
PARTITION p201610 VALUES [('2016-10-01 00:00:00'), ('2016-11-01 00:00:00')),
PARTITION p201611 VALUES [('2016-11-01 00:00:00'), ('2016-12-01 00:00:00')),
PARTITION p201612 VALUES [('2016-12-01 00:00:00'), ('2017-01-01 00:00:00')),
PARTITION p201701 VALUES [('2017-01-01 00:00:00'), ('2017-02-01 00:00:00')),
PARTITION p201702 VALUES [('2017-02-01 00:00:00'), ('2017-03-01 00:00:00')),
PARTITION p201703 VALUES [('2017-03-01 00:00:00'), ('2017-04-01 00:00:00')),
PARTITION p201704 VALUES [('2017-04-01 00:00:00'), ('2017-05-01 00:00:00')),
PARTITION p201705 VALUES [('2017-05-01 00:00:00'), ('2017-06-01 00:00:00')),
PARTITION p201706 VALUES [('2017-06-01 00:00:00'), ('2017-07-01 00:00:00')),
PARTITION p201707 VALUES [('2017-07-01 00:00:00'), ('2017-08-01 00:00:00')),
PARTITION p201708 VALUES [('2017-08-01 00:00:00'), ('2017-09-01 00:00:00')),
PARTITION p201709 VALUES [('2017-09-01 00:00:00'), ('2017-10-01 00:00:00')),
PARTITION p201710 VALUES [('2017-10-01 00:00:00'), ('2017-11-01 00:00:00')),
PARTITION p201711 VALUES [('2017-11-01 00:00:00'), ('2017-12-01 00:00:00')),
PARTITION p201712 VALUES [('2017-12-01 00:00:00'), ('2018-01-01 00:00:00')),
PARTITION p201801 VALUES [('2018-01-01 00:00:00'), ('2018-02-01 00:00:00')),
PARTITION p201802 VALUES [('2018-02-01 00:00:00'), ('2018-03-01 00:00:00')),
PARTITION p201803 VALUES [('2018-03-01 00:00:00'), ('2018-04-01 00:00:00')),
PARTITION p201804 VALUES [('2018-04-01 00:00:00'), ('2018-05-01 00:00:00')),
PARTITION p201805 VALUES [('2018-05-01 00:00:00'), ('2018-06-01 00:00:00')),
PARTITION p201806 VALUES [('2018-06-01 00:00:00'), ('2018-07-01 00:00:00')),
PARTITION p201807 VALUES [('2018-07-01 00:00:00'), ('2018-08-01 00:00:00')),
PARTITION p201808 VALUES [('2018-08-01 00:00:00'), ('2018-09-01 00:00:00')),
PARTITION p201809 VALUES [('2018-09-01 00:00:00'), ('2018-10-01 00:00:00')),
PARTITION p201810 VALUES [('2018-10-01 00:00:00'), ('2018-11-01 00:00:00')),
PARTITION p201811 VALUES [('2018-11-01 00:00:00'), ('2018-12-01 00:00:00')),
PARTITION p201812 VALUES [('2018-12-01 00:00:00'), ('2019-01-01 00:00:00')),
PARTITION p201901 VALUES [('2019-01-01 00:00:00'), ('2019-02-01 00:00:00')),
PARTITION p201902 VALUES [('2019-02-01 00:00:00'), ('2019-03-01 00:00:00')),
PARTITION p201903 VALUES [('2019-03-01 00:00:00'), ('2019-04-01 00:00:00')),
PARTITION p201904 VALUES [('2019-04-01 00:00:00'), ('2019-05-01 00:00:00')),
PARTITION p201905 VALUES [('2019-05-01 00:00:00'), ('2019-06-01 00:00:00')),
PARTITION p201906 VALUES [('2019-06-01 00:00:00'), ('2019-07-01 00:00:00')),
PARTITION p201907 VALUES [('2019-07-01 00:00:00'), ('2019-08-01 00:00:00')),
PARTITION p201908 VALUES [('2019-08-01 00:00:00'), ('2019-09-01 00:00:00')),
PARTITION p201909 VALUES [('2019-09-01 00:00:00'), ('2019-10-01 00:00:00')),
PARTITION p201910 VALUES [('2019-10-01 00:00:00'), ('2019-11-01 00:00:00')),
PARTITION p201911 VALUES [('2019-11-01 00:00:00'), ('2019-12-01 00:00:00')),
PARTITION p201912 VALUES [('2019-12-01 00:00:00'), ('2020-01-01 00:00:00')),
PARTITION p202001 VALUES [('2020-01-01 00:00:00'), ('2020-02-01 00:00:00')),
PARTITION p202002 VALUES [('2020-02-01 00:00:00'), ('2020-03-01 00:00:00')),
PARTITION p202003 VALUES [('2020-03-01 00:00:00'), ('2020-04-01 00:00:00')),
PARTITION p202004 VALUES [('2020-04-01 00:00:00'), ('2020-05-01 00:00:00')),
PARTITION p202005 VALUES [('2020-05-01 00:00:00'), ('2020-06-01 00:00:00')),
PARTITION p202006 VALUES [('2020-06-01 00:00:00'), ('2020-07-01 00:00:00')),
PARTITION p202007 VALUES [('2020-07-01 00:00:00'), ('2020-08-01 00:00:00')),
PARTITION p202008 VALUES [('2020-08-01 00:00:00'), ('2020-09-01 00:00:00')),
PARTITION p202009 VALUES [('2020-09-01 00:00:00'), ('2020-10-01 00:00:00')),
PARTITION p202010 VALUES [('2020-10-01 00:00:00'), ('2020-11-01 00:00:00')),
PARTITION p202011 VALUES [('2020-11-01 00:00:00'), ('2020-12-01 00:00:00')),
PARTITION p202012 VALUES [('2020-12-01 00:00:00'), ('2021-01-01 00:00:00')),
PARTITION p202101 VALUES [('2021-01-01 00:00:00'), ('2021-02-01 00:00:00')),
PARTITION p202102 VALUES [('2021-02-01 00:00:00'), ('2021-03-01 00:00:00')),
PARTITION p202103 VALUES [('2021-03-01 00:00:00'), ('2021-04-01 00:00:00')),
PARTITION p202104 VALUES [('2021-04-01 00:00:00'), ('2021-05-01 00:00:00')),
PARTITION p202105 VALUES [('2021-05-01 00:00:00'), ('2021-06-01 00:00:00')),
PARTITION p202106 VALUES [('2021-06-01 00:00:00'), ('2021-07-01 00:00:00')),
PARTITION p202107 VALUES [('2021-07-01 00:00:00'), ('2021-08-01 00:00:00')),
PARTITION p202108 VALUES [('2021-08-01 00:00:00'), ('2021-09-01 00:00:00')),
PARTITION p202109 VALUES [('2021-09-01 00:00:00'), ('2021-10-01 00:00:00')),
PARTITION p202110 VALUES [('2021-10-01 00:00:00'), ('2021-11-01 00:00:00')),
PARTITION p202111 VALUES [('2021-11-01 00:00:00'), ('2021-12-01 00:00:00')),
PARTITION p202112 VALUES [('2021-12-01 00:00:00'), ('2022-01-01 00:00:00')),
PARTITION p202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00')),
PARTITION p202202 VALUES [('2022-02-01 00:00:00'), ('2022-03-01 00:00:00')))
DISTRIBUTED BY HASH(`apply_no`) BUCKETS 4
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "4",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_smt_zib_activity_customer` (
  `id` int(11) NOT NULL ,
  `create_time` datetime NULL ,
  `callid` varchar(32) NULL ,
  `market_order_id` bigint(20) NULL ,
  `order_type` varchar(64) NULL DEFAULT "CALL_NEW" ,
  `batch_no` varchar(255) NULL DEFAULT "" ,
  `activity_id` varchar(32) NULL ,
  `activity_name` varchar(128) NULL DEFAULT "" ,
  `activity_start_time` datetime NULL ,
  `activity_end_time` datetime NULL ,
  `activity_type` varchar(16) NULL ,
  `customer_id` varchar(64) NULL DEFAULT "" ,
  `balance` decimal(10, 2) NULL ,
  `product_id` varchar(32) NULL DEFAULT "" ,
  `customer_name` varchar(64) NULL ,
  `customer_mobile` varchar(16) NULL DEFAULT "" ,
  `invite_code` varchar(64) NULL ,
  `invite_amount` varchar(16) NULL ,
  `invite_id` varchar(32) NULL ,
  `customer_groupid` varchar(32) NULL ,
  `dynamic_batch_no` varchar(64) NULL ,
  `call_start_time` datetime NULL ,
  `call_end_time` datetime NULL ,
  `status` int(11) NULL ,
  `order_status` tinyint(4) NULL DEFAULT "0" ,
  `order_recover_count` int(11) NULL DEFAULT "0" ,
  `biz_id` bigint(20) NULL ,
  `rule_name` varchar(32) NULL DEFAULT "" ,
  `create_userid` int(11) NULL ,
  `update_time` datetime NULL ,
  `update_userid` int(11) NULL ,
  `is_delete` tinyint(4) NULL ,
  `is_missing_customer` tinyint(4) NULL ,
  `commodity_number` int(11) NULL ,
  `is_finance` varchar(8) NULL ,
  `week_time` int(11) NULL ,
  `idle_begin_time` datetime NULL ,
  `idle_end_time` datetime NULL ,
  `month_rate` varchar(32) NULL ,
  `apply_port` varchar(255) NULL ,
  `apply_channel` varchar(32) NULL ,
  `contract_number` varchar(32) NULL ,
  `offer_effective_begin_time` datetime NULL ,
  `offer_effective_end_time` datetime NULL ,
  `call_count` int(11) NULL DEFAULT "0" ,
  `answer_count` int(11) NULL DEFAULT "0" ,
  `business_line` varchar(63) NULL DEFAULT "" ,
  `product_no` varchar(63) NULL DEFAULT "" ,
  `business_category` varchar(63) NULL DEFAULT "" ,
  `label_category` varchar(63) NULL DEFAULT "" ,
  `customer_group` varchar(63) NULL DEFAULT "" ,
  `last_assigned_time` datetime NULL ,
  `customer_mobile_enc` varchar(128) NULL ,
  `customer_name_enc` varchar(128) NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(64) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `create_time`, `callid`, `market_order_id`)
PARTITION BY RANGE(`create_time`)
(PARTITION p202111 VALUES [('2021-10-01 00:00:00'), ('2021-11-01 00:00:00')),
PARTITION p202112 VALUES [('2021-11-01 00:00:00'), ('2021-12-01 00:00:00')),
PARTITION p202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00')),
PARTITION p202202 VALUES [('2022-02-01 00:00:00'), ('2022-03-01 00:00:00')))
DISTRIBUTED BY HASH(`callid`, `market_order_id`) BUCKETS 8
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "8",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_smt_zib_assign_detail` (
  `id` int(11) NOT NULL ,
  `create_time` datetime NULL ,
  `market_order_id` bigint(20) NULL ,
  `assign_recordid` int(11) NULL DEFAULT "0" ,
  `activity_detailid` int(11) NULL ,
  `vgroup_id` int(11) NULL ,
  `user_id` int(11) NULL ,
  `create_userid` int(11) NULL ,
  `update_time` datetime NULL ,
  `update_userid` int(11) NULL ,
  `is_delete` tinyint(4) NULL ,
  `call_mobile` varchar(30) NULL ,
  `call_mobile_enc` varchar(128) NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(64) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `create_time`, `market_order_id`)
PARTITION BY RANGE(`create_time`)
(PARTITION p202111 VALUES [('2021-10-01 00:00:00'), ('2021-11-01 00:00:00')),
PARTITION p202112 VALUES [('2021-11-01 00:00:00'), ('2021-12-01 00:00:00')),
PARTITION p202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00')),
PARTITION p202202 VALUES [('2022-02-01 00:00:00'), ('2022-03-01 00:00:00')))
DISTRIBUTED BY HASH(`market_order_id`) BUCKETS 10
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "10",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);
CREATE TABLE `model_smt_zib_activity_customer_order` (
  `id` int(11) NOT NULL ,
  `create_time` datetime NULL ,
  `callid` varchar(32) NULL DEFAULT "" ,
  `batch_no` varchar(128) NULL DEFAULT "" ,
  `uniqueid` varchar(64) NULL DEFAULT "" ,
  `customer_mobile` varchar(16) NULL DEFAULT "" ,
  `activity_id` varchar(32) NULL DEFAULT "" ,
  `activity_name` varchar(128) NULL DEFAULT "" ,
  `activity_type` varchar(16) NULL DEFAULT "" ,
  `activity_source_system` varchar(16) NULL DEFAULT "" ,
  `activity_start_time` datetime NULL ,
  `activity_end_time` datetime NULL ,
  `product_code` varchar(32) NULL DEFAULT "" ,
  `customer_groupid` varchar(32) NULL DEFAULT "" ,
  `invite_id` varchar(32) NULL DEFAULT "" ,
  `invite_order` varchar(64) NULL DEFAULT "" ,
  `task_type` varchar(16) NULL DEFAULT "" ,
  `contract_no` varchar(32) NULL DEFAULT "" ,
  `offer_effective_begin_time` datetime NULL ,
  `offer_effective_end_time` datetime NULL ,
  `intellect_temp_id` varchar(32) NULL DEFAULT "" ,
  `intellect_call_start_time` datetime NULL ,
  `intellect_call_expand_param` varchar(4096) NULL ,
  `business_line` varchar(63) NULL DEFAULT "" ,
  `product_no` varchar(63) NULL DEFAULT "" ,
  `business_category` varchar(63) NULL DEFAULT "" ,
  `label_category` varchar(63) NULL DEFAULT "" ,
  `customer_group` varchar(63) NULL DEFAULT "" ,
  `status` tinyint(4) NULL DEFAULT "0" ,
  `order_status` tinyint(4) NULL DEFAULT "0" ,
  `is_delete` tinyint(4) NULL DEFAULT "0" ,
  `customer_mobile_enc` varchar(128) NULL DEFAULT "" ,
  `create_user_id` int(11) NULL ,
  `update_user_id` int(11) NULL ,
  `update_time` datetime NULL ,
  `etl_time` datetime NULL ,
  `etl_record_source` varchar(64) NULL ,
  `etl_record_type` int(11) NULL 
) ENGINE=OLAP
UNIQUE KEY(`id`, `create_time`, `callid`)
PARTITION BY RANGE(`create_time`)
(PARTITION p202108 VALUES [('0000-01-01 00:00:00'), ('2021-09-01 00:00:00')),
PARTITION p202109 VALUES [('2021-09-01 00:00:00'), ('2021-10-01 00:00:00')),
PARTITION p202110 VALUES [('2021-10-01 00:00:00'), ('2021-11-01 00:00:00')),
PARTITION p202111 VALUES [('2021-11-01 00:00:00'), ('2021-12-01 00:00:00')),
PARTITION p202112 VALUES [('2021-12-01 00:00:00'), ('2022-01-01 00:00:00')),
PARTITION p202201 VALUES [('2022-01-01 00:00:00'), ('2022-02-01 00:00:00')))
DISTRIBUTED BY HASH(`create_time`, `callid`) BUCKETS 8
PROPERTIES (
"replication_num" = "3",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "MONTH",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.start" = "-2147483648",
"dynamic_partition.end" = "2",
"dynamic_partition.prefix" = "p",
"dynamic_partition.replication_num" = "3",
"dynamic_partition.buckets" = "8",
"dynamic_partition.create_history_partition" = "false",
"dynamic_partition.history_partition_num" = "-1",
"dynamic_partition.hot_partition_num" = "0",
"dynamic_partition.start_day_of_month" = "1",
"in_memory" = "false",
"storage_format" = "V2"
);

#############################################建视图语句######################################################
CREATE VIEW `dwd_telemarketing_cust_market_register_dtl_v` AS SELECT  `a`.`unique_id`                                                AS `unique_id` ,`a`.`customer_mobile_enc`                                      AS `customer_mobile_enc` ,`a`.`activity_id`                                              AS `activity_id` ,`a`.`activity_name`                                            AS `activity_name` ,`a`.`batch_no`                                                 AS `batch_no` ,`a`.`create_time`                                              AS `create_time` ,`b`.`apply_no`                                                 AS `apply_no` ,`b`.`apply_time`                                               AS `apply_time` ,`b`.`status`                                                   AS `status` ,`d`.`org_leader_name`                                          AS `org_name` ,`d`.`group_leader_name`                                        AS `group_name` ,`d`.`user_empno`                                               AS `user_empno` ,`d`.`user_name`                                                AS `user_name` ,get_json_string(`a`.`intellect_call_expand_param`,'$.user_id') AS `user_id` ,`d`.`effe_iden_time`                                           AS `effe_iden_time` ,`c`.`list_allocation_time`                                     AS `list_allocation_time` FROM ( SELECT  `a12`.`unique_id`                   AS `unique_id` ,`a11`.`customer_mobile_enc`         AS `customer_mobile_enc` ,`a11`.`batch_no`                    AS `batch_no` ,`a11`.`activity_id`                 AS `activity_id` ,`a11`.`activity_name`               AS `activity_name` ,`a11`.`create_time`                 AS `create_time` ,`a11`.`callid`                      AS `callid` ,`a11`.`intellect_call_expand_param` AS `intellect_call_expand_param` FROM `model_smt_zib_activity_customer_order` a11 LEFT OUTER JOIN ( SELECT  `a121`.`unique_id`  AS `unique_id` ,`a121`.`mobile_enc` AS `mobile_enc` FROM `model_resu_info` a121 WHERE `a121`.`unique_id` IS NOT NULL AND trim(`a121`.`unique_id`) != '' UNION ALL SELECT  `a122`.`unique_id`  AS `unique_id` ,`a122`.`mobile_enc` AS `mobile_enc` FROM `model_resu_info_back` a122 WHERE `a122`.`unique_id` IS NOT NULL AND trim(`a122`.`unique_id`) != '' ) a12 ON `a11`.`customer_mobile_enc` = `a12`.`mobile_enc` WHERE `a11`.`create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `a11`.`activity_source_system` = 'MANUAL' AND `a11`.`activity_type` IN ('ZCWSQ_CLM_SS', 'ZCWSQ_CLM_LX') AND (`a11`.`uniqueid` IS NULL OR trim(`a11`.`uniqueid`) = '') UNION ALL SELECT  `a2`.`uniqueid`                    AS `unique_id` ,`a2`.`customer_mobile_enc`         AS `customer_mobile_enc` ,`a2`.`batch_no`                    AS `batch_no` ,`a2`.`activity_id`                 AS `activity_id` ,`a2`.`activity_name`               AS `activity_name` ,`a2`.`create_time`                 AS `create_time` ,`a2`.`callid`                      AS `callid` ,`a2`.`intellect_call_expand_param` AS `intellect_call_expand_param` FROM `model_smt_zib_activity_customer_order` a2 WHERE `a2`.`create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `a2`.`activity_source_system` = 'MANUAL' AND `a2`.`activity_type` IN ('ZCWSQ_CLM_SS', 'ZCWSQ_CLM_LX') AND `a2`.`uniqueid` IS NOT NULL AND trim(`a2`.`uniqueid`) != '' UNION ALL SELECT  `a3`.`customer_id`         AS `unique_id` ,`a3`.`customer_mobile_enc` AS `customer_mobile_enc` ,`a3`.`batch_no`            AS `batch_no` ,`a3`.`activity_id`         AS `activity_id` ,`a3`.`activity_name`       AS `activity_name` ,`a3`.`create_time`         AS `create_time` ,`a3`.`callid`              AS `callid` ,''                         AS `intellect_call_expand_param` FROM `model_smt_zib_activity_customer` a3 WHERE `a3`.`create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `a3`.`order_type` = 'AI_TO_MANUAL' ) a LEFT OUTER JOIN ( SELECT  `unique_id`  AS `unique_id` ,`apply_no`   AS `apply_no` ,`apply_time` AS `apply_time` ,`status`     AS `status` FROM `model_yllpa_main` WHERE `apply_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) ) b ON `a`.`unique_id` = `b`.`unique_id` LEFT OUTER JOIN ( SELECT  `c1`.`id`          AS `id` ,`c1`.`callid`      AS `callid` ,`c1`.`activity_id` AS `activity_id` ,`c2`.`create_time` AS `list_allocation_time` FROM ( SELECT  `callid`          AS `callid` ,`id`              AS `id` ,`activity_id`     AS `activity_id` ,`market_order_id` AS `market_order_id` FROM `model_smt_zib_activity_customer` WHERE `create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `activity_name` REGEXP '未申请|注册未|非注册|MOT|注册' ) c1 INNER JOIN ( SELECT  `market_order_id` AS `market_order_id` ,`create_time`     AS `create_time` FROM ( SELECT  `market_order_id`                                                                         AS `market_order_id` ,`create_time`                                                                             AS `create_time` ,row_number() OVER (PARTITION BY `call_mobile_enc` ORDER BY `create_time` ASC NULLS FIRST) AS `row_order` FROM `model_smt_zib_assign_detail` WHERE `create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `user_id` != 99999999 ) c21 WHERE `c21`.`row_order` = 1 ) c2 ON `c1`.`market_order_id` = `c2`.`market_order_id` ) c ON `a`.`callid` = `c`.`callid` AND `a`.`activity_id` = `c`.`activity_id` LEFT OUTER JOIN ( SELECT  `d1`.`activity_customerid`                                                                                                       AS `activity_customerid` ,`user_empno`                                                                                                                     AS `user_empno` ,`user_name`                                                                                                                      AS `user_name` ,`d1`.`org_name_od`                                                                                                               AS `group_leader_name` ,CASE WHEN (`d3`.`org_leader_name` IS NULL OR `d3`.`org_leader_name` = '') AND `d1`.`org_name_od` = '' THEN '' WHEN `d3`.`org_leader_name` IS NULL OR `d3`.`org_leader_name` = '' THEN `d1`.`org_name_od`  ELSE `d3`.`org_leader_name` END AS `org_leader_name` ,CASE WHEN `action_first_code` IN ('C01','C02') THEN `d1`.`create_time`  ELSE '' END                                              AS `effe_iden_time` FROM ( SELECT  `activity_customerid` AS `activity_customerid` ,`create_time`         AS `create_time` ,`action_first_code`   AS `action_first_code` ,`org_name_od`         AS `org_name_od` ,`user_empno`          AS `user_empno` ,`user_name`           AS `user_name` ,`rnk`                 AS `rnk` FROM ( SELECT  `activity_customerid`                                                                             AS `activity_customerid` ,`create_time`                                                                                     AS `create_time` ,`action_first_code`                                                                               AS `action_first_code` ,`user_empno`                                                                                      AS `user_empno` ,`user_name`                                                                                       AS `user_name` ,CASE WHEN `org_name` LIKE '%（代）%' THEN substr(`org_name`,1,instr(`org_name`,'（') - 1)  ELSE substr(`org_name`,1,char_length(`org_name`) - 1) END AS `org_name_od` ,row_number() OVER (PARTITION BY `activity_customerid` ORDER BY `first_dial_time` ASC NULLS FIRST) AS `rnk` FROM `model_smt_zib_call_result` WHERE `create_time` >= date_sub(date_format(now(), '%Y-%m-%d'), 10) AND `user_empno` != 'ICC_yuce' AND `activity_name` REGEXP '未申请|注册未|非注册|MOT|注册' ) t WHERE `rnk` = 1 ) d1 LEFT OUTER JOIN ( SELECT  `icc_account` AS `icc_account` ,`name`        AS `group_leader_name` ,`org_name`    AS `group_name` FROM `model_smt_mdm_user` WHERE `position_id` = 4 AND `status` = 1 GROUP BY  `icc_account` ,`name` ,`org_name` ) d2 ON `d1`.`org_name_od` = `d2`.`group_leader_name` LEFT OUTER JOIN ( SELECT  `name`     AS `org_leader_name` ,`org_name` AS `org_name` FROM `model_smt_mdm_user` WHERE `position_id` = 5 AND `status` = 1 GROUP BY  `name` ,`org_name` ) d3 ON `d2`.`group_name` = `d3`.`org_name` ) d ON `c`.`id` = `d`.`activity_customerid`;

1 reply

morningman Jan 29, 2022
Collaborator

This bug has been fixed in #7931

ChPi · 2021-11-26T03:28:22Z

ChPi
Nov 26, 2021

系统版本：0.14.0 Release
报错信息：
执行查询sql
具体表现：be挂了
操作系统：centos 7
是否能复现以及复现步骤：生产环境be挂了，同一个sql在其他环境执行成功，难以在其他环境复现，应该跟数据有关。后面doris升级到最新看看
相关Issue：
be info
fail to allocate mem via posix_memalign, res=12, errmsg=Cannot allocate memory
be out
@ 0x7f8efe205400 (unknown)
@ 0x7f8efe31bc40 __memcpy_ssse3
@ 0x11b25d1 doris::Tuple::deep_copy()
@ 0x17f44b3 doris::DataStreamSender::Channel::add_row()
@ 0x17f4acb doris::DataStreamSender::send()
@ 0x119b5c8 doris::PlanFragmentExecutor::open_internal()
@ 0x119bbc0 doris::PlanFragmentExecutor::open()
@ 0x111c18e doris::FragmentExecState::execute()
@ 0x111f4b6 doris::FragmentMgr::_exec_actual()
@ 0x112540c std::_Function_handler<>::_M_invoke()
@ 0x12aeef2 doris::ThreadPool::dispatch_thread()
@ 0x12a6d58 doris::Thread::supervise_thread()
@ 0x7f8efdfbaea5 start_thread
@ 0x7f8efe2cd96d __clone
core dump
#0 0x00007f357a4c1c40 in __memcpy_ssse3 () from /lib64/libc.so.6
#1 0x00000000011b25d1 in deep_copy (convert_ptrs=, pool=0x1fa901f40, desc=..., dst=,
this=) at /root/doris/be/src/runtime/tuple.cpp:76
#2 doris::Tuple::deep_copy (this=, desc=..., pool=0x1fa901f40, convert_ptrs=convert_ptrs@entry=false)
at /root/doris/be/src/runtime/tuple.cpp:59
#3 0x00000000017f44b3 in deep_copy (pool=, desc=..., this=)
at /root/doris/be/src/runtime/tuple.h:76
#4 doris::DataStreamSender::Channel::add_row (this=0x21ab738c0, row=row@entry=0x21a4dc000)
at /root/doris/be/src/runtime/data_stream_sender.cpp:262
#5 0x00000000017f4acb in doris::DataStreamSender::send (this=0x201c7f380, state=, batch=0x201f2e580)
at /root/doris/be/src/runtime/data_stream_sender.cpp:501
#6 0x000000000119b5c8 in doris::PlanFragmentExecutor::open_internal (this=this@entry=0x1fa64a370)
at /root/doris/be/src/runtime/plan_fragment_executor.cpp:310
#7 0x000000000119bbc0 in doris::PlanFragmentExecutor::open (this=this@entry=0x1fa64a370)
at /root/doris/be/src/runtime/plan_fragment_executor.cpp:253
#8 0x000000000111c18e in doris::FragmentExecState::execute (this=0x1fa64a300)
at /root/doris/be/src/runtime/fragment_mgr.cpp:219
#9 0x000000000111f4b6 in doris::FragmentMgr::_exec_actual(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>) (this=0x65ed2c0, exec_state=..., cb=...)
at /root/doris/be/src/runtime/fragment_mgr.cpp:423
#10 0x000000000112540c in __invoke_impl<void, void (doris::FragmentMgr::&)(std::shared_ptrdoris::FragmentExecState, std::function<void(doris::PlanFragmentExecutor)>), doris::FragmentMgr*&, std::shared_ptrdoris::FragmentExecState&, std::function<void(doris::PlanFragmentExecutor*)>&> (__t=@0x219d463b0: 0x65ed2c0, __f=
@0x219d46370: (void (doris::FragmentMgr::)(doris::FragmentMgr * const, std::shared_ptrdoris::FragmentExecState, std::function<void(doris::PlanFragmentExecutor)>)) 0x111f490 <doris::FragmentMgr::_exec_actual(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>)>) at /usr/include/c++/7.3.0/bits/invoke.h:73
---Type to continue, or q to quit---
#11 __invoke<void (doris::FragmentMgr::&)(std::shared_ptrdoris::FragmentExecState, std::function<void(doris::PlanFragmentExecutor)>), doris::FragmentMgr*&, std::shared_ptrdoris::FragmentExecState&, std::function<void(doris::PlanFragmentExecutor*)>&> (__fn=
@0x219d46370: (void (doris::FragmentMgr::)(doris::FragmentMgr * const, std::shared_ptrdoris::FragmentExecState, std::function<void(doris::PlanFragmentExecutor)>)) 0x111f490 <doris::FragmentMgr::_exec_actual(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>)>) at /usr/include/c++/7.3.0/bits/invoke.h:95
#12 __call<void, 0, 1, 2> (__args=..., this=0x219d46370) at /usr/include/c++/7.3.0/functional:632
#13 operator()<> (this=0x219d46370) at /usr/include/c++/7.3.0/functional:718
#14 std::_Function_handler<void (), std::_Bind_result<void, void (doris::FragmentMgr::(doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>))(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>)> >::_M_invoke(std::_Any_data const&) (__functor=...)
at /usr/include/c++/7.3.0/bits/std_function.h:316
#15 0x00000000012aeef2 in operator() (this=0x1fa91cad8) at /usr/include/c++/7.3.0/bits/std_function.h:706
#16 run (this=0x1fa91cad0) at /root/doris/be/src/util/threadpool.cpp:42
#17 doris::ThreadPool::dispatch_thread (this=0x6706000)
at /root/doris/be/src/util/threadpool.cpp:548
#18 0x00000000012a6d58 in operator() (this=0x67d2018) at /usr/include/c++/7.3.0/bits/std_function.h:706
#19 doris::Thread::supervise_thread (arg=0x67d2000) at /root/doris/be/src/util/thread.cpp:385
#20 0x00007f357a160ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f357a47396d in clone () from /lib64/libc.so.6

0 replies

mylifetostart · 2021-12-15T12:07:00Z

mylifetostart
Dec 15, 2021

flink程序使用stream load导入doris，be节点有一个挂了，启动之后，发现flink程序不写入doris，也不报错。必须重启flink任务，数据又重新写入。是不是http连接，在重启be节点之后，没法再连接be?

1 reply

deng-jeffer Jan 8, 2025

flink程序使用stream load导入doris，be节点有一个挂了，启动之后，发现flink程序不写入doris，也不报错。必须重启flink任务，数据又重新写入。是不是http连接，在重启be节点之后，没法再连接be?

你好,最新我也遇到了这样的问题,请问你是这样的解决的呢?

zyw8136 · 2021-12-24T06:40:30Z

zyw8136
Dec 24, 2021

系统版本：0.15.0

报错信息：
Doris的routine load稳定复现crash（librdkakfa导致的）

具体表现：
librdkakfa断掉连接，导致be挂。

操作系统：
centOS 7

平台：x86

我这里目前routine load一段时间就会出现这样的coredump：

2 replies

morningman Dec 27, 2021
Collaborator

挂掉的时候，kafka测是否有什么特殊的操作？这是个已知问题，但是并没有找到可以文档复现的操作步骤。。

zyw8136 Dec 27, 2021

Kafka当时就是在接收flink处理出来的数据和doris通过routine load消费数据，其他的特殊操作还真的没有做！但是每次的core dump都是提示“Closing connection due to nodename change”，但是这个操作当时没有做这个操作！

lymaimei · 2022-08-04T07:55:08Z

lymaimei
Aug 4, 2022

系统版本：1.1.0-rc05
报错信息：日志未发现明显异常
执行了什么操作导致的报错：
执行sql如下：
insert into t
select max(c1)..max(c100) from d inner join b on ..
group by d.gc1,d.gc2..
单线程单个sql执行时间约10s，同时启动16个线程提交sql（插入数据到不同的表）到fe后，fe无响应，之后客户端工具(HeidiSql)执行其他sql亦无响应
同时启动2个无异常，同时启动8个有几率出现(并发时单个sql执行时间约60s)，循环启动每个线程前都Sleep1秒无异常。

具体表现：fe无响应
fe_host:8030/Session 页面显示查询时间持续递增超过20分钟未结束

操作系统：Ubuntu 18.04.3 LTS

机器配置：1FE 2BE 8核32G

平台：x86

可复现

4 replies

dataalive Aug 9, 2022

FE no response , JVM heap size 4G or ?

lymaimei Aug 12, 2022

FE no response , JVM heap size 4G or ?

-Xmx20480m

dataalive Aug 13, 2022

这个FE 无响应的时候可以打个jstack，或者加下微信：warpdrive1987 线下分析下。

lymaimei Sep 6, 2022

Fixed,should be updated on 1.1.2

wenwj0 · 2022-08-09T08:32:08Z

wenwj0
Aug 9, 2022

系统版本：1.0.0
数据：1.5T TPC-H(用tools/tpch-tools生成)

机器配置：CPU核数：44core、内存384G、4节点
平台：x86

操作：执行doris/tools/tpch-tools/queries/q1.sql等多条SQL；
参数配置：set exec_mem_limit=357698037760;
具体表现：
be进程挂掉，界面显示
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason

dmesg -T 显示
[Tue Aug 9 11:46:05 2022] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/,task=doris_be,pid=1111674,uid=0

[Tue Aug 9 11:46:05 2022] Out of memory: Killed process 1111674 (doris_be) total-vm:25801980kB, anon-rss:17316216kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:39496kB oom_score_adj:0

[Tue Aug 9 11:46:06 2022] oom_reaper: reaped process 1111674 (doris_be), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

是否能复现：是
备注：相同的环境测试100G数据量的没有OOM报错；1.5T数据量有部分TPC-H的SQL能跑通。

3 replies

LOVEGISER Aug 9, 2022

这个问题应该在1.1修复了，本质是内存没限制住。升级到1.1现象应该是当大sql执行内存不足时直接提示内存不够。而不会引起be挂掉。

wenwj0 Aug 12, 2022

更换为1.1版本之后还是出现了OOM
[Fri Aug 12 09:38:20 2022] Out of memory: Kill process 39374 (doris_be) score 915 or sacrifice child
[Fri Aug 12 09:38:20 2022] Killed process 39374 (doris_be) total-vm:563596196kB, anon-rss:497038368kB, file-rss:0kB, shmem-rss:0kB

dataalive Aug 13, 2022

这个我们比较奇怪的啊，内存384G，怎么这里提示都快500g了，方便远程的话加下微信：warpdrive1987 分析下

lymaimei · 2022-12-21T09:54:14Z

lymaimei
Dec 21, 2022

系统版本：1.2.0
报错信息：日志未发现明显异常
执行了什么操作导致的报错：
执行sql如下：
CREATE TABLE TP
(
PERIOD DATE,
startPeriod VARCHAR(10)
)
UNIQUE KEY(PERIOD, startPeriod)
DISTRIBUTED BY HASH(PERIOD) BUCKETS 10 PROPERTIES("replication_num" = "1")
CREATE TABLE TP2
(
PERIOD DATE,
startPeriod VARCHAR(10)
)
UNIQUE KEY(PERIOD, startPeriod)
DISTRIBUTED BY HASH(PERIOD) BUCKETS 10 PROPERTIES("replication_num" = "1")
TP 和 TP2 插入相同数据
SELECT * FROM TP2
INNER JOIN TP ON TP.PERIOD = TP2.PERIOD
无结果
具体表现：INNER JOIN ON DATE 无结果

操作系统：Ubuntu 18.04.3 LTS

机器配置：1FE 2BE 8核32G

平台：x86

0 replies

[ Award ] Stability Problem Collection #7071

Uh oh!

Uh oh!

Replies: 9 comments · 13 replies

Uh oh!

Uh oh!

Uh oh!

yangzhg Nov 23, 2021 Collaborator

Uh oh!

Uh oh!

Uh oh!

morningman Nov 23, 2021 Collaborator

Uh oh!

Uh oh!

Uh oh!

morningman Jan 29, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morningman Dec 27, 2021 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 9 comments 13 replies

yangzhg Nov 23, 2021
Collaborator

morningman Nov 23, 2021
Collaborator

morningman Jan 29, 2022
Collaborator

morningman Dec 27, 2021
Collaborator