Linux环境Doris数据库部署异常问题
1.FE
1.1 Unknown system variable ‘character_set_database’
mysql> status
--------------
mysql Ver 8.0.25 for Linux on x86_64 (MySQL Community Server - GPL)
Connection id: 0
Current database:
Current user: 'admin'@'192.168.0.xxx'
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.7.99 Doris version doris-2.0.8-d083188cdd
Protocol version: 10
Connection: 192.168.0.xxx via TCP/IP
ERROR 1193 (HY000): errCode = 2, detailMessage = Unknown system variable 'character_set_database'
ERROR 2014 (HY000): Commands out of sync; you can't run this command now
Client characterset: utf8mb4
Server characterset: utf8mb4
TCP port: 9034
Binary data as: Hexadecimal
--------------
跟MySQL数据库的版本有关系，可以忽略。
1.2 notify new FE type transfer: UNKNOWN
修改配置文件后，元数据已存在，需要删除元数据：
# 问题
WARN (UNKNOWN fe_6031ab79_c88a_405f_b212_ec8b52178864(-1)|1)
[Env.notifyNewFETypeTransfer():2421] notify new FE type transfer: UNKNOWN
# 删除元数据
rm -rf doris-meta
1.3 mysql_load_server_secure_path
# 问题
mysql> LOAD DATA INFILE '/path/file.txt' INTO TABLE tableName;
ERROR 1105 (HY000): errCode = 2,
detailMessage = Load local data from fe local is not enabled.
If you want to use it, plz set the `mysql_load_server_secure_path` for FE to be a right path.
# fe的配置文件添加路径 /fe/conf/fe.conf
mysql_load_server_secure_path = /path/
指定LOCAL表示读取客户端文件.不指定表示读取FE服务端本地文件. 导入FE本地文件的功能默认是关闭的, 需要在FE节点上设置mysql_load_server_secure_path来指定安全路径, 才能打开该功能。
1.4 Only unique table could be updated
# 问题
java.sql.SQLException: errCode = 2, detailMessage = Only unique table could be updated.
# 主键模型
UNIQUE KEY(`user_id`, `username`)
PROPERTIES (
"enable_unique_key_merge_on_write" = "true"
);
1.5 too many filtered rows
# 问题
Failed to execute sql: java.sql.SQLException: (conn=4) errCode = 2, detailMessage = [INTERNAL_ERROR]too many filtered rows
0# std::_Function_handler<void (doris::RuntimeState*, doris::Status*), doris::StreamLoadExecutor::execute_plan_fragment(std::shared_ptr<doris::StreamLoadContext>)::$_0>::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::Status*&&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:357
1# doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360
2# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701
3# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
4# doris::Thread::supervise_thread(void*) at /var/local/ldb_toolchain/bin/../usr/include/pthread.h:562
5# ?
6# ?
with load id 92db959a-87e3-4e8a-b34b-5ab0333ce0c0
# 通过使用以下命令查看详细的报错信息
show load warnings where label='92db959a-87e3-4e8a-b34b-5ab0333ce0c0'
Reason: column_name[imei], the length of input is too long than schema. first 32 bytes of input str: [2023-10-10 15:59:14] schema length: 15; actual length: 19; . src line [];
2.BE
2.1 Have not get FE Master heartbeat yet
W0416 09:24:00.378782 3838800 olap_server.cpp:667] Have not get FE Master heartbeat yet
讲BE添加到集群即可：
ALTER SYSTEM ADD BACKEND "be_host_ip:heartbeat_service_port";
2.2 overcommit_memory: 2, expect is 1
这个问题是关于Linux操作系统的内存管理参数overcommit_memory。当overcommit_memory设置为2时，表示允许分配超过物理内存和交换空间总和的内存。而期望的值是1，表示不允许分配超过物理内存和交换空间总和的内存。在这种情况下，内存限制检查将交给Doris Allocator处理，否则即使剩余内存充足，Backend（BE）也可能会崩溃。
# 问题
/proc/sys/vm/overcommit_memory: 2, expect is 1, memory limit check is handed over to Doris Allocator, otherwise BE may crash even with remaining memory
# 解决
echo 1 > /proc/sys/vm/overcommit_memory