背景
同事反馈 dev环境的yapi不能登录，看了一下是同事两年前用helm搭建的。单副本使用。
排查发现是后端数据库mongodb数据库挂掉。
root@dev-k8s-master03:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mongodb ClusterIP 10.101.87.7 <none> 27017/TCP 2y39d
yapi-svc NodePort 10.108.180.80 <none> 3000:31781/TCP 2y39d
故障现象
2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory
2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] options: { config: "/opt/bitnami/mongodb/conf/mongodb.conf", net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/opt/bitnami/mongodb/tmp" } }, processManagement: { fork: false, pidFilePath: "/opt/bitnami/mongodb/tmp/mongodb.pid" }, security: { authorization: "enabled" }, setParameter: { enableLocalhostAuthBypass: "false" }, storage: { dbPath: "/bitnami/mongodb/data/db", directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/opt/bitnami/mongodb/logs/mongodb.log", quiet: false, verbosity: 0 } }
2024-04-12T09:52:26.258+0000 W STORAGE [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.
2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-04-12T09:52:26.259+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] WiredTiger metadata corruption detected
2024-04-12T09:52:26.390+0000 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
2024-04-12T09:52:26.390+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701
2024-04-12T09:52:26.390+0000 F - [initandlisten]
2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory
2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] options: { config: “/opt/bitnami/mongodb/conf/mongodb.conf”, net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: “/opt/bitnami/mongodb/tmp” } }, processManagement: { fork: false, pidFilePath: “/opt/bitnami/mongodb/tmp/mongodb.pid” }, security: { authorization: “enabled” }, setParameter: { enableLocalhostAuthBypass: “false” }, storage: { dbPath: “/bitnami/mongodb/data/db”, directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, logRotate: “reopen”, path: “/opt/bitnami/mongodb/logs/mongodb.log”, quiet: false, verbosity: 0 } }
2024-04-12T09:52:26.258+0000 W STORAGE [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.
2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2024-04-12T09:52:26.259+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options
2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] WiredTiger metadata corruption detected
2024-04-12T09:52:26.390+0000 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair
2024-04-12T09:52:26.390+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701
2024-04-12T09:52:26.390+0000 F - [initandlisten]
分析&思路
看前面的E级别的log信息，存储引擎读取root page失败。unable to read root page from file:WiredTiger.wt
找到这个mongo挂载的nfs存储，先将数据库文件做一个备份。
单副本的mongo并且一直不断重启，网上搜到的使用mogod --repair执行修复 根本不能进入到容器中执行的。
在同一个pod中使用跟mongo同样的镜像做一个二次封装，命名为mongodebug 把这个镜像放在mongo的pod中，使得这个pod存在两个容器，然后挂载同一个共享存储，进入到mongodebug执行mongod命令进行修复
实施
必要条件
挂载持久卷存储的容器路径/bitnami/mongodb/data/db
制作mongodebug容器镜像
目的：需要一个可执行的mongd命令
拉镜像，运行，找到容器启动脚本，覆盖掉。
防止读取同一份配置文件不能启动
root@master01:~# docker run -it bitnami/mongodb:4.0.14-debian-9-r24 /bin/bash
10:11:48.80
10:11:48.80 Welcome to the Bitnami mongodb container
10:11:48.81 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
10:11:48.81 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
10:11:48.81 Send us your feedback at containers@bitnami.com
10:11:48.81
I have no name!@7b9781de0eed:/$
I have no name!@7b9781de0eed:/$
I have no name!@7b9781de0eed:/$ ls
bin bitnami boot dev entrypoint.sh etc home lib lib64 libbitnami.sh libcomponent.sh libfile.sh libfs.sh liblog.sh libmongodb.sh libnet.sh libos.sh libservice.sh libvalidations.sh libversion.sh licenses media mnt opt postunpack.sh proc root run run.sh sbin setup.sh srv sys tmp usr var
I have no name!@7b9781de0eed:/$ cat entrypoint.sh
#!/bin/bash
# shellcheck disable=SC1091
set -o errexit
set -o nounset
set -o pipefail
# Load libraries
. /libbitnami.sh
. /libmongodb.sh
# Load MongoDB env. variables
eval "$(mongodb_env)"
print_welcome_page
if [[ "$*" = "/run.sh" ]]; then
info "** Starting MongoDB setup **"
/setup.sh
info "** MongoDB setup finished! **"
fi
echo ""
exec "$@"
I have no name!@7b9781de0eed:/$ cat run.sh
#!/bin/bash
# shellcheck disable=SC1091
set -o errexit
set -o nounset
set -o pipefail
. /libmongodb.sh
. /libos.sh
# Load MongoDB env. variables
eval "$(mongodb_env)"
cmd=$(command -v mongod)
flags=("--config=$MONGODB_CONF_FILE")
if [[ -n "${MONGODB_EXTRA_FLAGS:-}" ]]; then
read -r -a extra_flags <<< "$MONGODB_EXTRA_FLAGS"
flags+=("${extra_flags[@]}")
fi
info "** Starting MongoDB **"
if am_i_root; then
exec gosu "$MONGODB_DAEMON_USER" "${cmd}" "${flags[@]}"
else
exec "${cmd}" "${flags[@]}"
fi
I have no name!@7b9781de0eed:/$ cmd=$(command -v mongod)
I have no name!@7b9781de0eed:/$ echo $cmd
/opt/bitnami/mongodb/bin/mongod
I have no name!@7b9781de0eed:/$ echo $MONGODB_CONF_FILE
/opt/bitnami/mongodb/conf/mongodb.conf
I have no name!@7b9781de0eed:/$ ls
bin bitnami boot dev entrypoint.sh etc home lib lib64 libbitnami.sh libcomponent.sh libfile.sh libfs.sh liblog.sh libmongodb.sh libnet.sh libos.sh libservice.sh libvalidations.sh libversion.sh licenses media mnt opt postunpack.sh proc root run run.sh sbin setup.sh srv sys tmp usr var
I have no name!@7b9781de0eed:/$ ps -ef
UID PID PPID C STIME TTY TIME CMD
1001 1 0 0 10:11 pts/0 00:00:00 /bin/bash
1001 25 1 0 10:25 pts/0 00:00:00 ps -ef
I have no name!@7b9781de0eed:/$ exit
exit
dockerfile文件
root@master01:/data/mongodb# cat Dockerfile
FROM bitnami/mongodb:4.0.14-debian-9-r24
USER root
RUN echo "tail -f /etc/hosts" > /setup.sh
RUN echo "tail -f /etc/hosts" > /run.sh
构建镜像&推送到一个具有公共读的镜像仓库中
root@master01:/data/mongodb# docker build -t xxx/xxx/mongodb-debug:v1 .
测试
root@master01:/data/mongodb# docker run -it xxx/xxx/mongodb-debug:v1 /bin/bash
11:15:17.58
11:15:17.58 Welcome to the Bitnami mongodb container
11:15:17.58 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
11:15:17.58 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
11:15:17.58 Send us your feedback at containers@bitnami.com
11:15:17.58
root@8af93d798c63:/# ls /bitnami/
mongodb
root@8af93d798c63:/# cd bitnami/
root@8af93d798c63:/bitnami# ls
mongodb
编辑部署文件
编辑mongodb的deployment文件，将自定义的mongodb镜像做为第二个容器镜像，并配置pvc
kubectl edit deployment mongodb
containers:
- name: mongodbdebug
image: xxx/istio_test/mongodb-debug:v1
volumeMounts:
- name: data
mountPath: /bitnami/mongodb
- name: mongodb
image: xxx/xxx/mongodb:4.0.14-debian-9-r24
进入自定义的容器内部执行修复
root@dev-k8s-master03:/data# kubectl exec -it mongodb-7f97f799d9-qz82v -c mongodbdebug -- bash
root@mongodb-7f97f799d9-qz82v:/# ls bitnami/mongodb/data/db/
WiredTiger collection-11--5099670552419067160.wt collection-4-5162049135609455324.wt index-1--8902867768542019650.wt index-2--5099670552419067160.wt index-30--5099670552419067160.wt index-44--5099670552419067160.wt index-56--5099670552419067160.wt index-7-5162049135609455324.w
.........
root@mongodb-7f97f799d9-qz82v:/bitnami/mongodb/data/db# mongod --dbpath /bitnami/mongodb/data/db --repair
2024-04-12T11:23:29.167+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] MongoDB starting : pid=40 port=27017 dbpath=/bitnami/mongodb/data/db 64-bit host=mongodb-7f97f799d9-qz82v
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] db version v4.0.14
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] git version: 1622021384533dade8b3c89ed3ecd80e1142c132
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] allocator: tcmalloc
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] modules: none
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] build environment:
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] distmod: debian92
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] distarch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] target_arch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/bitnami/mongodb/data/db" } }
2024-04-12T11:23:29.175+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-04-12T11:23:29.175+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31648M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T11:23:29.284+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
省略日志.......
2024-04-12T11:23:34.236+0000 W STORAGE [initandlisten] Modifications made by repair:
2024-04-12T11:23:34.236+0000 W STORAGE [initandlisten] WiredTiger metadata salvaged
2024-04-12T11:23:34.236+0000 I STORAGE [initandlisten] finished checking dbs
2024-04-12T11:23:34.236+0000 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2024-04-12T11:23:34.238+0000 I STORAGE [initandlisten] Shutting down session sweeper thread
2024-04-12T11:23:34.238+0000 I STORAGE [initandlisten] Finished shutting down session sweeper thread
2024-04-12T11:23:34.378+0000 I STORAGE [initandlisten] shutdown: removing fs lock...
2024-04-12T11:23:34.380+0000 I CONTROL [initandlisten] now exiting
2024-04-12T11:23:34.380+0000 I CONTROL [initandlisten] shutting down with code:0
新的报错
mongodb提示读取数据没权限，看了一下是由于构建镜像时候用了root用户执行命令，导致修改后的文件的所有者是root，chown xxx mongodb数据库文件即可
验证
root@dev-k8s-master03:/data# kubectl get pod
NAME READY STATUS RESTARTS AGE
mongodb-68d99d8cf9-wlcn2 1/1 Running 0 2d16h
yapi-pod-6f889c4b68-6xgqc 1/1 Running 0 2d18h
yapi也能正常登录了。
refer
mongodb社区
https://www.mongodb.com/community/forums/t/my-mongodb-not-working-properly-wt-error-non-specific-wiredtiger-error/145434/4
官网修复意外断电导致的mongo报错
https://www.mongodb.com/zh-cn/docs/manual/tutorial/recover-data-following-unexpected-shutdown/
mongodb镜像层
https://hub.docker.com/layers/bitnami/mongodb/4.0.14-debian-9-r24/images/sha256-fd839bcba360b2f470d003706f21ad4b8b1306be0b74e3ab936ff69b72f2e5d3?context=explore