[Bug] ModuleNotFoundError: No module named 'transformers_modules.InternLM2'
Traceback (most recent call last):
File "/workspace/xtuner/xtuner/tools/train.py", line 360, in <module>
main()
File "/workspace/xtuner/xtuner/tools/train.py", line 353, in main
runner = RUNNERS.build(cfg)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 196, in build_runner_from_cfg
runner = runner_cls.from_cfg(args) # type: ignore
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 423, in from_cfg
runner = cls(
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 403, in __init__
self.register_hooks(default_hooks, custom_hooks)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 1430, in register_hooks
self.register_custom_hooks(custom_hooks)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 1410, in register_custom_hooks
self.register_hook(hook)
File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/_flexible_runner.py", line 1310, in register_hook
hook_obj = HOOKS.build(hook)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/workspace/xtuner/xtuner/engine/hooks/dataset_info_hook.py", line 24, in __init__
self.tokenizer = BUILDER.build(tokenizer)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/opt/conda/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 915, in from_pretrained
tokenizer_class = get_class_from_dynamic_module(class_ref, pretrained_model_name_or_path, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 553, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module, force_reload=force_download)
File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 250, in get_class_in_module
module_spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/root/.cache/huggingface/modules/transformers_modules/0.83/tokenization_internlm2_fast.py", line 35, in <module>
from .tokenization_internlm2 import InternLM2Tokenizer
ModuleNotFoundError: No module named 'transformers_modules.0'解决方法：
https://github.com/InternLM/InternLM/issues/756
如果是调用本地模型，最内层目录的名称中不能包含“."的符号
这是transformers加载动态包时候的限制