在运行 control_robot.py 的时候会启用相机拍摄，lerobot 中封装好了两种相机类型：realsense 和 opencv
realsense 直接使用他们的脚本就可以，但需要在 lerobot/robot_devices/robots/configs.py 中修改相机 serial_number
由于我们设备采用的是 Logitech C922x 相机，所以需要稍微修改一下通讯方式，采用 opencv+usb 的方式读取图像数据
目录
1 相机文件适配
1.1 intelrealsense.py
首先，因为相机不同嘛，直接运行肯定会报错，但相机文件具体改哪个呢，那就运行一下看看错误到哪一步：
Traceback (most recent call last):
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/cameras/intelrealsense.py", line 315, in connect
profile = self.camera.start(config)
RuntimeError: No device connected
Traceback (most recent call last):
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/scripts/control_robot.py", line 437, in <module>
control_robot()
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/configs/parser.py", line 227, in wrapper_inner
response = fn(cfg, *args, **kwargs)
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/scripts/control_robot.py", line 418, in control_robot
teleoperate(robot, cfg.control)
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/utils.py", line 42, in wrapper
raise e
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/utils.py", line 38, in wrapper
return func(robot, *args, **kwargs)
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/scripts/control_robot.py", line 235, in teleoperate
control_loop(
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/datasets/image_writer.py", line 36, in wrapper
raise e
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/datasets/image_writer.py", line 29, in wrapper
return func(*args, **kwargs)
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/control_utils.py", line 227, in control_loop
robot.connect()
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/robots/manipulator.py", line 290, in connect
self.cameras[name].connect()
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/cameras/intelrealsense.py", line 325, in connect
camera_infos = find_cameras()
File "/home/robot/Documents/lerobotnew/lerobot/lerobot/common/robot_devices/cameras/intelrealsense.py", line 67, in find_cameras
raise OSError(
OSError: Not a single camera was detected. Try re-plugging, or re-installing librealsense and its python wrapper pyrealsense2, or updating the firmware.根据报错 OSError: Not a single camera was detected，我们就能定位使用的 realsense 文件位置：lerobot/lerobot/common/robot_devices/cameras/intelrealsense.py
可以看到文件夹中有 intelrealsense.py 和 opencv.py 文件，就是调用不同的类，这就说明已经都帮我们写好了基础脚本
这时候就有两种思路可以将 realsense 专用代码改成通用的 Logitech（或其它 UVC）相机接口：
- 简单粗暴：直接用 OpenCV 的 VideoCapture，把所有跟 pyrealsense2、IntelRealSenseCamera 有关的代码删掉
- 高级一点：复用项目里的 OpenCVCamera 类重新封装
基本原理：OpenCVCameraConfig + 对应的 OpenCVCamera 完全可以拿来当做一个通用的 UVC（比如 Logitech）摄像头接口。核心思路就是把原先对 RealSense 专用的那一套——IntelRealSenseCameraConfig + IntelRealSenseCamera 换成通用的 OpenCV 实现
- 现在有 OpenCVCameraConfig(camera_index, fps, width, height, color_mode, …)，和 RealSense 那套同名字段一一对应
- 所有必需的信息（设备索引、帧率、分辨率、RGB/BGR、旋转）都在 OpenCVCameraConfig 里
同时，为了简单粗暴的最简调用，即不更改其他脚本，只修改 intelrealsense.py：
- 重写 IntelRealSenseCamera 类，使其继承并委托给通用的 OpenCVCamera
- 这样的话无论在 utils.py 里如何导入，都能拿到 OpenCV 驱动下的相机： IntelRealSenseCameraConfig.serial_number 被当作索引（如 /dev/videoN）使用
- 其他参数（fps, width, height, color_mode, rotation, mock）都一一映射给 OpenCVCameraConfig
- 直接用原来的配置（把相机 type 保留为 "intelrealsense"，并把 serial_number: 0）就能无缝切换到 Logitech 等 UVC 相机了
重写后的 intelrealsense.py 如下：
#!/usr/bin/env python
# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
'''
This file contains utilities for recording frames from generic UVC cameras (e.g., Logitech) using OpenCV.
'''
import argparse
import concurrent.futures
import logging
import shutil
import time
from pathlib import Path
import numpy as np
from PIL import Image
from lerobot.common.robot_devices.cameras.configs import OpenCVCameraConfig
from lerobot.common.robot_devices.cameras.opencv import OpenCVCamera
from lerobot.common.robot_devices.utils import (
RobotDeviceAlreadyConnectedError,
RobotDeviceNotConnectedError,
busy_wait,
)
from lerobot.common.utils.utils import capture_timestamp_utc
def save_image(img_array, camera_index, frame_index, images_dir):
try:
img = Image.fromarray(img_array)
path = images_dir / f"camera_{camera_index}_frame_{frame_index:06d}.png"
path.parent.mkdir(parents=True, exist_ok=True)
img.save(str(path), quality=100)
logging.info(f"Saved image: {path}")
except Exception as e:
logging.error(f"Failed to save image for camera {camera_index} frame {frame_index}: {e}")
def save_images_from_cameras(
images_dir: Path,
camera_indices: list[int] | None = None,
fps=None,
width=None,
height=None,
record_time_s=2,
mock=False,
):
'''
Initializes all the cameras and saves images to the directory.
Useful to visually identify the camera associated to a given index.
'''
if camera_indices is None or len(camera_indices) == 0:
camera_indices = [0]
cameras = []
for idx in camera_indices:
print(f"Setting up camera index={idx}")
config = OpenCVCameraConfig(
camera_index=idx,
fps=fps,
width=width,
height=height,
color_mode="rgb",
mock=mock,
)
camera = OpenCVCamera(config)
camera.connect()
print(
f"OpenCVCamera(index={idx}, fps={camera.fps}, width={camera.width}, height={camera.height}, color_mode={camera.color_mode})"
)
cameras.append(camera)
images_dir = Path(images_dir)
if images_dir.exists():
shutil.rmtree(images_dir)
images_dir.mkdir(parents=True, exist_ok=True)
print(f"Saving images to {images_dir}")
frame_index = 0
start_time = time.perf_counter()
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
while True:
now = time.perf_counter()
for i, camera in enumerate(cameras):
image = camera.read() if fps is None else camera.async_read()
if image is None:
print("No Frame")
continue
executor.submit(
save_image,
image,
camera_indices[i],
frame_index,
images_dir,
)
if fps is not None:
dt_s = time.perf_counter() - now
busy_wait(max(0, 1 / fps - dt_s))
if time.perf_counter() - start_time > record_time_s:
break
print(f"Frame: {frame_index:04d}\tLatency (ms): {(time.perf_counter() - now) * 1000:.2f}")
frame_index += 1
finally:
print(f"Images have been saved to {images_dir}")
for camera in cameras:
camera.disconnect()1.2 configs.py
修改相机配置文件，将 camera_index 参数由序列号改为实际 OpenCV 识别的索引号，因为我是直接改的 IntelRealSenseCameraConfig 类，所以只需要改 serial_number：
cameras: dict[str, CameraConfig] = field(
default_factory=lambda: {
"cam_low": IntelRealSenseCameraConfig(
serial_number=0,
fps=30,
width=640,
height=480,
),
"cam_left_wrist": IntelRealSenseCameraConfig(
serial_number=2,
fps=30,
width=640,
height=480,
),
"cam_right_wrist": IntelRealSenseCameraConfig(
serial_number=4,
fps=30,
width=640,
height=480,
),
}
)以上修改完后，完成相机更换的全部脚本配置
2 其他细节
运行以下命令，可以确定可用相机，它会扫一遍 /dev/video*，并输出哪些是活跃的：
python lerobot/common/robot_devices/cameras/opencv.py
如果运行出现如下报错，是端口选择错误，0-2-4-6端口有相机，是3个外接相机和一个笔记本自带相机，所以要排查掉一个并选择对的端口：
[ WARN:0@0.562] global cap.cpp:215 open VIDEOIO(V4L2): backend is generally available but can't be used to capture by name
Linux detected. Finding available camera indices through scanning '/dev/video*' ports
Camera found at index /dev/video6
Camera found at index /dev/video4
Camera found at index /dev/video2
Camera found at index /dev/video0所以创建一个简单的 Python 脚本快速测试并可视化 ，命名为camera_test.py：
import cv2
def test_camera(port):
cap = cv2.VideoCapture(port)
if not cap.isOpened():
print(f"Camera at {port} could not be opened.")
return
print(f"Displaying camera at {port}. Press 'q' to quit.")
while True:
ret, frame = cap.read()
if not ret:
print("Failed to capture frame.")
break
cv2.imshow(f'Camera at {port}', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
if __name__ == "__main__":
ports = ["/dev/video0", "/dev/video2", "/dev/video4", "/dev/video6"]
for port in ports:
test_camera(port)运行脚本开始测试，会依次打开每个端口对应的相机画面
- 每次打开窗口时，观察显示的画面，记录相应的端口和对应的摄像头
- 按下键盘上的 q 键关闭当前摄像头窗口，自动切换至下一个摄像头进行测试
可以看到图像后逐一排查
如果遇到相机权限问题，则运行：
sudo chmod 666 /dev/video*