使用vllm部署neo4j的text2cypher-gemma-2-9b-it-finetuned-2024v1模型

使用vllm部署neo4j的text2cypher-gemma-2-9b-it-finetuned-2024v1模型

系统环境准备

由于使用的基于 nvcr.io/nvidia/cuda:12.1.1-cudnn8-runtime-ubuntu22.04 的 workbench，需要进行以下准备（其他系统环境可忽略）

ldconfig -p | grep libcudnn 找到 libcudnn 的so库，然后建立软链接：

ln -s /lib/x86_64-linux-gnu/libcudnn.so.8 /usr/local/cuda/lib64/libcudnn.so

将以下添加到 ~/.bashrc

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/cuda/lib64/:$LD_LIBRARY_PATH

安装可能用到的CUDA相关包：

apt-get install -y --no-install-recommends \
    cuda-cudart-dev-12-1=12.1.105-1 \
    cuda-command-line-tools-12-1=12.1.1-1 \
    cuda-minimal-build-12-1=12.1.1-1 \
    cuda-libraries-dev-12-1=12.1.1-1 \
    cuda-nvml-dev-12-1=12.1.105-1 \
    cuda-nvprof-12-1=12.1.105-1 \
    libnpp-dev-12-1=12.1.0.40-1 \
    libcusparse-dev-12-1=12.1.0.106-1 \
    libcublas-dev-12-1=12.1.3.1-1 \
    libnccl-dev=2.17.1-1+cuda12.1 \
    cuda-nsight-compute-12-1=12.1.1-1

避免驱动更新：

apt-mark hold cuda-cudart-dev-12-1 cuda-command-line-tools-12-1 cuda-minimal-build-12-1 cuda-libraries-dev-12-1 cuda-nvml-dev-12-1 cuda-nvprof-12-1 libnpp-dev-12-1 libcusparse-dev-12-1 libcublas-dev-12-1 libnccl-dev cuda-nsight-compute-12-1

安装系统依赖：

apt-get update
apt-get install wget unzip
apt-get install ffmpeg libsm6 libxext6

设置 git 代理

git config --system url."https://githubfast.com/".insteadOf https://github.com/

创建conda环境

pytorch

conda create -n py310torch python=3.10
conda activate py310torch
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

bitsandbytes

参考官方文档 https://huggingface.co/docs/bitsandbytes/v0.45.0/en/installation#cuda-compile 安装 bitsandbytes

# bitsandbytes require gcc>=12 for cuda>=12.0
apt update
apt install gcc-12 g++-12
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12 --slave /usr/bin/g++ g++ /usr/bin/g++-12
gcc --version

git clone --depth 1 --branch 0.45.0 https://github.com/bitsandbytes-foundation/bitsandbytes
cd bitsandbytes
pip install -r requirements-dev.txt -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
cmake -DCOMPUTE_BACKEND=cuda -S .
make -j 16
pip install .   # `-e` for "editable" install, when developing BNB (otherwise leave that out)

添加 export BNB_CUDA_VERSION=121 到 ~/.bashrc 后，打开新的shell，执行