1.拉取官方镜像
地址: https://hub.docker.com/r/pytorch/torchserve/tags
docker pull pytorch/torchserve:0.7.1-gpu
2. docker启动指令
CPU
docker run --rm -it -d -p 8380:8080 -p 8381:8081 --name torch-server -v /path/model-server/extra-files:/home/model-server/extra-files -v /path/model-server/model-store:/home/model-server/model-store pytorch/torchserve:0.7.1-gpu
GPU
docker run --rm -it -d --gpus all -p 8380:8080 -p 8381:8081 --name torch-server -v /path/model-server/extra-files:/home/model-server/extra-files -v /path/model-server/model-store:/home/model-server/model-store pytorch/torchserve:0.7.1-gpu
/home/model-server/model-store 是docker映射地址，不能更改
进入容器，可以发现各个端口的意义，8080是通信访问接口，8081是管理服务配置接口，8082是服务监控接口
3. 打包模型文件
3.1 使用框架中脚本或者自己写脚本将模型转为torchscript（.pt）
3.2 torchscript转.mar文件
(1) run_hander.py
from xx_model_handler import KnowHandler
_service = KnowHandler()
def handle(data, context):
try:
if not _service.initialized:
print('ENTERING INITIALIZATION')
_service.initialize(context)
if data is None:
return None
data = _service.preprocess(data)
data = _service.inference(data)
data = _service.postprocess(data)
return data
except Exception as e:
raise Exception("Unable to process input data. " + str(e))
(2) xx_model_handler.py
"""
ModelHandler defines a custom model handler.
"""
import torch
import os
import json
import logging
from transformers import BertTokenizer
class KnowHandler(object):
"""
A custom model handler implementation.
"""
def __init__(self):
super(KnowHandler, self).__init__()
self.initialized = False
def initialize(self, ctx):
"""
Initialize model. This will be called during model loading time
:param context: Initial context contains model server system properties.
:return:
"""
self.manifest = ctx.manifest
properties = ctx.system_properties
model_dir = properties.get("model_dir")
serialized_file = self.manifest["model"]["serializedFile"]
model_pt_path = os.path.join(model_dir, serialized_file)
self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
config_path = os.path.join(model_dir, "config.json")
with open(config_path,"r") as fr:
setup_config = json.load(fr)
self.model = torch.jit.load(model_pt_path, map_location=self.device)
self.tokenizer = BertTokenizer(setup_config["vocab_path"])
self.max_length = setup_config["max_length"]
self.initialized = True
# load the model, refer 'custom handler class' above for details
def preprocess(self, data):
"""
Transform raw input into model input data.
:param batch: list of raw requests, should match batch size
:return: list of preprocessed model input data
"""
# Take the input data and make it inference ready
preprocessed_data = data[0].get("data")
if preprocessed_data is None:
preprocessed_data = data[0].get("body")
inputs = preprocessed_data.decode('utf-8')
inputs = json.loads(inputs) # {"text": []}
return inputs
def inference(self, model_input):
"""
Internal inference methods
:param model_input: transformed model input data
:return: list of inference output in NDArray
"""
# Do some inference call to engine here and return output
text = model_input["text"]
inputs = self.tokenizer(
text,
max_length=self.max_length,
truncation=True,
padding='max_length',
return_tensors='pt'
)
#inputs = {k: torch.as_tensor(v, dtype=torch.int64) for k, v in inputs.items()}
for key, value in inputs.items():
if isinstance(value, torch.Tensor):
inputs[key] = value.to(self.device)
input_ids = inputs['input_ids']
token_type_ids = inputs['token_type_ids']
attention_mask = inputs['attention_mask']
logits = self.model(input_ids,attention_mask,token_type_ids)
return logits
def postprocess(self, inference_output):
"""
Return inference result.
:param inference_output: list of inference output
:return: list of predict results
"""
# Take output from network and post-process to desired format
postprocess_output = [inference_output.tolist()]
return postprocess_output
(3) config.json
{
"threshold": 0.8,
"max_length": 40
}
torch-model-archiver --model-name {name of model} --version {模型版本} --serialized-file {torchscript文件地址} --export-path {.mar文件存放地址} --handler run_handler.py --extra-files {其它文件如配置文件等} --runtime python3 -f
torch-model-archiver --model-name my_model --version 1.0 --serialized-file /path/mymodel.pt --export-path /home/model-server/model-store --handler run_handler.py --extra-files "xx_model_handler,utils.py,config.json,vocab.txt" --runtime python -f
–model-name: 模型的名称，后来的接口名称和管理的模型名称都是这个
–serialized-file: 模型环境及代码及参数的打包文件
–export-path: 本次打包文件存放位置
–extra-files: handle.py中需要使用到的其他文件
–handler: 指定handler函数。（模型名:函数名）
-f 覆盖之前导出的同名打包文件
4. torchserver配置接口
（1）查询已注册的模型
curl "http://localhost:8381/models"
（2）注册模型并为模型分配资源
将.mar模型文件注册，注意：.mar文件必须放在model-store文件夹下，即/path/model-server/model-store
curl -X POST "{ip:port}/models?url={.mar文件名}&model_name={model_name}&batch_size=8&max_batch_delay=10&initial_workers=1"
curl -X POST "localhost:8381/models?url=my_model.mar&model_name=my_model&batch_size=8&max_batch_delay=10&initial_workers=1"
（3）查看模型状态
curl http://localhost:8381/models/{model_name}
（4）删除注册模型
curl -X DELETE http://localhost:8381/models/{model_name}/{version}
5. 模型推理
response = requests.post('http://localhost:8380/predictions/{model_name}/{version}',data = data)
# -*- coding: utf-8 -*-
import requests
import json
text = ['xxxxx']
data = {'data':json.dumps({'text':text})}
print(data)
response = requests.post('http://localhost:8380/predictions/my_model',data = data)
print(response)
if response.status_code==200:
vectors = response.json()
print(vectors)
参考：
https://blog.51cto.com/u_16213661/8750698
https://blog.csdn.net/wangzitaotao/article/details/131101852
https://pytorch.org/serve/index.html
https://docs.aws.amazon.com/zh_cn/sagemaker/latest/dg/deploy-models-frameworks-torchserve.html