大模型的webui

发布于:2025-03-17 ⋅ 阅读:(14) ⋅ 点赞:(0)
export XXX_LLM_C=~/xcore-llm/build/Release
export CUDA_VISIBLE_DEVICES=2,3
export CUDACXX=/usr/local/cuda-12.3/bin/nvcc
nohup /usr/bin/python3 /home/src/api_server/api_server.py --model LLama2:7b-chat-hf_A800 --tokenizer /home/weight/Llama-2-7b-chat-hf/ --swap-space 40 --tensor-parallel-size 2 --port 8764 > ~/llm_output.log 2>&1 &

# 天数
docker run -it --privileged --cap-add=ALL --pid=host -p 8764:8764 --name tianshu -v /home/llm/:/root/llm/ -v /home/weight/:/home/weight/ llm_tianshu_4.0.0

Configure runtime environment according to the following article
http://git.xxx.com/xxx/xxx-llm

pip install orjson
pip install starlette
pip install fastapi
pip install transformers
pip install openai

export XXX_LLM_C=/root/xcore-llm/build/Release
export CUDA_VISIBLE_DEVICES=2,3

nohup python3 /root/xxx-llm/src/api_server/api_server.py --model LLama2:7B_MR-V100_XCoreLLM --tokenizer /home/weight/Llama-2-7b-chat-hf/ --swap-space 40 --tensor-parallel-size 2 --port 8764 > ~/xxxllm_output.log 2>&1 &

启动webui

docker run -d -p 3210:8080 -e OPENAI_API_KEY='EMPTY' -e OPENAI_API_BASE_URL='http://10.208.130.44:8764/v1' -e WEBUI_AUTH=False --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui-nvidia-xcorellm --restart always ghcr.io/open-webui/open-webui:main
docker run -d -p 3211:8080 -e OPENAI_API_KEY='EMPTY' -e OPENAI_API_BASE_URL='http://192.168.100.101:8764/v1' -e WEBUI_AUTH=False --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui-mr-xcorellm --restart always ghcr.io/open-webui/open-webui:main
docker run -d -p 3212:8080 -e OPENAI_API_KEY='EMPTY' -e OPENAI_API_BASE_URL='http://10.208.130.44:8765/v1' -e WEBUI_AUTH=False --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui-nvidia-vllm --restart always ghcr.io/open-webui/open-webui:main
docker run -d -p 3213:8080 -e OPENAI_API_KEY='EMPTY' -e OPENAI_API_BASE_URL='http://192.168.100.101:8765/v1' -e WEBUI_AUTH=False --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui-mr-vllm --restart always ghcr.io/open-webui/open-webui:main
docker run --rm -d -p 3000:3000 -e VLLM_URL=http://192.168.100.101:8765 ghcr.io/yoziru/nextjs-vllm-ui:latest

curl测试

curl http://120.133.77.251:12345/v1/completions    -H "Authorization: Bearer 2#xY9hJ8sQwR7tZ4pG6mK3nL5vB1@aC0dE"  -H "Content-Type: application/json"     -d '{
        "model": "deepseek-reasoner",
        "prompt": "特朗普是一个",
        "max_tokens": 20,
        "temperature": 0
    }'



curl http://120.133.77.251:12345/v1/chat/completions  -H "Authorization: Bearer 2#xY9hJ8sQwR7tZ4pG6mK3nL5vB1@aC0dE"  -H "Content-Type: application/json"    -d '{"model": "deepseek-reasoner","messages": [{"role": "user", "content": "以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n王利发是老舍戏剧作品____中的主人公。\nA. 《茶馆》\nB. 《龙须沟》\nC. 《方珍珠》\nD. 《女店员》\n答案: \nA\n以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n《等待戈多》>的作者是____。\nA. 加缪\nB. 贝克特\nC. 阿尔比\nD. 尤奈斯库\n答案: \nB\n以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n____创作了舞蹈《马赛曲》等作品,被现代舞蹈家们奉为精神领袖,誉为“现代舞之母”。\nA. 露丝·圣丹尼丝\nB. 安娜·巴普洛娃\nC. 玛莎·格雷姆\nD. 依莎多拉·邓肯\n答案: \nD\n以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n____创立了“叙事体戏剧”理论\nA. 布莱希特\nB. 萨特\nC. 格洛托夫斯基\nD. 贝克特\n答案: \nA\n以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n“伊莎多拉·邓肯”通过研究____创造了现代舞的表演风格。\nA. 奥托艺术\nB. 巴洛克艺术\nC. 古希腊艺术\nD. 拜占廷艺术\n答案: \nC\n以下是中国关于艺术学考试的单项选择题,请选出其中的正确答案。\n最早发展工业设计的北欧国家,开创北欧工业设计之先河的是____。\nA. 瑞典\nB. 芬兰\nC. 挪威\nD. 波兰\n答案:"}],"max_tokens": 100,"temperature": 0}'

curl http://120.133.77.251:12348/v1-openai/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer apikey_3a8c3974c7397c80_59e1f6ae4b9849b5873a59a4a7b0b6db" \
-d '{
  "seed": null,
  "stop": null,
  "temperature": 1,
  "top_p": 1,
  "max_tokens": 32768,
  "model": "deepseek-reasoner",
  "messages": [
    {
      "role": "user",
      "content": "niaho \n"
    }
  ]
}'

数据

context    batch      TTFT ms    TPS (without prefill)      TPS (with prefill)
128+1024   1          74.534     85.686                     85.236
128+1024   8          308.224    75.326                     73.722
128+1024   16         554.945    68.628                     66.223
128+1024   32         847.846    61.197                     58.284
128+1024   64         1429.486   52.311                     48.745
128+1024   128        2513.521   39.213                     35.722

In the tapestry of human history, threads of invention, exploration, and cultural fusion intertwine to form a narrative rich in complexity and diversity. From the ancient civilizations that laid the foundations of modern society such as the Egyptians, who mastered the art of irrigation and monumental architecture, to the Greeks, whose philosophical inquiries and democratic ideals still resonate today the march of progress has been relentless and multifaceted. The Roman Empire, with its unparalleled network of roads, legal systems, and military strategies, exemplified the heights of administrative and organizational prowess, setting the stage for the medieval period. This era, often misconstrued as dark and stagnant, was, in reality, a crucible of innovation and cross-cultural exchange, as seen in the Silk Road's bustling trade and the scholarly pursuits within the Islamic Golden Age. The Renaissance, a rebirth of art and knowledge, emerged from the depths of the Middle Ages, casting a spotlight on humanism and reviving interest in classical antiquity. This period saw luminaries like Leonardo da Vinci and Michelangelo break new ground in art and science, embodying the spirit of inquiry and the pursuit of excellence. The Industrial Revolution marked another pivotal turn, transforming societies with the power of steam, mechanization, and later, electricity. The rapid pace of change brought about by inventors and entrepreneurs like James Watt and Thomas Edison reshaped daily life, economies, and the environment, ushering in the modern era. The 20th century, with its world wars, technological advancements, and social movements, highlighted humanity's capacity for both destruction and profound creativity. The advent of the digital age, led by visionaries such as Steve Jobs and Tim Berners-Lee, has connected the world in ways previously unimaginable, making information and communication virtually instantaneous. As we gaze into the future, the challenges of climate change, resource scarcity, and technological ethics loom large. Yet, the story of human progress is one of resilience and innovation. The quest for sustainable development, the exploration of space, and the advancements in artificial intelligence and biotechnology present new frontiers for humanity to explore. This narrative, weaving through millennia, showcases the indomitable spirit of human curiosity and the relentless pursuit of knowledge. As we stand on the shoulders of giants, we look forward to writing the next chapters of this ongoing saga, hopeful that the future will reflect the best of our collective endeavors. In this continuing saga, the emphasis shifts towards a more interconnected and technologically advanced world. The 21st century has ushered in an era where the digital realm and artificial intelligence not only augment human capabilities but also challenge our ethical frameworks and societal norms. Innovations in machine learning and AI have the potential to revolutionize industries, from healthcare, where personalized medicine and robotic surgery become the norm, to environmental science, with predictive models and solutions for combating climate change. However, these advancements are not without their quandaries. The digital divide and issues of privacy, surveillance, and data security raise critical questions about the kind of society we wish to cultivate. The ethical use of technology, equitable access to its benefits, and the safeguarding of individual rights become paramount in ensuring that progress does not come at the cost of human dignity or freedom. Simultaneously, the specter of climate change and environmental degradation demands a reevaluation of our relationship with the natural world. The pursuit of sustainable living, renewable energy sources, and green technologies is more than an environmental imperative it's a moral one. As we harness the power of wind, solar, and bioengineering, we embark on a path that seeks harmony with the planet, aspiring to leave a livable earth for future generations. Moreover, the exploration of space represents the boundless curiosity of the human spirit. The successful landing of rovers on Mars, the search for extraterrestrial life, and plans for human colonization of other planets speak to our unyielding desire to understand our place in the universe. These endeavors not only push the boundaries of science and technology but also ignite imaginations, inspiring a sense of wonder and possibility. Yet, amidst these strides in science and technology, the cultural and social fabric of humanity continues to evolve. The global exchange of ideas, facilitated by the internet, has led to a cultural renaissance where art, music, and literature transcend geographical boundaries, fostering a global community that celebrates diversity yet acknowledges the challenges of integration and identity. As we navigate the complexities of the 21st century, the story of humanity is one of adaptation and resilience. The challenges we face, from ethical dilemmas posed by technology to the existential threat of climate change, are daunting. However, history has shown that in moments of crisis, there lies opportunity for growth and transformation. The future is a canvas yet to be painted, and it is up to us, collectively, to decide the colors and shapes of the world to come. With wisdom gleaned from the past and an eye towards the future, we step forward, ready to contribute our verses to the enduring epic of human civilization. As the narrative of humanity unfurls into the latter ha



curl 127.0.0.1:12345/generate -d '{
"prompt": "In the tapestry of human history, threads of invention, exploration, and cultural fusion intertwine to form a narrative rich in complexity and diversity. From the ancient civilizations that laid the foundations of modern society such as the Egyptians, who mastered the art of irrigation and monumental architecture, to the Greeks, whose philosophical inquiries and democratic ideals still resonate today the march of progress has been relentless and multifaceted. The Roman Empire, with its unparalleled network of roads, legal systems, and military strategies, exemplified the heights of administrative and organizational prowess, setting the stage for the medieval period. This era, often misconstrued as dark and stagnant, was, in reality, a crucible of innovation and cross-cultural exchange, as seen in the Silk Road's bustling trade and the scholarly pursuits within the Islamic Golden Age. The Renaissance, a rebirth of art and knowledge, emerged from the depths of the Middle Ages, casting a spotlight on humanism and reviving interest in classical antiquity. This period saw luminaries like Leonardo da Vinci and Michelangelo break new ground in art and science, embodying the spirit of inquiry and the pursuit of excellence. The Industrial Revolution marked another pivotal turn, transforming societies with the power of steam, mechanization, and later, electricity. The rapid pace of change brought about by inventors and entrepreneurs like James Watt and Thomas Edison reshaped daily life, economies, and the environment, ushering in the modern era. The 20th century, with its world wars, technological advancements, and social movements, highlighted humanity's capacity for both destruction and profound creativity. The advent of the digital age, led by visionaries such as Steve Jobs and Tim Berners-Lee, has connected the world in ways previously unimaginable, making information and communication virtually instantaneous. As we gaze into the future, the challenges of climate change, resource scarcity, and technological ethics loom large. Yet, the story of human progress is one of resilience and innovation. The quest for sustainable development, the exploration of space, and the advancements in artificial intelligence and biotechnology present new frontiers for humanity to explore. This narrative, weaving through millennia, showcases the indomitable spirit of human curiosity and the relentless pursuit of knowledge. As we stand on the shoulders of giants, we look forward to writing the next chapters of this ongoing saga, hopeful that the future will reflect the best of our collective endeavors. In this continuing saga, the emphasis shifts towards a more interconnected and technologically advanced world. The 21st century has ushered in an era where the digital realm and artificial intelligence not only augment human capabilities but also challenge our ethical frameworks and societal norms. Innovations in machine learning and AI have the potential to revolutionize industries, from healthcare, where personalized medicine and robotic surgery become the norm, to environmental science, with predictive models and solutions for combating climate change. However, these advancements are not without their quandaries. The digital divide and issues of privacy, surveillance, and data security raise critical questions about the kind of society we wish to cultivate. The ethical use of technology, equitable access to its benefits, and the safeguarding of individual rights become paramount in ensuring that progress does not come at the cost of human dignity or freedom. Simultaneously, the specter of climate change and environmental degradation demands a reevaluation of our relationship with the natural world. The pursuit of sustainable living, renewable energy sources, and green technologies is more than an environmental imperative it's a moral one. As we harness the power of wind, solar, and bioengineering, we embark on a path that seeks harmony with the planet, aspiring to leave a livable earth for future generations. Moreover, the exploration of space represents the boundless curiosity of the human spirit. The successful landing of rovers on Mars, the search for extraterrestrial life, and plans for human colonization of other planets speak to our unyielding desire to understand our place in the universe. These endeavors not only push the boundaries of science and technology but also ignite imaginations, inspiring a sense of wonder and possibility. Yet, amidst these strides in science and technology, the cultural and social fabric of humanity continues to evolve. The global exchange of ideas, facilitated by the internet, has led to a cultural renaissance where art, music, and literature transcend geographical boundaries, fostering a global community that celebrates diversity yet acknowledges the challenges of integration and identity. As we navigate the complexities of the 21st century, the story of humanity is one of adaptation and resilience. The challenges we face, from ethical dilemmas posed by technology to the existential threat of climate change, are daunting. However, history has shown that in moments of crisis, there lies opportunity for growth and transformation. The future is a canvas yet to be painted, and it is up to us, collectively, to decide the colors and shapes of the world to come. With wisdom gleaned from the past and an eye towards the future, we step forward, ready to contribute our verses to the enduring epic of human civilization. As the narrative of humanity unfurls into the latter ha",
"max_tokens": 32,
"stream": false,
"do_sample":true,
"repetition_penalty": 1.00,
"temperature": 0.01,
"top_p": 0.001,
"top_k": 1,
"model": "qwen"
}'

网站公告

今日签到

点亮在社区的每一天
去签到