《RuoYi基于SpringBoot+Vue前后端分离的Java快速开发框架学习》系列博客_Part4_三模态融合

系列博客目录

目标

 实现三模态融合，将文本、图片、音频从前端上传，后端返回数据库数据。主要包含两种类别，电脑和弓。而且电脑和弓的数据在数据库中不同的表中，表的列名存在不同(仅最后一列不同)。

Step1:之前工作形成子组件

 之前已经实现了图文识别、音文识别、音图识别的前端页面，而当时的要求，比如图文识别就是用户上传图片，得到相应文字信息(来自数据库)。所以图文识别，就已经写好了上传图片的vue。同理上传音频的vue也写好了。这次直接把之前的组件当作子组件。

Step2:弥补缺失的文本子组件，同时举例如何子组件向父组件传数据

 通过下面代码可以看出，当点击按钮时，将文本框中内容传递给父组件。
文本子组件代码如下：

<template>
  <div class="app-container">
    <label for="textInput">请输入文本内容：</label>
    <textarea
      id="textInput"
      v-model="textContent"
      rows="5"
      placeholder="在此输入文本内容..."
      style="width: 100%; margin-bottom: 10px;"
    ></textarea>
    <el-button
      type="primary"
      @click="submitTextContent"
      :disabled="!textContent.trim()"
    >
      点击此按钮上传文本
    </el-button>
  </div>
</template>

<script>
export default {
  name: "TextInput",
  data() {
    return {
      textContent: "", // 文本输入框内容
    };
  },
  methods: {
    submitTextContent() {
      // 通过事件将文本内容传递给父组件
      this.$emit("fileUploaded", this.textContent.trim());//trim()去除文本两端的空白字符
      this.$message.success("文本提交成功！");
    },
  },
};
</script>

当上面子组件点击按钮后，调用父组件fileUploaded指定的函数。 <PictureToText v-if="showChild" @fileUploaded="handleImageUpload" />
父组件中的表格的表头是根据后端返回的resultType 动态变化的。
setTimeout()函数是在延迟后，执行里面的代码。
父组件核心代码如下：

<template>
  <div>
  <div class="container">
    <!-- 模态选择区域 目前不需要，因为要直接提供三个模态的上传 -->
<!--    <el-radio-group v-model="radio2">-->
<!--      <el-radio :label="3">图片</el-radio>-->
<!--      <el-radio :label="6">音频</el-radio>-->
<!--      <el-radio :label="9">文本</el-radio>-->
<!--    </el-radio-group>-->
    <!-- 图片上传区域 -->
    <!-- 动态显示子组件 -->
    <div class="child">
<!--      <PictureToText v-if="radio2 === 3 && showChild" @updateResult="handleResult"/>-->
      <PictureToText v-if="showChild" @fileUploaded="handleImageUpload" />
    </div>
    <!-- 音频上传区域 -->
    <div class="child">
<!--      <AudioToText v-if="radio2 === 6"/>-->
      <AudioToText v-if="showChild" @fileUploaded="handleAudioUpload" />
    </div>
    <!-- 文本上传区域 -->
    <div class="child">
      <TextUploader v-if="showChild" @fileUploaded="handleTextUpload"/>
    </div>

    <!-- 添加提交按钮 -->
<!--    <button v-if="showChild" @click="submitData">提交数据</button>-->
    <!-- 提交按钮，独占一行 -->
    <div class="button-container">
    <el-button
        v-if="showChild"
        type="primary"
        class="action-button"
        @click="submitData"
    >
      提交数据进行分析
    </el-button>
    </div>
  </div>
    <div>
    <el-table v-if="tableOfResultVisible" v-loading="loading"  element-loading-text="拼命分析中,预计加载结果需要等待大约10秒钟...." :data="jmwlList" >
<!--      <el-table-column type="selection" width="55" align="center" />-->
      <el-table-column label="交易id" align="center" prop="交易id" show-overflow-tooltip/>
      <el-table-column label="品牌" align="center" prop="品牌" show-overflow-tooltip/>
      <el-table-column label="商品名称" align="center" prop="商品名称" show-overflow-tooltip />
      <el-table-column label="评论用户名" align="center" prop="评论用户名" show-overflow-tooltip/>
      <el-table-column label="评论内容" align="center" prop="评论内容" show-overflow-tooltip/>
      <el-table-column label="赞数" align="center" prop="赞数" show-overflow-tooltip/>

      <el-table-column label="商品交易客体图片" align="center" >
        <template slot-scope="scope1">

          <img v-if="scope1.row.商品交易主体图片" :src="scope1.row.商品交易主体图片" prop="商品交易客体图片" style="width:100px;height:100px;cursor:pointer" @click="showImage(scope1.row.商品交易主体图片)"/>

          <img v-else src="./default.jpg" style="width:100px;height:100px"/>
          <el-dialog :visible="dialogVisible" @close="dialogVisible=false">
            <img :src="dialogImageUrl" style="width:300px;height:300px"/>
          </el-dialog>
        </template>
      </el-table-column>


      <el-table-column label="商品交易客体图片1" align="center" >
        <template slot-scope="scope1">

          <img v-if="scope1.row.商品交易客体图片1" :src="scope1.row.商品交易客体图片1" prop="商品交易客体图片1" style="width:100px;height:100px;cursor:pointer" @click="showImage(scope1.row.商品交易客体图片1)"/>

          <img v-else src="./default.jpg" style="width:100px;height:100px"/>
          <el-dialog :visible="dialogVisible" @close="dialogVisible=false">
            <img :src="dialogImageUrl" style="width:300px;height:300px"/>
          </el-dialog>
        </template>
      </el-table-column>
      <el-table-column label="产品质量" align="center" prop="产品质量" show-overflow-tooltip/>
      <el-table-column label="产品价格" align="center" prop="产品价格" show-overflow-tooltip/>
      <el-table-column label="客户评分" align="center" prop="客户评分" show-overflow-tooltip/>
      <el-table-column label="店铺总分" align="center" prop="店铺总分" show-overflow-tooltip/>
      <el-table-column label="店铺名称" align="center" prop="店铺名称" show-overflow-tooltip/>
      <el-table-column label="店铺链接" align="center" prop="店铺链接" show-overflow-tooltip/>
      <el-table-column label="评论类别" align="center" prop="评论类别" show-overflow-tooltip/>
      <el-table-column label="交易平台" align="center" prop="交易平台" show-overflow-tooltip/>
<!--      <el-table-column label="疑似假冒伪劣原因" align="center" prop="疑似假冒伪劣原因" show-overflow-tooltip/>-->
      <el-table-column v-if="resultType === 0" label="疑似假冒伪劣原因" align="center" prop="疑似假冒伪劣原因" show-overflow-tooltip/>
      <el-table-column v-else-if="resultType === 1" label="疑似禁限售原因" align="center" prop="疑似禁限售原因" show-overflow-tooltip/>
    </el-table>

    <!-- 条件显示，当数据总数大于 0 时显示分页组件 -->
    <!-- 数据总数，决定分页的总页数 -->
    <!-- 当前页码，使用.sync实现双向绑定 -->
    <!-- 每页显示的数据条数，使用.sync实现双向绑定 -->
    <!-- 当分页发生变化时触发的事件，调用getList函数 -->
    <pagination
      v-show="tableOfResultVisible"
      :total="total"
      :page.sync="queryParams.pageNum"
      :limit.sync="queryParams.pageSize"
      @pagination="getList"
    />

    </div>
  </div>
</template>

<script>
import PictureToText from "../../pictureToText/pictureToText/index.vue";
import AudioToText from "../../audioToText/audioToText/index.vue"
import TextUploader from "@/views/3MFusion/3MFusion/TextUploader.vue";
import {uploadAudioToText, uploadPictureToText, uploadTextToText} from "@/api/pictureToText/pictureToText";

export default {
  components: {TextUploader, PictureToText,AudioToText },
  name: "ThreeMFusion",
  data() {
    return {
      //用来判断哪一类文件已经上传
      textFile: null,
      imageFile: null,
      audioFile: null,

      showChild: true, // 控制子组件是否显示
      tableOfResultVisible: false,//控制结果表格是否显示
      // 遮罩层
      loading: true,
      // 总条数
      total: 0,
      // 假冒伪劣产品研究表格数据也可以显示禁限售表格数据，通过后端传过来的resultType动态参数决定
      jmwlList: [],
      // 弹出层标题
      title: "",
      // 表单参数
      form: {},
      // 表单校验
      rules: {
      }
    };
  },
  methods: {
    //显示哪一模态的文件已经上传
    handleTextUpload(file) {
      this.textFile = file;
    },
    handleImageUpload(file) {
      this.imageFile = file;
    },
    handleAudioUpload(file) {
      this.audioFile = file;
    },
    submitData() {
      this.loading = true;
      if (this.textFile) {
        this.tableOfResultVisible = true;
        this.showChild = false;
        console.log('提交文本数据:', this.textFile);
        uploadTextToText({ content: this.textFile }).then((response) => {
          if (response.code === 200) {
            setTimeout(() => {
              this.loading = false;
              this.jmwlList = response.rows;
              this.total = response.total;
              this.resultType = response.resultType;
            }, 5000);
          }
        });

      } else if (this.imageFile) {
        this.showChild = false;
        this.tableOfResultVisible = true;
        console.log('提交图片数据到3MFusion');
        uploadPictureToText(this.imageFile).then(response => {
          if (response.code === 200) {
            this.imgUrl=response.pictureUrl;
            console.log(this.imgUrl);
            setTimeout(() => {
              this.loading = false;
              this.jmwlList = response.rows;
              this.total = response.total;
              this.resultType = response.resultType;

            }, 5000);
          }
        });
      } else if (this.audioFile) {
        this.showChild = false;
        this.tableOfResultVisible = true;
        console.log('提交音频数据到3MFusion');
        uploadAudioToText(this.audioFile).then(response => {
          if (response.code === 200) {
            console.log("uploadAudio()已完成");
            this.audioUrl = response.audioUrl
            console.log(response.audioUrl);
            setTimeout(() => {
              this.loading = false;
              this.jmwlList = response.rows;
              this.total = response.total;
              this.resultType = response.resultType;
            }, 5000);
          }
        });
      } else {
        this.$message.error("没有可提交的数据！");
        console.warn('没有可提交的数据');
      }
    },
  }
};
</script>

Step3:后端代码需要根据上传的文件传给python服务器

    @PostMapping("/uploadAudioToText")
    public TableDataInfo insertUploadOfAudio(@RequestParam("audioName") MultipartFile file) throws Exception{
        List<Integer> idsList = new ArrayList<>();
        int resulttype = -1;
        if (!file.isEmpty())
        {
            LoginUser loginUser = getLoginUser();
            String audioPath = FileUploadUtils.upload(RuoYiConfig.getAudioPath(), file, MimeTypeUtils.MEDIA_EXTENSION);
            Upload upload=new Upload();
            upload.setUsername(loginUser.getUsername());
            upload.setAudioPath(audioPath);
            String pictureName=audioPath.substring(audioPath.lastIndexOf("/")+1,audioPath.length());
            upload.setPictureName(pictureName);
            if (iUploadService.insertUpload(upload)!=0)
            {
                try {
                    // 定义 URL
                    URL url = new URL("http://127.0.0.1:6000/api");
                    HttpURLConnection connection = (HttpURLConnection) url.openConnection();
                    connection.setRequestMethod("POST");
                    connection.setRequestProperty("Content-Type", "application/json; utf-8");
                    connection.setRequestProperty("Accept", "application/json");
                    connection.setDoOutput(true);
                    String number = String.valueOf(6);
                    String jsonInputString = "{\"audioPath\": \"" + audioPath + "\", \"number\": " + number + "}";
                    try (OutputStream os = connection.getOutputStream()) {
                        byte[] input = jsonInputString.getBytes("utf-8");
                        os.write(input, 0, input.length);
                    }
                    // 检查响应码
                    int responseCode = connection.getResponseCode();
                    if (responseCode == 200) {
                        // 读取响应内容
                        BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
                        StringBuilder response = new StringBuilder();
                        String line;
                        while ((line = reader.readLine()) != null) {
                            response.append(line);
                        }
                        reader.close();

                        // 打印完整的响应内容
                        System.out.println("Python 服务返回的数据: " + response);

                        // 将响应字符串解析为 JSON 数组
                        try {
//                            JSONArray jsonResponse = new JSONArray(response.toString());
                            JSONArray jsonResponse = JSONArray.parseArray(response.toString());
                            // 查看 jsonResponse 的长度
                            int length = jsonResponse.size();  // 获取 JSONArray 的长度
                            System.out.println("JSON 数组的长度: " + length);
                            // 获取第一个元素，应该是数字（0 或 1）
                            resulttype = jsonResponse.getIntValue(0);  // 这里获取 0 或 1
                            JSONArray idsArray = jsonResponse.getJSONArray(1);  // 获取 IDs 数组

                            // 打印 type 和 ids
                            System.out.println("type: " + resulttype);
                            System.out.println("ids: " + idsArray);

                            // 如果你需要将 ids 转换为 Java List，可以使用以下方法

                            for (int i = 0; i < idsArray.size(); i++) {
                                idsList.add(Integer.valueOf(idsArray.getString(i)));  // 添加每个 ID
                            }
                            System.out.println("IDs as List: " + idsList);

                        } catch (Exception e) {
                            e.printStackTrace();
                            System.err.println("解析 JSON 响应时发生错误.");
                        }

                    } else {
                        System.err.println("HTTP 请求失败，响应码: " + responseCode);
                    }
                    connection.disconnect();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
        if(resulttype==0){
            List<Jmwl> list = jmwlService.selectJmwlListByIdList(idsList);
            return getDataTable(list,resulttype);
        }else {
            List<Jxs> list = jxsService.selectJxsListByIdList(idsList);
            System.out.println("resulttype = " + resulttype);
            return getDataTable(list,resulttype);
        }
    }

Step4:python服务器进行分析

from database_server import get_ids_by_keyword  # 导入函数
from flask import Flask, jsonify, request
from detectKeyword import detect_keyword
from audio import getKeywordOfAudio
import sys
import os

# 将项目根目录添加到 sys.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))

from predict import get_keywordOfImage  # 导入 predict.py 中的函数

app = Flask(__name__)


@app.route('/api', methods=['POST'])
def api():

    data = request.json # 假设数据通过 JSON 格式传递
    file_type = data.get('number')
    print("file_type = ", file_type)

    # 根据类型处理不同逻辑
    if file_type == 3:
        # 图像处理逻辑
        image_path = data.get('imagePath')
        print("image_path+file_type: " + str(image_path) + str(file_type))
        keyword = get_keywordOfImage(image_path)
    elif file_type == 6:
        # 音频处理逻辑
        audioPath = data.get('audioPath')
        print("audioPath =" + str(audioPath))
        keyword = getKeywordOfAudio(audioPath)
        keyword = detect_keyword(keyword)
    else:
        # 处理不支持的类型
        contextOfText = data.get('context')
        keyword = detect_keyword(contextOfText)

    ids = get_ids_by_keyword(keyword)  # 调用函数查询
    print(f"与关键词 '{keyword}' 相关的 ID 列表: {ids}")  # 打印结果
    if keyword == "电脑":
        response = [0, ids]
    elif keyword == "弓":
        response = [1, ids]
    else:
        response = [None, ids]
    
    return jsonify(response)



if __name__ == "__main__":
    app.run(port=6000)

python服务器输出示例

file_type = 9
keyword = 弓
与关键词 ‘弓’ 相关的 ID 列表: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]

返回给java后端的数据示例如下，第一个0，代表电脑。

[0,[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]]