20241206-Windows 10下使用IDEA 2024.2.3(JDK 18.0.2.1)搭建Hadoop 3.3.6开发环境

Windows 10下使用IDEA 2024.2.3(JDK 18.0.2.1)搭建Hadoop 3.3.6开发环境

1. 配置好本地hadoop之后

2. idea 新建或导入 Maven 项目

3. 编写 pom.xml 文件: 有些版本和项目信息需要根据自己的项目进行调整

JDK 18.0.2.1
Hadoop 3.3.6

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <!-- 设置项目模型版本 -->
    <modelVersion>4.0.0</modelVersion>

    <!-- 项目坐标，groupId, artifactId 和 version -->
    <groupId>org.example</groupId>
    <artifactId>Student_performance_analysis_MapReduce</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <!-- 设置属性，定义JDK 18.0.2.1与Hadoop版本号 -->
        <java.version>18</java.version>
        <hadoop.version>3.3.6</hadoop.version>
        <!-- 设置编码方式 -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <!-- 依赖管理 -->
    <dependencies>
        <!-- Hadoop Common 依赖 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!-- Hadoop Client 依赖 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!-- 如果需要其他Hadoop模块，可以继续添加 -->
        <!-- 例如 Hadoop HDFS 依赖 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <!-- Hadoop MapReduce 依赖 -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

<!--        <dependency>-->
<!--            &lt;!&ndash;调整日志级别和格式&ndash;&gt;-->
<!--            <groupId>log4j</groupId>-->
<!--            <artifactId>log4j</artifactId>-->
<!--            <version>1.2.17</version>-->
<!--        </dependency>-->
        <dependency>
            <!-- 排除slf4j-log4j依赖 -->
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.25</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-reload4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <!-- 排除slf4j-reload4j依赖 -->
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-reload4j</artifactId>
            <version>1.7.36</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-reload4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>


    </dependencies>

    <build>
        <!-- 确保编译时使用的是 Java 18 -->
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

4. 下载 源代码和文档

5. 手动导入依赖包

4 里会显示配置的依赖

导入成功后会在外部库里看到一堆 hadoop 的文件库

Over！