kafka专题-1、集群安装

1、kafka集群安装

1、前期准备

1、需要3个服务器节点，并且节点都安装了java8

2、组件版本

2、部署步骤

1、下载并解压kafka

 cd /usr/local/soft
 wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/3.7.2/kafka_2.13-3.7.2.tgz
 tar -xzvf kafka_2.13-3.7.2.tgz
 cd kafka_2.13-3.7.2
 mv kafka_2.13-3.7.2 kafka #重命名

2、创建存储目录

 mkdir -p /usr/local/soft/kafka/log
 mkdir -p /usr/local/soft/kafka/zookeeper

3、修改配置文件，并且同步到dev2和dev3服务器上

 #切换到kafka目录下
 cd /usr/local/soft/kafka/conf 
 ​
 vim zookeeper.properties
 #设置数据存储目录
 dataDir=/usr/local/soft/kafka/zookeeper 
 #设置日志存储路径
 dataLogDir=/usr/local/soft/kafka/zookeeper/log 
 clientPort=2181
 maxClientCnxns=100
 tickTimes=2000
 initLimit=10
 syncLimit=5
 server.0=192.168.198.16:2888:3888
 server.1=192.168.198.17:2888:3888
 server.2=192.168.198.18:2888:3888

4、编写myid

 # 设置myid dev1设置为0，dev2设置为1 dev3设置为2
 vim /usr/local/soft/kafka/zookeeper/myid
 0

3、kafka安装

1、dev1节点需改

 # 进入到指定目录
 cd /usr/local/soft/kafka/config
 # 配置根据自己需求修改
 vim server.properties
 broker.id=0
 listeners=PLAINTEXT://192.168.198.16:9092
 advertised.listeners=PLAINTEXT://192.168.198.16:9092
 num.network.threads=3
 num.io.threads=8
 socket.send.buffer.bytes=102400
 socket.receive.buffer.bytes=102400
 socket.request.max.bytes=104857600
 log.dirs=/usr/local/soft/kafka/log
 num.partitions=1
 num.recovery.threads.per.data.dir=1
 offsets.topic.replication.factor=1
 transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1
 log.retention.hours=168
 log.retention.check.interval.ms=300000
 zookeeper.connect=192.168.198.16:2181,192.168.198.17:2181,192.168.198.18:2181
 zookeeper.connection.timeout.ms=18000
 group.initial.rebalance.delay.ms=0

2、dev2节点修改

 # 进入到指定目录
 cd /usr/local/soft/kafka/config
 # 配置根据自己需求修改
 vim server.properties
 broker.id=1
 listeners=PLAINTEXT://192.168.198.17:9092
 advertised.listeners=PLAINTEXT://192.168.198.17:9092
 num.network.threads=3
 num.io.threads=8
 socket.send.buffer.bytes=102400
 socket.receive.buffer.bytes=102400
 socket.request.max.bytes=104857600
 log.dirs=/usr/local/soft/kafka/log
 num.partitions=1
 num.recovery.threads.per.data.dir=1
 offsets.topic.replication.factor=1
 transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1
 log.retention.hours=168
 log.retention.check.interval.ms=300000
 zookeeper.connect=192.168.198.16:2181,192.168.198.17:2181,192.168.198.18:2181
 zookeeper.connection.timeout.ms=18000
 group.initial.rebalance.delay.ms=0

3、dev3节点修改

 # 进入到指定目录
 cd /usr/local/soft/kafka/config
 # 配置根据自己需求修改
 vim server.properties
 broker.id=2
 listeners=PLAINTEXT://192.168.198.18:9092
 advertised.listeners=PLAINTEXT://192.168.198.18:9092
 num.network.threads=3
 num.io.threads=8
 socket.send.buffer.bytes=102400
 socket.receive.buffer.bytes=102400
 socket.request.max.bytes=104857600
 log.dirs=/usr/local/soft/kafka/log
 num.partitions=1
 num.recovery.threads.per.data.dir=1
 offsets.topic.replication.factor=1
 transaction.state.log.replication.factor=1
 transaction.state.log.min.isr=1
 log.retention.hours=168
 log.retention.check.interval.ms=300000
 zookeeper.connect=192.168.198.16:2181,192.168.198.17:2181,192.168.198.18:2181
 zookeeper.connection.timeout.ms=18000
 group.initial.rebalance.delay.ms=0

4、服务启动

1、zk和kafka启动脚本同时复制到其余dev2和dev3节点上

 cd /usr/local/soft/kafka
 # 配置启动脚本
 vim kafkaStart.sh
 ​
 /usr/local/soft/kafka/bin/zookeeper-server-start.sh -daemon /usr/local/soft/kafka/config/zookeeper.properties 
 sleep 10 
 /usr/local/soft/kafka/bin/kafka-server-start.sh -daemon /usr/local/soft/kafka/config/server.properties 
 ​
 # 配置关闭脚本
 vim kafkaStop.sh
 /usr/local/soft/kafka/bin/zookeeper-server-stop.sh /usr/local/soft/kafka/config/zookeeper.properties &
 sleep 3 
 /usr/local/soft/kafka/bin/kafka-server-stop.sh /usr/local/soft/kafka/config/server.properties &
 ​
 # 对脚本赋权
 chmod u+x kafkaStart.sh
 chmod u+x kafkaStop.sh
 ​