Kafka集成Debezium监听postgresql变更

下载postgres的插件：https://debezium.io/documentation/reference/2.7/install.html

2.7版本支持postgresql12数据库。

debezium-connector-postgres-2.7.4.Final-plugin.tar.gz

上传插件并解压

mkdir /usr/local/kafka/kafka_2.12-2.2.1/connector
cd /usr/local/kafka/kafka_2.12-2.2.1/connector
tar -zxvf debezium-connector-postgres-2.7.4.Final-plugin.tar.gz

进入kafka配置目录配置连接器（单机kafka）

cd /usr/local/kafka/kafka_2.12-2.2.1/config
#修改配置
vi connect-standalone.properties

#配置kafka地址
bootstrap.servers=192.168.159.100:9092

#采集的数据存储到kafka的数据格式
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter

#是否存储schemas，存储则格式会复杂些
key.converter.schemas.enable=false
value.converter.schemas.enable=false

#插件的路径
plugin.path=/usr/local/kafka/kafka_2.12-2.2.1/connector

使用命令启动 Kafka Connect 服务（单机kafka）

/usr/local/kafka/kafka_2.12-2.2.1/bin/connect-standalone.sh -daemon /usr/local/kafka/kafka_2.12-2.2.1/config/connect-standalone.properties /usr/local/kafka/kafka_2.12-2.2.1/config/connect-file-source.properties /usr/local/kafka/kafka_2.12-2.2.1/config/connect-file-sink.properties

配置Debezium Connector：

pgsql修改配置文件：postgresql.conf

listen_addresses = '*'
#开启逻辑复制
wal_level = logical
# 允许多少个流复制协议连接过来，一个流复制协议会产生一个walsender进程，该数不能低于master的配置数量
max_wal_senders = 2	
max_replication_slots = 1

修改pg_hba.conf：

#添加
host    all             all             0.0.0.0/0                 trust
host    replication     all             0.0.0.0.0                 trust

注册连接器：

{
    "name": "my-postgresql-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.hostname": "192.168.159.103",
        "database.port": "15432",
        "database.dbname": "db_test",
        "database.user": "postgres",
        "database.password": "123456",
        "plugin.name": "pgoutput",
        "database.server.name": "server5",
        "topic.prefix": "linging"
    }
}

执行：

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 192.168.159.100:8083/connectors/ -d '{"name":"my-postgresql-connector","config":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector","database.hostname":"192.168.159.103","database.port":"15432","database.dbname":"db_test","database.user":"postgres","database.password":"123456","plugin.name":"pgoutput","database.server.name":"server5","topic.prefix":"linging"}}'

修改数据库表，查看kafka消息: