文章目录
一、 下载canal
分别下载 canal.deployer-1.1.4 和 canal.adapter-1.1.4
二、安装canal
1. 分别解压canal.deployer-1.1.4和canal.adapter-1.1.4
mkdir /software
tar -xzvf canal.deployer-1.1.4.tar.gz -C /software/canal
tar -xzvf canal.deployer-1.1.4.tar.gz -C /software/canal-adapter-1.1.4
2. MysqL相关配置
① MysqL开启binlog,修改 my.cnf
[MysqLd]
log-bin=MysqL-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MysqL replaction 需要定义,不要和 canal 的 slaveId 重复
查看是否成功
show variables like '%log_bin%'
② 创建canal用户,并赋权
# 创建用户
CREATE USER canal IDENTIFIED BY 'Canal@123456';
# MysqL8.0需要在创建用户后,先更改密码加密方式,再赋权
update user set plugin='MysqL_native_password' where user='canal';
grant all privileges ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
③ 查看binlog偏移量并记录
show master status;
④ 导出目标表历史数据,并在目标库创建表并导入 (请自行操作,注意时间格式)
3. 配置canal.deployer-1.1.4
配置instance.properties
注意:/software/canal/conf/example路径下,首次启动或者数据库变更后,请将此路径下除instance.properties以外的文件删除,否则可能报错
vim /software/canal/conf/example/instance.properties
## MysqL serverId , v1.0.26+ will autoGen
## v1.0.26版本后会自动生成slaveId,所以可以不用配置
# canal.instance.MysqL.slaveId=0
# 源数据库地址,此处请修改
canal.instance.master.address=127.0.0.1:3306
# binlog日志名称,此处请修改
canal.instance.master.journal.name=MysqL-bin.000005
# MysqL主库链接时起始的binlog偏移量,此处请修改
canal.instance.master.position=820633026
# MysqL主库链接时起始的binlog的时间戳
canal.instance.master.timestamp=
canal.instance.master.gtid=
# username/password
# 在MysqL服务器授权的账号密码,此处请修改
canal.instance.dbUsername=canal
canal.instance.dbPassword=Canal@123456
# 字符集
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
# table regex .*\\..*表示监听所有表 也可以写具体的表名,用,隔开
canal.instance.filter.regex=.*\\..*
# MysqL 数据解析表的黑名单,多个表用,隔开
canal.instance.filter.black.regex=
4. 配置canal.adapter-1.1.4
① 修改application.yml
vim /software/canal-adapter-1.1.4/conf/application.yml
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp # kafka rocketMQ
canalServerHost: 10.10.10.10:11111 #自己的canal-deployer启动的ip和端口,11111为默认端口
# zookeeperHosts: slave1:2181
# mqServers: 127.0.0.1:9092 #or rocketmq
# flatMessage: true
batchSize: 500
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
srcDataSources:
defaultDS:
# 换成自己MysqL源库的地址,此处请修改
url: jdbc:MysqL://127.0.0.1:3306/person?useUnicode=true
username: canal
password: Canal@123456
canalAdapters:
- instance: example # canal instance Name or mq topic name
groups:
- groupId: g1
outeradapters:
# - name: logger
- name: rdb
key: MysqL1
properties:
# 驱动类,如果是MysqL8.0,需要在lib中添加mysql-connector-java-8.0.27.jar,请自行搜索下载
jdbc.driverClassName: com.MysqL.jdbc.Driver
# 目标库的地址,此处请修改
jdbc.url: jdbc:MysqL://8.8.8.1:3306/person?useUnicode=true
jdbc.username: root
jdbc.password: root@123.com
- groupId: g2
outeradapters:
- name: hbase
properties:
# zookeeper地址,此处请修改
hbase.zookeeper.quorum: 8.8.8.5,8.8.8.6,8.8.8.7
hbase.zookeeper.property.clientPort: 2181
zookeeper.znode.parent: /hbase
② 添加MysqL同步配置文件,每张表新建一个文件,名称随意
vim /software/canal-adapter-1.1.4/conf/rdb/teacher.yml
dataSourceKey: defaultDS
destination: example
groupId: g1
outeradapterKey: MysqL1
concurrent: true
dbMapping:
database: person #源数据库
table: teacher #待同步表
targetTable: teacher #目标表
targetPk: #主键映射,左侧为源表主键,右侧为目标表主键
id: id
mapAll: true #全同步
commitBatch: 3000 # 批量提交的大小
vim /software/canal-adapter-1.1.4/conf/rdb/student.yml
dataSourceKey: defaultDS
destination: example
groupId: g1
outeradapterKey: MysqL1
concurrent: true
dbMapping:
database: person #源数据库
table: student #待同步表
targetTable: student #目标表
targetPk: #主键映射,左侧为源表主键,右侧为目标表主键
id: id
mapAll: true #全同步
commitBatch: 3000 # 批量提交的大小
③ 修改hbase同步配置
dataSourceKey: defaultDS
destination: example
hbaseMapping:
mode: STRING #NATIVE #PHOENIX
database: person # 数据库名
table: student # 数据库表名
hbaseTable: test:student # HBase表名
family: info # 默认统一Family名称
uppercaseQualifier: false # 字段名转大写, 默认为true
commitBatch: 3000 # 批量提交的大小
rowKey: vin # 复合字段rowKey不能和columns中的rowKey重复
columns:
id: info:id
name: info:name
excludeColumns: # 忽略字段
- score
三、启动
1. 启动canal.deployer-1.1.4
# 启动
sh /software/canal/bin/startup.sh
# 重启
sh /software/canal/bin/restart.sh
# 停止
sh /software/canal/bin/stop.sh
2. 启动canal.adapter-1.1.4
# 启动
sh /software/canal-adapter-1.1.4/bin/startup.sh
# 重启
sh /software/canal-adapter-1.1.4/bin/restart.sh
# 停止
sh /software/canal-adapter-1.1.4/bin/stop.sh
3. 查看log
# 查看canal.deployer-1.1.4运行log
tail -100f /software/canal/logs/example/example.log
# 查看canal.adapter-1.1.4运行log
tail -100f /software/canal-adapter-1.1.4/logs/adapter/adapter.log
四、问题
1. 用户赋权报错
SHOW GRANTS FOR 'canal'@'%' > 1141 - There is no such grant defined for user 'canal' on host '%'
解决方法:删除用户,重新创建,赋权
drop user 'canal'@'%';
CREATE USER canal IDENTIFIED BY 'Canal@123456';
# MysqL8.0需要在创建用户后,先更改密码加密方式,再赋权
update user set plugin='MysqL_native_password' where user='canal';
grant all privileges ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
原文地址:https://www.jb51.cc/wenti/3279180.html
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。