Joyber 的博客 - es
type: channel
link: https://blog.qqvbc.com/tag/es/
items: mysql数据库自动数据接入到 ES(elasticsearch)之工具logstash

mysql数据库自动数据接入到 ES(elasticsearch)之工具logstash
type: item
date: 2020-03-25T17:55:00+08:00
description: 使用 logstash 接入 ES 集群下载（一定要选择和elasticsearch服务版本一致的版本）：https://www.elastic.co/cn/downloads/past-releases#logstash下载 rpm 包安装:yum localinstall -y +包文件名称6.6版本的文档，主要查看怎么配置和怎么运行设置并运行Logstashhttps://www.elastic.co/guide/en/logstash/6.6/setup-logstash.html怎么使用jdbc插件导入数据库数据https://www.elastic.co/guide/en/logstash/6.6/plugins-inputs-jdbc.html使用JDBC导入mysql数据的话，需要先装好java，再去mysql官网下载jar包https://dev.mysql.com/downloads/connector/j/这里下载rpm包，使用yum localinstall -y 安装后，会在/usr/share/java目录产生mysql的连接器jar包配置文件实例：/etc/logstash/conf.d/xxx.confinput { stdin {} jdbc { jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/xxxxtablename?serverTimezone=UTC" jdbc_user => "xxxusername" jdbc_password => "xxxxxpassword" jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.15.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_paging_enabled => "true" jdbc_page_size => "2000" # 增量更新 use_column_value => "true" record_last_run => "true" tracking_column => "id" last_run_metadata_path => '/etc/logstash/conf.d/xxxx.log' # our query statement => "SELECT id,title,content FROM articles where id > :sql_last_value and status=1 ORDER BY id ASC" schedule => "* * * * * " } } output { stdout { codec => json_lines } elasticsearch { "hosts" => "127.0.0.1:9200" "index" => "indexname" "document_type" => "article" "document_id" => "%{id}" } }