数据接入 ES logstash

使用 logstash 接入 ES 集群

下载（一定要选择和elasticsearch服务版本一致的版本）：
https://www.elastic.co/cn/downloads/past-releases#logstash

下载 rpm 包

安装:

yum localinstall -y +包文件名称

6.6版本的文档，主要查看怎么配置和怎么运行
设置并运行Logstash
https://www.elastic.co/guide/en/logstash/6.6/setup-logstash.html
怎么使用jdbc插件导入数据库数据
https://www.elastic.co/guide/en/logstash/6.6/plugins-inputs-jdbc.html

使用JDBC导入mysql数据的话，需要先装好java，再去mysql官网下载jar包
https://dev.mysql.com/downloads/connector/j/

这里下载rpm包，使用yum localinstall -y 安装后，会在/usr/share/java目录产生mysql的连接器jar包

配置文件实例：/etc/logstash/conf.d/xxx.conf

input {
  stdin {}
  jdbc {
    jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/xxxxtablename?serverTimezone=UTC"
    jdbc_user => "xxxusername"
    jdbc_password => "xxxxxpassword"
    jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.15.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "2000"
    # 增量更新
    use_column_value => "true"
    record_last_run => "true"
    tracking_column => "id"
    last_run_metadata_path => '/etc/logstash/conf.d/xxxx.log'

    # our query
    statement => "SELECT id,title,content FROM articles where id > :sql_last_value and status=1 ORDER BY id ASC"
    schedule => "* * * * * "
    }
}
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "127.0.0.1:9200"
  "index" => "indexname"
  "document_type" => "article"
  "document_id" => "%{id}"
  }
}

版权属于：Joyber

本文链接：https://blog.qqvbc.com/default/261.html

转载时须注明出处及本声明