配置主机名与服务器的映射
对于CentOS来说在，在本地试验可能会遇到莫名其妙的问题，这一般是由于主机名不能正确识别导致。为了避免可能遇到的问题，首先查询本机主机名,
$ hostname
HOME
然后加入一条本地解析到/etc/hosts文件中
127.0.0.1 HOME
获取web数据写入kafka
input{
tcp{
port => 8888
#端口
mode => "server"
#操作模式,server:监听客户端连接, client:连接到服务器
host => "10.2.46.195"
#当mode为server, 指定监听地址, 当mode为client, 指定连接地址, 默认0.0.0.0
#ssl_enable => false
#是否启用SSL, 默认false ssl_cert => ""
#SSL证书路径
#ssl_extra_chain_certs => []
#将额外的X509证书添加到证书链中
#ssl_key => ""
#SSL密钥路径
#ssl_key_passphrase => "nil"
#SSL密钥密码, 默认nil
#ssl_verify => true
#核实与CA的SSL连接的另一端的身份
#tcp_keep_alive => false
#TCP是否保持alives
}
}
input{
udp{
buffer_size => 65536
#从网络读取的最大数据包大小, 默认65536
host => "10.2.46.195" #监听地址
port => 8888 #端口
queue_size => 2000 #在内存中保存未处理的UDP数据包的数量, 默认2000
workers => 2 #处理信息包的数量, 默认2
}
}
output {
#stdout { codec => rubydebug }
kafka {
bootstrap_servers => ["10.2.46.195:9092"]
topic_id =>["test"]
codec => plain {
format => "%{message}"
}
}
}
从kafka读取日志写入elasticsearch集群A
input {
#日志入的数据源
kafka {
bootstrap_servers => ["10.2.46.195:9092"]
#消费组必填
group_id => "test"
topics => ["test"]
consumer_threads => 5
decorate_events => true
codec => "json"
}
file{
#path => "/usr/local/logstatic_elasticsearch_kafka_data/*.xml"
#type => "sparkfile"
#start_position => "beginning"
#}
}
filter {
json {
source => "message"
}
date {
match => ["timestamp","UNIX_MS"]
target => "@timestamp"
}
}
output {
#stdout { codec => rubydebug }
elasticsearch {
hosts => ["10.2.46.196:9200","10.2.46.195:9200","10.2.46.195:9201","10.2.46.195:9202","10.2.46.195:9203"]
#cluster => es6
}
}
启动kafka服务器
./bin/kafka-server-start.sh ./config/server.properties
创建主题
bin/kafka-topics.sh --create --zookeeper 127.0.01:2181 --replication-factor 1 --partitions 1 --topic test
消费者
./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:9092 --topic test
新版本(2.0.)
./bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test
生产者
./bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
详情查询
- ELK配置
- http://toutiao.com/item/6639600006435176964/
- kafka+zookeeper集群
- https://www.toutiao.com/i6640963938471117316/
- 应用日志通过logback 发送到日志收集器 logstash
- http://toutiao.com/item/6640335482557301255/
閱讀更多 養碼場007 的文章
關鍵字: ElasticSearch JSON CentOS
