Zeek stopped collecting logs (or Kibana stopped showing zeek logs)

Kibana stopped showing zeek conn.log and/or event.dataset : connection. New to the platform and I don’t know where it all of a sudden broke. See screenshots for reference.

I typically get relatively the same amount of suricata alerts and zeek counts on the Network SIEM tab. and using the same timeframe, zero zeek.

Not sure if this is related but I checked systemctl status zeek and shows a fatal: parameter inet_interface: no local interface found for ::1 error. I tried restarting zeek but no joy, tried restarting rock, also no joy.

Thanks for posting the screenshots. Thats really helpful. Zeek looks fine. That error is because the system isn’t configured to send email, but Zeek likes to send hourly summaries.

My guess is kafka or logstash. Post the output of the following and we can narrow it down.

# show data pipeline status
sudo systemctl status kafka zookeeper logstash
# Show all TCP/IP sockets and their processes
sudo ss -plant
# Show disk usage
sudo df -h
# show rock rpm version
sudo yum info rock 

I got it to work after rebooting a couple times. All the missing data showed up; there was no break in collection, Kibana just wasn’t showing it. Then last night, it stopped working again. Here’s the output as requested:

Status of Kafka, Zookeeper and LogStash

● kafka.service - Kafka
   Loaded: loaded (/usr/lib/systemd/system/kafka.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kafka.service.d
           └─override.conf
   Active: active (running) since Sat 2020-04-25 18:53:07 UTC; 1 day 22h ago
 Main PID: 2248 (java)
   CGroup: /system.slice/kafka.service
           └─2248 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xlog:gc:/var/log/kafka/kafkaServer-gc.log -Xlog:gc* -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -D...

Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,phases     ] GC(815)   Evacuate Collection Set: 18.8ms
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,phases     ] GC(815)   Post Evacuate Collection Set: 1.5ms
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,phases     ] GC(815)   Other: 0.3ms
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,heap       ] GC(815) Eden regions: 45->0(45)
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,heap       ] GC(815) Survivor regions: 6->6(7)
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,heap       ] GC(815) Old regions: 425->427
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,heap       ] GC(815) Humongous regions: 129->129
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,metaspace  ] GC(815) Metaspace: 37567K->37567K(1083392K)
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc            ] GC(815) Pause Young (Normal) (G1 Evacuation Pause) 603M->561M(1024M) 20.834ms
Apr 27 17:12:36 RockNSM kafka-server-start.sh[2248]: [166768.071s][info][gc,cpu        ] GC(815) User=0.05s Sys=0.00s Real=0.02s

● zookeeper.service - Zookeeper
   Loaded: loaded (/usr/lib/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-04-25 18:53:00 UTC; 1 day 22h ago
 Main PID: 1832 (java)
   CGroup: /system.slice/zookeeper.service
           └─1832 /bin/java -cp /usr/share/java/zookeeper/zookeeper-3.4.14.jar:/usr/share/java/zookeeper/slf4j-log4j12-1.7.25.jar:/usr/share/java/zookeeper/slf4j-api-1.7.25.jar:/usr/share/java/zookeeper/netty-3.10.6.Final.jar:/usr/share/java/zookeeper/log4j-1.2.17.jar:/usr/share/java/zookeeper/jline-0.9.94.jar:/usr/share/java/zookeeper/audienc...

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-04-25 18:52:56 UTC; 1 day 22h ago
 Main PID: 1319 (java)
   CGroup: /system.slice/logstash.service
           └─1319 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/uran...

Apr 27 17:09:43 RockNSM logstash[1319]: [2020-04-27T17:09:43,967][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x46261723>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,968][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x2e9c00fb>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,969][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x57162ea7>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,971][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1cd5c011>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,972][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x69bcd8cd>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,989][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x290e9291>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,991][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x42b5d256>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,992][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x1d5456d9>], :response=>{"index"=>{"_inde
Apr 27 17:09:44 RockNSM logstash[1319]: [2020-04-27T17:09:43,996][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x478f7a7c>], :response=>{"index"=>{"_inde
Apr 27 17:10:33 RockNSM logstash[1319]: [2020-04-27T17:10:33,914][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ecs-suricata-network-2020.04.27", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x47982673>], :response=>{"index"=>{"_inde

TCP/IP sockets and their process:

State       Recv-Q Send-Q               Local Address:Port                              Peer Address:Port              
LISTEN      0      128                              *:22                                           *:*                  
LISTEN      0      50                               *:44151                                        *:*                  
LISTEN      0      100                      127.0.0.1:25                                           *:*                  
LISTEN      0      128                              *:443                                          *:*                  
LISTEN      0      128                    192.168.1.27:5601                                         *:*                  
LISTEN      0      50                               *:9092                                         *:*                  
LISTEN      0      50                               *:2181                                         *:*                  
LISTEN      0      5                        127.0.0.1:5800                                         *:*                  
LISTEN      0      128                      127.0.0.1:6379                                         *:*                  
LISTEN      0      50                               *:46287                                        *:*                  
LISTEN      0      128                              *:9200                                         *:*                  
LISTEN      0      128                              *:80                                           *:*                  
LISTEN      0      128                    192.168.1.27:1234                                         *:*                  
LISTEN      0      128                              *:9300                                         *:*                  
ESTAB       0      0                      192.168.1.27:43430                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50756              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56836              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56814              
ESTAB       0      0                        127.0.0.1:44104                                127.0.0.1:47761              
ESTAB       0      0                        127.0.0.1:56746                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:56858                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:50618                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43528              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43412              
ESTAB       0      0                        127.0.0.1:56144                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:57184                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39904              
ESTAB       0      0                      192.168.1.27:50426                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43430              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39906              
ESTAB       0      0                        127.0.0.1:39852                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:55208                                127.0.0.1:47762              
ESTAB       0      0                        127.0.0.1:56876                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:33644                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39870              
ESTAB       0      0                        127.0.0.1:56704                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39828              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39882              
ESTAB       0      0                        127.0.0.1:57164                                127.0.0.1:6379               
ESTAB       0      0                      192.168.1.27:43424                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:56248                                127.0.0.1:6379               
ESTAB       0      0                      192.168.1.27:50732                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56866              
ESTAB       0      0                        127.0.0.1:56948                                127.0.0.1:9092               
TIME-WAIT   0      0                      192.168.1.27:9092                               192.168.1.27:50630              
ESTAB       0      0                        127.0.0.1:42654                                127.0.0.1:47763              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43532              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56876              
ESTAB       0      0                      192.168.1.27:43412                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:56264                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56814                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33664              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57124              
ESTAB       0      0                      192.168.1.27:43404                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39854              
ESTAB       0      0                      192.168.1.27:50616                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:9200                                192.168.1.4:57782              
ESTAB       0      0                      192.168.1.27:50438                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56746              
ESTAB       0      0                        127.0.0.1:39854                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:56782                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39902              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39852              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56248              
ESTAB       0      0                        127.0.0.1:56856                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:33646                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56946                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56444              
ESTAB       0      0                        127.0.0.1:57460                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56866                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:43432                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:33660                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:2181                                 127.0.0.1:43232              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57466              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43416              
ESTAB       0      0                        127.0.0.1:39828                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56852              
ESTAB       0      0                      192.168.1.27:43472                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:39826                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:43232                                127.0.0.1:2181               
ESTAB       0      0                      192.168.1.27:50730                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50438              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43534              
ESTAB       0      0                        127.0.0.1:33652                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56176              
ESTAB       0      0                      192.168.1.27:9200                                192.168.1.4:57774              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33656              
ESTAB       0      0                        127.0.0.1:56278                                127.0.0.1:6379               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43472              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39868              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50498              
ESTAB       0      0                        127.0.0.1:56174                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33652              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39878              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56142              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56950              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56430              
ESTAB       0      0                      192.168.1.27:50498                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:43394                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39826              
ESTAB       0      0                        127.0.0.1:56820                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57460              
ESTAB       0      0                        127.0.0.1:33642                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33646              
ESTAB       0      0                        127.0.0.1:33664                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:33640                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:44086                                127.0.0.1:47761              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39844              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33654              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56144              
ESTAB       0      0                      192.168.1.27:50510                              192.168.1.27:9092               
ESTAB       0      0                      192.168.1.27:50754                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33660              
ESTAB       0      0                      192.168.1.27:9200                                192.168.1.4:57770              
ESTAB       0      0                        127.0.0.1:55188                                127.0.0.1:47762              
ESTAB       0      0                        127.0.0.1:56172                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56962              
ESTAB       0      0                        127.0.0.1:56430                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33642              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56264              
ESTAB       0      0                      192.168.1.27:50756                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:56960                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33662              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43404              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43536              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43448              
ESTAB       0      0                        127.0.0.1:56836                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50436              
ESTAB       0      0                        127.0.0.1:56874                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39874              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39880              
ESTAB       0      0                        127.0.0.1:57466                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56862              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43526              
ESTAB       0      0                      192.168.1.27:43448                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39876              
ESTAB       0      0                        127.0.0.1:56142                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56854                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50618              
ESTAB       0      0                        127.0.0.1:57464                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56854              
ESTAB       0      0                        127.0.0.1:33654                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33666              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56946              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56956              
ESTAB       0      0                        127.0.0.1:56862                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:39856                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:56962                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:56444                                127.0.0.1:6379               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50730              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56292              
ESTAB       0      0                        127.0.0.1:56850                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:56852                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43394              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50616              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43410              
ESTAB       0      0                        127.0.0.1:33658                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57164              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56958              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43530              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56782              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56856              
ESTAB       0      0                        127.0.0.1:44052                                127.0.0.1:47761              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33640              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56178              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56704              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50744              
ESTAB       0      0                        127.0.0.1:39868                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:56958                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:57126                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57126              
ESTAB       0      0                        127.0.0.1:56952                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43432              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33658              
ESTAB       0      0                      192.168.1.27:50734                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56820              
ESTAB       0      0                        127.0.0.1:56950                                127.0.0.1:9092               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56948              
ESTAB       0      0                        127.0.0.1:33662                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56850              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50754              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50732              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50510              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56172              
ESTAB       0      0                        127.0.0.1:57462                                127.0.0.1:6379               
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43484              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56874              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39872              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50734              
ESTAB       0      0                      192.168.1.27:22                              192.168.41.102:58380              
TIME-WAIT   0      0                      192.168.1.27:9092                               192.168.1.27:50668              
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57462              
ESTAB       0      0                        127.0.0.1:57124                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56952              
ESTAB       0      0                      192.168.1.27:43416                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57464              
ESTAB       0      0                        127.0.0.1:39844                                127.0.0.1:9200               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:33644              
ESTAB       0      0                        127.0.0.1:33666                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56178                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56960              
ESTAB       0      0                        127.0.0.1:9200                                 127.0.0.1:39856              
ESTAB       0      0                      192.168.1.27:43410                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56278              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:50426              
ESTAB       0      0                      192.168.1.27:9092                               192.168.1.27:43424              
ESTAB       0      0                      192.168.1.27:43484                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:33656                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:57184              
ESTAB       0      0                        127.0.0.1:9092                                 127.0.0.1:56858              
ESTAB       0      0                      192.168.1.27:50436                              192.168.1.27:9092               
ESTAB       0      0                        127.0.0.1:56292                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:6379                                 127.0.0.1:56174              
ESTAB       0      0                        127.0.0.1:56176                                127.0.0.1:6379               
ESTAB       0      0                        127.0.0.1:56956                                127.0.0.1:9092               
ESTAB       0      0                      192.168.1.27:50744                              192.168.1.27:9092               
LISTEN      0      50              [::ffff:127.0.0.1]:9600                                      [::]:*                  
LISTEN      0      128                           [::]:47761                                     [::]:*                  
LISTEN      0      128                           [::]:47762                                     [::]:*                  
LISTEN      0      128                           [::]:47763                                     [::]:*                  
LISTEN      0      128                           [::]:47764                                     [::]:*                  
ESTAB       0      0             [::ffff:192.168.1.27]:43530                     [::ffff:192.168.1.27]:9092               
ESTAB       0      0               [::ffff:127.0.0.1]:39904                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0             [::ffff:192.168.1.27]:43526                     [::ffff:192.168.1.27]:9092               
ESTAB       0      0               [::ffff:127.0.0.1]:47761                       [::ffff:127.0.0.1]:44086              
ESTAB       0      0               [::ffff:127.0.0.1]:39882                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:47762                       [::ffff:127.0.0.1]:55208              
ESTAB       0      0               [::ffff:127.0.0.1]:39902                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:39880                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:47761                       [::ffff:127.0.0.1]:44052              
ESTAB       0      0               [::ffff:127.0.0.1]:39878                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0             [::ffff:192.168.1.27]:43532                     [::ffff:192.168.1.27]:9092               
ESTAB       0      0               [::ffff:127.0.0.1]:39870                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0             [::ffff:192.168.1.27]:43536                     [::ffff:192.168.1.27]:9092               
ESTAB       0      0               [::ffff:127.0.0.1]:39872                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:47762                       [::ffff:127.0.0.1]:55188              
ESTAB       0      0               [::ffff:127.0.0.1]:39876                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:39874                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:39906                       [::ffff:127.0.0.1]:9200               
ESTAB       0      0               [::ffff:127.0.0.1]:47761                       [::ffff:127.0.0.1]:44104              
ESTAB       0      0             [::ffff:192.168.1.27]:43528                     [::ffff:192.168.1.27]:9092               
ESTAB       0      0               [::ffff:127.0.0.1]:47763                       [::ffff:127.0.0.1]:42654              
ESTAB       0      0             [::ffff:192.168.1.27]:43534                     [::ffff:192.168.1.27]:9092

Disk Usage:

Filesystem                             Size  Used Avail Use% Mounted on
devtmpfs                               6.9G     0  6.9G   0% /dev
tmpfs                                  6.9G     0  6.9G   0% /dev/shm
tmpfs                                  6.9G  679M  6.2G  10% /run
tmpfs                                  6.9G     0  6.9G   0% /sys/fs/cgroup
/dev/mapper/rocknsm-root                15G  6.5G  8.6G  43% /
tmpfs                                  6.9G  748K  6.9G   1% /tmp
/dev/mapper/rocknsm-data                91G   69G   23G  76% /data
/dev/mapper/rocknsm-data_stenographer  200G  178G   22G  89% /data/stenographer
/dev/mapper/rocknsm-home                20G   33M   20G   1% /home
/dev/sda1                              509M  160M  350M  32% /boot
/dev/mapper/rocknsm-var_log             15G  8.3G  6.8G  55% /var/log
/dev/sda2                              512M  7.6M  505M   2% /boot/efi
tmpfs                                  1.4G     0  1.4G   0% /run/user/1000

Rock RPM version

Installed Packages
Name        : rock
Arch        : noarch
Version     : 2.5.0
Release     : 2
Size        : 294 k
Repo        : installed
From repo   : anaconda
Summary     : Network Security Monitoring collections platform
URL         : http://rocknsm.io/
License     : BSD
Description : ROCK is a collections platform, in the spirit of Network Security Monitoring.

Thanks for your help!

Ill have to look at the logstash one later but the issue with kafka and zookeeper is usually one starts before the other. Zookeeper needs to start first and then kafka.