Need Help Deploying Auditbeat and Filebeat to hosts in my network

I have a single node Rock NSM deployed on an ESXi in my homelab network.

I downloaded the package onto the respective host, configured the auditbeat.yml with the following

output.elasticsearch:
 hosts: ["192.168.1.27:9200"]

setup.kibana:
  host: "192.168.1.27:5601"

When I run
sudo auditbeat setup
I get “connection refused” error
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.1.27:9200: Get http://192.168.1.27:9200: dial tcp 192.168.1.27:9200: connect: connection refused]

I edited my RockNSM node’s /etc/elasticsearch/elasticsearch.yml file to change network.hosts to
network.host: 0.0.0.0
to allow connection other than the default localhost.

it seems to still refuse the connection.

Is there any special network configuration that I need to do with RockNSM outside of a normal ELK deployment to enable distributed deployment of beats agents onto my host machines?

Please help. Thank you

you may need to open the firewall via sudo firewall-cmd --zone=public --add-port=9200/tcp --permanent and then sudo firewall-cmd --reload

Thanks @koelslaw. the firewall changes below helped.

I had to do a little more tweaking to make it work.

first:
in addition to setting my network.hosts in /etc/elasticsearch/elasticsearch.yml to

 network.host: 0.0.0.0

I had to add the following lines:

  discovery.seed_hosts: ["host1"]
  cluster.initial_master_nodes: ["node-1"]

note: “host1” = node.name

and comment out the following:

#discovery.type: single-node

That worked. I tested it with netcat and succeeded.

ran into another problem with Kibana when I ran the sudo auditbeat setup
I get:

    Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

    Index setup finished.
    Loading dashboards (Kibana must be running and reachable)
    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to 
    http://192.168.1.27:5601/api/status fails: fail to execute the HTTP GET request: Get 
    http://192.168.1.27:5601/api/status: dial tcp 192.168.1.27:5601: connect: connection refused. 
    Response: .

I looked around online and found that I have to define server.host in /etc/kibana/kibana.yml to the Kibana’s IP address otherwise its default is localhost.

I edited the kibana.yml and added server.host: 192.168.1.27 to look like below:

server.port: 5601
server.name: "RockNSM"
server.host: 192.168.1.27
server.defaultRoute: "/app/kibana#/dashboard/6151e9d0-bf83-11e9-85bb-3b744f61312d"
elasticsearch.hosts: "http://127.0.0.1:9200"

I ran sudo auditbeat setup on the host machine and voila! it worked!

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

or at least it told me it worked… HOWEVER…

Now I can’t connect to Kibana. Kibana is running, even when i restart it, everything is running. I go to the IP/app/kibana#/ and I get a 503 error.

I do a netstat -ano and I get a TIME_WAIT on the state of my connection

I went back to the kibana.yml file and tried to change this

elasticsearch.hosts: "http://127.0.0.1:9200"

to this:

elasticsearch.hosts: "http://192.168.1.27:9200"

and that didn’t work… :pensive:

I changed the .yml file back to normal and commented out the server.host

server.port: 5601
server.name: "RockNSM"
#server.host: 192.168.1.27
server.defaultRoute: "/app/kibana#/dashboard/6151e9d0-bf83-11e9-85bb-3b744f61312d"
elasticsearch.hosts: "http://127.0.0.1:9200"

I got my Kibana dashboard back but the my auditbeat host is gone. back to square 1!

I’m once again out of things to try. looking for guidance. Just trying to deploy beats to hosts. Thanks!

-Java

Can you try and set kibana server.host to 0.0.0.0 and see if that works? Did you specify a DHCP or Static during setup?

I was able to get the beats to work, finally using the above steps that I mentioned.

for some reason, after I changed the server.host to Rock’s IP (192.168.1.27), Kibana threw a 503 error when accessing the Kibana Url.

The Fix was that I had to include the port number to the URL: https://:5601/app/kibana#/ and it all worked!

Thanks for the help! I should edit the above post to reflect the all the steps should someone run into the same issue.

-Java

Lighttpd config may to be restarted or changes in the config file need to be made. So they match the changes for kibana

Brother, I have tried your fix to get winlogbeats to connect and I cannot get kibana to connect to load dashboards during setup. I can reach kibana through my browser from the computer I am attempting to load beats on.

My error:

PS C:\Program Files\Winlogbeat> .\winlogbeat.exe setup
Overwriting ILM policy is disabled. Set setup.ilm.overwrite:true for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
.\winlogbeat.exe : Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://192.168.0.28:5601/api/status fails:
fail to execute the HTTP GET request: Get http://192.168.0.28:5601/api/status: dial tcp 192.168.0.28:5601: connectex: A connection attempt failed
because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to
respond… Response: .
At line:1 char:1

  • .\winlogbeat.exe setup
  •   + CategoryInfo          : NotSpecified: (Exiting: error ...d.. Response: .:String) [], RemoteException
      + FullyQualifiedErrorId : NativeCommandError
    
    

I changed the elastic net host to 0.0.0.0
I added the additional lines.
I commented out the discovery type
This got elastic to connect.
I modded kibana.yml adding server.host

@koelslaw got any suggestions?

Just solved the problem. I added 5601 to the public zone on the firewall. I thought I had already added it. That’s how this thing works, you can’t fix your own problem until after you publicly ask for help.

1 Like

Thanks for circling back with the solution.

Remember, that 5601 is now exposed and not behind the SSL reverse proxy, so that’s an attack surface you may want to control.

Once you’ve run setup --dashboards, you may want to consider shutting that port down as I don’t think Winlogbeat should need that after the dashboards are loaded as it writes to Logstash or Elasticsearch.

firewall-cmd --remove-port=5601/tcp \
 --permanent && firewall-cmd --reload

Andy, what would be a better solution to this problem?

Sorry, I wasn’t clear.

You can certainly do what you did to load the Index Patterns and the dashboards. Just remember to remove that opening on the FW using the firewall-cmd command, once you’ve run those Beat setup functions, to close up that attack surface.

5601 isn’t needed for the Beats to communicate once the setup is completed.

Ah Andy, thanks for the clarity. Closing the opening using the fw cmd was clear. I am just wondering if there is a better answer then a short term opening of the fw. Thanks for your assistance.

Nope, I think for getting the Dashboards and Index Patterns loaded via *beat setup, you’ve got to be able to reach Kibana via the API over port 5601.

I’m sure there is a very elegant way of reconfiguring Kibana or using some proxy chaining, but the way you’ve done it, is how I’ve done it.