Adding HP Intel Quad NIC to RockNSM


#1

Problem:

Neither RockNSM 2.1 nor 2.2 will install and run with HP Intel Quad NIC installed on motherboard.

  • Elasticsearch fails to load and install fails at that point.

Affected Hardware:

  • Motherboard: BE170-77EN-427B
    • AMD® Bald Eagle RX-427BB, Quad Core, 4M Cache, 2.7GHz (3.6GHz), 35W
    • Bultin NICs:
      • Intel® I210
      • Realtek RTL8111EP
  • Memory
    • Crucial 16GB Kit 2x8GB DDR3 DDR3L -1600 CL11
  • Additional PCIe NIC
    • HP NC365T PCIe Quad Port Server NIC

Solution:

  • Install RockNSM without quad NIC
  • Install Quad NIC
  • Create bond Interface
    • sudo nmcli con add type bond con-name bond0 ifname bond0 mode 4
    • NOTE: Mode 4 = Dynamic Link Aggregation
  • Disable IPv4 and IPv6 networking on Bond0
    • sudo nmcli connection modify bond0 ipv4.method disabled
    • sudo nmcli connection modify bond0 ipv6.method ignore
  • Set bond interface as promiscuous
    • sudo ip link set bond0 promisc on
  • Create slave Interfaces
    • sudo nmcli con add type bond-slave ifname enp1s0f0 master bond0
    • sudo nmcli con add type bond-slave ifname enp1s0f1 master bond0
    • sudo nmcli con add type bond-slave ifname enp1s0f2 master bond0
    • sudo nmcli con add type bond-slave ifname enp1s0f3 master bond0
  • Bring up all slave interfaces:
    • sudo nmcli connection up bond-slave-enp1s0f0
    • sudo nmcli connection up bond-slave-enp1s0f1
    • sudo nmcli connection up bond-slave-enp1s0f2
    • sudo nmcli connection up bond-slave-enp1s0f3
  • Bring up the bond interface
    • sudo nmcli con up bond0
  • Edit interface in bro conf
    • sudo vi /etc/bro/node.cfg
    • Edit: interface=af_packet::bond0
  • Edit interface in Suricata conf
    • sudo vi /etc/suricata/rocknsm-overrides.yaml
    • Edit: - interface: bond0
  • Edit interface in Stenographer conf
    • sudo vi /etc/stenographer/config.[original interface name]
      • Be sure to do ls /etc/stenographer/ to check the interface name.
    • Edit: , "Interface": "bond0"
  • NOTE: I did NOT edit the name of the interface section or file name in anything. It seems RockNSM and the components themselves use these names and files as a reference elsewhere and when I edited them before everything broke.
  • Restart Rock
    • sudo rock_stop
    • sudo rock_start
  • Log onto Kibana and check Timelion and Discover

Dealing with Bro logging notice.note: CaptureLoss::Too_Much_Loss

If the Ethernet interface on a Bro worker is not properly configured Bro may be unable to capture an entire IP packet. Some NICs offload the reassembly of traffic into "superpackets" so that fewer packets are passed up the stack (e.g. "TCP segmentation offload", or "generic segmentation offload"). This causes the capturing application to observe packets much larger than the MTU of the interface from which they were captured, and may interfere with the maximum packet capture length, or snaplen. Therefore it’s a good idea to disable an interface’s offloading features.

Quick jump into su and a shell script based on the mentioned article:

sudo su
for i in rx tx sg tso ufo gso gro lro; do ethtool -K enp1s0f1 $i off; done
exit

I’ve had no more Bro CaptureLoss messages since running this for each interface. Note that this has to applied at every reboot. The text also discusses how to add a boot script that does this.

Reference links:
Creating bond interface: https://www.thegeekdiary.com/centos-rhel-7-how-to-create-an-interface-bonding-nic-teaming-using-nmcli/

Conf files to edit interfaces in RockNSM (and original idea about how to fix the problem with bonding):
https://handlers.sans.org/gbruneau/rockNSM_2.1.htm#_Sensor_Monitoring_Two

Script to automate most of above:

#Create bond Interface
sudo nmcli con add type bond con-name bond0 ifname bond0 mode 4
#Disable IPv4 and IPv6 networking on Bond0
sudo nmcli connection modify bond0 ipv4.method disabled
sudo nmcli connection modify bond0 ipv6.method ignore
#Set bond interface as promiscuous
sudo ip link set bond0 promisc on
#Create slave Interfaces
sudo nmcli con add type bond-slave ifname enp1s0f0 master bond0
sudo nmcli con add type bond-slave ifname enp1s0f1 master bond0
sudo nmcli con add type bond-slave ifname enp1s0f2 master bond0
sudo nmcli con add type bond-slave ifname enp1s0f3 master bond0
#Bring up all slave interfaces
sudo nmcli connection up bond-slave-enp1s0f0
sudo nmcli connection up bond-slave-enp1s0f1
sudo nmcli connection up bond-slave-enp1s0f2
sudo nmcli connection up bond-slave-enp1s0f3
#Bring up the bond interface
sudo nmcli con up bond0
#Disaable offloading (causes packet loss in Bro)
for i in rx tx sg tso ufo gso gro lro; do ethtool -K enp1s0f0 $i off; done
for i in rx tx sg tso ufo gso gro lro; do ethtool -K enp1s0f1 $i off; done
for i in rx tx sg tso ufo gso gro lro; do ethtool -K enp1s0f2 $i off; done
for i in rx tx sg tso ufo gso gro lro; do ethtool -K enp1s0f3 $i off; done