Rocknsm-data at 100% disk utilization

I recently tried to log into the Kibana interface for my RockNSM installation, and there was no data available. I ran sudo rockctl status, and noticed that there were several services that had failed. After some research, I realized that the rocknsm-data was a 100% usage. I’m curious to know if there is a way to delete some data from that folder (shown below in the details) and then to cause it to overwrite data at about 90% utilization, in the way that Stenographer does. If I can’t get this installation working correctly, then I will just rebuild the server. However, I would like to know of a way that I can prevent this from happening in the future. I’ve seen some folks talk about creating an automated rollover of Elastic Search’s ILM, but I can no longer even log into Kibana to make any of that happen. I’ve also tried to perform a sudo rockctl destroy, but oddly ironically, I am told that there is not enough space to do this.

Details:

[adminuser@myrocksvr~]$ sudo rockctl start
[sudo] password for adminuser:
ZEEK: starting…
Job for zeek.service failed because the control process exited with error code. See “systemctl status zeek.service” and “journalctl -xe” for details.

STENOGRAPHER: starting…
DOCKET: starting…
SURICATA: starting…

ELASTICSEARCH: starting…
Job for elasticsearch.service failed because the control process exited with error code. See “systemctl status elasticsearch.service” and “journalctl -xe” for details.

KIBANA: starting…
ZOOKEEPER: starting…
KAFKA: starting…
LIGHTTPD: starting…
FSF: starting…
FILEBEAT: starting…
LOGSTASH: starting…
STENOGRAPHER@ENP1S0F1: starting…
STENOGRAPHER@ENP3S0F0: starting…
STENOGRAPHER@ENP3S0F1: starting…
STENOGRAPHER@ENP3S0F2: starting…
STENOGRAPHER@ENP3S0F3: starting…
DOCKET-CELERY-IO: starting…
DOCKET-CELERY-QUERY: starting…

[adminuser@myrocksvr~]$ systemctl status zeek.service

● zeek.service - Zeek Network Intrusion Detection System (NIDS)
Loaded: loaded (/usr/lib/systemd/system/zeek.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:27 UTC; 45s ago
Process: 24037 ExecStart=/usr/bin/zeekctl deploy (code=exited, status=1/FAILURE)

Mar 08 05:10:26 my-rock-svr.mydomain.comsystemd[1]: Starting Zeek Network Intrusion Detection System (NIDS)…
Mar 08 05:10:27 my-rock-svr.mydomain.comzeekctl[24037]: Warning: ZeekControl plugin uses legacy BroControl API. Use
Mar 08 05:10:27 my-rock-svr.mydomain.comzeekctl[24037]: ‘import ZeekControl.plugin’ instead of ‘import BroControl.plugin’
Mar 08 05:10:27 my-rock-svr.mydomain.comzeekctl[24037]: Error: failed to write file: [Errno 28] No space left on device: ‘/data/zeek/spool/.zeekctl-config.sh.tmp’
Mar 08 05:10:27 my-rock-svr.mydomain.comzeekctl[24037]: Error: cannot acquire lock: [Errno 28] No space left on device: ‘/data/zeek/spool/lock.24037’
Mar 08 05:10:27 my-rock-svr.mydomain.comzeekctl[24037]: Error: Unable to get lock
Mar 08 05:10:27 my-rock-svr.mydomain.comsystemd[1]: zeek.service: control process exited, code=exited status=1
Mar 08 05:10:27 my-rock-svr.mydomain.comsystemd[1]: Failed to start Zeek Network Intrusion Detection System (NIDS).
Mar 08 05:10:27 my-rock-svr.mydomain.comsystemd[1]: Unit zeek.service entered failed state.
Mar 08 05:10:27 my-rock-svr.mydomain.comsystemd[1]: zeek.service failed.

[adminuser@myrocksvr~] systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─override.conf Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:28 UTC; 1min 54s ago Docs: https://www.elastic.co Process: 24074 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p {PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 24074 (code=exited, status=1/FAILURE)

Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: Error: A fatal exception has occurred. Program will exit.
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd-entrypoint[24074]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd[1]: Failed to start Elasticsearch.
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd[1]: Unit elasticsearch.service entered failed state.
Mar 08 05:10:28 my-rock-svr.mydomain.comsystemd[1]: elasticsearch.service failed.

[adminuser@myrocksvr~] sudo rockctl status ZEEK: Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:27 UTC; 2min 23s ago STENOGRAPHER: Active: active (exited) since Mon 2021-03-08 04:40:29 UTC; 32min ago DOCKET: Active: active (running) since Mon 2021-03-08 05:10:30 UTC; 2min 19s ago SURICATA: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago ELASTICSEARCH: Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:28 UTC; 2min 21s ago KIBANA: Active: active (running) since Mon 2021-03-08 04:40:31 UTC; 32min ago ZOOKEEPER: Active: active (running) since Mon 2021-03-08 04:40:31 UTC; 32min ago KAFKA: Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:28 UTC; 2min 21s ago LIGHTTPD: Active: failed (Result: exit-code) since Mon 2021-03-08 05:10:28 UTC; 2min 21s ago FSF: Active: active (running) since Mon 2021-03-08 04:40:31 UTC; 32min ago FILEBEAT: Active: active (running) since Mon 2021-03-08 04:40:31 UTC; 32min ago LOGSTASH: Active: active (running) since Mon 2021-03-08 04:40:31 UTC; 32min ago STENOGRAPHER@ENP1S0F1: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago STENOGRAPHER@ENP3S0F0: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago STENOGRAPHER@ENP3S0F1: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago STENOGRAPHER@ENP3S0F2: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago STENOGRAPHER@ENP3S0F3: Active: active (running) since Mon 2021-03-08 04:40:29 UTC; 32min ago DOCKET-CELERY-IO: Active: failed (Result: start-limit) since Mon 2021-03-08 05:10:29 UTC; 2min 20s ago DOCKET-CELERY-QUERY: Active: failed (Result: start-limit) since Mon 2021-03-08 05:10:29 UTC; 2min 21s ago [adminuser@myrocksvr~]

[adminuser@myrocksvr~]$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/rocknsm-root 157209600 6496372 150713228 5% /
devtmpfs 32955744 0 32955744 0% /dev
tmpfs 32969056 0 32969056 0% /dev/shm
tmpfs 32969056 902740 32066316 3% /run
tmpfs 32969056 0 32969056 0% /sys/fs/cgroup
tmpfs 32969056 63140 32905916 1% /tmp
/dev/mapper/rocknsm-var_log 26201600 26201580 20 100% /var/log
/dev/mapper/rocknsm-home 157209600 33020 157176580 1% /home
/dev/mapper/rocknsm-data 3911374848 3911374828 20 100% /data
/dev/sda2 1038336 141208 897128 14% /boot
/dev/sda4 1046516 7696 1038820 1% /boot/efi
/dev/mapper/rocknsm-data_stenographer
3911374848 3480546184 430828664 89%
/data/stenographer

tmpfs 6593812 0 6593812 0% /run/user/1000
[adminuser@myrocksvr~]$

for steganographer to curb disk space usage you actually have to configure multiple files, not just one, to read the same value of disk space % to keep free. I set mine to 30% and this works great.