Kibana broken after 'yum upgrade'

Kibana shows a lot of error messages which all have the same message

[esaggs] > Unexpected token < in JSON at position 0

this after performing ‘yum upgrade’ and landing at version 7.11 for elasticsearch, kibana and a number of beats

i cannot find a way to troubleshoot it or analyse it or fix it

possible causes reported are

  • permissions
  • broken code (though no break is expected for 7.11)

a rollback to 7.10 is not possbile, tried it, elasticsearch logs say no go

I just found this for Kibana / Index Patterns, not sure if suitable as root cause

ecs-*

Time field: ‘@timestamp
Default
This page lists every field in the ecs-* index and the field’s associated core type as recorded by Elasticsearch. To change a field type, use the Elasticsearch Mapping API(opens in a new tab or window)

Mapping conflict
A field is defined as several types (string, integer, etc) across the indices that match this pattern. You may still be able to use these conflict fields in parts of Kibana, but they will be unavailable for functions that require Kibana to know their type. Correcting this issue will require reindexing your data.

this for Name ssh.client

maybe implied by Migrating to 7.11 | Elasticsearch Reference [7.11] | Elastic

FINALLY found something tangible

When ‘Inspect’ ‘View:Requests’ ‘Response’ all Error fields are are found to be generated because of

{
“message”: “Unexpected token < in JSON at position 0”,
“code”: “STREAM”
}

Found a number of things.

  • Errors with kafka shutdown
  • Errors in logstash

After a very messy procedure of redeploying the configuration the elasticsearch stores are now no longer empty.

I continue updating this issue here

The reported issue was fixed and documented how in the mentioned thread in ‘Getting Started’