Zitified Beats Error "Failed to publish events caused by: EOF"

I am trying out the zitified Beats but can’t establish a connection to my logstash instance.
This kind of looks like the connection was successfully established but beats throws an error because the underlying TCP connection was terminated. Any ideas?

Logs:

Jan 22 17:46:45 server auditbeat[20550]: {"log.level":"info","@timestamp":"2023-01-22T17:46:45.240+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://my.ziti.de:9000)) established","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:45 server auditbeat[20550]: {"log.level":"error","@timestamp":"2023-01-22T17:46:45.246+0100","log.logger":"logstash","log.origin":{"file.name":"logstash/async.go","file.line":280},"message":"Failed to publish events caused by: EOF","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:45 server auditbeat[20550]: {"log.level":"info","@timestamp":"2023-01-22T17:46:45.246+0100","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":181},"message":"Drop batch","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:45 server auditbeat[20550]: {"log.level":"error","@timestamp":"2023-01-22T17:46:45.262+0100","log.logger":"logstash","log.origin":{"file.name":"logstash/async.go","file.line":280},"message":"Failed to publish events caused by: client is not connected","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:45 server auditbeat[20550]: {"log.level":"info","@timestamp":"2023-01-22T17:46:45.262+0100","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":181},"message":"Drop batch","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:46 server auditbeat[20550]: {"log.level":"error","@timestamp":"2023-01-22T17:46:46.674+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":176},"message":"failed to publish events: client is not connected","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:46 server auditbeat[20550]: {"log.level":"info","@timestamp":"2023-01-22T17:46:46.674+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":139},"message":"Connecting to backoff(async(tcp://my.ziti.de:9000))","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:46 server auditbeat[20550]: {"log.level":"info","@timestamp":"2023-01-22T17:46:46.682+0100","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":147},"message":"Connection to backoff(async(tcp://my.ziti.de:9000)) established","service.name":"auditbeat","ecs.version":"1.6.0"}
Jan 22 17:46:46 server auditbeat[20550]: {"log.level":"error","@timestamp":"2023-01-22T17:46:46.687+0100","log.logger":"logstash","log.origin":{"file.name":"logstash/async.go","file.line":280},"message":"Failed to publish events caused by: EOF","service.name":"auditbeat","ecs.version":"1.6.0"}

You’ve done all the “normal” troubleshooting? Usually that boils down to making sure the identity you’re using on the beats side has access to edge routers and the service using policy-advisor: ziti edge policy-advisor identities -q looking for the Dial: Y for your service…

You’ve verified that the far side is accepting requests for that intercepts specified etc, right?

In my experience and “EOF” like that is usually the overlay saying “this traffic is not accepted” or “this traffic has no where to go” or “this traffic tried to go somewhere, but the far end rejected it”… It’s that kind of error. So when I see “EOF” type errors, I think, “something is not listening and is immediately rejecting my traffic”.

A couple of suggestions:

  • verify your config (per @TheLumberjack)
  • make sure your environment variable is set (ZITI_IDENTITIES=)
  • try another zitified agent filebeat or metricbeat with the same setup