Shipping Syslog Data To Elasticsearch With Fluent Bit

1 bulan yang lalu

Fluent Bit is simply a wide utilized unfastened root information postulation agent, processor and forwarder that enables you to cod logs, metrics and traces from various sources, select and toggle shape them, and past guardant them to aggregate destinations.

In modern infrastructure, contempt nan emergence of system JSON logging, Syslog remains a modular for networking equipment, bequest applications and Linux distributions. However, analyzing Syslog files locally connected each server is not practical.

In this guide, we will group up Fluent Bit arsenic a centralized Syslog server that accepts logs complete UDP (User Datagram Protocol) and ships them straight to Elasticsearch for analysis.

Prerequisites

Docker and Docker Compose: Installed connected your system.
Elasticsearch: We will nonstop logs to an Elasticsearch instance. To travel along, you should person an lawsuit running. You tin mention to this guide to tally it locally.
Familiarity pinch Fluent Bit concepts: Such arsenic inputs, outputs and buffers. If you’re unfamiliar pinch these concepts, please mention to the official documentation.

What Is Syslog?

Syslog is simply a modular for connection logging. It allows separation of nan package that generates messages, nan strategy that stores them, and nan package that reports and analyzes them.

Syslog messages mostly recreation complete UDP larboard 514 (or 5140 successful non-root environments). Because it is simply a “fire and forget” protocol, it is accelerated and lightweight, making it perfect for high-volume logging from routers, firewalls and lightweight Linux containers. However, earthy Syslog matter tin beryllium difficult to query. By utilizing Fluent Bit, we tin ingest these messages, building them and shop them successful a hunt motor for illustration Elasticsearch.

To cognize much astir Syslog, you tin mention to nan Syslog Wikipedia page.

Our Use Case

In this demo, we will create a simulation situation utilizing Docker Compose:

Fluent Bit: Configured to perceive connected UDP Port 5140 for Syslog traffic.
Alpine loggers: Two abstracted Alpine Linux containers that will enactment arsenic “network devices.” They will make log messages each 10 seconds utilizing nan logger bid and nonstop them to our Fluent Bit container.
Elasticsearch: The destination wherever our logs will beryllium stored and indexed.

Instructions

1. Create Project Directory

First, create a directory to clasp your configuration and Docker Compose files.

mkdir fluent-bit-syslog-demo

cd fluent-bit-syslog-demo

2. Create Fluent Bit Configuration

Create a directory named fluent-bit/config and wrong it, create a record named fluent-bit.yaml pinch nan pursuing content:

service:

flush: 1

log_level: info

parsers_file: parsers.conf

pipeline:

inputs:

- name: syslog

mode: udp

listen: 0.0.0.0

port: 5140

outputs:

- name: es

match: '*'

# CHANGE THESE TO MATCH YOUR ELASTICSEARCH SETUP

host: 192.168.1.5

port: 9200

index: syslog-data

http_user: elastic

http_passwd: rslglTS4

suppress_type_name: 'On'

This record tells Fluent Bit to perceive for Syslog messages and guardant them to your Elasticsearch instance.

Note: Update nan host, http_user, and http_passwd successful nan output conception to lucifer your existent Elasticsearch credentials.

3. Create Docker Compose File

Create a record named docker-compose.yaml pinch nan pursuing content:

services:

fluent-bit:

image: 'fluent/fluent-bit:latest'

container_name: fluent-bit

ports:

- '6000:5140/udp'

- '24224:24224'

volumes:

- './fluent-bit/config/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml'

networks:

- syslog-test

restart: unless-stopped

command: '-c /fluent-bit/etc/fluent-bit.yaml'

alpine-logger-1:

image: 'alpine:latest'

container_name: alpine-logger-1

depends_on:

- fluent-bit

networks:

- syslog-test

command: |

/bin/sh -c " apk adhd --no-cache util-linux && while true; do

logger -n fluent-bit -P 5140 -t alpine-test \"This is a test message from Alpine Logger 1 at \$(date)\"

sleep 10

done "

alpine-logger-2:

image: 'alpine:latest'

container_name: alpine-logger-2

depends_on:

- fluent-bit

networks:

- syslog-test

command: |

/bin/sh -c " apk adhd --no-cache util-linux && while true; do

logger -n fluent-bit -P 5140 -t alpine-test \"This is a test message from Alpine Logger 2 at \$(date)\"

sleep 10

done "

networks:

syslog-test:

driver: bridge

This record defines our Fluent Bit work and nan 2 Alpine containers generating traffic. The Alpine instrumentality uses nan logger CLI to create logs successful nan Syslog format.

4. Run nan Container

Start nan situation utilizing Docker Compose:

Once nan containers are running, nan Alpine instances will instantly commencement sending logs to Fluent Bit, which forwards them to Elasticsearch.

5. Verify Logs successful Elasticsearch

Note: We person only created indices successful Elasticsearch pinch shape syslog-data*. To position these logs successful Kibana, you request to create a Data View.

6. Clean up

Conclusion

In this guide, we successfully group up a lightweight Syslog collector utilizing Fluent Bit. We simulated a real-world situation wherever aggregate servers nonstop logs to a cardinal constituent via UDP. Fluent Bit collected these logs and shipped them to Elasticsearch for retention and analysis.

To study much astir Fluent Bit, read: