34 logstash with filebeat integration

⚙️ 1️⃣ Install Filebeat on the Linux Machine

# Download the deb package from Elastic official site
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.1.5-amd64.deb

# Install the package
sudo dpkg -i filebeat-9.1.5-amd64.deb

---

🌐 2️⃣ Prepare Nginx Logs (Dummy Logs)

🔹 Install Nginx:

sudo apt update
sudo apt install nginx -y

🔹 Check if the service is running:

sudo systemctl status nginx

Expected result: active (running)

🔹 Test the default page:

curl http://localhost

You should see the default Nginx HTML page content.

📁 Log file locations:

/var/log/nginx/access.log
/var/log/nginx/error.log

To monitor logs in real time:

sudo tail -f /var/log/nginx/access.log

---

⚙️ 3️⃣ Configure Filebeat to Send Logs to Logstash

Open Filebeat configuration:

sudo nano /etc/filebeat/filebeat.yml

🔸 (A) Input Configuration:

filebeat.inputs:
- type: filestream
  id: nginx-access-logs
  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
  ignore_older: 0
  close_inactive: 0

📝 Note: Enabling the Nginx module automatically activates input collection, so you can later comment this section to avoid duplicate data.

---

🔸 (B) Logstash Output Configuration:

#output.logstash:
   hosts: ["192.168.1.16:5045"]

---

🔸 (C) Enable Nginx and System Modules:

filebeat version
sudo filebeat modules list
sudo filebeat modules enable nginx

---

🔸 (D) Configure the Nginx Module:

Open the module configuration:

sudo nano /etc/filebeat/modules.d/nginx.yml

Ensure the following settings:

- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

✅ Note: After enabling the module, comment out the filebeat.inputs section in filebeat.yml to prevent duplicate data:

# filebeat.inputs:
# - type: filestream
#   enabled: true
#   paths:
#     - /var/log/nginx/access.log
#     - /var/log/nginx/error.log

---

🧩 4️⃣ Configure Logstash to Receive Filebeat Data

Open the pipeline configuration:

sudo nano /etc/logstash/conf.d/nginx-pipeline.conf

Add the following configuration:

input {
  beats {
    port => 5045
    tags => ["nginx"]
  }
}

output {
  elasticsearch {
    hosts => ["https://192.168.1.16:9200"]
    index => "nginx-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "3lSq=GCEWU1ygpW_cEkl"
    ssl_enabled => true
    ssl_verification_mode => "none"
  }
}

---

🔍 5️⃣ Test the Logstash Configuration

Verify configuration correctness:

sudo -u logstash /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/nginx-pipeline.conf

✅ Expected result: Configuration OK

---

🚀 6️⃣ Send Logs from Filebeat to Logstash

🔹 Ensure Logstash is running:

sudo systemctl status logstash

Expected result: active (running)

🔹 Enable and start Filebeat:

sudo systemctl enable filebeat
sudo systemctl start filebeat

🔹 Check Filebeat output:

sudo journalctl -u filebeat -f

You should see:

Connected to logstash host: 192.168.1.16:5045
Successfully published events

To generate a large number of logs for testing:

for i in {1..10000}; do curl -s http://localhost/ > /dev/null; done

🔹 Check Logstash logs:

sudo tail -f /var/log/logstash/logstash-plain.log

Expected line:

[INFO ] Successfully processed events

---

🧠 7️⃣ Verify in Elasticsearch

curl -k -u elastic:3lSq=GCEWU1ygpW_cEkl -X GET "https://localhost:9200/_cat/indices?v"

Expected output:

open nginx-logs-2025.10.19 ...

---

📊 8️⃣ View Data in Kibana

Open Kibana → Discover → Create Data View

Enter:

nginx-logs*

📈 You should now see live log data, confirming that the entire setup is working perfectly:

✅ Filebeat reads logs from /var/log/nginx/access.log\ ✅ Logstash receives and processes the data\ ✅ Elasticsearch stores it in a dedicated index\ ✅ Kibana displays the results

---

✅ Final Result

The pipeline was successfully implemented:\ Filebeat → Logstash → Elasticsearch → Kibana

Linux/Nginx log collection is now running efficiently. 🎯

---

---

---