Step-by-step guide to monitor logs with Grafana, Loki, and FluentBit

[rubens-facilit]

After struggling for a while to monitor my Coolify apps using Grafana, I finally made it! Here’s a step-by-step tutorial on how to do it with Loki, Grafana, and FluentBit. I have not find any straightforward instructions, so I hope you find it helpful.

---

1. Deploy Loki and Grafana Services

• Deploy Grafana and Loki in a Docker Compose resource:
  Note: Replace the Grafana admin password with a strong one.

  version: '3'
  services:
    loki:
      image: grafana/loki:latest
      ports:
        - "3100:3100"
      volumes:
        - loki-data:/loki
      command: -config.file=/etc/loki/local-config.yaml
      networks:
        - loki
  
    grafana:
      image: grafana/grafana:latest
      ports:
        - "3000:3000"
      environment:
        - GF_SECURITY_ADMIN_USER=admin
        - GF_SECURITY_ADMIN_PASSWORD=password
        - GF_AUTH_ANONYMOUS_ENABLED=false
      volumes:
        - grafana-data:/var/lib/grafana
      networks:
        - loki
      depends_on:
        - loki
  
  volumes:
    loki-data:
    grafana-data:
  
  networks:
    loki:

---

2. Configure FluentBit on the Server

• Update the FluentBit configuration:
  Go to Servers >> your server >> Log Drains.
  Paste the following into "Custom FluentBit Configuration". Replace the Host value with the name of your Loki container.

  [SERVICE]
      Flush             5
      Log_Level         debug
  
  [INPUT]
      Name              forward
      Listen            0.0.0.0
      Port              24224
      Tag               docker
  
  [OUTPUT]
      Name              loki
      Match             *
      Host              loki-mcs40ccs40cmcs40ccs40c
      Port              3100
      Labels            job=fluentbit, container_name=$CONTAINER_NAME, container_id=$CONTAINER_ID
      Label_Keys        $container_name,$container_id
      Line_Format       json

• Mark "enabled" and then click save.

---

3. Connect the Loki Container to the FluentBit Network

• Connect the Loki container to the log-drains network:

  docker network connect log-drains_default loki-mcs40ccs40cmcs40ccs40c

Again, replace the Loki container name with yours.

---

4. Set Environment Variable in Each Service and enable Log Drains

For single Docker resourcers

• Define the variable:
  In each service container, set the following environment variable:

  FLUENTD_ADDRESS=localhost:24224

• Go to Advanced >> Drain Logs and activate it.

• Restart each container after setting this.

For Docker Compose resourcers

Add this to each service of your docker-compose file:

logging:
  driver: fluentd
  options:
    fluentd-address: "localhost:24224"

Redeploy.

---

5. Access Grafana and Explore Logs

• Open Grafana:
  Visit http://<server_ip>:3000 in your browser.
• Enable Loki Data source:
  Go to Connections >> Data Sources >> Loki. URL: http://loki-mcs40ccs40cmcs40ccs40c:3100. Save and Test.
• Test:
  Go to Connections >> Data Sources >> Loki >> Explore and view your logs.

---

6. Troubleshooting

• If you do not see any logs:
  Check the FluentBit logs by running:

  docker logs coolify-log-drain

If the logs appear correctly, everything is working and you can now create dashboards to monitor your application logs.

[rubens-facilit]

Attention: It seems that this doesn't work for services deployed via Docker Compose, unfortunately. Does anyone have a clue why, and how to solve it?

[rubens-facilit]

Attention: It seems that this doesn't work for services deployed via Docker Compose, unfortunately. Does anyone have a clue why, and how to solve it?

Could be related to #2915

[rubens-facilit]

Ok, got it!
For docker-compose resources, we need to add this to each service:

  logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"

I'll update the first post.