Export logs to ElasticSearch/Kibana

In this tutorial, you will learn how to export Devbox logs into ElasticSearch/Kibana stack using aidbox-cli.
Configure ElasticSearch & Kibana
So first of all, let's add to Devbox docker-compose file Elastic & Kibana:
1
services:
2
  ...
3
  elasticsearch:
4
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.0
5
    volumes:
6
      - ./esdata:/usr/share/elasticsearch/data
7
    environment:
8
      - cluster.name=aidbox-cluster
9
      - bootstrap.memory_lock=true
10
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
11
      - discovery.type=single-node
12
    ulimits:
13
      memlock:
14
        soft: -1
15
        hard: -1
16
    ports:
17
      - "9200:9200"
18
  kibana:
19
    links:
20
      - "elasticsearch:elasticsearch"
21
    image: docker.elastic.co/kibana/kibana:7.0.0
22
    ports:
23
      - "5601:5601"
24
    environment:
25
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200
26
  ....
Copied!
Configure Aidbox & aidbox-cli 
Next step is to configure Devbox logging into the file system. We have to mount logs volume and provide AIDBOX_LOGS env variable set to path to logs file:
1
services:
2
  devbox:
3
    image: "${DEVBOX_IMAGE}"
4
    depends_on:
5
      - "devbox-db"
6
    links:
7
      - "devbox-db:database"
8
    ports:
9
      - "8888:8888"
10
    volumes:
11
    - "./logs:/logs"
12
    environment:
13
      ....
14
      AIDBOX_LOGS: '/logs/devbox'
15
  devbox-db:
16
    ...
17
  logexp:
18
    links:
19
      - "elasticsearch:elasticsearch"
20
    image: 'aidbox/aidbox-cli:0.0.1-RC1'
21
    command: ["/aidbox", "es", "logs", "-l", "http://elasticsearch:9200/logs", "-f", "/logs/devbox"]
22
    volumes:
23
    - "./logs:/logs"
Copied!
In the volumes section we mount logs directory as a volume to devbox (Devbox)  service and do the same for aidbox-cli (logexp) service. Devbox will log into /logs/devbox file and aidbox-cli will read this file and send logs into elasticsearch service.
Logger use elastic url prefix  http://elasticsearch:9200/logs and send logs into index like thishttp://lasticsearch:9200/logs-2019-08-01/logs , i.e. index prefix-<date> will be used as an index name.  You can use pattern logs-* to search in indexes in kibana.
Resulting docker-compose file should similar to  docker-compose.yaml. Do not forget to replace LICENSE_ID & KEY with your license information.
Start Devbox and all services
Now it's time to start your services:
1
docker-compose -d up
Copied!

Elastic Logs and Monitoring Integration

Log analysis and visualization tutorial

Last modified 1yr ago

Copy link

Contents

Configure ElasticSearch & Kibana

Configure Aidbox & aidbox-cli

Start Devbox and all services