Distributed logs have very important role in Micro-service architecture and ELK stack is one of the most the popular technology.
In ELK stack,
- E stands for ElasticSearch, this service store logs and index logs for searching/visualizing data. Usually it is running on 9200 port.
- L stands for LogStash which is a server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.”. The lightweight version of LogStash is beats, in this article I will show how to install filebeat DaemonSet in kubernetes and collect logs from nodes then send data to ElasticSearch
- K stands for Kibana, it is a website dashboard where you can search/analyse and visualize your logs
In this article I write detail steps to install ELK in Kubernetes, It can work with any cloud platform (EKS, AKS or GCP)
- Install custom resource definitions and the operator with its RBAC rules from ECK
kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml
2. Create elasticsearch.yaml file to deploy an Elasticsearch cluster, in this example I disable SSL but it is not recommend for production deployment.
3. Deploy by run command
kubectl apply -f elasticsearch.yaml
4. After then command, ECK will create an elastic cluster with username “elastic”, to get password run command
Result:
5. To test cluster from your pc, run port-forward command to forward port 9200
kubectl port-forward service/your-cluster-name-es-http 9200
Open browser, run http://localhost:9200 with username: elastic, password generated from command in step 4
6. Install Kibana dashboard to visualize your log, firstly create kibana.yaml deployment to link with Elasticseach cluster
7. Deploy kibana by command
kubectl apply -f kibana.yaml
8. Run port-forward command to test kibana in your pc
kubectl port-forward service/kibana-name-kb-http 5601
Test in your browser by serve url https://localhost:5601 by user: elastic, password generated from step 4
9. Next we need to install filebeat DaemonSet to collect all pod’s logs from kubernetes
10. To test whether filebeat can connect to your elasticsearch, go to DaemonSet log to check connection with elasticsearch
After finish step 10, you can go to Kibana to start analyze your logs, enjoy it!
