ELK Documentation

Installing and Configuring Elasticsearch

Installing and Configuring Kibana

Installing Nginx

Installing and Configuring Filebeat on Client servers

Browsing the Kibana Dashboard

Installing and Configuring Elasticsearch

sudo apt-get update

sudo apt-get upgrade

sudo apt-get install software-properties-common apt-transport-https

sudo add-apt-repository ppa:webupd8team/java

apt install default-jre

java -version

EC:-

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

apt-get update

apt-get install elasticsearch

cd /etc/elasticsearch/

Vim elasticsearch.yml

1. uncomment the ‘network.host’ line and change the value to ‘localhost’, and uncomment the default port for elasticsearch ‘http.port’:

———

network.host: localhost

http.port: 9200

———

systemctl start elasticsearch

systemctl enable elasticsearch

We will test our Elasticsearch service by sending an HTTP request:

curl -X GET "localhost:9200"

root@ip-172-31-9-134:/etc/elasticsearch# curl -X GET "localhost:9200"

{

"name" : "ip-172-31-9-134",

"cluster_name" : "elasticsearch",

"cluster_uuid" : "UwCwj42oTSSYCLlcTeqIYw",

"version" : {

"number" : "7.6.2",

"build_flavor" : "default",

"build_type" : "deb",

"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",

"build_date" : "2020-03-26T06:34:37.794943Z",

"build_snapshot" : false,

"lucene_version" : "8.4.0",

"minimum_wire_compatibility_version" : "6.8.0",

"minimum_index_compatibility_version" : "6.0.0-beta1"

},

"tagline" : "You Know, for Search"

}

Installing and Configuring Kibana

apt install kibana

cd /etc/kibana/

Vim kibana.yml

uncomment the ‘server.port’, ‘server.host’, and ‘elasticsearch.hosts’

server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["http://localhost:9200%22]

systemctl enable kibana

systemctl start kibana

Installing Nginx

apt install nginx apache2-utils

cd /etc/nginx/sites-available

rm /etc/nginx/sites-enabled/default

/etc/nginx/sites-enabled

Vim /etc/nginx/sites-enabled/kibana

server {

listen 80;

server_name 172.31.9.134;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/.htpasswd;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

}

1. htpasswd -c /etc/nginx/.htpasswd admin

New password:

Re-type new password:

Adding password for user admin

Tw4:Zz3g9m

http://52.66.253.185/status

1. 4 Installing and Configuring Logstash

apt-get install logstash

Here we are going to generate SSL certificate key to secure log transfer from file beat client. Modify the “hosts” file before creating the SSL certificate.

sudo vim /etc/hosts

Add the following line to file. Make sure to change IP and server name to yours.

172.31.31.158 elk-server elk-server

When done, save and exit the file.

Now change directory to Logstash.

sudo cd /etc/logstash/

Create a folder for SSL:

sudo mkdir ssl

Generate SSL certificate. Change elk-server to your server name in the below command.

sudo openssl req -subj '/CN=elk-server/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout ssl/logstash-forwarder.key -out ssl/logstash-forwarder.crt

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ssl/logstash-forwarder.key -out ssl/logstash-forwarder.crt

Create following files inside “/etc/logstash/conf.d”.

sudo cd /etc/logstash/conf.d/

create a filebeat-input file using vim.

sudo vim filebeat-input.conf

Add the following lines to it.

input {

beats {

port => 5443

type => syslog

ssl => true

ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"

ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"

}

}

Save and close the file and create a new configuration file.

sudo vim syslog-filter.conf

Add the following contents to it.

filter {

if [type] == "syslog" {

grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ]

}

date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

}

}

Save and exit the file. Create elasticsearch output file.

sudo vim output-elasticsearch.conf

Add the following lines to it.

output {

elasticsearch { hosts => ["localhost:9200"]

hosts => "localhost:9200"

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}

}

Syntax verify:-

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

.

.

.

[2020-04-28T12:40:01,901][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

If all ok, Let’s enable Logstash on boot and start the service:

sudo systemctl enable logstash.service

sudo systemctl start logstash.service

curl -v --cacert ca.crt https://elk-server:5443

Installing and Configuring Filebeat on Client servers

sudo wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Install “apt-transport-https” and add repo.

sudo apt-get install apt-transport-https

sudo echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

Update repo and install Filebeat.

sudo apt-get update

sudo apt-get install filebeat

vim /etc/filebeat/filebeat.yml

enabled: true

paths:

- /var/log/*.log

- /home/ubuntu/.pm2/logs

Uncomment the following lines:

output.logstash:

hosts: ["filebeat.airmeet.com:5044"]

json.keys_under_root: true

json.add_error_key: true

Comment Elasticsearch:

1. output.elasticsearch:

1. Array of hosts to connect to.

1. hosts: ["localhost:9200"]

sudo systemctl enable filebeat.service

sudo systemctl start filebeat.service

Browsing the Kibana Dashboard

Enter the created user name and password.

You should see the Kibana Welcome page. Click “Explore my Own” button.

You should be directed to the Kibana Home Page.

Click “Discover” on the left side. Click “Create index pattern”.