The Construction and Deployment of ELK Real Time Log Analysis Platform

1. The Initial Contact with ELK

1.1 Why ELK?

Logs mainly include system logs, application logs and security logs. The system operation and development personnel can log to understand the hardware and software information of the server, check the errors in the configuration process and the causes of the errors. The log can understand the server’s load, performance security, and take timely corrective actions to correct errors. In general, logs are stored separately on different devices. If you manage dozens or hundreds of servers, you’re still using the traditional method of logging each machine in turn. It feels cumbersome and inefficient. It is imperative that we use centralized logging management, such as open source syslog, to collect and summarize log collection on all servers. Centralized management after logging, log statistics and retrieval has become a more troublesome thing, normally we use grep, awk and wc Linux commands can realize retrieval and statistics, but to demand a higher query, sorting and statistics requirements and the huge machine quantity still use this way to avoid a bit overwhelmed.

Selection and Use of the Hadoop Yarn Scheduler

1. the introduction

Yarn has the role of resource management and task scheduling in the Hadoop ecosystem. A simple understanding of the architecture of the Yarn is given before discussing its constructors.