Outcomes

Creating Appsec metrics and visualisation

View the original Working Session content
 

Logging as the Basis of SOC (Security Operations Centre) Monitoring

  • A log (file) is a collection of records or events
  • Logs are a critical part of any system giving you an insight into the working of a system

What Challenges are Associated with Logging?

  • Historically, logs have been (often indecipherable) lines of text intended for offline human analysis of what went wrong
  • Logs can be large and correlation can be painful
  • Managing logs and accessing them can get complicated with multiple hosts
  • Searching for an error across hundreds of log files on hundreds of servers is difficult without good tools

A common approach to this problem is to setup a centralised logging solution so that multiple logs can be aggregated in a central location

How is Centralized Logging different from Traditional Logging?

  • Logs are collected at a central server
  • Parsing becomes simpler since data is accessible at a single location
  • A common issue across multiple hosts/services can be identified by correlating specific time frames

What is the ELK stack?

  • Elasticsearch, Logstash and Kibana (E, L, and K) + Beats
  • Different open source modules working together
  • Helps users/admins to collect, analyse and visualize data in (near) real-time
  • Each module fits based on your use case and environment

What is Elasticsearch?

  • Distributed and highly available search engine, written in Java and uses Groovy (painless scripting)
  • Scale can come from bigger servers (vertical scale, or scaling up) or from more servers (horizontal scale, or scaling out)
  • Built on top of Lucene
  • Multi-tenant with Multi types and a set of APIs
  • Document-oriented providing (near) real time search

What is Logstash?

  • Tool for managing events and logs written in Ruby
  • Centralized data processing of all types of logs

What are the 3 Main Components of Logstash?

  • Input: Passing logs to process them into machine understandable format
  • Filter: Set of conditions to perform specific action on an event
  • Output: Decision maker for processed events/logs

What is Kibana?

  • Powerful front-end dashboard written in JavaScript
  • Browser based analytics and search dashboard for Elasticsearch
  • Flexible analytics & visualisation platform
  • Provides data in the form of charts, graphs, counts, maps, etc. in real-time

What is Beats?

  • Lightweight shippers for Elasticsearch & Logstash
  • Capture all sorts of operational data like logs or network packet data
  • Can send logs to either Elasticsearch or Logstash

How do ELK and Beats Work Together?

  • Beats = collect, parse, and ship
  • Logstash = enrich and transport
  • Elasticsearch = search and analyse
  • Kibana = explore and visualize

References

Additional/External References

Session organiser(s)

Imran Mohammed A Imran Mohammed A

Participants

Francois Raynaud Francois Raynaud , Timo Pagel Timo Pagel , Jim Newman Jim Newman , Abhinav Sejpal Abhinav Sejpal , Avi Douglen Avi Douglen , Dominik de Smit Dominik de Smit , Jannik Hollenbach Jannik Hollenbach , Pedro Laguna Pedro Laguna , Peter Turczak Peter Turczak , Sotiraki Sima Sotiraki Sima , Yasmin Martin Yasmin Martin Ernesto Bethencourt Ernesto Bethencourt , Harmeet Singh Harmeet Singh , Pascal Schulz Pascal Schulz , Paul Cutting Paul Cutting , Sophie Tonnoir Sophie Tonnoir , Subash Subash , Sugumaran Uppili Sugumaran Uppili , Vinod Anandan Vinod Anandan

Attached materials: