Event Monitoring

Kuan Cheang | November 18, 2020

Introudction

Event logs data becomes more important than before since the big data concept has been introduced all over the world. People is trying find valuable information from the data and use it to predict something that may happen in the coming future or prevent something happens again. Therefore, harvesting event data from different kind of network devices, databse or application/system is the first milestone of your big data.

Central Logs Overview

Data streaming and Storage Management

If your company has thousand of computers or network devices such as IoT devices, switches or firewalls. Your may find a trouble about how to harvesting all the event data from these devices and create a central repository for storing all the event data in a single place. Besides, data should be transformed in your required format and performed a data cleansing to reduce the volume usage while streaming the data to your central repository.

Event data can be divided into structured or un-structured. Structured data can be read and understand easily. You don’t need to do much for this kind of data. However, some applications/system may have its own format and store in binary format. You have to use their tool to export the event data or parse it by yourself. It may be take a lot of efforts in order to massage these data and stream it to your central repository. For example, Windows event is stored in a log file with binary format. The log data will required you to parse it and format it as Json, XML or any structural format.

Binary to File

Your data should be compressed and archived after a period of time. Saving the disk space will help you to lower your resource cost since streaming data is keep flowing in every second. A resource management strategy is used to define a lifecycle for your data so it can reduce the risk of running out of disk space.

Data Corelationship

Event data can be co-related to each other for getting better understand what happens during an attack event. For example, a suspicious behavior was found at a specified time. Now, you have to design a rule for detecting this kind of behavior to prevent it happens again. Besides, it should notify the administrator once it has found a behavior that is similar to the previous one.

Macau central view