Towards Real-time SIEM Based Network Monitoring and Intrusion Detection Through Advanced Event Normalization
Today, many vital functions of a modern society rely on complex IT systems. As this dependence is only set to grow in the future, the ability of administrators to monitor the operations and security of their increasingly more sophisticated network landscapes has become ever more vital. Existing event driven passive network monitoring systems such as Security Information and Event Management (SIEM) systems have largely focused on search and text analytics (for unstructured event data integration) to monitor IT systems using their events. The use of such monitoring systems has been proven highly effective in postmortem analysis and investigation of incidents. However, the index and search paradigm they employ inherently presents a number of performance and accuracy limitations which caps the effectiveness of incident detection, environment monitoring and advanced event correlation in these systems. Additionally, the growing need for real-time complex event analytics has become apparent, which these systems can not address compellingly. As part of this thesis, a new SIEM system named Real-time Event Analytics and Monitoring System (REAMS) was developed which makes use of advanced event processing and normalization paradigms. As part of REAMS, a new process is introduced for event format detection with a constant computational complexity. A Unified Event Representation Model (UERM) (viz. Object Log Format (OLF)) is proposed and implemented which allows events to be expressed in a fully structured way while maintaining semantics (i.e. meaning, context, intention). A dynamic ECMAscript compatible scripting environment is used to implement targeted logic-enabled event parsing as well as a targeted stateless signature and pattern matching engine for detecting incidents and violations. A number of complex correlation algorithms are implemented using REAMS, e.g. network topology discovery and service mapping, network traffic classification, and attack path detection. An approach for event field type optimizations of network related events is also implemented and shown to improve the accuracy of data mining/machine learning algorithms in network environments. The work presented in this thesis demonstrates that advanced event normalization and targeted analytics paradigms can enable passive event driven monitoring systems (e.g. SIEM systems) to execute complex analytical tasks in real-time.