iLSE: An Intelligent Web-based System for Log Structuring and Extraction

Abstract

Analysing software log files has become a challenging task due to the diversity in file structure and the nonstandardisation of log syntax. During the process of extracting log data, it is required to manually decode the log syntax and interpret data semantics, which can become tedious and is often error-prone if not performed carefully. Contemporary log analysis software tools do exist in the market and most of them offer numerous options to analyse log files, however, their sheer focus is on providing log management solutions instead of log analysis capabilities. In particular, none of them offers a generic parsing and extracting solution that can discover hidden data structures, a critical and effortful task in log analysis. We thereby devise such a solution that is able to automatically identify hidden patterns in a given log file and extract useful information by generalising the patterns. The solution is implemented as an intelligent Web-based system known as iLSE (intelligent Log Structuring and Extraction) whose users are not required to possess fluent programming skills. This paper presents a reference architecture for the system as well as a comparison study on how the system performs against contemporary log analysis systems.