Index logging versus index-free logging

When searching for a log management solution, some providers are claiming the advantages of index logging, and others claiming the advantages of index-free logging. Which one is the best? In my opinion, it depends on different factors. In this article, I will try to bring some light to this.  

Index logging

With this method, the data is being processed upfront. It creates different tables that combine different indexing options. As a result, when querying the database, the result will appear instantly. Moreover, even when managing a vast amount of data, the index-logging leads to instant query results. Certainly, this method will induce a high-speed response. However, we must take into consideration that this upfront processing needs more CPU & RAM from the server. In addition, the servers need more disk space because of the tables created by the indexing. Also, you need to define upfront the searchable elements and/or correlations.

Index-free logging

In contrast, with index-free logging, the data is stored as a table on the fly. The ingestion process is quicker and does not consume additional CPU, RAM, or disk space. Furthermore, there are several mechanisms to speed up the queries results: containers, data compression & data aggregation. On the whole, thanks to the current state of the art, this method can provide real-time responses to heavy queries. Undoubtedly, the results will not appear at the same speed as index logging, but still, it seems likely that this experience is enough in most cases. Moreover, everything is searchable.

Conclusion

I’d say that depending on the use cases, the complexity of the data sets, and their sizes, the optimal approach could be one or the other. Evidently, index-free logging is less costly in terms of hardware. Not only it can ingest data faster with fewer resources but also unindexed tables can provide deeper analytics procedures. What I mean is that when you are investigating an issue, with unindexed tables you can drill down by any of the available dimensions.  So, if you are analyzing stored data, it seems like the best approach. Despite this, there may be some environments where the amount and complexity of data are so vast that the query response time may be too long. For these cases, the index logging will solve any performance issue. As a result of the indexing, you might not be able to drill down or follow different post-analysis troubleshooting. Nevertheless, if you already set the parameters that you want to analyze and correlate, index logging is the best method to get real-time results.

The Viewtinet approach

The vision of Viewtinet has always been to create a flexible platform suitable for several and different use cases. Having said that, why limiting to one logging method when you can have both? The Viewtinet style is to be easy to use and intuitive. Viewtinet has already integrations with most of the IT data sources in the market. The templates in the Visual Smart Data Broker have already taken into consideration the optimal logging method. So from the user experience, Viewtinet is already providing the most efficient configuration. With new or custom data sources, what Viewtinet does by default is index-free logging. This way, the system administrators can benefit from having all the data stored, to create any report, dashboard, and to drill down. After that, in case a report or several reports appear with some delay, with the visual interface, you can click the indexing option to switch from index-free logging to index-logging. This way, you may not know upfront how you are going to analyze the data. You will have all the possibilities. Then, once you design your dashboard you can optimize the performance by indexing specific reports if needed.