Software Usage Intelligence
Our client provides a cloud-based platform that helps leading software vendors gain insight into how their solutions are being used. Specifically, the platform enables detecting which features are used most and least, visualise user paths, and see what challenges they face. This data allows software companies to know what really matters to end users in order to supply a more valuable product.
The client was maintaining an existing intelligence solution basically aimed at desktop applications. The client’s idea was to create a brand-new on-premises solution for both web and desktop applications using a newer technology stack. The buyers of the solution were expected to be able to integrate it into their products to obtain information on how users interact with them, what features they use and how exactly this process takes place. The expected system needed to be easy to install, maintain and update. Also, the system had to be flexible and allow for various metrics from client applications to be gathered and used afterwards when building reports.
Client SDK Components
Storage has been divided into relational and nonrelational databases. The relational database is used in the regular way to store information such as user credentials, preferences, report structures and their settings and filters. PostgreSQL was the simple choice to cover everything we needed on this front.
Choosing which nonrelational data engine to use was not easy. After an investigation, we agreed to use the time series database InfluxDB to store incoming event data. The schemaless approach seemed very attractive, as the client application could send undetermined data, and we were required to store it all. We implemented a proof-of-concept, and it worked well on smaller volumes of data. However, once we created performance tests, it became clear that InfluxDB could not handle our workload profile.
After some further investigation, we settled on ClickHouse. It is a column-oriented database, but it uses a strict schema, unlike InfluxDB. Therefore, we created a bunch of general-purpose columns that clients must bind to the metrics they gather. ClickHouse has demonstrated high performance under the required workload profile.
Two dedicated server-side components were created. One of them accepts, validates and normalises incoming event data before finally pushing it to ClickHouse via an intermediate buffer. Another component is responsible for the authentication/authorisation of users, handling incoming requests for reports, building the reports and sending them to the UI component. Both server components share ClickHouse and PostgreSQL as data access levels.
The UI has been built as a single-page application with a React library. It is the starting point for end users to observe and manipulate the gathered event data. The user interface allows users to go through predefined reports, apply various filters to the overall data, copy reports and rename them to keep an eye on them later. The D3.js library has been utilised to display data with bar, stack, line, pie and other types of charts, including Sankey diagrams, to show users’ paths in the application.
The diagram of the system’s final components looked like this:
We have created a heterogeneous system to be used on-premises by the customers. It consists of several client- and server-side components to support the entire data flow. The system supports versioning and is delivered using Docker. The scalability level was confirmed using performance and stress testing. Nevertheless, there remain ways to improve the scalability using ClickHouse’s clustering features in the future.
Related CasesRead all
UX Audit for a Country Risk Platform
Improving UX for an online platform, bringing new traffic and subscribers from the main product.
Vehicle Traffic Control System
The system intended to control and manage the movement of vehicles between parking areas and show their current position.
Price Reporting and Analytics System
Development of a solution that represents pricing and news data in a convenient and flexible way.