Evolution of the search algorithm for a proprietary insights platform 

The client came to DigitalMara with the need to scale their data engineering team to accelerate development. They already had a simple ETL (extract, transform, load) platform with some front-end parts and wanted to extend its capabilities, to improve search accuracy and speed. As one of the main processes in data warehouse management, ETL must be able to efficiently handle raw data from external resources, extract the relevant data then clean and transform it into the form needed for subsequent use.

Data needed for analysis is collected from various sources. The main problem with this type of information is the wide variety in format, syntax, and content. For example, work responsibilities and positions of candidates can be described in numerous ways: a similar role may be titled chief marketing officer or director of marketing, trend analysis or market research. For proper matchmaking, it is necessary to consider these discrepancies. In addition, fragmented data needs to be classified, organized, and correctly interpreted for visualization to users.

DigitalMara has been involved in improving the search and data processing components within the DRAX platform, particularly in the improvement of the Search API (Application Programming Interface). This API is connected to the ETL and makes it possible to conduct accurate and detailed full-text search in large data volumes and, even more importantly, to create indexes that lead to more convenient and faster search results for the system’s users. If the data received is incomplete, incorrect, or distorted, additional processing is performed to clean up the data and bring it into its proper form, while unverifiable data is deleted.