Client asked us to develop a scoring system for businesses in already running application. System must be intelligent, reliable and fast. Must be categorized into different sections and new sections can be added in future.
Application holds millions of businesses records. Traditional SQL base query take too much time to search. First customer has to search for their desired business and then he/she can run quality control engine to view business scores.
After R & D we suggested client to use amazon elastic search service. Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It utilizes a JSON based REST API to refer to Lucene features.
So for the first step we migrated all of our businesses data to elastic search server. After implementation of elastic search on quality control engine, business search became very effective, robust and fast.
Before scoring we had to gather new data from our data partners. Then we analyzed them, preprocess them and afterwards saved to our data warehouse for future use. Each search involves lots of requests so it not a very cost effective solution. Alternatively, we set a data refresh period. After reaching every refresh time frame system automatically updates searched businesses in our database. We used repository design pattern for decoupling database operations and business logic.
Next, scoring was also a complicated process, so we had to analyze thousands of data points and then compile reports that were easy to understand by customers. We categorized each report into different sections and sub sections for better understanding and future enhancements.
After doing above process scoring system became very reliable and fast. Customers can see their desired business scores with in no time.