Recommender System using AWS Amazon Redshift

Requirement

We needed system to generate recommendations at run time for related resource items based on the item being viewed. The recommendation system needed to consider various inputs like user preferences, priority/sponsored items, relevance, ranking etc. The legacy system based on local MySql database was facing performance issues as the data footprint was expanding with time. Therefore it was proposed to use third party cloud based data warehouse that was scalable.

Amazon Redshift

Amazon Redshift is a third party cloud based data warehouse. It can scale from few hundred gigabytes to more than petabyte. This enables faster query performance for data analysis applications. These features along with the familiarity with the other AWS products already being used made Amazon Redshift preferred choice for recommendation system.

Solution Implemented

There were two aspects to Amazon Redshift integration, data synchronization with Redshift and recommendation generation for word press application.

Data Synchronization

In order to grantee that the generated recommendations are update to date, the changes to local data needed to be kept in sync with Redshift. In order to achieve this, a separate word press plugin was developed. It contained the classes that encapsulated the business logic for core, entities, factories, models, repositories and services. Any CRUD action on local data was synced with Redshift using the above mentioned interfaces.

Recommendation Generation

The word press plugin was extended to include the business logic to perform data analysis queries. This had to incorporate inputs like user preferences, priority/sponsored items, relevance, ranking etc. Also separate PostgreSQL driver connection was setup for Redshift and integrated throughout the site. Resource recommendation was generated based on associated subject areas, education levels, file downloads, files, views and comments.

Results

Robust, responsive and reliable recommendation system was delivered. It successfully overcome the performance and scalability issues faced by legacy system. Users were served with targeted recommendation based on preferences, priority/sponsored items, relevance, ranking etc. This increased usability of the application for user and increased their satisfaction level, encouraging more active users on the site.