You are currently viewing Implementing Caching and Pagination to a Web Based Application
implementing-caching-and-pagination-to-a-web-based-application

Implementing Caching and Pagination to a Web Based Application

  • Post published:January 27, 2017

vteam #679 was required to implement caching and pagination of a UI, within tabs and across tabs, to decrease latency in the client’s UX as per given requirements. The client’s project was a Django based web application, able to sift through the vast amounts of information available to users through social media, news and blogs to identify trends and patterns. vteams engineer was expected to apply pagination on different pages of the web application and also implement caching on different pages (Content Data, Ego Frequency) to decrease latency in the UX. To ensure the security of data/analytics – wrap and control access to API URLs would be integrated into the admin panel.

Caching

In existing client’s application, data from social media was gathered and analyzed to display top results for hashtags, word pairs and most frequented domains.

Previously, for similar requests, the same data was reciprocated as displayed in the first instance the request was made. Making the data inaccurate at the time of each new request being made as it was not displaying real-time results.

This problem was solved by caching the analyzed data. Because for each repeated request, data analysis took too much time and the results were inaccurate. To apply caching, redis-cloud add-on was used on heroku. Some default settings for cache are given below, add them in settings.py file:

CACHES = {
        'default': {
            'BACKEND': 'redis_cache.RedisCache',
            'LOCATION': '%s:%s' % ( 'redis_url', 'port_#'),
            'OPTIONS': {
                'PASSWORD': 'password',
                'DB': 0,
            }}}

Next step is to apply caching using .py file:

url_key = 'name'
data = cache.get(url_key + 'data')
if not data:
    # do analysis on data
    # get results
    # return data
    cache.set(url_key + 'data', data, 900)
else:
    results = data

The .py file code, will first look into the cache for required results. If the results are found in cache then there is no wastage of time on analysis of data. If data is not found in cache then the data will be analyzed following which it will be cached against each different request.

Pagination

Django provides a few classes that help manage paginated data or data that’s split across several pages, with “Previous/Nextlinks. These classes can be found in django/core/paginator.py.

vteams engineer was required to implement pagination on the existing project’s code and used a very generic django approach. The page limit was set to 20 results as per requirements.

The code for implementation of pagination is below:

from django.core.paginator import Paginator
limit_per_page = 20
objects = [ data_in_list ]
p = Paginator(objects, limit_per_page)
page1 = p.page(1)
# count number of objects in paginator
p.count

p.page_range” was used to calculate the total number of pages. Similarly, different methods of Paginator were used according to the given requirements.

Security & Authorization of API calls

To secure API endpoints that engineer used for the Client Area, the following steps were followed:

  1. The application server will send a request to the API server, with client ID, in encryption form
  2. The API server will provide a token against that user in response to the application server
  3. The application server will provide the token to client side (JS application)
  4. The JS application will always send a request to the API server with that token
  5. The API server will respond to the request only if that token is valid and registered to the same user for which the data query is being made. Otherwise it will return with ‘401 Authorization required
  6. The API server will also renew tokens and return them with the same response
  7. The next time the JS application will use new token
  8. The same steps from 4 to 7 will keep repeating
    .

The given steps can be implemented as follows:

In models.py file, some necessary fields for authorization of API were defined and different properties on the concerned fields were applied as per project requirements:

class AuthToken(models.Model):
 auth_token = models.CharField(max_length=1000, null=False)
 id = models.IntegerField(null=False, blank=False, db_index=True)
 date_created = models.DateTimeField(default=timezone.now, null = True)

In views.py file, methods for verification and generation of token on different stages were written:

# generation of tokens
def api_view(func):
 def func_wrapper(request, *args, **kwargs):
  # get token and id from GET request
  # verified token and get user against that token
  # generated token and saved it in db
  # returned response according to verification and generation of tokens
# securing API
def get_client_token(request,id=None):
 # creation of secured token for Site User object which represents a Client
 # generation of key
 # encryption/decryption of key
 # generation and verification of token
 # return response after verification

Conclusion

The given solution resulted in 15 min (900 sec) decrease time saving results in the cache and an overall significant decrease in latency of the UX.