A paid utility to find rankings of websites was required by the client of vteams #409. The purpose of the required utility was to designate websites on Google as well as on Yahoo and Bing.
The major challenge faced by vteams engineer Tariq Mahmood was that search engines doesn’t provide any APIs to find ranking of websites against keywords.
The utility was developed using Yii 1, jqPlot, phpexcel, Bootstrap and Yii email extension. A crawler was written by Tariq to get the required results on first 5 pages from search engine against a keyword. After that, the HTML returned by search engines was parsed to find the ranking of a required domain.
He was using CURL to get search results. Even so, search engine detected some bots sending those search requests showing an intermediate CAPTCHA screen instead of search results. He did some R&D and then implemented the following solutions to fake as an actual browser:
- Faked a different browser/OS for each CURL request
- Added a random interval of 1-10 seconds between two CURL requests
- Sent cookies stored by Google on system in CURL request to ensure that bot is never detected
An interface to add/edit/delete domain in the utility was provided in addition to multiple keywords for different domains. Admin could find rankings by a domain (all keywords for that particular domain) or a specific keyword for a domain. Moreover, if they wanted to search for any keyword that was not stored in the database then they could perform quick ranking search.
The history of rankings was maintained and charts were used to show history trends of how site ranking is increasing or decreasing. A functionality was added to group keywords in categories for a domain so that rankings could be found by category. The utility also had a feature to download ranking reports in excel format as per the selected criteria. A comparison feature was added to compare rankings of similar business domains for the same keyword.