• Miscellaneous
  • < 1

How To Sync 24 TB Of Data In No Time?

  • POSTED ON
  • September 29, 2014
  • POSTED BY
  • Muhammad Ahmad
  • POSTED ON September 29, 2014
  • POSTED BY Muhammad Ahmad

BitTorrent allows for much more reliable DNS Failover capacity Our client operates a web portal linking to a large and growing media database mostly in the form of mp3 files—24 Terabytes and counting. Wanting to enhance their back-end performance and create much-needed redundancy, they created vteam #377 to design and implement a system that: Keeps

BitTorrent allows for much more reliable DNS Failover capacity

Our client operates a web portal linking to a large and growing media database mostly in the form of mp3 files—24 Terabytes and counting. Wanting to enhance their back-end performance and create much-needed redundancy, they created vteam #377 to design and implement a system that:

  1. Keeps stored files readily available to the application.
  2. Replicates the entire contents of their data center.
  3. Supports DNS Failover, in the event of one server cluster crashing, another one can take its place immediately.

The solutions included:

Network Attached Storage (NAS) Sync: The network operations center (NOC) decided on a multi-drive Raid-60 setup for file storage.

Second Server Cluster: Acquired by the client and the NOC helped us get an exact replica of the hardware setup working and online.

Replication: Rsync is the standard way of syncing the data, but 24TB using Rsync would take 24 days. Unsurprisingly, no one was willing to go that route. Instead, we followed the best practices of the Fortune 500 and used BitTorrent to sync both NAS locations. It simultaneously syncs data while verifying the integrity of the files and it can do so in one or both directions.

DNS Failover: The client purchased and deployed the hardware.

MySQL database replication: Replication was done in real-time between the two clusters. This diagram illustrates the process:

DNS_Failover

ABOUT THE AUTHOR

Muhammad Ahmad

0 Comments

Leave a Reply

More Related Article
We provide tips and advice on delivering excellent customer service, engaging your customers, and building a customer-centric business.