(858) 586 7777 | About | Testimonials | Contact
vteams vteams vteams vteams
  • How does it work?
    • Startup Process
    • Your First Day
  • Technologies
    • Hire PHP Developer
    • Hire App Developer
    • Hire JavaScript Developer
    • Hire ROR Developer
    • Hire IOS Developer
    • Hire .NET Developer
    • Hire AI Developer
    • Hire Robotics Engineer
  • Sample Budgets
  • Meet The Team
  • Experiments
  • Captain’s Log
  • Blog
vteams vteams
  • How does it work?
    • Startup Process
    • Your First Day
  • Technologies
    • Hire PHP Developer
    • Hire App Developer
    • Hire JavaScript Developer
    • Hire ROR Developer
    • Hire IOS Developer
    • Hire .NET Developer
    • Hire AI Developer
    • Hire Robotics Engineer
  • Sample Budgets
  • Meet The Team
  • Experiments
  • Captain’s Log
  • Blog
Blog
  1. vteams
  2. Blog
  3. How To Sync 24TB Of Data In No Time?
Sep 29

How To Sync 24TB Of Data In No Time?

  • September 29, 2014

Symphonic---24TB-of-DataBitTorrent allows for much more reliable DNS Failover capacity

Our client operates a web portal linking to a large and growing media database mostly in the form of mp3 files—24 Terabytes and counting. Wanting to enhance their back-end performance and create much needed redundancy, they created vteam #377 to design and implement a system that:

  1. Keeps stored files readily available to the application.
  2. Replicates the entire contents of their data center.
  3. Supports DNS Failover, in the event of one server cluster crashing, another one can take its place immediately.

The solutions included:

Network Attached Storage (NAS) Sync: The network operations center (NOC) decided on a multi-drive Raid-60 setup for file storage.

Second Server Cluster: Acquired by the client and the NOC helped us get an exact replica of the hardware setup working and on-line.

Replication: rsync is the standard way of syncing the data, but 24TB using rsync would take 24 days. Unsurprisingly, no one was willing go that route. Instead, we followed the best practices of the Fortune 500 and used BitTorrent to sync both NAS locations. It simultaneously syncs data while verifying the integrity of the files and it can do so in one or both directions.

DNS Failover: Client purchased and deployed the hardware.

MySQL database replication: Replication was done in real-time between the two clusters. This diagram illustrates the process:

DNS_Failover

  • Facebook
  • Twitter
  • Tumblr
  • Pinterest
  • Google+
  • LinkedIn
  • E-Mail

Comments are closed.

SEARCH BLOG

Categories

  • Blog (490)
  • Captain's Log (1)
  • Closure Reports (45)
  • Experiments (7)
  • How-To (56)
  • Implementation Notes (148)
  • Learn More (156)
  • LMS (8)
  • Look Inside (10)
  • Operations Log (12)
  • Programmer Notes (20)
  • R&D (14)
  • Rescue Log (4)
  • Testimonials (25)
  • Uncategorized (4)

RECENT STORIES

  • GitHub Actions- Automate your software workflows with excellence
  • Yii Framework – Accomplish Repetitive & Iterative Projects with Ease
  • A Recipe for CRM Software Development
  • Are Agile and DevOps the same?
  • The Data Scientist’s Toolset

ARCHIVES

In Short

With the vteams model, you bypass the middleman and hire your own offshore engineers - they work exclusively for you. You pay a reasonable monthly wage and get the job done without hassles, re-negotiations, feature counts or budget overruns.

Goals for 2020

  • Open development center in Australia
  • Complete and Launch the Robot
  • Structural changes to better address Clients' needs

Contact Us

Address: NEXTWERK INC.
6790 Embarcadero Ln, Ste 100,
Carlsbad, CA 92011, USA

Tel: (858) 586 7777
Email: fahad@nextwerk.com
Web: www.vteams.com

© 2020 vteams. All Rights Reserved.

Content Protection by DMCA.com