Using AWS S3 and Glacier as a Backup Solution

vteam #367 was hired to work on the client’s application which is a leading Learning Management System (LMS) for corporate training, higher education, and K-12 school districts. This LMS emphasizes a personal learning approach based on collaborative and interactive technology to develop life-long learners.

The client wanted to provide his customers with a feature for file archive (move file from S3 to Glacier) functionality. As the client’s application had hundreds of users, it was impossible to keep backups of trash files on the main server. So, it was decided to use AWS Glacier services along with AWS S3 to resolve the archive issue.

The Challenge

As the number of users continued to increase on the site, it was becoming very difficult to manage and protect the archive of deleted files.

Since you cannot use the Amazon S3 API to move specific files into Glacier, the client wanted to:

1. Avoid first downloading files from S3 server using S3 API call
2. Then make another Glacier API call to upload file on Glacier for the file deletion process (because it was time consuming)
   .

So, client asked vteam #367 to find a way to achieve both functionalities using a single API call. It was a challenge to make a single API call to move deleted files from S3 into Glacier.

The Solution

After a lot of research, vteams engineer found a way to handle this issue; the solution was to move deleted files into an S3 bucket named “trash”, after setting a trash flag on the database, whenever the user deletes a file on the site. The following code was used to move files:

	$this -> s3Handle -> copyObject(array(
		'Bucket'		=> $this->bucket.'-delete',
		'Key'			=> $cdnName,
		'CopySource'	=> $this -> bucket."/".$cdnName,
	));

In the administration panel of the AWS management tool, vteams engineer defined life-cycle rules that archive files from Amazon S3 bucket to Amazon Glacier (screen shot given below):

S3 will evaluate the life cycle policies for each of the buckets and will appropriately archive objects in Glacier. After the object has been successfully archived using the Glacier storage option, the object’s data will be removed from S3 but its index entry will remain as-is. The S3 storage class of an object, that has been archived in Glacier, will be set to GLACIER. Thus, this solution saved users’ wait time by avoiding multiple S3 and Glacier API calls.

Conclusion

If you’re searching for a cheap and reliable storage solution on AWS Glacier, this project is an evidence about how intelligently the files could be moved from S3 into Glacier without effecting users. It is a secure, efficient, reliable and by far the cheapest solution available for backing up data using AWS S3 and Glacier.