Project-2 : Automating S3 Bucket Copying with Terraform

Recently I needed to create an exact copy of an existing S3 bucket in my AWS account. I first did this manually using the AWS CLI, but wanted to automate it using Infrastructure as Code with Terraform.

Manually Copying the S3 Bucket

My first step was to copy the bucket manually using the AWS CLI. The bucket I wanted to copy was named my-existing-bucket. Here is the command I used:

aws s3 sync s3://my-existing-bucket s3://my-new-bucket

This recursively copies all objects from my-existing-bucket into a new bucket called my-new-bucket.

While this accomplished the goal, it was manual. I really wanted to define this copy process in Terraform to keep my infrastructure automated and codified.

Creating the Terraform Configuration

I created the following Terraform configuration to copy the S3 bucket:

provider "aws" {
  region = "us-east-1" 
}

resource "aws_s3_bucket" "new_bucket" {
  bucket = "my-new-bucket"
}

resource "null_resource" "copy_bucket" {
  provisioner "local-exec" {
    command = "aws s3 sync s3://my-existing-bucket s3://${aws_s3_bucket.new_bucket.id}"
  }
}

Here's what each section does:

• The aws provider is configured to use the us-east-1 region

• A new S3 bucket resource new_bucket is defined

• A null_resource executes the aws s3 sync command to copy the bucket

• The new bucket name is dynamically referenced to sync to

When applied, this Terraform configuration will create my-new-bucket and populate it by syncing all content from my-existing-bucket.

Copying to Another AWS Account

If I needed to copy the S3 bucket to another AWS account, I can simply add the --profile option to specify a different AWS CLI profile that has permissions on the destination account:

aws s3 sync --profile acct2 s3://my-existing-bucket s3://my-new-bucket

And in Terraform:

command = "aws s3 sync --profile acct2 s3://my-existing-bucket s3://${aws_s3_bucket.new_bucket.id}"

Automating infrastructure operations like copying S3 buckets with Terraform is a great way to keep your resources codified! Let me know if you have any other questions.