Copy Data From DigitalOcean Spaces to AWS S3 Bucket

Introduction

DigitalOcean Spaces is a cloud-based object storage service that provides object storage through a web service interface and API. It allows you to store any amount of data at any time. In some cases, you will need to move or copy data from one storage provider to another. In this case, you can choose SnapShooter's S3 Sync service to copy your data from one S3 compatible storage to another.

SnapShooter offers an S3 Sync tool that helps you to copy or move data from DigitalOcean Spaces to AWS S3 and another storage provider. With SnapShooter, you can configure and schedule the S3 Sync service to sync data between multiple buckets and keep different buckets updated at a specific interval of time.

In this post, we will show you how to copy data from DigitalOcean Spaces to the AWS S3 bucket using SnapShooter.

Create an S3 Bucket in AWS

First, you will need to create an S3 bucket in AWS to store the data copying from an existing DigitalOcean Spaces. Follow the below steps to create a bucket on AWS S3:

Step 1 - Login to Amazon Management Console, search for S3 and open the S3 service as shown below.

Step 2 - Click on the Create bucket button. You should see the General configuration screen:

Step 3 - Type your bucket name, select your region, select object ownership, block all public access, and click on the Create bucket button. Once the bucket is created, you should see the following screen:

Create a New IAM User

After creating an S3 bucket, you must create a new IAM user and assign permission to access both S3 buckets. Follow the below steps to create an IAM user:

Step 1 - Log in to AWS Management Console, search for IAM from the top services menu, and click on Users in the left sidebar. You should see the IAM user screen:

Step 2 - Click on the Add users button to add a new IAM user. You should see the following screen:

Step 3 - Type your username, select "Access Type Programmatic access", then click on the Next: Permissions:

Step 4 - Leave all default options and click on the Next: Tags:

Step 5 - Leave all default options and click on the Next: Review:

Step 6 - Review your configuration and click on the Create user button. You should see the AWS access key ID and secret access key on the following screen:

Step 7 - Note down the Access key ID and Secret access key in a safe location. You will need both keys to attach the AWS S3 to SnapShooter. Now, click on the Close button. You should see the following screen:

Step 8 - Click on your newly created user. You should see the following screen:

Step 9 - Next, you will need to create a policy for the S3 bucket. Click on Add inline policy. You should see the Create Policy screen:

Step 10 - Choose JSON Editor and add the following contents to grant access permission on the S3 bucket:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:s3:::aws-destination-bucket-snapshooter/*",
"arn:aws:s3:::aws-destination-bucket-snapshooter"
]
}
]
}

Step 11 - Click on the Review policy button. You should see the following screen:

Step 12 - Provide your policy name and click on the Create policy button to save the policy.

Add a Server to Snapshooter

You will also need to add a server to the SnapShooter. The S3 Sync service uses a server to copy data from DigitalOcean Spaces to the AWS S3 bucket.

Follow the below steps to add a server to Snapshooter:

Step 1 - First, go to the SnapShooter website and register for their 14-day free trial.

Step 2 - On the SnapShooter dashboard, click on the Servers in the left sidebar. You should see the following screen:

Step 3 - Click on the Connect Server. You should see the following screen:

Step 4 - Copy and paste the code shown in the above image to your server:

curl -sSL "https://ingestor.snapshooter.com/api/server/add?token=eyJpdiI6Imp6Y3diYi91eXIxalk1czRrQlBHSkE9PSIsInZhbHVlIjoieEFhY0RnMWlHOUdubTgwbWQxOGVzUT09IiwibWFjIjoiMDBhODY3MTkwZWJhYzFiMTJjNzdlYzczNmEyNTJmYjk2NzQ5NzBmZmUwODgxOGI5MWJjNmRlYmM2OTQyYTlhNiIsInRhZyI6IiJ9&key=5895" | bash

Once your server is connected to the Snapshooter, you will get the following output:

Welcome to SnapShooter Server Setup
Environment Detected: OS: linux, OS Type: amd64
Installing SSH public key
Creating temp SSH key file
grep: /root/.ssh/authorized_keys: No such file or directory
cp: cannot stat '/root/.ssh/authorized_keys': No such file or directory
Creating authorized_keys backup /root/.ssh/authorized_keys.bak-1653474994
Key Installed /root/.ssh/authorized_keys
Removing temp SSH key file
Scanning SSH config
Checking 22
matched (22)
SSH test confirmed, Server Added to SnapShooter ...

Step 5 - Go back to your web browser, you should see your added server on the following screen:

Attache Both DigitalOcean Space and WS S3 Bucket to Snapshooter

Next, you will need to attach DigitalOcean Spaces and AWS S3 bucket to the SnapShooter.

Click here to attach the DigitalOcean S3 Spaces to the SnapShooter You should see the following screen:

Again, click here to attach the AWS S3 bucket to the SnapShooter You should see the following screen:

Choose a Friendly Name, Select AWS S3, select region, type your access key and secret access key, type your AWS S3 bucket name, and press the Attach External Storage button.

Create a Backup Job to Copy the Content of DigitalOcean Spaces to AWS S3 Bucket

Now, you will need to create a backup job to copy data from DigitalOcean Spaces to the AWS S3 bucket at specific intervals of time. Follow the below steps to create a backup job:

Step 1 - On the SnapShooter dashboard, click on the Backup Jobs in the left pane. You should see the following screen:

Step 2 - Click on the Create New Jobs. You should see the following screen:

Step 3 - Click on the S3 Sync, you should see the following screen:

Step 4 - Provide your backup job name and click on the Continue button. You should see the following screen:

Step 5 - Select your server and click on the Next button. You should see the following screen:

Step 6 - Define the directory path in the destination bucket, select your source DigitalOcean Space and click on the Test button. You should see the following screen:

Step 7 - Click on the Save and Next button. You should see the following screen:

Step 8 - Select your destination AWS S3 bucket and click on the Set Storage button. You should see the following screen:

Step 9 - Define your backup schedule and click on the Set Schedule and Finish button. You should see the following screen:

Step 10 - Click on the Backup Now button to start copying data from DigitalOcean Spaces to the AWS S3 bucket. Once the backup process is completed successfully, you should see the following screen:

You can also verify your backup by accessing the AWS S3 bucket. You should see your all data on the following screen:

Conclusion

In this guide, you learned how to copy data from DigitalOcean Spaces to the AWS S3 bucket using SnapShooter. SnapShooter offers a simple and user-friendly web interface to manage all backups and restore them easily in the event of data loss.

SnapShooter support many S3 providers on the marketplace, including AWS S3, DigitalOcean Spaces, Backblaze, Upcloud, Scaleway, Minio, and custom endpoints. If you have object storage missing let us know we are happy to add any. We believe everyone should keep a copy of their data offsite and secure.