Moving Large Amounts of Data into Amazon S3

Moving data into Amazon S3 is not only a question of size. The best transfer method depends on whether the migration is a one-time event or an ongoing process, whether a reliable network is available, and how quickly the data must arrive.

This is Part 14 of the AWS Cloud Practitioner Study Notes series. Part 11 explains S3 fundamentals, Part 13 covers storage classes, and this note focuses on the path data takes before it becomes an S3 object.

The decision at a glance

Requirement	Recommended option	Why
Small or moderate upload over a healthy network	S3 console, CLI, or SDK	Simple API-based transfer
Large individual objects over a network	S3 multipart upload	Parts can upload in parallel and retry independently
Scheduled or recurring file synchronisation	AWS DataSync	Managed online transfer with verification and task scheduling
One-time migration with limited bandwidth	AWS Snowball Edge	Physical appliance avoids sending all data over the internet
Multi-petabyte or exabyte-scale migration	AWS Snowmobile, where available	Physical transport at a much larger scale
Ongoing private network path to AWS	AWS Direct Connect	Dedicated connectivity to AWS public services, including S3

The important distinction is between a transfer tool and a network connection. DataSync and Snowball Edge move or copy data. Direct Connect provides a network path; an application or service still has to transfer the data to S3 over that connection.

1. S3 uploads: console, CLI, SDK, and APIs

For a small dataset or a normal application upload, use Amazon S3 directly. The AWS Management Console is convenient for occasional manual work. The AWS CLI, SDKs, and S3 APIs are better for automation and repeatable processes.

Typical examples include:

• Uploading a few gigabytes of documents
• Sending website images and user-generated files
• Copying daily exports from an application
• Running a script that uploads objects to a bucket

For larger individual objects, use S3 multipart upload. It divides one object into parts that can be uploaded independently and in parallel. If one part fails, that part can be retried without restarting the whole object. AWS recommends multipart upload for objects around 100 MB or larger, and objects larger than 160 GB must be uploaded through the CLI, SDK, or S3 REST API rather than the console.

Multipart upload improves resilience and throughput, but it is not a replacement for a migration service. It is still an online transfer over the available network. Applications should also complete or abort multipart uploads properly, because unfinished parts remain stored until they are cleaned up.

2. AWS DataSync: recurring online transfers

AWS DataSync is a managed, high-speed file and object transfer service. It can move data between supported on-premises storage and AWS services such as Amazon S3, Amazon EFS, and Amazon FSx.

DataSync is a strong fit when the source remains active during migration or when data must continue synchronising after the first copy. Useful capabilities include:

• Scheduled and repeatable transfer tasks
• Incremental copying of changed data
• Data integrity verification using checksums
• Encryption in transit
• Transfers from file systems such as NFS, SMB, and HDFS

Imagine a company that keeps its file server running while moving data to S3. A DataSync task can perform an initial transfer, then run nightly to copy changes. This is different from a one-time appliance shipment: the network connection remains part of the migration design.

Choose DataSync when the question emphasises:

• Ongoing synchronisation
• Scheduled transfers
• A hybrid environment
• An active on-premises file store
• Online migration with validation

DataSync still needs network connectivity between the source, the DataSync service, and the destination. It improves the managed transfer workflow; it does not make an unavailable network unnecessary.

3. AWS Snowball Edge: large offline migrations

AWS Snowball Edge is a physical appliance that AWS ships to the customer. The customer copies local data onto the device, returns it, and AWS imports the data into the requested AWS destination, such as Amazon S3.

It is useful when a dataset is large but the internet connection is too slow, expensive, or unreliable for a practical online transfer. A 300 TB migration over a constrained link may take far longer than copying the same data over the local network and shipping a device.

The workflow is:

1. Create an import job in the AWS Snow Family Management Console.
2. AWS ships an encrypted Snowball Edge device to the source location.
3. Copy the data to the device using the supported tools or S3 adapter.
4. Return the device to AWS.
5. AWS transfers the data into S3.

Snowball Edge is usually the exam answer for a large, one-time migration where bandwidth is insufficient. It is not automatically the right choice just because the data is large. If the source must synchronise every night, DataSync is a better match. If the requirement is a permanent dedicated path, Direct Connect is the relevant service.

4. AWS Snowmobile: massive physical migration

AWS Snowmobile is designed for extremely large migrations, such as multi-petabyte media archives or data-centre decommissioning. It uses a secure, truck-transported container that acts as a network-attached data store at the customer site before the data is returned to AWS and loaded into S3.

Snowmobile is a specialised option rather than a normal self-service upload method. Availability and planning are region- and engagement-dependent, so an organisation should discuss the requirement with AWS. For CLF-C02-style questions, the key recognition pattern is:

• Multi-petabyte or exabyte-scale data
• A data-centre shutdown or massive media migration
• Physical transport is more practical than network transfer

For a migration measured in tens or hundreds of terabytes, Snowball Edge is generally the more relevant Snow Family concept. For a continuously changing dataset, an online service such as DataSync may still be required alongside the bulk migration.

5. AWS Direct Connect: dedicated connectivity

AWS Direct Connect links an organisation’s network to an AWS Direct Connect location. A public virtual interface can provide access to public AWS services such as Amazon S3. This gives the organisation a more predictable network path than relying entirely on the public internet.

Direct Connect is appropriate when the requirement is ongoing enterprise connectivity, not simply a single transfer. Examples include:

• Daily backups sent from a data centre to S3
• Continuous hybrid application traffic
• Predictable bandwidth for recurring data movement
• A private connection requirement for compliance or network design

Direct Connect does not itself copy files into S3. The organisation still uses S3 APIs, CLI commands, SDKs, DataSync, or another transfer process over the connection. It also requires planning and provisioning, so it is usually not the fastest answer for a one-time migration with a short deadline.

Online versus offline transfer

Use this mental model when choosing between the services:

Question	Online approach	Offline approach
Is a usable network available?	Yes	Limited, slow, or unavailable
Does the source keep changing?	DataSync or repeated S3 transfers	Bulk appliance followed by an online delta sync
What determines speed?	Network bandwidth and transfer tooling	Local copy speed plus shipping and AWS import time
Typical AWS choices	S3 CLI/SDK, multipart upload, DataSync, Direct Connect	Snowball Edge or Snowmobile

For a very large migration, a blended strategy can be sensible: use Snowball Edge for the initial bulk copy, then use DataSync or another online process to transfer changes made while the appliance was in transit.

Common CLF-C02 exam cues

• A few GB over the internet: S3 console, CLI, or SDK
• A large object that needs resilient network upload: S3 multipart upload
• Recurring or scheduled synchronisation: AWS DataSync
• Hundreds of TB and limited bandwidth: AWS Snowball Edge
• Multiple PB or exabyte-scale physical migration: AWS Snowmobile
• Dedicated ongoing connection to AWS: AWS Direct Connect

The words “large amount of data” alone are not enough. Look for the second clue: recurring synchronisation points to DataSync, a dedicated connection points to Direct Connect, and a physical shipment points to the Snow Family.

Conclusion

Start with direct S3 uploads when the dataset and network are manageable. Use multipart upload for large individual objects. Choose DataSync when the migration is online, repeatable, or incremental. Choose Snowball Edge when a one-time large transfer makes network upload impractical, and Snowmobile for specialised multi-petabyte or exabyte-scale engagements. Use Direct Connect when the real requirement is predictable, dedicated connectivity for ongoing traffic.

The strongest migration design often combines these options: physically move the bulk data, then synchronise the remaining changes over a managed online path.

Sources

• What is AWS DataSync?
• How AWS DataSync works
• Large data migration with Snowball Edge
• What is Snowball Edge?
• What is Direct Connect?
• Uploading and copying objects using multipart upload