com.amazonaws.services.s3.transfer
Class TransferManager

java.lang.Object
  com.amazonaws.services.s3.transfer.TransferManager

public class TransferManager
extends Object

High level utility for managing transfers to Amazon S3.

TransferManager provides a simple API for uploading content to Amazon S3, and makes extensive use of Amazon S3 multipart uploads to achieve enhanced throughput, performance and reliability.

When possible, TransferManager attempts to use multiple threads to upload multiple parts of a single upload at once. When dealing with large content sizes and high bandwidth, this can have a significant increase on throughput.

TransferManager is responsible for managing resources such as connections and threads; share a single instance of TransferManager whenever possible. TransferManager, like all the client classes in the AWS SDK for Java, is thread safe.

Using TransferManager to upload options to Amazon S3 is easy:

 AWSCredentials myCredentials = new BasicAWSCredentials(...);
 TransferManager tx = new TransferManager(myCredentials);
 Upload myUpload = tx.upload(myBucket, myFile.getName(), myFile);

 while (myUpload.isDone() == false) {
     System.out.println("Transfer: " + myUpload.getDescription());
     System.out.println("  - State: " + myUpload.getState());
     System.out.println("  - Progress: " + myUpload.getProgress().getBytesTransfered());
     // Do work while we wait for our upload to complete...
     Thread.sleep(500);
 }
 

Note: Transfers are stored in memory. If the JVM is restarted, previous transfers are no longer accessible. If needed, clean up any multipart uploads that are incomplete.

Constructor Summary

TransferManager(AmazonS3 s3)
Constructs a new TransferManager, specifying the client to use when making requests to Amazon S3.

TransferManager(AmazonS3 s3, ThreadPoolExecutor threadPool)
Constructs a new TransferManager specifying the client and thread pool to use when making requests to Amazon S3.

TransferManager(AWSCredentials credentials)
Constructs a new TransferManager and Amazon S3 client using the specified AWS security credentials.

Method Summary

void	abortMultipartUploads(String bucketName, Date date) Aborts any multipart uploads that were initiated before the specified date.
<X extends AmazonWebServiceRequest> X	appendUserAgent(X request, String userAgent)
Download	download(GetObjectRequest getObjectRequest, File file) Schedules a new transfer to download data from Amazon S3 and save it to the specified file.
Download	download(String bucket, String key, File file) Schedules a new transfer to download data from Amazon S3 and save it to the specified file.
MultipleFileDownload	downloadDirectory(String bucketName, String keyPrefix, File destinationDirectory) Downloads all objects in the virtual directory designated by the keyPrefix given to the destination directory given.
AmazonS3	getAmazonS3Client() Returns the underlying Amazon S3 client used to make requests to Amazon S3.
TransferManagerConfiguration	getConfiguration() Returns the configuration which specifies how this TransferManager processes requests.
void	setConfiguration(TransferManagerConfiguration configuration) Sets the configuration which specifies how this TransferManager processes requests.
void	shutdownNow() Forcefully shuts down this TransferManager instance - currently executing transfers will not be allowed to finish.
Upload	upload(PutObjectRequest putObjectRequest) Schedules a new transfer to upload data to Amazon S3.
Upload	upload(String bucketName, String key, File file) Schedules a new transfer to upload data to Amazon S3.
Upload	upload(String bucketName, String key, InputStream input, ObjectMetadata objectMetadata) Schedules a new transfer to upload data to Amazon S3.
MultipleFileUpload	uploadDirectory(String bucketName, String virtualDirectoryKeyPrefix, File directory, boolean includeSubdirectories) Uploads all files in the directory given to the bucket named, optionally recursing for all subdirectories.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TransferManager

public TransferManager(AWSCredentials credentials)

Constructs a new TransferManager and Amazon S3 client using the specified AWS security credentials.

TransferManager and client objects may pool connections and threads. Reuse TransferManager and client objects and share them throughout applications.

TransferManager and all AWS client objects are thread safe.

Parameters:

credentials - The AWS security credentials to use when making authenticated requests.

TransferManager

public TransferManager(AmazonS3 s3)

Constructs a new TransferManager, specifying the client to use when making requests to Amazon S3.

TransferManager and client objects may pool connections and threads. Reuse TransferManager and client objects and share them throughout applications.

TransferManager and all AWS client objects are thread safe.

Parameters:

s3 - The client to use when making requests to Amazon S3.

TransferManager

public TransferManager(AmazonS3 s3,
                       ThreadPoolExecutor threadPool)

Constructs a new TransferManager specifying the client and thread pool to use when making requests to Amazon S3.

TransferManager and client objects may pool connections and threads. Reuse TransferManager and client objects and share them throughout applications.

TransferManager and all AWS client objects are thread safe.

Parameters:

s3 - The client to use when making requests to Amazon S3.

threadPool - The thread pool in which to execute requests.

Method Detail

setConfiguration

public void setConfiguration(TransferManagerConfiguration configuration)

Sets the configuration which specifies how this TransferManager processes requests.

Parameters:

configuration - The new configuration specifying how this TransferManager processes requests.

getConfiguration

public TransferManagerConfiguration getConfiguration()

Returns the configuration which specifies how this TransferManager processes requests.

Returns:

The configuration settings for this TransferManager.

getAmazonS3Client

public AmazonS3 getAmazonS3Client()

Returns the underlying Amazon S3 client used to make requests to Amazon S3.

Returns:

The underlying Amazon S3 client used to make requests to Amazon S3.

upload

public Upload upload(String bucketName,
                     String key,
                     InputStream input,
                     ObjectMetadata objectMetadata)
              throws AmazonServiceException,
                     AmazonClientException

Schedules a new transfer to upload data to Amazon S3. This method is non-blocking and returns immediately (i.e. before the upload has finished).

When uploading options from a stream, callers must supply the size of options in the stream through the content length field in the ObjectMetadata parameter. If no content length is specified for the input stream, then TransferManager will attempt to buffer all the stream contents in memory and upload the options as a traditional, single part upload. Because the entire stream contents must be buffered in memory, this can be very expensive, and should be avoided whenever possible.

Use the returned Upload object to query the progress of the transfer, add listeners for progress events, and wait for the upload to complete.

If resources are available, the upload will begin immediately. Otherwise, the upload is scheduled and started as soon as resources become available.

Parameters:

bucketName - The name of the bucket to upload the new object to.

key - The key in the specified bucket by which to store the new object.

input - The input stream containing the options to upload to Amazon S3.

objectMetadata - Additional information about the object being uploaded, including the size of the options, content type, additional custom user metadata, etc.

Returns:

A new Upload object to use to check the state of the upload, listen for progress notifications, and otherwise manage the upload.

Throws:

AmazonClientException - If any errors are encountered in the client while making the request or handling the response.

AmazonServiceException - If any errors occurred in Amazon S3 while processing the request.

upload

public Upload upload(String bucketName,
                     String key,
                     File file)
              throws AmazonServiceException,
                     AmazonClientException

Schedules a new transfer to upload data to Amazon S3. This method is non-blocking and returns immediately (i.e. before the upload has finished).

The returned Upload object allows you to query the progress of the transfer, add listeners for progress events, and wait for the upload to complete.

If resources are available, the upload will begin immediately, otherwise it will be scheduled and started as soon as resources become available.

Parameters:

bucketName - The name of the bucket to upload the new object to.

key - The key in the specified bucket by which to store the new object.

file - The file to upload.

Returns:

A new Upload object which can be used to check state of the upload, listen for progress notifications, and otherwise manage the upload.

Throws:

AmazonClientException - If any errors are encountered in the client while making the request or handling the response.

AmazonServiceException - If any errors occurred in Amazon S3 while processing the request.

upload

public Upload upload(PutObjectRequest putObjectRequest)
              throws AmazonServiceException,
                     AmazonClientException

Schedules a new transfer to upload data to Amazon S3. This method is non-blocking and returns immediately (i.e. before the upload has finished).

Use the returned Upload object to query the progress of the transfer, add listeners for progress events, and wait for the upload to complete.

If resources are available, the upload will begin immediately. Otherwise, the upload is scheduled and started as soon as resources become available.

Parameters:

putObjectRequest - The request containing all the parameters for the upload.

Returns:

A new Upload object to use to check the state of the upload, listen for progress notifications, and otherwise manage the upload.

Throws:

AmazonClientException - If any errors are encountered in the client while making the request or handling the response.

AmazonServiceException - If any errors occurred in Amazon S3 while processing the request.

download

public Download download(String bucket,
                         String key,
                         File file)

Schedules a new transfer to download data from Amazon S3 and save it to the specified file. This method is non-blocking and returns immediately (i.e. before the data has been fully downloaded).

Use the returned Download object to query the progress of the transfer, add listeners for progress events, and wait for the download to complete.

Parameters:

bucket - The name of the bucket containing the object to download.

key - The key under which the object to download is stored.

file - The file to download the object's data to.

Returns:

A new Download object to use to check the state of the download, listen for progress notifications, and otherwise manage the download.

Throws:

AmazonClientException - If any errors are encountered in the client while making the request or handling the response.

AmazonServiceException - If any errors occurred in Amazon S3 while processing the request.

download

public Download download(GetObjectRequest getObjectRequest,
                         File file)

Schedules a new transfer to download data from Amazon S3 and save it to the specified file. This method is non-blocking and returns immediately (i.e. before the data has been fully downloaded).

Use the returned Download object to query the progress of the transfer, add listeners for progress events, and wait for the download to complete.

Parameters:

getObjectRequest - The request containing all the parameters for the download.

file - The file to download the object data to.

Returns:

A new Download object to use to check the state of the download, listen for progress notifications, and otherwise manage the download.

Throws:

AmazonClientException - If any errors are encountered in the client while making the request or handling the response.

AmazonServiceException - If any errors occurred in Amazon S3 while processing the request.

downloadDirectory

public MultipleFileDownload downloadDirectory(String bucketName,
                                              String keyPrefix,
                                              File destinationDirectory)

Downloads all objects in the virtual directory designated by the keyPrefix given to the destination directory given. All virtual subdirectories will be downloaded recursively.

Parameters:

bucketName - The bucket containing the virtual directory

keyPrefix - The key prefix for the virtual directory, or null for the entire bucket. All subdirectories will be downloaded recursively.

destinationDirectory - The directory to place downloaded files. Subdirectories will be created as necessary.

uploadDirectory

public MultipleFileUpload uploadDirectory(String bucketName,
                                          String virtualDirectoryKeyPrefix,
                                          File directory,
                                          boolean includeSubdirectories)

Uploads all files in the directory given to the bucket named, optionally recursing for all subdirectories.

S3 will overwrite any existing objects that happen to have the same key, just as when uploading individual files, so use with caution.

Parameters:

bucketName - The name of the bucket to upload objects to.

virtualDirectoryKeyPrefix - The key prefix of the virtual directory to upload to. Use the null or empty string to upload files to the root of the bucket.

directory - The directory to upload.

includeSubdirectories - Whether to include subdirectories in the upload. If true, files found in subdirectories will be included with an appropriate concatenation to the key prefix.

abortMultipartUploads

public void abortMultipartUploads(String bucketName,
                                  Date date)
                           throws AmazonServiceException,
                                  AmazonClientException

Aborts any multipart uploads that were initiated before the specified date.

This method is useful for cleaning up any interrupted multipart uploads. TransferManager attempts to abort any failed uploads, but in some cases this may not be possible, such as if network connectivity is completely lost.

Parameters:

bucketName - The name of the bucket containing the multipart uploads to abort.

date - The date indicating which multipart uploads should be aborted.

Throws:

AmazonServiceException

AmazonClientException

shutdownNow

public void shutdownNow()

Forcefully shuts down this TransferManager instance - currently executing transfers will not be allowed to finish. Callers should use this method when they either:

• have already verified that their transfers have completed by checking each transfer's state
• need to exit quickly and don't mind stopping transfers before they complete.

Callers should also remember that uploaded parts from an interrupted upload may not always be automatically cleaned up, but callers can use abortMultipartUploads(String, Date) to clean up any upload parts.

appendUserAgent

public <X extends AmazonWebServiceRequest> X appendUserAgent(X request,
                                                             String userAgent)