Using AzCopy to transfer a large number of files between Azure Blob Storage Containers

Azure Blob Storage is Microsoft’s primary cloud storage solution and can handle massive amounts of unstructured data. Whether you need to store images, videos, documents, or log files, it offers a secure and scalable platform for online file storage that you can trust.

Within an Azure Blob Storage account, you can configure one or ‘containers’ for your data. These containers allow you to organize files, set permission levels, control retention periods, and more. When developing and maintaining cloud applications that leverage Azure Blob Storage, you may encounter situations where you need to copy a large number of files from one container to another. This could be as part of a backup strategy or an initial system data migration perhaps.

When it comes to copying files between containers, several options are available, such as the Azure Portal, Azure Storage Explorer, or Azure PowerShell cmdlets. However, the AzCopy command-line utility, with its support for bulk operations makes it ideal for large-scale data migration and will be the focus of this article.

The sections that follow will provide an introduction to the AzCopy program, explaining how to get started and some of the key things to be aware of when it comes to transferring a large number of files between containers.

Getting started

To get started with AzCopy, first download it from the Microsoft Learn website.

If your Operating System is Windows-based, you’ll usually want to download the Windows 64-bit zip file.

Note that I will assume you are using a Windows device from this point on, however, the general concepts of using AzCopy once things are set up should apply regardless of your Operating System.

After the download completes, extract the contents of the archive file to a suitable location and you should find two files in the extracted directory, as follows.

• azcopy.exe
• NOTICE.txt

azcopy.exe is the command-line executable and NOTICE.txt contains the details of third-party licenses.

If you plan to use AzCopy often, you may want to consider placing the program in a permanent installation location, such as C:\Program Files\AzCopy and adding azcopy.exe to your System PATH. This will allow AzCopy to be accessed from the command line more readily.

To use AzCopy from the command line, you can open a Command Prompt or your preferred terminal and navigate to the location you extracted the directory containing AzCopy to using the cd command e.g. as follows.

cd C:\Users\Jonathan\Downloads\AzCopy\azcopy_windows_amd64_10.28.0

Note that an easy way to get started quickly is to navigate to the folder containing the azcopy.exe file, then type ‘cmd’ into the File Explorer directory text bar and press the Enter/Return key. The Command Prompt should open with the current directory set to your current File Explorer location.

After opening the Command Prompt and navigating to the correct working directory, you can type ‘azcopy’ and press Enter/Return to display some help on how to use the AzCopy program/command, which should look similar to the following.

AzCopy 10.28.0
Project URL: github.com/Azure/azure-storage-azcopy
 
AzCopy is a command line tool that moves data into and out of Azure Storage.
To report issues or to learn more about the tool, go to github.com/Azure/azure-storage-azcopy
 
The general format of the commands is: 'azcopy [command] [arguments] --[flag-name]=[flag-value]'.
 
Usage:
  azcopy [command]
 
Available Commands:
  bench          Performs a performance benchmark
  completion     Generate the autocompletion script for the specified shell
  copy           Copies source data to a destination location
  doc            Generates documentation for the tool in Markdown format
  env            Shows the environment variables that you can use to configure the behavior of AzCopy.
  help           Help about any command
  jobs           Sub-commands related to managing jobs
  list           List the entities in a given resource
  login          Log in to Microsoft Entra ID to access Azure Storage resources.
  logout         Log out to terminate access to Azure Storage resources.
  make           Create a container or file share.
  remove         Delete blobs or files from an Azure storage account
  set-properties Given a location, change all the valid system properties of that storage (blob or file)
  sync           Replicate source to the destination location
 
Flags:
      --cap-mbps float                      Caps the transfer rate, in megabits per second. Moment-by-moment throughput might vary slightly from the cap. If this option is set to zero, or it is omitted, the throughput isn't capped.
  -h, --help                                help for azcopy
      --log-level string                    Define the log verbosity for the log file, available levels: DEBUG(detailed trace), INFO(all requests/responses), WARNING(slow responses), ERROR(only failed requests), and NONE(no output logs). (default 'INFO'). (default "INFO")
      --output-level string                 Define the output verbosity. Available levels: essential, quiet. (default "default")
      --output-type string                  Format of the command's output. The choices include: text, json. The default value is 'text'. (default "text")
      --skip-version-check                  Do not perform the version check at startup. Intended for automation scenarios & airgapped use.
      --trusted-microsoft-suffixes string   Specifies additional domain suffixes where Azure Active Directory login tokens may be sent.  The default is '*.core.windows.net;*.core.chinacloudapi.cn;*.core.cloudapi.de;*.core.usgovcloudapi.net;*.storage.azure.net'. Any listed here are added to the default. For security, you should only put Microsoft Azure domains here. Separate multiple entries with semi-colons.
  -v, --version                             version for azcopy
 
Use "azcopy [command] --help" for more information about a command.

Now we are ready to log in and start exploring what is possible with AzCopy!

Logging in

Before we can start copying files with AzCopy, we need to log in.

To do this, execute the login command, as shown below.

azcopy login

After executing the above command, you should see a prompt similar to the following.

INFO: Authentication is required. To sign in, open the webpage https://aka.ms/devicelogin and enter the code ABCDEFG12 to authenticate.

As instructed, navigate to https://aka.ms/devicelogin via your web browser and enter the code provided, then enter your login credentials to authenticate with Microsoft Entra ID. After entering your credentials, you should be prompted with the question “Are you trying to sign in to Azure Storage AzCopy?” with the option to continue. After continuing, you should see a message confirming that you have signed in.

Following a successful login, you should see a message like the one shown below in the console.

INFO: Login succeeded.

Now we’re ready to start executing other AzCopy commands.

Copying

After logging in, we can start copying files between containers, and the copy command can be used to do this.

Assuming you have a container named ‘source_container’ and a second container that you want to copy all files to named ‘destination_container’, the command below will perform the copy operation for you.

azcopy copy "https://[source_account].blob.core.windows.net/[source_container]" "https://[destination_account].blob.core.windows.net/[destination_container]"

Note that you will need to replace the text in square brackets above with the appropriate account names and container names. The ‘source_account’ and ‘destination_account’ should be set to the same value if you are copying files between containers within the same Storage Account.

The above command is the most basic one you can execute and uses all of the default options, which are mostly fine. However, you may want to consider adding the --recursive parameter which will copy all files from sub-directories within the container, as shown below.

azcopy copy "https://[account].blob.core.windows.net/[source_container]/[path/to/blob]" "https://[account].blob.core.windows.net/[destination_container]/[path/to/blob]" --recursive

After executing the command, you should see the console output something similar to what is shown below.

INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
 
Job 12345678-90ab-cdef-1234-567890abcdef has started
Log file is located at: /path/to/azcopy.log
 
0.1 %, 1000 Done, 0 Failed, 1032530 Pending, 0 Skipped, 56 Total, 100.5 MiB/s, 10s

When transferring a large volume of files, after some time has passed, you may see a message similar to the following.

INFO: Reducing progress output frequency to 60s, because there are over 1,000,000 files.

It is important to be aware that although AzCopy is fast at copying files between containers, the copy operation will still place considerable strain on the local network when transferring a large volume of data. You will need to plan for this before kicking things off and remember that the more bandwidth you have the better.

Resuming a failed job

While AzCopy tries its best to transfer all files successfully (retrying up to 20 times) you can expect a portion of files to fail when performing a copy operation that is expected to last several hours or days. Additionally, the overall copy operation could potentially fail due to a timeout or some other reason.

When this happens, you could run the original copy command from the previous section, but this will start a new job and begin copying all of the files again.

Alternatively, you could run the copy command with the --overwrite=false parameter name/value specified, which would at least skip over files that have already been transferred, but again this would be a new job and would not be a significantly faster option, as it still starts a new job and has to process all of the files again.

What you should do instead is resume the failed job so that the copy command can pick up where it left off and thereby complete the operation much faster.

To do this, you should first execute the following command to list all AzCopy jobs.

azcopy jobs list

This will display a list of Existing Jobs within the console, similar to the following.

Existing Jobs
JobId: f9e8d7c6-b5a4-3210-fedc-ba9876543210
Start Time: Thursday, 27-Feb-25 15:20:45 GMT
Status: Failed
Command: azcopy copy "https://my-account.blob.core.windows.net/source-container" "https://my-account.blob.core.windows.net/destination-container" --recursive
 
JobId: 92257d06-66a1-4b35-a49d-c1c5f9afd8f0
Start Time: Thursday, 27-Feb-25 14:56:23 GMT
Status: Failed
Command: azcopy copy "https://my-account.blob.core.windows.net/source-container" "https://my-account.blob.core.windows.net/destination-container" --recursive

Note that AzCopy stores the details of previous jobs in the following location: C:\Users\<YourUsername>\.azcopy

After identifying the failed job, you can then execute the following command to resume it.

azcopy jobs resume [job_id]

Note that you will need to replace [job_id] with the ‘JobID’ value from the jobs list before executing the command.

After you execute the resume command, you should see that the job resumes, picking up from where it left off and saving lots of valuable time and resources.

If the resumed job fails again at a later point for any reason, you can execute the resume command as many times as required until the overall job has been completed.

One last thing to be aware of is that there can be occasions when the copy operation stalls and needs to be restarted. In these cases, you can use the Ctrl + C keyboard shortcut to cancel the copy command. You should be prompted if you are sure you want to cancel when doing this, allowing the job to be safely resumed afterwards using the same approach outlined above.

Summary

In this article, I have explained the essential knowledge you will need to get started with using the AzCopy command-line utility to transfer a large number of files between Azure Blob Storage Containers.

I covered the scenario where the entire contents of a container need to be transferred from one Azure Blob Storage container to another container, either within the same Storage Account or a separate one. However, it is important to note that AzCopy supports many other cases, such as uploading files from a local directory to a container and even copying files from other cloud storage solutions such as Amazon Web Services S3 or Google Cloud Storage to an Azure Blob Storage Container.

All of these additional use cases and the parameters to use can be found in the AzCopy GitHub Wiki which is a great reference point. You can also check out the Microsoft Docs for further practical examples.