How-To: Copy an EBS-Backed AMI from one region to another one

Goal:

The goal of this document is to describe the process for copying an Amazon EC2 EBS-Backed AMI from one region to another.
We have published an opensource project (CloudyScripts) to automatically perform this operation, but we get a lot of questions about it. We will continue to improve Cloudyscripts to better answer all specific issues but here goes the manual HowTo. ;)

Outline:

  1. Create an archive of the AMI file-system in the source region
  2. Copy the archive in the target region
  3. Create a volume containing the file-system
  4. Create a new EBS-backed AMI in the target region

Requirements:

  • A running instance with SSH access in both source and target region. I suggest to launch an Amazon 32bits micro instance-type from Basic 32-bit Amazon Linux AMI 2010.11.1 Beta in the source and target region.
    NB: On Linux, I’d suggest to export JAVA_HOME and EC2_HOME variables (they also could be added in the command line).
  • A temporary SSH key:
    I suggest to use a 1024 bits RSA key, that can be generated as follows (under a Linux system): 


    linux-box-source:~$ ssh-keygen -b 1024 -C "Temporary SSH Key" -t rsa -f temp_ssh_key
    linux-box-source:~$ ls temp_ssh_key*
    temp_ssh_key temp_ssh_key.pub

  • Get AWS account parameters from your AWS account page in the Security Credentials part:
  • Private Key file: AWS_EC2_PRIVATE_KEY.pem
  • Certificate Key file: AWS_EC2_CERT.pem
    NB: These files can be downloaded from the Access Credentials table, X509 Certificates tab.
  • Get all the parameters you might need from the EBS-Backed AMI:
  • AMI ID: AMI-XXXXXXXX
  • EBS Volume size: VOL_SIZE
  • Architecture: AMI_ARCH
  • Kernel ID (AKI): AKI_XXXXXXXX
  • Root Device: ROOT_DEVICE (usually /dev/sda1)
  • Find the Kernel ID (AKI) that will be used for registering your new EBS-Backed AMI in the target region
  • This is described at the end of the document.

NB: In this document, we will use to a Linux system (for command line) and Amazon Console (for Console Administration), but feel free to use your preferred tools. ;)
NB: The LABEL part is useful for some Linux distributions (such as RedHat, Fedora, Suse, OpenSuse) as they use labeled file system in their fstab file and in the root option of their GRUB configuration file.

Step 1: Create an archive of the AMI in the Source Region

  • Using Amazon console:
  • Launch an instance of the AMI (a micro instance-type should be enough)
  • Create a snapshot of the EBS volume
  • Create a volume from this newly created snapshot
  • Stop the instance
  • Launch an instance in the same availability zone of the EBS volume
  • Attached the newly created EBS volume to the running instance in that region and check the name of the device (/dev/sdx)
  • Using you favorite SSH tool, connect to the running instance in the source region and get root access:
  • Get the device label:

    [root@amazon-linux ~]# e2label /dev/sdx
  • Create a mount point:

    [root@amazon-linux ~]# mkdir /mnt/ebs_volume 

     

  • Mount the device on that mount point:

    [root@amazon-linux ~]# mount /dev/sdx /mnt/ebs_volume/
  • Create an archive of the device:

    [root@amazon-linux ~]# tar -zcpvf /mnt/system-YYYY-MM-DD.tar.gz /mnt/ebs_volume/
  • Using Amazon console:
  • Detached the newly created EBS volume to the running instance in that region

NB: If needed you can get root access as follows:

[ec2-user@amazon-linux ~]$ sudo su -
[root@amazon-linux ~]#

NB: Avoid bzip2 compression if you use a small CPU instance.

NB: Please note the file system type (ext2, ext3, …), as it will be reused in the target region while creating a new file system

NB: Please note the permissions while creating file system from archive (-p option)

Step 2: Copy the archive to the Target Region

  • Copy the temporary SSH Private Key on your running instance in the source region:
  • Using you favorite SSH tool to copy the file.
    This could be done as follows on a Linux system:

    linux-box-source:~$ scp -i ssh_key_4_amazon-linux_source.pem temp_ssh_key ec2-user@amazon-linux_source.compute.amazonaws.com:~/
  • Using the Amazon console, launch an instance in the target region (be careful with the availability zone)
  • Add the temporary SSH public key to the authorized key, and reload SSH daemon:
  • Using you favorite SSH tool, connect to the running instance in the target region and get root access
  • Edit file .ssh/authorized_keys and add you public temporary SSH key contained in file temp_ssh_key.pub
  • Reload SSH daemon

    [root@amazon-linux ~]# /etc/init.d/sshd reload
    Reloading sshd: [ OK ] 

     

  • Copy the archive from the source region to the target region as follows:

    [root@amazon-linux ~]# scp -i /home/ec2-user/temp_ssh_key /mnt/system-2011-01-20.tar.gz ec2-user@amazon-linux_target.compute.amazonaws.com:~/

Step 3: Create a volume containing the file-system in the Target Region

  • Using Amazon console:
  • Create an EBS volume in the target region in the same availability zone as your running instance and of the same size of the EBS volume of your EBS-Backed AMI in the source region
  • Attach that volume to the running instance in the target zone
  • Using you favorite SSH tool, connect to the running instance in the target region and get root access:
  • Create a file-system on the device (same file system as the one used by your EBS-Backed AMI):
    [root@amazon-linux ~]# mke2fs -t ext3 /dev/sdx
  • Set device label: LABEL

    [root@amazon-linux ~]# e2label /dev/sdx LABEL
  • Create a mount point:

    [root@amazon-linux ~]# mkdir /mnt/ebs_volume
  • Mount the device on that mount point:

    [root@amazon-linux ~]# mount /dev/sdx /mnt/ebs_volume/
  • Extract the archive to that mount point:

    [root@amazon-linux ~]# tar -zxpvf /mnt/system-YYYY-MM-DD.tar.gz -C /
  • UnMount the device:

    [root@amazon-linux ~]# umount /mnt/ebs_volume/
  • Using the Amazon console:
  • Detach the newly created EBS volume to the running instance in that region

NB: If needed you can get root access as follows:

[ec2-user@amazon-linux ~]$ sudo su -
[root@amazon-linux ~]#

NB: Be careful with the permissions while extracting the file system from archive (-p option)

Step4: Create an EBS-Backed AMI in the Target Region

  • Using the Amazon console, create a snapshot of the volume:
  • Register a new AMI from the previously created snapshot, using Amazon EC2 API tools:

    linux-box-source:~/ec2-api-tools-1.3-62308$ ./bin/ec2-register --private-key AWS_EC2_PRIVATE_KEY.pem --cert AWS_EC2_CERT.pem -v -H --region TARGET_REGION -a AMI_ARCH -s SNAP-XXXXXXXX -d 'CopyAMI Generated' -n 'AMI_DESCRIPTION' --root-device-name /dev/sda1 --kernel AKI-XXXXXXXX

NB: The kernel ID AKI-XXXXXXX is found by the procedure at the end of this post

Cleanup

  • Using the Amazon console:
  • Terminate the running instance in the source region
  • Cleanup the EBS volume created in the source region
  • Cleanup the snapshot created in the source region
  • Terminate the running instance in the target region
  • Cleanup the EBS volume created in the target region
  • Cleanup the snapshot created in the target region

Find the right Kernel ID (AKI) for registering your EBS Backed AMI

Outline:

  1. Find the description of the original AMI used to create your EBS-Backed AMI in the source region
  2. Look for the same AMI in the target region using the description of the original AMI
  3. Get the Kernel ID (AKI) of the corresponding AMI in the target region

HowTo:

  • Retrieve all the AMIs in a specific region that use a specific KernelID (AKI):

    linux-box-source:~/ec2-api-tools-1.3-62308$ ./bin/ec2-describe-images --private-key AWS_EC2_PRIVATE_KEY.pem --cert AWS_EC2_CERT.pem -v -H --region SOURCE_REGION -a -F kernel-id=AKI-XXXXXXXX
  • Retrieve a KernelID in a specific region that is used by a specific AMI:

    linux-box-source:~/ec2-api-tools-1.3-62308$ ./bin/ec2-describe-images --private-key AWS_EC2_PRIVATE_KEY.pem --cert AWS_EC2_CERT.pem -v -H --region SOURCE_REGION -a -F name="*AMI_NAME*"
  • Example: Suppose I made an AMI using a FreeBSD AMI: FreeBSD/EC2 9.0-CURRENT 2011-01-04 in US-West and I want to copy that AMI to US-East
  • I just have to retrieve the KernelID to use in the target region, as follows:

    debian-secludit:~/ec2-api-tools-1.3-62308# JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20 EC2_HOME=/root/ec2-api-tools-1.3-62308 ./bin/ec2-describe-images -K ../amazon-ec2-keys/key-BK54TI5LZQMBWA3GVE4XXFYMCWSUCGVY.pem -C ../amazon-ec2-keys/cert-BK54TI5LZQMBWA3GVE4XXFYMCWSUCGVY.pem -H --region us-east-1 -a -F name="*FreeBSD*"
    Type ImageID Name Owner State Accessibility ProductCodes Architecture ImageType KernelId RamdiskId Platform RootDeviceType VirtualizationType Hypervisor
    IMAGE ami-c01aeca9 118940168514/FreeBSD/EC2 9.0-CURRENT 2010-12-12 118940168514 available public i386 machine aki-407d9529 ebs paravirtual xen
    BLOCKDEVICEMAPPING /dev/sda1 snap-409c852a 1
    BLOCKDEVICEMAPPING /dev/sdb snap-ce948da4 9
    IMAGE ami-a0fc0dc9 118940168514/FreeBSD/EC2 9.0-CURRENT 2010-12-29 118940168514 available public i386 machine aki-407d9529 ebs paravirtual xen
    BLOCKDEVICEMAPPING /dev/sda1 snap-7127841c 1
    BLOCKDEVICEMAPPING /dev/sdb snap-291ab944 9
    IMAGE ami-f4db2a9d 118940168514/FreeBSD/EC2 9.0-CURRENT 2011-01-01 118940168514 available public i386 machine aki-407d9529 ebs paravirtual xen
    BLOCKDEVICEMAPPING /dev/sda1 snap-dbe855b6 1
    BLOCKDEVICEMAPPING /dev/sdb snap-2fe35e42 9
    IMAGE ami-8cce3fe5 118940168514/FreeBSD/EC2 9.0-CURRENT 2011-01-04 118940168514 available public i386 machine aki-407d9529 ebs paravirtual xen
    BLOCKDEVICEMAPPING /dev/sda1 snap-1df57270 1
    BLOCKDEVICEMAPPING /dev/sdb snap-e1fe798c 9
  • I could register my new AMI using the following KernelID: aki-407d952

/fred

About these ads

21 thoughts on “How-To: Copy an EBS-Backed AMI from one region to another one

  1. Pingback: Amazon EC2 Copy AMI and Snapshot: CloudyScripts updated « Elastic Security

  2. Pingback: Amazon US West Oregon Region Support in CloudyScripts « Elastic Security

  3. After spending a couple of days of trying to figure out why I was not able to use this process successfully, I discovered the issue.
    Some background – I created an instance in Region A using the Amazon 64 bit Linux AMI. Installed a whole bunch of stuff on it over a couple of months before we decided we wanted to switch to region B. I created an AMI of my instance and I reached Step 4 of the outline above with no problems before i hit a wall
    Turns out the base Amazon 64 bit Linux AMI in region A (off of which I had created my AMI) has a different kernel id from the 64 bit Linux AMI in region B. The above procedure relies on the base AMI having the same kernel id across regions …
    If it is relevant – Region A above = ap-southeast-1 and region B = us-west-1.
    you can do ec2-describe-images |and compare the 2 base images to see what I mean. Sigh – looks like i will have to rebuild from scratch

    • Hi Anand,

      Thanks for sharing your experience with this procedure.

      The above procedure relies on the base AMI having the same kernel id across regions

      We apologize if we have not been clear enough, but this is NOT true. The kernel ID are NOT shared between AWS Region. It seems that you missed the part on “Find the right Kernel ID (AKI) for registering your EBS Backed AMI“, which explains how to find the new kernel ID (AKI) in the target region. Before registering the new AMI as describe in Step 4, you have to find the right kernel ID in the target region.

      Hope this helps.
      /fred

  4. Hi Fred,
    Thanks for the pointer. Actually, I forgot to mention that I had indeed tried the “Finding the right Kernel ID” step. Using the resultant Kernel id from the target did create the AMI but the instances launched from it failed the accessibility status check and I was unable to connect to it. Just to make sure I hadn’t made a mistake in following the steps outlined, i tried twice more with the same end result.
    I will do some more investigation and report back

  5. Happy to report that it all worked out – i had screwed up the file system specification (used ext4 instead of ext3)
    Many thanks for this writeup.

  6. A few small things to keep in mind for others using this outline
    1. You can find out the kernel id to use by simply by using the Amazon console in case your base image was one of the standard ones provided by Amazon
    2. You may see a few tar warnings – keep an eye out to see if there is anything strange there. It may be better to redirect the tar output to a file so that you can look through it later
    3. It was (for me) a bit non trivial to determine the file system type. The default at this time appears to be ext3 for Amazon base images
    Regards
    /anand

  7. * Launch an instance of the AMI (a micro instance-type should be enough)
    * Create a snapshot of the EBS volume
    * Create a volume from this newly created snapshot
    * Stop the instance
    * Launch an instance in the same availability zone of the EBS volume
    * Attached the newly created EBS volume to the running instance in that region and check the name of the device (/dev/sdx)

    On that last step, when I go to mount it on the new (temp) instance it does not allow me to mount the device (/dev/sdx) however it does allow me to mount the partition (/dev/sdx1).

    I do not see anything specifying the partition and my AMI has 2 (a / and a swap) should i continue the process just mounting /dev/sdx1 and ignore the swap partition (/dev/sdx2)

    • Hi Mike,

      Yes, just mount /dev/sdx1 (which is your root partition ‘/’) instead of the /dev/sdx one.

      Pay attention that in the target region you will have to create the same partition table (/dev/sdx1 for ‘/’ root partition and /dev/sdx2 for swap partition) using fdisk command in Step 3.
      In this particular case, you also will have to create the swap space using ‘mkswap’ command.

      Hope this helps.
      /fred

  8. Pingback: the echothis blog » Migrating an Amazon server to a new region

  9. Pingback: 手動copy AMI到不同Region | Linda's murmur

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s