It is hard to take ebs backup manually each day as system administrator should log in to account daily. This bash script gives the power to automate that process. This automated ebs snapshots generating script reduce administrative task such it can defined how long backup should be kept on AWS account by configuring backup retention period, so Script automatically remove old backups for you. Also you add “cost center values” and “tag” to the snapshot, this could be useful to identify it and calculate snapshot storage cost later.
Pre Requirements
1) Script is runs on Linux environment
you can keep small EC2 for this, if you don’t have one.
2) Install AWS CLI
please refer this http://docs.aws.amazon.com/cli/latest/userguide/installing.html
3) Configure IAM credentials
Please refer above doc. It will guide you how to do it properly.
** IAM should have permission to create and delete snapshots at least.
Below is full script with sample cron job. I have explained code in detail at the end.
|
1 2 3 |
#Backup EBS volume #30 4 * * * sh /home/ubuntu/scripts/snapshots/prod3/prod3.sh >> /home/ubuntu/scripts/snapshots/prod3/cron.log 2>&1 # |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
#!/bin/bash DATE_TIME=$(date +"%Y%m%d%H%M") retention_days="3" retention_date_in_seconds=`date +%s --date "$retention_days days ago"` logfile="/home/ubuntu/scripts/snapshots/prod3/log" volume_info="/home/ubuntu/scripts/snapshots/prod3/volume_info.txt" snapshot_info="/home/ubuntu/scripts/snapshots/prod3/snapshot_info.txt" ##### Volume information of the Server ##################### instance_id=i-xxxxxxxa os_vol="vol-xxxxxxxb" data_vol="vol-xxxxxxxc" os_snapshot_tag="prod3-sda1-$DATE_TIME" data_snapshot_tag="prod3-sdb-$DATE_TIME" description="auto_created_prod3_snap" echo " " >> $logfile echo "Today is $(date)" >> $logfile echo "Retention date in seconds : $retention_date_in_seconds" >> $logfile ############################ #### Creating Sanpshots #### ############################ echo "" echo "Today is $(date)" echo "Creating sanpshots ..." echo "Creating snapshot" >> $logfile os_cs_output=$(/usr/local/aws/bin/aws ec2 create-snapshot --volume-id $os_vol --description $description --output text) os_snapshot_id=$(echo $os_cs_output | sed -n 's/^.*\([s][n][a][p]-[^ ]*\).*/\1/p'); echo "OS sanpshot: $os_snapshot_id" >> $logfile os_ct_output=$(/usr/local/aws/bin/aws ec2 create-tags --resources $os_snapshot_id --tags Key=Name,Value=$os_snapshot_tag Key='Cost Center',Value=cloud Key='Cost Center:Type',Value=cloud-snap) echo "sleep 30 seconds" sleep 30 echo "start data snapshot" data_cs_output=$(/usr/local/aws/bin/aws ec2 create-snapshot --volume-id $data_vol --description $description --output text) data_snapshot_id=$(echo $data_cs_output | sed -n 's/^.*\([s][n][a][p]-[^ ]*\).*/\1/p'); echo "Data sanpshot: $data_snapshot_id" >> $logfile data_ct_output=$(/usr/local/aws/bin/aws ec2 create-tags --resources $data_snapshot_id --tags Key=Name,Value=$data_snapshot_tag Key='Cost Center',Value=cloud Key='Cost Center:Type',Value=cloud-snap) echo "finish snapshot creation" echo "finish snapshot creation" >> $logfile echo "sleep 30 seconds" sleep 30 ############################## #### Deleting old snapshots ## ############################## echo "Start deleting sanpshot" echo "Start deleting sanpshot" >> $logfile rm $volume_info --force /usr/local/aws/bin/aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=$instance_id --query Volumes[].VolumeId --output text | tr '\t' '\n' > $volume_info 2>&1 rm $snapshot_info --force for vol_id in $(cat $volume_info) do /usr/local/aws/bin/aws ec2 describe-snapshots --output=text --filters "Name=volume-id,Values=$vol_id" "Name=description,Values=$description" --query Snapshots[].SnapshotId | tr '\t' '\n' | sort | uniq >> $snapshot_info 2>&1 done for snapshot_id in $(cat $snapshot_info) do echo "Checking $snapshot_id..." snapshot_date=$(/usr/local/aws/bin/aws ec2 describe-snapshots --output=text --snapshot-ids $snapshot_id --query Snapshots[].StartTime | awk -F "T" '{printf "%s\n", $1}') snapshot_date_in_seconds=`date "--date=$snapshot_date" +%s` #if (( $snapshot_date_in_seconds <= $retention_date_in_seconds )); then if [ "$snapshot_date_in_seconds" -lt "$retention_date_in_seconds" ];then echo "Deleting snapshot: $snapshot_id created at: $snapshot_date_in_seconds ..." >> $logfile /usr/local/aws/bin/aws ec2 delete-snapshot --snapshot-id $snapshot_id else echo "Not deleting snapshot: $snapshot_id created at: $snapshot_date_in_seconds ..." >> $logfile fi done echo "finish deleting sanpshot" >> $logfile echo "end" |
Bash script explanation.
|
1 2 3 4 5 6 7 8 |
#!/bin/bash DATE_TIME=$(date +"%Y%m%d%H%M") retention_days="3" retention_date_in_seconds=`date +%s --date "$retention_days days ago"` logfile="/home/ubuntu/scripts/snapshots/prod3/log" volume_info="/home/ubuntu/scripts/snapshots/prod3/volume_info.txt" snapshot_info="/home/ubuntu/scripts/snapshots/prod3/snapshot_info.txt" |
retention_days –> How long back should be kept on. Remember AWS will bill you for this. So keep retention days as short as possible.
logfile –> script output, kind of debug
volume_info –> EC2 volume ID are kept here for instance named “prod3”
snapshot_info –> Sanpshot ID are kept here for instance named “prod3”
|
1 2 3 4 5 |
instance_id=i-xxxxxxxa os_vol="vol-xxxxxxxb" data_vol="vol-xxxxxxxc" os_snapshot_tag="prod3-sda1-$DATE_TIME" data_snapshot_tag="prod3-sdb-$DATE_TIME" |
instance_id –> unique ID of the EC2 instance.
os_vol,data_vol –> instance has 2 volume you can define any amount of volumes. example instance was named as “prod3”. so changed that as well.
|
1 2 3 4 |
os_cs_output=$(/usr/local/aws/bin/aws ec2 create-snapshot --volume-id $os_vol --description $description --output text) os_snapshot_id=$(echo $os_cs_output | sed -n 's/^.*\([s][n][a][p]-[^ ]*\).*/\1/p'); echo "OS sanpshot: $os_snapshot_id" >> $logfile os_ct_output=$(/usr/local/aws/bin/aws ec2 create-tags --resources $os_snapshot_id --tags Key=Name,Value=$os_snapshot_tag Key='Cost Center',Value=cloud Key='Cost Center:Type',Value=cloud-snap) |
os_cs_output –> text base (you can see command is append with “–output text” ) output result after creating snapshot. This is for the volume which contains OS. you have to create other volumes if there any. for example I create separate variable for data volume (data_cs_output). cs stands for “create snaphost” 🙂
os_snapshot_id –> grab the snapshot ID.
os_ct_output –> tag the snapshot with cost center value, we already have snapshot ID. This is extra step. you can skip If you wish. ct stands for “create tag”
Now time for delete old snapshots
|
1 2 |
rm $volume_info --force /usr/local/aws/bin/aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=$instance_id --query Volumes[].VolumeId --output text | tr '\t' '\n' > $volume_info 2>&1 |
Empty the volume_info file first. and then insert all volume ID to it.
|
1 2 3 4 5 |
rm $snapshot_info --force for vol_id in $(cat $volume_info) do /usr/local/aws/bin/aws ec2 describe-snapshots --output=text --filters "Name=volume-id,Values=$vol_id" "Name=description,Values=$description" --query Snapshots[].SnapshotId | tr '\t' '\n' | sort | uniq >> $snapshot_info 2>&1 done |
Empty the snapshot_info file and insert all snapshot created for volumes mentioned on volume_info file.
I feels, I explain a lot in here. Do comment here if you have any doubt.