by Jason Kölker /
Technical, AWS /
Aug 7, 2017 /
Comments
GPU Research Instances
AWS GPU instances are a great way to perform data manipulation and
tinker with machine learning without the expense of a dedicated research
machine for the casual data scientist. While the minimum instance type
(p2.xlarge) is nothing to sneeze at, its larger brethren (the g3 instances)
host a step above in terms of CPU and memory. However the p2.xlarge is
$0.9/hour and the smallest g3 is $1.14/hour. If tinkering is the goal, the
expenditure may give pause.
Spot Instances
Spot instances are a great way to save vs on-demand pricing, but
come at the cost that the OS is not persistent. Data that must be transported
between coding sessions needs to live on an external EBS volume.
For the lazy (or time constrained) researcher, the thought of spending
30-60 minutes every time they desire to play becomes cumbersome. Tools
like Ansible and Terraform can automate the initial
provisioning; however, a spot instance may be terminated at any time
if the current price exceeds its bid price. An alternative is to use
a custom ami, but every spot instance will start back from where that ami
was created.
Keeping a pet in a cattle yard
Using a clever hack, we can turn a spot instance into a persistent instance.
The flow is.
Create an on-demand (or spot if brave and the bid is high enough) with
a large EBS root volume.
Install and configure it as you like.
Terminate the instance but keep the EBS volume.
Boot a spot instance with the same (or similar enough) ami.
Attach the EBS volume to the instance
Modify the grub config to boot the kernel with the root= param set to
the other volume.
Reboot and ssh back into the server.
For this example, the latest Fedora 26 ami was used.
Setting up the persistent volume
After setting up an instance with all the required research packages and
perhaps setting up the environment(s) like Jupyter , a custom script
needs to be installed to run on shutdown. This is due to cloud-init
writing the mac address to the /etc/sysconfig/network-scripts/ifcfg-* files.When this image is booted up on another instance, the mac address will
of course be different. A static [ENI][8] could be carried around from
instance to instance, but it is easier to just delete the files.
/usr/bin/clear-network
#!/bin/bash
for f in $(ls /etc/sysconfig/network-scripts/ifcfg-* | grep -v lo);
do
rm -f ${f}
done /etc/sysconfig/systemd/clear-network.service
[Unit]
Description=Remove /etc/sysconfig/network-scripts/ifcfg-* on shutdown
[Service]
Type=oneshot
RemainAfterExit=true
ExecStop=/usr/bin/clear-network
[Install]
WantedBy=multi-user.target Place the files as specified then run
chmod +x /usr/bin/clear-network
systemctl start clear-network.service
systemctl enable clear-network.service
Terminate the instance making sure to preserve the EBS root volume.
Automating the Petification
A script can get written to automate the launching of a spot instance with
a user-data script that will transform the spot instance to boot from
the saved volume.
$HOME/bin/spotter
#!/bin/bash
service_tag=spotter
role_tag=gpu_research
spot_fleet_request_validity="5min"
spot_fleet_role_name=aws-ec2-spot-fleet-role
spot_price_history_time="1day"
preboot_image_id=ami-50f9d935
instance_type=p2.xlarge
instance_type=g3.4xlarge
public_key_name=ohio-gpu
ssh_username=fedora
ssh_port=314
echo -n "Discovering Volume... "
volume_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-volumes \
--filters Name=tag:service,Values=${service_tag} \
Name=tag:role,Values=${role_tag} \
--output=json)
volume_data=$(echo "${volume_data}" | jq .Volumes[0])
if [ "x${volume_data}" == "xnull" ]; then
echo "Error discovering volume"
exit 10
fi
volume_id=$(echo "${volume_data}" | jq -r .VolumeId)
if [ "x${volume_id}" == "x" ]; then
echo "Error discovering volume id"
exit 10
fi
volume_az=$(echo "${volume_data}" | jq -r .AvailabilityZone)
if [ "x${volume_az}" == "x" ]; then
echo "Error discovering volume availability zone"
exit 10
fi
echo "done"
echo -n "Discovering IAM instance Profile... "
profile_data=$(aws --profile ${service_tag}_${role_tag} \
iam get-instance-profile \
--instance-profile-name "${role_tag}" \
--output=json)
profile_data=$(echo "${profile_data}" | jq .InstanceProfile)
if [ "x${profile_data}" == "x" ]; then
echo "Error discovering instance profile"
echo "Make sure there is an instance profile named ${role_tag}"
exit 10
fi
profile_arn=$(echo "${profile_data}" | jq -r .Arn)
if [ "x${profile_arn}" == "x" ]; then
echo "Error discovering instance profile arn"
exit 10
fi
echo "done"
echo -n "Discovering IAM spot fleet role... "
fleet_role_data=$(aws --profile ${service_tag}_${role_tag} \
iam get-role \
--role-name "${spot_fleet_role_name}" \
--output=json)
fleet_role_data=$(echo "${fleet_role_data}" | jq .Role)
if [ "x${fleet_role_data}" == "x" ]; then
echo "Error discovering spot fleet role"
echo "Make sure there is an spot fleet role named ${spot_fleet_role_name}"
exit 10
fi
fleet_role_arn=$(echo "${fleet_role_data}" | jq -r .Arn)
if [ "x${fleet_role_arn}" == "x" ]; then
echo "Error discovering spot fleet role arn"
exit 10
fi
echo "done"
echo -n "Discovering VPC... "
vpc_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-vpcs \
--filters Name=is-default,Values=true \
--output=json)
vpc_data=$(echo "${vpc_data}" | jq .Vpcs[0])
if [ "x${vpc_data}" == "xnull" ]; then
echo "Error discovering default vpc"
exit 10
fi
vpc_id=$(echo "${vpc_data}" | jq -r .VpcId)
if [ "x${vpc_id}" == "x" ]; then
echo "Error discovering vpc id"
exit 10
fi
echo "done"
echo -n "Discovering Subnet... "
subnet_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-subnets \
--filters Name=vpc-id,Values="${vpc_id}" \
Name=availability-zone,Values="${volume_az}" \
--output=json)
subnet_data=$(echo "${subnet_data}" | jq .Subnets[0])
if [ "x${subnet_data}" == "xnull" ]; then
echo "Error discovering subnet"
exit 10
fi
subnet_id=$(echo "${subnet_data}" | jq -r .SubnetId)
if [ "x${subnet_id}" == "x" ]; then
echo "Error discovering subnet id"
exit 10
fi
echo "done"
echo -n "Discovering Security Group... "
sg_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-security-groups \
--filters Name=tag:service,Values=${service_tag} \
Name=tag:role,Values=${role_tag} \
--output=json)
sg_data=$(echo "${sg_data}" | jq .SecurityGroups[0])
if [ "x${sg_data}" == "xnull" ]; then
echo "Error discovering security group"
exit 10
fi
sg_id=$(echo "${sg_data}" | jq -r .GroupId)
if [ "x${sg_id}" == "x" ]; then
echo "Error discovering security group id"
exit 10
fi
echo "done"
echo -n "Generating Userdata... "
read -r -d '' user_data <<"EOF"
#!/bin/bash
dnf update
dnf install -y jq python-pip
pip install awscli
function swap_root_volume {
echo "Swapping root volume"
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
ZONE=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)
REGION=${ZONE::-1}
aws ec2 attach-volume --volume-id ${VOLUME_ID} \
--instance-id ${INSTANCE_ID} \
--device /dev/sdf \
--region ${REGION} || exit -1
while ! lsblk /dev/xvdf
do
echo "waiting for device to attach"
sleep 5
done
# Ready up for the swap
DEVICE=/dev/xvdf1
NEW_UUID=$(uuidgen)
e2fsck -f -p ${DEVICE}
tune2fs-U ${NEW_UUID} ${DEVICE}
EXISTING=$(df --output=source / | tail -n1)
OLD_UUID=$(blkid -s UUID -o value ${EXISTING})
sed -i "s/${OLD_UUID}/${NEW_UUID}/g" /boot/grub/grub.conf
sed -i "s/${OLD_UUID}/${NEW_UUID}/g" /boot/grub2/grub.cfg
}
EOF
user_data+="
SERVICE_TAG=${service_tag}
ROLE_TAG=${role_tag}
VOLUME_ID=${volume_id}
VOLUME_AZ=${volume_az}
swap_root_volume
shutdown -r now
"
user_data=$(echo "${user_data}" | base64 | tr -d '\n')
echo "done"
echo -n "Generating Instance Spec... "
read -r -d '' instance_spec <<EOF
{
"ImageId" : "${preboot_image_id}",
"InstanceType": "${instance_type}",
"KeyName" : "${public_key_name}",
"EbsOptimized": true,
"Placement": {
"AvailabilityZone": "${volume_az}"
},
"IamInstanceProfile": {
"Arn": "${profile_arn}"
},
"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"VolumeType": "gp2",
"VolumeSize": 8
}
}
],
"NetworkInterfaces": [
{
"DeviceIndex": 0,
"SubnetId": "${subnet_id}",
"Groups": [ "${sg_id}" ],
"AssociatePublicIpAddress": true
}
],
"UserData" : "${user_data}"
}
EOF
echo "done"
echo -n "Fetching spot price history... "
spot_price_history_start=$(date -u -d "-${spot_price_history_time}" \
+%Y-%m-%dT%H:%M:%SZ)
spot_price_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-spot-price-history \
--start-time "${spot_price_history_start}" \
--instance-types "${instance_type}" \
--filters Name=availability-zone,Values="${volume_az}" \
Name=product-description,Values=Linux/UNIX \
--output=json)
spot_price_data=$(echo "${spot_price_data}" | jq .SpotPriceHistory)
if [ "x${spot_price_data}" == "xnull" ]; then
echo "Error collecting spot price data"
exit 10
fi
echo "done"
if hash gnuplot 2>/dev/null; then
IFS= gnuplot_plot_data=$(echo "${spot_price_data}" |
jq -r '.[] | "\(.Timestamp) \(.SpotPrice)"')
gnuplot <<EOF
set terminal dumb
\$spot_price << EOD
$gnuplot_plot_data
EOD
set autoscale fix
set title "Spot Bid Price History"
set ylabel "Price"
set xlabel "Time"
set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S.000Z"
set grid
set nokey
plot \$spot_price using 1:2 pt "*"
EOF
else
echo ''
echo "gnuplot not found in PATH, consider installing for price graphing."
echo "Only showing last 10 prices"
echo ''
echo "${spot_price_data}" |
jq -r '.[0:10][] | "\(.Timestamp)\t\(.SpotPrice)"'
fi
spot_prices=$(echo "${spot_price_data}" |
jq '[ .[] | .SpotPrice | tonumber ]')
echo "Min Price: $(echo "${spot_prices}" | jq 'min')"
echo "Max Price: $(echo "${spot_prices}" | jq 'max')"
echo "Average Price: $(echo "${spot_prices}" | jq 'add/length')"
echo -n "Standard Deviation: "
echo "${spot_prices}" | jq '(map(.*.)|add/length)-pow(add/length;2)|sqrt'
echo ''
read -r -p "Spot bid price: " spot_price
echo -n "Generating Spot Fleet Request... "
fleet_req_valid_until=$(date -u -d "+${spot_fleet_request_validity}" \
+%Y-%m-%dT%H:%M:%SZ)
read -r -d '' spot_fleet_req <<EOF
{
"IamFleetRole": "${fleet_role_arn}",
"AllocationStrategy": "lowestPrice",
"TargetCapacity": 1,
"SpotPrice": "${spot_price}",
"ValidUntil": "${fleet_req_valid_until}",
"TerminateInstancesWithExpiration": false,
"LaunchSpecifications": [ ${instance_spec} ],
"Type": "request"
}
EOF
echo "done"
echo -n "Requsting Spot Fleet... "
fleet_req_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 request-spot-fleet \
--spot-fleet-request-config "${spot_fleet_req}" \
--output=json)
fleet_req_id=$(echo "${fleet_req_data}" | jq -r .SpotFleetRequestId)
if [ "x${fleet_req_id}" == "x" ]; then
echo "Error requesting spot fleet"
echo "${fleet_req_data}"
exit 10
fi
echo "done"
echo ''
echo "Spot Fleet Request ID: ${fleet_req_id}"
echo ''
echo -n "Waiting for instance to come online... "
while true; do
fleet_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-spot-fleet-instances \
--spot-fleet-request-id "${fleet_req_id}" \
--output=json)
fleet_data=$(echo "${fleet_data}" | jq .ActiveInstances[0])
if [ "x${fleet_data}" == "xnull" ]; then
sleep 5
continue
fi
instance_id=$(echo "${fleet_data}" | jq -r .InstanceId)
if [ "x${instance_id}" == "x" ]; then
sleep 5
continue
else
break
fi
done
echo "done"
echo ''
echo "Instance ID: ${instance_id}"
echo ''
echo -n "Discoverying Public IP Address... "
instance_data=$(aws --profile ${service_tag}_${role_tag} \
ec2 describe-instances \
--instance-ids "${instance_id}" \
--output=json)
instance_data=$(echo "${instance_data}" | jq .Reservations[0])
instance_data=$(echo "${instance_data}" | jq .Instances[0])
if [ "x${instance_data}" == "xnull" ]; then
echo "Error discovering public ip address"
echo "${instance_data}"
exit 10
fi
public_ip=$(echo "${instance_data}" | jq -r .PublicIpAddress)
if [ "x${public_ip}" == "x" ]; then
echo "Error getting public ip address"
exit 10
fi
echo "done"
echo ''
echo "Public IP: ${public_ip}"
echo ''
read -r -p "ssh to instance? [Y/n] " response
case "$response" in
[nN][oO]|[nN])
exit 0
;;
*)
ssh -i ~/.aws/${public_key_name}.pem \
-o "ConnectTimeout=5" \
-q \
-p ${ssh_port} \
${ssh_username}@"${public_ip}"
ssh_exit=$?
while [ $ssh_exit -ne 0 ] || [ $ssh_exit -ne 130 ]
do
ssh -i ~/.aws/${public_key_name}.pem \
-o "ConnectTimeout=5" \
-q \
-p ${ssh_port} \
${ssh_username}@"${public_ip}"
ssh_exit=$?
done
;;
esac
read -r -p "destroy instance? [Y/n] " response
case "$response" in
[nN][oO]|[nN])
exit 0
;;
*)
aws --profile ${service_tag}_${role_tag} \
ec2 terminate-instances \
--instance-ids "${instance_id}"
;;
esac
This script requires that elements needed (VPC, security group, volume,
instance profile, etc) be name and or tagged. Check out the script for
details. The iam polices can be found below
aws-ec2-spot-fleet-role
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"ec2:DescribeImages",
"ec2:DescribeSubnets",
"ec2:RequestSpotInstances",
"ec2:TerminateInstances",
"ec2:DescribeInstanceStatus",
"iam:PassRole"
],
"Resource": ["*"]
}]
} instance-profile
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1500333055000",
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:DescribeVolumes"
],
"Resource": [
"*"
]
}
]
}