AWS GPU Instances: Turning Spot Instances Into Persistent Instances
GPU Research Instances
AWS GPU instances are a great way to perform data manipulation and tinker with machine learning without the expense of a dedicated research machine for the casual data scientist. While the minimum instance type (p2.xlarge) is nothing to sneeze at, its larger brethren (the g3 instances) host a step above in terms of CPU and memory. However the p2.xlarge is $0.90/hour and the smallest g3 is $1.14/hour. If tinkering is the goal, the expenditure may give pause.
Spot Instances
Spot instances are a great way to save compared to on-demand pricing, but come at the cost that the operating system is not persistent. Data that must be transported between coding sessions needs to live on an external EBS volume.
For the lazy (or time-constrained) researcher, the thought of spending 30-60 minutes every time they desire to play becomes cumbersome. Tools like Ansible and Terraform can automate the initial provisioning; however, a spot instance may be terminated at any time if the current price exceeds its bid price. An alternative is to use a custom AMI, but every spot instance will start back from where that AMI was created.
Keeping a pet in a cattle yard
Using a clever hack, we can turn a spot instance into a persistent instance. The flow is:
- Create an on-demand (or spot if you’re brave and the bid is high enough) with a large EBS root volume.
- Install and configure it as you like.
- Terminate the instance but keep the EBS volume.
- Boot a spot instance with the same (or similar enough) AMI.
- Attach the EBS volume to the instance
- Modify the grub config to boot the kernel with the root= param set to the other volume.
- Reboot and ssh back into the server.
For this example, the latest Fedora 26 AMI was used.
Setting up the persistent volume
After setting up an instance with all the required research packages and
perhaps setting up the environment(s) like Jupyter, a custom script
needs to be installed to run on shutdown. This is due to cloud-init
writing the mac address to the /etc/sysconfig/network-scripts/ifcfg-* files. When this image is booted up on another instance, the mac address will
of course be different. A static ENI could be carried around from
instance to instance, but it is easier to just delete the files.
/usr/bin/clear-network
/etc/sysconfig/systemd/clear-network.service
Place the files as specified then run
chmod +x /usr/bin/clear-network
systemctl start clear-network.service
systemctl enable clear-network.service
Terminate the instance making sure to preserve the EBS root volume.
Automating the Petification
A script can get written to automate the launching of a spot instance with
a user-data script that will transform the spot instance to boot from
the saved volume.
$HOME/bin/spotter
This script requires that elements needed (VPC, security group, volume, instance profile, etc.) be name and or tagged. Check out the script for details. The IAM polices can be found below: