August 3, 2023
@Eduardo Lugo
AI tools are everywhere now and in order to have them do their thing with acceptable performance they need to use the power of GPU enabled instances, in this article I'll tell you how we are achieving this.
The Omniarcs team is building and supporting the The little AI assistant that could - Locus Extension. **So to see how the application of this post is running in production, please download and try Locus!
“Karpenter is an open-source node provisioning project built for Kubernetes. Adding Karpenter to a Kubernetes cluster can dramatically improve the efficiency and cost of running workloads on that cluster”
The idea of Karpenter in short is that instead of having a node group for your Kubernetes Cluster you let Karpenter manage that by giving it options of what type of instances and resources can it use for that node group that can scale depending on the requirements of your pods.
Here is a terraform example of setting an EKS cluster with Karpenter enabled
The interesting part of Karpenter are the provisioners, we’ll get into that later.
Bottlerocket is a Linux-based open-source operating system that is purpose-built by Amazon Web Services for running containers.
We use Bottlerocket in Karpenter as the family of AMIs our provisioners use, you can read about it here, but basically it is the AMI that Karpenter will use to create the instance that the pods will run on.
You have some options here, you can use the nvidia gpu operator or the nvidia device plugin, for now we have choosen the plugin
Now lets get into the details of each bit
This is the blueprint of what your instances will look like, so we can define, instance size, cpu, family, generation, architecture, etc. You can be as specific or generic as you like, make sure you use Karpenter docs to define stuff that is not contradicting itself like an arch that does not work on a specific instance type (it happens!)