The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
๐ง ๐ง๐ต๐ถ๐ป๐ธ๐ถ๐ป๐ด ๐ผ๐ณ ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ฎ๐ป ๐๐๐ ๐ผ๐ป ๐๐๐ฏ๐ฒ๐ฟ๐ป๐ฒ๐๐ฒ๐?
Hereโs a rough outline of what it takes
๐๐ผ๐ป๐๐ฎ๐ถ๐ป๐ฒ๐ฟ๐ถ๐๐ฒ ๐๐ต๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น โ Start with something like a quantized Llama2, Mistral, or a custom fine-tuned model.
Use a lightweight serving framework (like text-generation-inference, vLLM, or TGI) and wrap it in a Docker container.
๐๐ฃ๐จ ๐๐ฐ๐ต๐ฒ๐ฑ๐๐น๐ถ๐ป๐ด โ Use node selectors or taints/tolerations to schedule pods on GPU-enabled nodes
๐๐๐๐ผ๐๐ฐ๐ฎ๐น๐น๐ถ๐ป๐ด โ Use KEDA or HPA to scale pods based on requests per second or GPU utilization. LLM workloads are spiky, so dynamic scaling saves $$.
๐๐ฃ๐ ๐๐ฎ๐๐ฒ๐๐ฎ๐ / ๐๐ผ๐ฎ๐ฑ ๐๐ฎ๐น๐ฎ๐ป๐ฐ๐ฒ๐ฟ โ Expose your model via a gateway (like Istio, NGINX, or even API Gateway in hybrid setups).
#LLM #Kubernetes #DevOps #MLOps #CloudNative #K8s #OpenSource
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
