This article takes note of useful information about training, fine-tuning, serving and deploying Large Language Models (LLMs).

Fine-tune

Embedding models

Fine-tune the BGE emedding models for optimized RAG performance

deployment

Basic considerations for deploying LLMs

EKS + vLLM

https://aws.amazon.com/blogs/machine-learning/deploy-meta-llama-3-1-8b-on-aws-inferentia-using-amazon-eks-and-vllm/

It is said EKS is as easy as ECS to maintain.

Sagemaker

Check out my notebook on Deployment of the DeepSeek-R1-Distilled-7B model to Sagemaker.

It is pretty straightforward to deploy models from huggingface to Sagemaker.

Pro: Easy to train, fine-tune and deploy. Con: Costy and not customizable compared to EKS or ECS on EC2. Not ideal to serce production traffic.

EC2

https://nlpcloud.com/deploy-llama-2-mistral-and-mixtral-on-aws-ec2-with-vllm.html

Fairly easy to deploy on EC2, but not scalable.

ECS on EC2 instances

https://generativeai.pub/llm-deployment-in-the-cloud-a-practical-guide-435db7fda3b7

The most scable solution of deploying LLMs in my opinion.