This original CPU exercise is foundation sample 1 of 4. The four samples retain their code and numbering, separate from the redesigned nine-week track. They do not establish validation of the new week 3 GPU lab.
That’s where we’re going.
The redesigned course aims to build and operate your own bounded model platform: accept requests, use GPU capacity deliberately, measure performance, and recover when something fails. This sample starts with the CPU model runner; a working sample does not establish production readiness.
You aren’t paying a model provider for each token. You’re running the model yourself. That still means paying for compute, storage, networking, and idle capacity—and taking responsibility for the service. Whether it costs less depends on the workload.
Your application → Your HTTP API → Model on your GPU
↑ │
└── response ─────┘
Around it: access control, queues, limits, metrics, recoveryBut before any of that, we need a model that can answer one question. Let’s start there.
First, get the files.
Open a model repository on Hugging Face and look at its Files tab. There’s no running chatbot in that directory. There are files that a program can load.
- Configuration: dimensions, layer counts, and the model type. Transformers uses this to choose and construct the architecture; the implementation is Python code in the library.
- Weights: the learned numbers, here in
model.safetensors. Larger models often split them across several files. - Tokenizer files: how text becomes token IDs and back. For this text model, we need the matching tokenizer.
- Chat template: how user and assistant messages are arranged for this instruction model. It may live in tokenizer configuration or a separate file. A base model doesn’t necessarily have one.
We’ll use SmolLM2-135M-Instruct, a small model we can run on a CPU. Its answers will be limited; it’s useful here because we can inspect the whole process without renting a GPU. When choosing another model, read its model card, license, and hardware requirements first.
Download once. Load when you start.
This page works as a standalone exercise. To use the existing lesson 1 lab and continue through lessons 2–4, download the lab files. Unzip them, open a terminal in the inference-labs folder, create the environment, and run these commands. They use the same model and inference steps shown below.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python lessons/01-local-inference/download_model.py
python lessons/01-local-inference/run_model.pyIf you use that route, skip the standalone setup and file creation below; read on to understand what the scripts do.
To follow along, use Python 3.13 in a terminal. Create a folder for this exercise and run these commands inside it. The first download needs internet access and several hundred megabytes of disk space.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install torch==2.14.0 transformers==5.17.0 huggingface-hub==1.33.0Save this as download_model.py, then run python download_model.py. The revision pins the exact files used in this lesson.
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="HuggingFaceTB/SmolLM2-135M-Instruct",
revision="12fd25f77366fa6b3b4b768ec3050bf629380bac",
local_dir="models/smollm2",
allow_patterns=["*.json", "*.txt", "model.safetensors"],
)Now the files are in models/smollm2. Nothing is answering requests yet. Downloading puts bytes on disk. Loading constructs the model and brings its weights into memory. Inference uses that loaded model to compute output.
Make those files answer you.
Save the following as run_model.py in the same folder. Run python run_model.py, then type a short question such as “What is gravity?” Type quit to stop.
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "models/smollm2"
tokenizer = AutoTokenizer.from_pretrained(path, local_files_only=True)
model = AutoModelForCausalLM.from_pretrained(path, local_files_only=True)
while True:
prompt = input("You: ")
if prompt.strip().lower() == "quit":
break
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)
output = model.generate(**inputs, max_new_tokens=100)
new_tokens = output[0][inputs["input_ids"].shape[-1]:]
print("Model:", tokenizer.decode(new_tokens, skip_special_tokens=True))The two from_pretrained calls read local files. The model loads before the loop, so we keep using it for each new prompt. There’s no hosted generation API behind this code.
"What is gravity?"
↓ chat template: arrange the message and assistant prefix
↓ tokenizer: turn that text into token IDs
↓ model.generate: compute and select new tokens, repeatedly
↓ decode: turn the new token IDs into text
An answer in your terminalThe chat template supplies the format this model expects. We avoid adding special tokens a second time when tokenizing. Generation returns the input tokens along with the continuation, so we slice off the input before printing. max_new_tokens=100 caps the continuation; it doesn’t guarantee a complete or correct answer.
Try a second question. Then quit and start the program again. Notice which work happens once per process and which work happens once per prompt.
You send a second prompt.
What needs to happen again?
The Python process is still running, and your first response has finished. Choose the best explanation.
Keeping the weights in memory doesn’t keep a conversation. Each loop iteration builds a fresh message list. To include earlier turns, your application would need to store them and send them as part of the next input.
It answers. Is it a server yet?
Not yet. Another application can’t call this terminal loop. The next step is to keep the model loaded inside an HTTP service, accept a prompt in a request, and return the generated text in a response.
From foundation sample 2 · Your first model API
Which request runs the model?
The server has already loaded the model. Read these two requests and make a prediction before you check.
GET /healthPOST /generate
{"prompt": "Say hello."}Check your answer
Only POST /generate calls the model. GET /health returns a fixed status. It tells you the app can respond, but it doesn’t test generation speed or answer quality.
In the lab, you run both requests and inspect the code and metrics to see the difference.
Now send ten requests at once.
This is where the interesting work begins. Our loop handles one prompt at a time and waits for the whole answer. Putting an HTTP endpoint around it doesn’t automatically solve scheduling, memory limits, or overload.
- While one answer is generating, where do the other requests wait?
- Can we send tokens as they arrive instead of making the client wait?
- How much memory do the weights and each active request need?
- When does batching help, and what happens to latency?
- What should happen when a client disconnects or the queue fills?
- How do we measure useful throughput, answer quality, and cost?
We’ll start with a simple implementation so you can see what it does. Then we can measure its limits and understand why serving engines use techniques such as continuous batching and KV-cache management. Transformers also offers more advanced generation tools; our starting point deliberately keeps the request flow simple.
The same reasoning carries into GPU deployment, evaluation, and post-training: change something, measure what happened, and decide whether it helped the workload you actually care about.
You’ve got the first piece.
Files on disk became a model in memory, and that model produced an answer. Next we turn it into a service and start measuring it.
The first four labs are ready. Later GPU, evaluation, post-training, and serving material is planned for the course.
Lesson 2: Give your model an API →Further reading: Hugging Face chat templates and continuous batching.