This is part of the original four-sample CPU sequence, separate from the redesigned nine-week track. Sample numbers are not week numbers.
Here’s the result we want.
POST /generate
{"prompt": "Say hello.", "max_new_tokens": 10}
HTTP 200
{"response": "Hello! How can I help you?"}In lesson 1, you typed into the same terminal that ran the model. That works for you sitting at your laptop. But what if a web application needs an answer? It needs an address to call, a format for its request, and a response it can read.
That’s the job of our API. We’ll keep the inference steps and replace terminal input with an HTTP request. The example answer above shows the response shape; your small model may produce different words.
Use the same files for all four lessons.
Download the lab source, unzip it, and open a terminal inside the resulting inference-labs folder. This is the project root: you should see requirements.txt and lessons. The download contains code, not model weights. The walkthrough uses Python 3.13 on macOS.
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python lessons/01-local-inference/download_model.pyIf you already have the course repository and a working environment, use that folder instead. If you followed the standalone lesson 1 example, copy its complete models/smollm2 folder into this project’s models folder to reuse the download. Otherwise run the downloader above.
A server is a process that keeps listening.
Stop the lesson 1 loop with quit. In your server terminal, run:
source .venv/bin/activate
python -m uvicorn server:app --app-dir lessons/02-simple-server --host 127.0.0.1 --port 8000Wait for Application startup complete. Uvicorn is the HTTP server. server:app tells it to import the app object from this lesson’s server.py. FastAPI matches requests to Python functions. Transformers does the model computation.
Terminal 2: client → HTTP request → Uvicorn → FastAPI route
↓
loaded CPU model
↓
Terminal 2: client ← JSON response ← generated textThe server’s lifespan function loads the tokenizer and model before it begins serving. Each request reuses them. We use one serving process here; adding workers would also mean additional model copies and memory use. FastAPI’s lifespan documentation explains this startup pattern.
Ask whether it’s alive. Then ask it to think.
curl --connect-timeout 5 --max-time 5 http://127.0.0.1:8000/health
curl --connect-timeout 5 --max-time 60 \
http://127.0.0.1:8000/generate \
-H "Content-Type: application/json" \
-d '{"prompt":"Give me three names for a coffee shop.","max_new_tokens":50}'The first call returns {"status":"ok"}. It doesn’t generate anything. The second sends JSON to /generate, waits for generation, and returns a response string. Watch the server terminal: you should see both requests.
127.0.0.1 means this computer. Port 8000 identifies the listening service. The path selects the operation. Curl’s -d sends a body and makes this a POST request; the content-type header tells the server the body is JSON.
Read the route without losing the model.
Open lessons/02-simple-server/server.py. Start at GenerateRequest: it requires a string prompt and defaults to 100 new tokens. Then follow generate. It formats a message, tokenizes it, calls model.generate, slices off the input tokens, and decodes the continuation. You’ve already seen that sequence.
The new part is the boundary: FastAPI turns incoming JSON into a request object and the returned dictionary into JSON. Measurement helpers live in observability.py so you can follow the inference code first.
The health check works. Have we proved generation works?
Check your reasoning
No. We’ve proved the application can answer that endpoint. A real generation request tests a different path. Even a successful answer doesn’t prove good quality or enough capacity for your users.
Send a request the server can’t accept.
curl -i --connect-timeout 5 --max-time 5 \
http://127.0.0.1:8000/generate \
-H "Content-Type: application/json" -d '{}'Expect 422 with validation details. The request reached the application, but it did not enter the generation handler. This is different from a connection refusal, where there may be no listener at all. Ordinary curl can exit successfully even for an HTTP error; -i makes the HTTP status visible.
Visit http://127.0.0.1:8000/docs in your browser. Expand POST /generate, choose Try it out, and execute a request. That browser interface is another client of the same API.
We have an API. Now who can use it?
For now, only programs on this computer can reach this loopback listener. It has no authentication, bounded request size, or deliberate queueing policy. Keep it local. In the next lesson, we’ll call it from three different clients, measure what they did, and optionally move one client to another computer.