Run Local LLMs in C# with Ollama

You want a language model on your machine.
C# talks to it.
Same IChatClient interface you will later point at OpenAI or Azure.
Zero API keys. Zero Python. Zero Azure.

---

What you need

• .NET 8+
• Ollama (install it)
• A few GB free
• phi3:mini (fits in 8 GB RAM)
• Any editor

---

1. Fire up the model

ollama serve          # if it complains the port is taken, leave it alone
ollama pull phi3:mini
ollama run phi3:mini  # sanity check

That’s it. The model lives on your disk now.

---

2. One-line project

dotnet new console -o LocalLlm
cd LocalLlm
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp

---

3. One-shot request

using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient client = new OllamaApiClient(
    new Uri("http://localhost:11434/"),
    "phi3:mini");

var response = await client.GetResponseAsync(
    "Explain async/await in C# in three short sentences.");

Console.WriteLine(response);

First call is slow. Ollama is loading the model into memory. After that it flies.

---

4. Stream the tokens

await foreach (var update in client.GetStreamingResponseAsync(
    "Explain async/await in C# in three short sentences."))
{
    Console.Write(update.Text);
}

No waiting for the full answer. Tokens appear as they are born.

---

5. Keep a real conversation

var history = new List<ChatMessage>();

history.Add(new(ChatRole.User, "Explain async/await in C# in three short sentences."));

var first = new StringBuilder();
await foreach (var u in client.GetStreamingResponseAsync(history))
{
    Console.Write(u.Text);
    first.Append(u.Text);
}
history.Add(new(ChatRole.Assistant, first.ToString()));

history.Add(new(ChatRole.User, "Now show a tiny C# method that uses that."));

var second = new StringBuilder();
await foreach (var u in client.GetStreamingResponseAsync(history))
{
    Console.Write(u.Text);
    second.Append(u.Text);
}
history.Add(new(ChatRole.Assistant, second.ToString()));

The model only remembers what you send back. History is the memory.

---

When things break

---

Bonus: Run Ollama in Docker

Want isolation? Put Ollama in a container.

CPU only

docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

NVIDIA GPU

docker run -d \
  --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

Pull the model inside the container:

docker exec -it ollama ollama pull phi3:mini

Your C# code stays exactly the same.
It still talks to http://localhost:11434.

Optional docker-compose.yml

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    # Uncomment for NVIDIA GPU
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  ollama:

Then:

docker compose up -d
docker exec -it ollama ollama pull phi3:mini

Same endpoint. Same C# client. Cleaner isolation.

---

That’s the whole loop:
local model → standard interface → stream or chat → ship.

No cloud. No bill. Just C# and a model that lives on your machine (or in a container).