When I looked at the latest version of SSMS 22, I found that there’s now an option to add your own model. You’re no longer required to use GitHub Copilot.
In an earlier blog, I posted about how you can add a model in Microsoft Foundry. So this time, I’m going to host one on my local machine.
Why would you need this?
There are a few reasons why you may want to run your model locally. You may want to do some experiments without wanting to wonder about the cost. Or, you may be worried about where your data goes when you write a prompt. The latter is something I’m hearing more; customers who want to make sure their data doesn’t go out of their control and influence.
In these situations, running your own model can be a solution. Now, Ollama isn’t a contender for GPT, Claude or Mistral for instance. These commercial models have the huge data centers to back them up, something that’s quite hard to achieve normally. Especially when you’ve just got a regular laptop to work with. So let’s see what we can do here!
Installing Ollama
Since Ollama is a major player in this area, I decided to give it a try.
irm https://ollama.com/install.ps1 | iex
This is the command to install Ollama.

This took about 5 minutes on my machine. But the tool is nothing without a model.
Load a model
Next, load the model.
ollama run llama3
This triggered the download of the default model.

In this step, the model is downloaded and made available on your machine.
After about 5 minutes, you’ll see this.

Let’s see what this model knows
When you see this screen, the CLI interface is working. Let’s see how it replies to some ego-centric questions.

Well, I know my place in the universe again. Never mind, let’s move on to more serious stuff, like getting it to work in SSMS.
SSMS
If you want to use this in SSMS, you have to change a setting first.

Note the blue information; restart the application.
After the restart, when you open the Chat, you can open the Agent dropdown and go to Manage agents

This opens a new window where you can select Models. At the bottom of that window, you can find the option to add a model provider.

I can now add a provider:

When I select that one, I can see the Endpoint URL (localhost), and I have to acknowledge that the data is sent over HTTP, not HTTPS. In a very theoretical situation, someone listening in on my internal traffic could eavesdrop. Also, note that you’re not limited to Ollama. You can also use Microsoft Foundry, Anthropic, OpenAI or a custom endpoint. It really is up to you.

Click Add, and you’ll see it in the list as another model. Let’s start a new chat and see what happens.
New Chat
To make sure you can use the model, close the chat window and reopen it. Next, select your local model.

The first question, like my ego check, didn’t work well.

And yes, the SSMS Sidecar chat relies on tools. And, this model clearly doesn’t support tools. I have now learned that that’s one of the differences.
So, I decided to move to Mistral.

And when I manage my models, I can now see that it supports tools.

I also added the DeepSeek cloud model to check whether the issue was on my side.
Let’s see what the model can do, but also add some clarity on what it’s running on.
Putting it to work
Now, before I show the results, let me be very open and show you the machine I’m working with.



You could say it’s a high-end business laptop, not the most suitable option to run an AI model, maybe.
The Mistral model I downloaded isn’t the largest one; with 4 GB, you could even say it’s a small one.
So, the result should be treated as such; it’s no match for the big models.
Generate a query

This looks like something that could work, so let’s insert it at the Cursor.
Oh dear

Cool, this hallucinated like mad. It got a few things correct and then went completely off the rails.
I asked it to try again; all I could do was laugh.

It’s clear that this model isn’t suited for SQL. So, let’s clear that one up and move on.
Remove the bad models
To keep your environment tidy, you can remove the models that you no longer need, or want.
ollama rm modelname
A simple line of code.

The removel is near-instant on my machine. It freed up a lot of memory as well.
So, let’s see if the largest model can do something.
Does size matter?
As a last test, I decided to pick the Llama 3 70b model. This one is 40 GB, and maybe this helps more in understanding what it has to do with SQL.

Downloading this one took me about 40 minutes. And the result?

It doesn’t understand tool calling, apparently.
I then chose to use gpt-oss, a GPT based model that should help with reasoning.
Well…

So… hallucinations galore!
Is it the prompt?
To verify it wasn’t me, but it’s the model that’s being silly, I fed the same prompt to Claude Fable 5.1 from my GitHub Copilot subscription. The model was chosen because it was at the top of the list.
It came up with a working query and, though technically correct, provided the wrong result.

The schema’s, tables, and columns are all valid.

But this isn’t how to calculate the number of shifts.
Now, the above part might tell you that it’s all bad. As far as my testing goes, yes. But, I didn’t try all the models that are available. Also, I’m not an AI genius, I may have missed something that’s obvious to you. Don’t forget to let me know!
Take aways
For the models I’ve tested, none were able to return valid SQL, or even understand that they can connect to my local instance. I may have missed some configuration, though I suspect that it’s got more to do with either the capabilities and training of the model.
The Ollama website is a bit hazy when it comes to which models have certain capabilities. You really need to try them out.
Also, be aware that some models will route you to a cloud environment; the exact scenario that we’d like to prevent.
Maybe my laptop has something to with this too, it’s not the smartest kid on the block
The models that didn’t work:
- gpt-oss
- gemma3
- llama3:70b
- mistral
- llama3
Finding the correct model that can actually understand SQL Server proved harder than expected. Maybe someone can come up with one that does just that; SQL Server.
Knowing how to run Ollama is good to know. If you want to experiment with the Vector data type, this can be a very good starting point. It’s free, your data stays on your machine, and you can experiment without a bill going through the roof. Though, depending on your PC and the model you run, your PC may want to go through the roof. Or melt down the floor.
The most important Ollama commands
- ollama serve: Start Ollama
- ollama run: run a model; if it’s not on your machine, it will pull it from the registery
- ollama stop: stop a running model
- ollama rm: remove a model
- ollama list: show all the models you have
- ollama ps: show the running models