Do You Need a Bigger Model, or Better Retrieval?

The date was not missing.

The model had already listed that file.

Then it opened the wrong contract and told us the date did not exist.

Nine calls. Six files. One wrong contract. A confident no.

You know this one. You uploaded the file. You can see it in the knowledge base. You can open it in another tab, scroll to the line, and read it with your own eyes. The chat says it isn't there.

So you rephrase the question. Same answer. You upload the file again. Same answer. And the thought arrives on its own: this thing is broken.

From where you are sitting, that is a fair conclusion. Usually, though, something smaller is in the way. Maybe the import didn't stick. Maybe knowledge bases only half-work. Maybe this is just what RAG is like. Maybe the model is too small, maybe this 8B needed to be a 70B, maybe the local setup was the mistake and this wants a frontier model behind an API key.

Every one of those guesses costs a download, a GPU, or a bill, and not one of them has been checked yet. The check is free: which text actually reached the chat?

Here is the answer up front. If the right passage never reached the model, the problem is retrieval, and a bigger model will only read the same wrong chunk more fluently. If the passage arrived and the answer is still wrong, the problem is the model.

If the model is picking its own searches, that is the model too, because choosing a query is a capability rather than a setting. And sometimes finding the line was only step one. If you need to add up a column, check one file against another, or hand someone a finished report, no amount of searching gets you there. That job needs a shell, not a better index.

Telling them apart takes a minute. Open the stored text, open the Sources row, and if tools ran, open the tool calls. Here is what each one shows you.

What the model actually did#

We ran this on a stock Open WebUI instance with six company files attached as a knowledge base: an operations handbook, an office lease, a cloud hosting agreement, an incident report, one quarter's planning notes, and the year's operating expenses export. About 1,500 words in total. The chat model was Qwen3-30B-A3B-Instruct, 30B parameters with roughly 3B of them active per token. The question: when is notice due to stop the analytics subscription renewing?

Watch what it did. Three of its nine calls searched filenames for a fact that is not a filename. One grepped a four-phrase alternation against a file whose column is headed Non-renewal notice by. Then it opened the cloud hosting agreement, which is the wrong contract, and reported that no document in the knowledge base contains the date.

Nine tool calls, then a confident negative. The unfolded log shows every query the model chose

One message later, the same model on the same files quoted the exact line. The follow-up named no answer, only a method: search the whole knowledge base instead of one file, grep for a single plain word, and run the semantic query too.

That's where most bad answers over your own documents get decided, before the model composes a word. Either the text was mangled on its way out of your files, or the wrong search ran against it. Both are visible. Both have fixes you can apply this afternoon, and one of them no model swap can touch.

Open WebUI gives a model four ways to reach your files: the retrieval pipeline, a set of search tools, kb_exec as a shell over the knowledge base, and Open Terminal as a shell over a real working directory. Three of them hand back chunks, lines, or file text. Open Terminal does all of that and keeps going. It reads the raw files, adds the numbers up, checks its own answer, and hands you a chart, a report, or a spreadsheet.

Keep an eye on that pipeline. It's the one that scores perfect at the end.

Start with the extracted text#

When you add a file to a knowledge base, it rides a conveyor belt. The industry calls that machinery Retrieval Augmented Generation (RAG). Text gets extracted from the file, split into chunks, and turned into embeddings, the numerical fingerprints that let similar meanings be found without shared words. Ask a question and it gets embedded the same way, then the closest chunks come back. The chat model reads those chunks. It never reads your documents.

The first stage decides everything after it. Mangle extraction and nothing downstream recovers, because the chunks were cut from text that was already wrong. You know these failures: the scanned PDF that extracts as empty pages, the table flattened into word soup, the two-column layout read straight across. Every one of them happens before a single embedding exists. Handing the files to the model directly doesn't route around it either, because the search tools read the same extracted text the knowledge base stored. One bad extraction breaks every path into your files.

So before you touch the chat model, go look at what the extractor produced. You can read it yourself. Open Workspace, then Knowledge, open the knowledge base, and click the file.

The handbook as the knowledge base stored it. Every path to your files reads this text

If what you see is wrong, swap the stage out. The Content Extraction Engine setting, in Admin Settings under Documents, points at dedicated extraction services instead, and two of its options, Tika and Docling, run self-hosted beside the instance. Clean digital text is fine on the local default. Scans, tables, and complex layouts are where a dedicated extractor earns its keep, and it asks nothing of the chat model.

When the model picks its own searches#

Extraction decides what text exists. Next question: who decides which of it reaches the model?

With native function calling on, a model in Open WebUI gets tools alongside whatever chunks retrieval picked, so it can search on its own behalf. That's the default in v0.11.1, so an untouched model already works this way. The Function Calling option that controls it sits under Advanced Params when you edit the model in Workspace under Models, where Legacy is the opt-out for models that can't drive tools.

A model searching on its own behalf is what people mean by agentic search, and attaching a knowledge base to the chat puts six tools in its hands. One runs the same semantic search the conveyor belt runs. One greps for exact text across the files and returns matching lines with their line numbers. One searches filenames. One lists what a knowledge base holds. Two open a file and read it back. The pipeline stops being the path and becomes one option among several, and the model picks.

Picking is exactly where our test failed. The index held the answer. The files were fine. Nothing was broken. All nine of those calls were queries the model chose, and every one of them was plausible. Searching filenames is a reasonable opening move in a knowledge base you've never seen. non-renewal notice date is a reasonable thing to grep for when someone asks about a non-renewal notice date. It just isn't what the file says, and grep doesn't care how reasonable your pattern was.

The fix took one message, and the point was not a magic word. It was forcing the model to search wider, use simpler terms, and cross-check exact search against semantic search.

The same model, the same files, one message later. Two calls, and the date comes back cited to the handbook

That's the trade you're making. A model that can look for itself gains abilities no fixed pipeline has. It can quote a line exactly, follow a reference from one file into another, and notice a result is empty and try another way. In exchange you inherit a new kind of failure, and it's a nasty one. The model decides what to search for, and a search that returns nothing looks exactly like a fact that doesn't exist.

Giving the model a shell#

There's a sharper way to hand a model the same files, and it'll look familiar to anyone who lives in a terminal.

Setting ENABLE_KB_EXEC=true swaps the browsing tools for kb_exec, a single tool that speaks filesystem. The model runs ls to list a knowledge base and cat -n to read a file with line numbers. It runs head and tail to take the first or last lines, and grep to search, with the flags you'd expect from a shell. -r searches every file in the base, -i ignores capitalisation. Output is capped, and pipes are how the model stays under the cap.

All of it stays sandboxed to the knowledge bases the user could already open. The semantic query tool stays alongside kb_exec, so meaning-based search is still one call away when you don't know the exact words.

The bet is simple. Models have seen a lifetime of shell transcripts, so one interface they already know should beat several bespoke tools they have to learn from a schema. Given the right commands, it delivers exactly what a terminal user expects.

Given the case-insensitive flag, two kb_exec calls are enough: a recursive grep and a numbered read, then the row quoted with its notice date

One missing flag decided it.

Left to itself, the model reached for grep -r "analytics" . Recursive flag on, case-insensitive flag off. The vendor in the handbook is written Example Analytics, with a capital A. The only lowercase analytics in those six files sits in the planning notes, in a sentence that tells you to check the handbook for the deadline. So the model read the planning notes, found a pointer instead of a date, stopped there rather than following the pointer to the handbook, and answered that no date was given.

Same failure as the multi-word grep. New syntax.

grep -ri was right there. Nothing about the interface failed. A stronger model is more likely to reach for -i, and to follow the pointer into the handbook instead of stopping at it. Those are judgment calls, and a better model makes better ones. That is the swap you make on every agentic path. Your retrieval problem quietly becomes a model problem.

ENABLE_KB_EXEC ships off by default, for the same reason the retrieval defaults start low. Putting a sharper instrument in a model's hands is a decision an administrator should make deliberately.

Giving the model a directory#

Every path so far hands the model text and stops. But you wanted that notice date because something depends on it. You get the date, then you open the export yourself, build the chart, write the summary, and paste the figure into a report for someone else. The answer was never the deliverable. It was step one, and every step after it was yours.

So we gave the model a directory instead.

The pipeline returns chunks. The search tools return lines. kb_exec returns file contents. All three stop at text that you then have to act on. Open Terminal gives the model a working directory, a runtime, and a place to put the result, so it can do the acting. It also reaches further. kb_exec can only see knowledge bases the user could already open. A directory has whatever you put in it, including files that never went through extraction at all.

Finishing the job is the obvious win. What people miss is that it also searches better. kb_exec gives the model a fixed set of commands and a cap on the output. A directory gives it the real ones. It can grep with any flag it likes, print line numbers with -n and quote the file back at that line, pipe through sort and uniq to find the row it wants, and read anything in the directory without waiting on an extractor. Ask which file and which line a clause sits on and it can show you both.

And when a question isn't a search at all, it still has Python. Anything you'd reach for a script to answer, it can answer, because it can write the script.

With an Open Terminal server connected, you can point a chat at a real working directory. The server is a single container that runs beside your instance, and the connection form lives in Settings under Integrations. Set the connection's Chat Uploads to Filesystem, and a file dropped into the message box lands in that directory instead of in the conversation. That switch is new in v0.11.1 of Open WebUI. Before it, an attached file went to Open WebUI's own storage, and the shell the model was standing in could never touch it.

The Edit Terminal Connection form, Chat Uploads set to Filesystem

The default image ships Python with pandas, matplotlib, python-docx, and openpyxl, so a dropped spreadsheet can be aggregated, charted, and handed back as a file.

Start with the plainest run. We dropped in the operating expenses export, the same CSV the knowledge base holds, and asked for a bar chart of spending by category. No extraction, no chunks, no embeddings. The model read the raw file off disk and returned a picture, which makes this the one path a mangled extraction can't spoil.

The expenses export dropped into the chat lands in the terminal's working directory. The model writes a script, runs it, and the chart comes back as a card in the reply

Retrieval can't do that. Its output is text by construction, so the numbers arrive as a passage rather than a picture.

The second run asks for something no line in the file states. Which cost category rose in every single month is a fact the data implies rather than records, so it has to be computed. Before answering, the model wrote a pandas script into the working directory and ran it, comparing each month to the one before.

The model verifies the trend with its own pandas check before answering, then charts it

The category it found is software subscriptions. That's the same line the operations handbook attributes to SUB-2026-114, the contract whose notice date the model couldn't find at the start. The handbook claims that contract is the only one growing. The terminal reached the same conclusion from the raw export, without reading the handbook at all. One document asserts it, the other proves it.

Retrieval hands a model a chunk of text and hopes it reads the numbers correctly. Here the model computed the answer and checked it against the file before speaking.

In an earlier take, it skipped the script and answered from the text anyway. It invented three of the six totals. Payroll came back as 60,200. The file says 68,600.

When it ran the code, every figure matched. That is the gap no amount of retrieval tuning closes. A model reading numbers out of a passage is easier to trip up, and a model running sum() over the column is not.

The last two runs end somewhere the knowledge base path can't reach. We asked for a report, and then for a spreadsheet.

The same export turned into expense-report-2026.docx, built with python-docx and handed back for download

The same export as a two-sheet XLSX summary. The spreadsheet renders in the reply, and one click flips it to the second sheet

A knowledge base answer ends at a paragraph, and that is where your afternoon starts. This one ended at a chart, a checked trend, a DOCX report, and a two-sheet XLSX you can send to someone. Same six files, same small model. The difference is that the job finished inside the chat.

Which path won#

Four paths, each reaching further than the last. So the terminal should have the best record and the pipeline the worst.

The pipeline went four for four. Every agentic path fumbled at least once.

We asked the same question four ways, holding the model, the files, and the wording constant. Six small files, and uneven run counts: four attempts on the pipeline, one or two on each of the others. Read it as a demonstration, not a benchmark. What matters is why the pipeline won, and that part holds at any size.

With no search tools in play, the retrieval pipeline answered correctly on all four attempts. It quoted the vendor row and cited the handbook at 79.83% relevance. The browsing tools took nine calls and named the wrong contract. The default configuration, which hands a model both the tools and the chunks, took seventeen calls and named the wrong contract too. Given the shell, the model spent a run answering from a pointer instead of a date.

The retrieval path answering the same question, with the citation opened on the chunk it used

The pipeline won because it doesn't choose.

A fixed retrieval stage can't pick the wrong file, can't run a case-sensitive grep, and can't decide that an empty result means a fact does not exist. Every rung of the ladder hands the model more reach.

Reach is not accuracy.

What the terminal earns at the top of that ladder is everything the other three do, plus what none of them can. It inspects the raw file, runs code against it, checks its own answer, and produces the thing you actually needed. It also answers the questions searching can't. You cannot find a total by searching for it. Someone has to add the numbers up.

Which means the pipeline is worth tuning rather than escaping, and every setting sits in Admin Settings under Documents.

The shipped defaults are deliberately small: chunks of 1,000 characters with 100 of overlap, the top 3 chunks per question, and a small embedding model running on the instance itself. Roughly 3,000 characters of context per question. That is on purpose. Open WebUI runs on everything from a spare laptop to a rack, and defaults tuned for the biggest deployments would quietly break the smallest ones.

The Retrieval section on a stock instance. Hybrid Search off, Top K at 3

Raising those limits is a settings pass, and it has an order.

Change one thing at a time, ask the same question again, and judge the setting by the opened Sources row rather than by the final answer alone.

The same section with Hybrid Search on. The reranker fields and the Relevance Threshold appear with the toggle

When a collection is small enough to fit in a model's context, skip retrieval altogether. A file attached in chat can be flipped to Using Entire Document, and the Full Context Mode toggle in the same Documents settings does it for the whole instance. Search exists because context runs out. Where it doesn't, the best retrieval is none.

A file attached in chat, flipped to Using Entire Document. The model gets the whole file instead of chunks

Not every bad answer is a tuning problem. Swap the embedding model after documents are indexed and the stored embeddings stop matching new questions, because embeddings only compare against embeddings from the same embedding model. Retrieval quietly returns junk and raises no error. One click on Reindex Knowledge and Memory Vectors at the bottom of the same tab fixes it. It rebuilds every knowledge base and each per-file index, so give it time, and budget for it if your embedding model bills per call. If a knowledge base that used to answer well went strange after a settings change, reindex before blaming the chat model.

One boundary runs the other way. Across ten thousand documents an index does work no shell loop can match, and embeddings still find the paragraph whose words you can't name. We tested six files, not ten thousand. The terminal shines where the files are few, fresh, and specific. None of these paths retires the others.

What to fix, in order#

So the next time a knowledge base disappoints you, walk these steps in this order.

1. Read what extraction produced. Open Workspace, then Knowledge, open the knowledge base, and click the file. If the text arrived mangled, that's your problem and no model swap will fix it. Switch the Content Extraction Engine in Admin Settings under Documents, then delete the mangled file and upload it again, because a reindex reuses the old text.
2. Check which searches ran. The tool-call log sits above the reply, and a click unfolds each call. If the wrong search fired, or none did, that's where the problem lives. A sharper instruction fixes one conversation.
3. Decide whether the job is retrieval or workspace work. ENABLE_KB_EXEC=true gives the model one search interface instead of six tools. Open Terminal gives it a working directory when the job is to inspect raw files, calculate, verify, chart, or produce a file rather than find a line.
4. Tune the pipeline where it fell short. If Sources missed the answer, raise Top K. If exact names or table labels matter, turn on Hybrid Search. If more chunks add noise, use a Relevance Threshold or a reranker. If the answer is split across chunk boundaries, adjust chunk size or overlap. If a higher Top K changed nothing at all, check the model's context length, because chunks that overflow the window never reach the model. If answers went strange after an embedding model change, reindex first.
5. Test the chat model last, and with evidence. The Sources row under a reply shows what retrieval delivered, and a click opens each passage. The unfolded tool calls from step 2 show what every search returned.

That last step answers the title. If the right passage never reached the model, fix retrieval, because a bigger model will only read the same wrong chunk more fluently. If the right passage reached the model and the answer is still wrong, change the model. If finding the line was only ever step one, neither one is your bottleneck, and what you want is a directory.

That middle case is real. The number comes back garbled, or a line you needed verbatim comes back paraphrased. The chat model failed with the answer in hand, and parameters fix that. Once you can see the passage sitting in the Sources row, pulling a stronger model stops being a guess and becomes a change you can defend.

There's a second reason to reach for a stronger model, and our test walked into it twice. Hand a model the search tools and it has to choose the query, choose the flags, and decide when to stop. Every one of those is a capability rather than a setting. A stronger model is likelier to grep -i, and to follow the pointer into the second file. If you're committed to an agentic path, that's a real reason to run one.

A fixed pipeline answered correctly four times out of four while driven by the same small model that fumbled the question nine ways, because it never asked that model to choose. Every failure in this post came from a choice, and the choices only exist once the model is the one doing the searching.

The date came from retrieval. The work came from the directory.