Install local AI

This beginner setup uses AnythingLLM Desktop on Windows 11. The aim is not to tune an AI laboratory. It is to get one useful local document workflow working, know where its data live, and be able to prove that the important processing remains local.

Before you download anything

Check Windows Settings → System → About and note the installed RAM. You do not need a specialist gaming computer, but model size matters.

A sensible starting point is:

Computer	Suggested local model	Model file	Practical guidance
16 GB RAM	Qwen3-8B-GGUF Q4_K_M	about 5.03 GB	Good starting model. Expect slower responses if running mainly on CPU. Keep context moderate so Windows has memory left.
32 GB RAM	Qwen3-14B-GGUF Q4_K_M	about 9 GB	Better reasoning and extraction while remaining realistic on a normal desktop. CPU inference can still be slow.
32 GB RAM + about 8 GB NVIDIA VRAM	Qwen3-8B-GGUF Q4_K_M	about 5.03 GB	The smaller model is often the more pleasant choice because its weights fit comfortably within an 8 GB GPU, leaving room for runtime overhead. Try the 14B model later if quality matters more than speed.

These are working recommendations, not minimum-memory guarantees. Context size, GPU offload, other applications and the particular runtime all change memory use.

Qwen3 is useful here because the current 8B and 14B instruction models have broad multilingual support and are available as official GGUF files. That does not mean te reo Māori accuracy is guaranteed. Preserve source text and check Māori spelling rather than allowing the model to silently “correct” it.

Official model pages:

• Qwen3-8B-GGUF — choose Qwen3-8B-Q4_K_M.gguf, about 5.03 GB.
• Qwen3-14B-GGUF — choose Qwen3-14B-Q4_K_M.gguf, about 9 GB.

Install AnythingLLM Desktop

1. Download AnythingLLM Desktop from the official site. The current Windows x64 installer is also available directly from AnythingLLM.
2. Run the downloaded .exe.
3. Install it for the current user, not all users. This matches the vendor's current Windows guidance.
4. Open AnythingLLM Desktop.
5. Choose a local model option during the setup rather than supplying an OpenAI, Anthropic, Google or other cloud API key.

AnythingLLM targets Windows 11 and currently includes built-in local LLM support powered through Ollama, so the beginner path does not require a separate terminal-based Ollama installation.

Pick the language model

The easiest first test is to allow AnythingLLM to install a suitable local model for your machine. Once the application is working, you can move to an exact model such as Qwen3-8B or Qwen3-14B through its supported local-provider options.

If you use LM Studio as the model provider:

1. Install LM Studio.
2. Search for the official Qwen repository and download the Q4_K_M file that fits your computer.
3. Load the model locally.
4. Start LM Studio's local server.
5. In AnythingLLM, select LM Studio as the local LLM provider and point it at the local server.

This is a useful arrangement when you want AnythingLLM's document workspaces but LM Studio's clearer model management.

Use a multilingual embedding model for mixed collections

The embedding model is not the model that writes the final answer. It turns pieces of document text into numeric representations so the software can retrieve passages related to your question.

AnythingLLM's lightweight default embedder is all-MiniLM-L6-v2, which its documentation describes as primarily intended for English documents. For collections containing both English and te reo Māori, a stronger current option is Qwen3-Embedding-0.6B-GGUF Q8_0.

The official Qwen model card describes support for more than 100 languages, a 32K context length and multilingual/cross-lingual retrieval. The Q8_0 GGUF is about 639 MB. This makes it much more practical than using a multi-billion-parameter embedding model on an ordinary desktop.

• Qwen3-Embedding-0.6B-GGUF
• AnythingLLM LM Studio embedder setup
• AnythingLLM Ollama embedder setup

For the very first test you can keep the AnythingLLM default embedder so there are fewer moving parts. Before indexing an important mixed-language collection, switch to the multilingual embedder and then re-index the documents. Embeddings made by different models are not interchangeable.

Find the local data

On a standard Windows install, AnythingLLM Desktop stores its data under:

C:\Users\<your-user-name>\AppData\Roaming\anythingllm-desktop\storage

The current vendor documentation identifies these useful parts:

storage\
  documents\      parsed document content
  lancedb\        local vector database
  vector-cache\   cached document embeddings
  models\         locally stored LLM and embedding model files
  anythingllm.db  application database
  direct-uploads\ files dropped directly into chats
  logs\           application logs

Do not delete these folders while the application is running. The point of looking is to understand what the application means by local and what needs to be considered when a workspace must later be removed.

Check that the selected model is local

Open the AnythingLLM settings and inspect the language-model provider. A local setup should identify a local provider such as AnythingLLM's built-in local model, Ollama or LM Studio. A provider such as OpenAI, Anthropic, Gemini, OpenRouter or another hosted API is not local simply because AnythingLLM itself is installed on your computer.

Do the same for the embedding provider. If you want the document index created locally, the embedding model must also be local.

Perform the Wi-Fi test

After the model and embedding files are fully downloaded:

1. Create a small test workspace.
2. Add a public text PDF.
3. Ask a question and confirm you can open or identify the supporting source.
4. Close web pages that could confuse the test.
5. Turn off Wi-Fi or disconnect the network cable.
6. Ask a different question about the same PDF.
7. Confirm that document retrieval and the answer still work.

If it fails, check whether the selected language model, embedder, OCR service, web tool or plugin requires an external connection.

This is more useful than simply trusting a “private” or “local” label.

Keep the first workspace ordinary

Use a public report for the first setup. Do not make unpublished whānau material the file that discovers a configuration mistake. Once the workflow is understood and the local test passes, decide what information is appropriate for the kaupapa using your own governance and tikanga.

Next, search your own documents.

Sources

• AnythingLLM Windows installation
• AnythingLLM desktop system requirements
• AnythingLLM desktop storage
• LM Studio system requirements
• LM Studio offline operation
• Qwen3-8B official GGUF
• Qwen3-14B official GGUF
• Qwen3-Embedding-0.6B official model

Last verified: 22 August 2026

Continue after installation

Once the software opens successfully, move to Search your documents rather than spending time tuning settings without a research question. If you are still deciding between applications, return to Choose software.