WWDC Quick Look đź’“ By SwiftGGTeam
中文 EN 日本語
Run local agentic AI on the Mac using MLX

Run local agentic AI on the Mac using MLX

MLXAgentic AILocal LLMDistributed InferenceM5 Neural Accelerator
Watch original video

Highlight

Through the MLX framework and MLX-LM Server, Apple provides a four-layer local agentic AI stack that lets the Mac run complete agentic workflows without a network connection or API key. The M5 Neural Accelerator improves prompt-processing speed to four times that of M4, while Thunderbolt RDMA support makes distributed inference across multiple Macs up to three times faster.

Core Content

From Chat to Agents: The Loop Is the Bottleneck

The traditional way to interact with an LLM is simple: you send one message, and the model replies with text. If you need to run commands, modify files, or call APIs, you do all of that manually. (00:32)

Agents change this pattern. After receiving a task, the agent first asks the model what to do next, then calls tools to execute that step, such as running commands, reading files, or calling APIs. It observes the result, feeds that result back into the model, and enters the next decision cycle. This loop continues until the task is complete. (00:45)

When that loop runs in the cloud, every call crosses the network, API costs accumulate quickly, and sensitive data leaves the local machine. Apple’s solution is to move the entire loop onto the Mac: data stays on the machine, the agent is available anytime, and usage cost drops to zero. (01:10)

The Four-Layer Local Agentic Stack

Apple defines a four-layer architecture, from low-level hardware acceleration to agent tooling at the top:

Layer 1: MLX. An open-source array-computation framework designed for Apple silicon. It handles low-level computation, Metal acceleration, and memory management. (02:49)

Layer 2: MLX-LM. Provides model loading, running, quantization, and fine-tuning. It supports thousands of models on HuggingFace and provides both CLI tools and Python APIs. (03:06)

Layer 3: MLX-LM Server. An OpenAI-compatible HTTP server that exposes local models through a standard API. It supports structured tool calling and reasoning models, so it can directly replace cloud LLM APIs. (03:28)

Layer 4: Agent. Any framework or tool that supports the OpenAI Chat Completions protocol can connect to it: Xcode, OpenCode, Pi Agent, custom scripts, and more. Because the interface is standardized, the agent does not know or care whether the model is running locally or in the cloud. (03:53)

This stack is already widely adopted. Popular tools such as Ollama, LM Studio, and vLLM all use MLX underneath. (04:13)

Details

Build a Local Agent in Three Steps

It only takes three commands to go from zero to a running local agent. (04:36)

Step 1: Install MLX-LM

pip install mlx-lm

Step 2: Start the local server

mlx_lm.server --model mlx-community/Qwen-3.5-4B-8bit

Use a small model first to verify the environment. After the server starts, it listens on a local port, loads the model, and accepts requests.

Step 3: Test with curl

curl -X POST \
  http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default_model","messages":[{"role":"user","content":"Hello!"}]}'

Key points:

  • pip install mlx-lm installs all required dependencies in one step
  • --model specifies the model ID on HuggingFace, and the first run downloads it automatically
  • The interface is fully compatible with OpenAI /v1/chat/completions, so existing code can switch to local execution by changing the URL

Configure OpenCode to Use Local MLX

Using OpenCode as an example, configure the agent to point at the local server. (05:18)

{
  "$schema": "https://opencode.ai/config.json",
  "model": "mlx/default_model",
  "small_model": "mlx/default_model",
  "provider": {
    "mlx": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "MLX (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "default_model": {
          "name": "Default MLX Model"
        }
      }
    }
  }
}

Key points:

  • baseURL points to the local MLX-LM Server address
  • The npm field specifies the OpenAI-compatible SDK adapter
  • Both model and small_model point to the local model, ensuring every conversation and tool call goes through the local model
  • After configuration, every agent interaction and every tool call is processed through the local model

M5 Neural Accelerator: Four Times Faster Prompt Processing

Agentic workflows have a special bottleneck: after every tool result, the model must process the full context again before it can enter the next round of reasoning. A typical agent session may contain hundreds of thousands of tokens, and most of the work is prompt processing, or prefill, rather than generating new text. (05:51)

The M5 chip introduces a dedicated Neural Accelerator, and MLX can call it directly for matrix multiplication. Matrix multiplication on M5 is four times faster than on M4. Combined with MLX’s optimized multiplication and attention kernels, prompt processing improves by nearly the same proportion. (06:12)

That means an agent can read codebases and process tool results almost four times faster. Developers do not need extra parameters or code changes; MLX automatically selects the optimal kernel. (06:42)

Continuous Batching: Multiple Agents Run in Parallel Without Waiting

In real-world scenarios, one agent often needs to start multiple subagents at the same time, each handling a separate task: one reads documentation, one searches code, and one writes tests. Multiple requests hit the local model concurrently. (06:57)

MLX-LM Server solves this with Continuous Batching. Instead of processing requests one by one, it dynamically merges newly arriving requests into the active batch and computes them together. New requests do not have to wait for the current batch to finish before joining. (07:18)

The result is that all subagents can execute concurrently without getting stuck in a queue, keeping the whole workflow moving.

Distributed Inference: Run Huge Models Across Multiple Macs

Even a single Mac with 512GB of unified memory may not be able to fit some ultra-large models. The latest DeepSeek model has 1.6 trillion parameters, and the weights alone require more than 800GB of memory. (07:49)

MLX supports connecting multiple Macs over Thunderbolt or Ethernet and slicing the model across different machines. (08:08)

Startup command:

mlx.launch --hostfile hosts.json \
  --backend jaccl \
  /remote/path/to/mlx_lm.server \
  --model mlx-community/Qwen-3.5-122B-A3B-8bit

Key points:

  • --hostfile specifies the cluster node configuration, including connection information for each node
  • --backend jaccl uses the Jaccl communication backend
  • The model is automatically sharded across all available devices, while the rest of the workflow remains unchanged
  • macOS 26.2 and later support Thunderbolt RDMA, providing low-latency, high-bandwidth communication
  • A four-node distributed inference setup can deliver up to a threefold speedup

For agents, this matters in two ways: first, you can run larger models that do not fit on a single machine; second, distributed prompt processing directly accelerates the context-processing step inside the agent loop. (08:17)

Demo: From PR Summaries to Xcode Integration

The speaker showed three practical scenarios:

PR summary: A local agent fetches recent pull requests from the MLX repository, summarizes the changes, and flags items that need attention. The model runs locally; only GitHub CLI commands go over the network. (01:40)

Building a SwiftUI app from scratch: In an empty Xcode project, the agent builds an iPad drawing app. It inspects the directory structure, creates a plan, writes code, invokes xcodebuild, and fixes build errors. Two minutes later, it produces a running drawing app. When asked to add rounded pen caps, the agent modifies the code, rebuilds it, and verifies the result in the simulator. (09:29)

Direct Xcode integration: In Xcode Intelligence settings, the speaker adds a “Locally Hosted” provider with port 8080. Xcode connects directly to the local MLX server. The speaker intentionally introduces a bug and asks the model to locate and fix it. Within seconds, the model finds the issue, inspects nearby code, writes the fix, then builds and runs the app to verify it. (11:47)

Key Takeaways

Local code review assistant

What to do: Deploy a Mac mini as a local code review server inside the team. Every submitted PR triggers a local agent to perform static analysis and logic checks.

Why it is worth doing: Code never leaves the company network, eliminating compliance risk. MLX-LM Server’s OpenAI-compatible interface makes it straightforward to migrate existing review tools.

How to start: Load a code-specialized model such as CodeQwen with mlx_lm.server, then write a Git hook or CI script that calls the local /v1/chat/completions endpoint.

Offline intelligent IDE plug-in

What to do: Build a fully offline code completion and refactoring plug-in for VS Code or Xcode.

Why it is worth doing: The M5 Neural Accelerator makes local prompt processing feel close to a cloud experience, while keeping zero network latency, zero usage fees, and zero data leakage risk.

How to start: On the plug-in side, use any OpenAI SDK, change baseURL to http://127.0.0.1:8080/v1, and set the model name to default_model.

Multi-Mac home compute pool

What to do: Connect several Macs at home or in the office with Thunderbolt cables and form a private distributed inference cluster.

Why it is worth doing: A single Mac can run 7B models easily, but a 122B model requires multiple machines. Thunderbolt RDMA makes the communication latency low enough to be practical, and four machines can reach a threefold speedup.

How to start: Prepare a hosts.json file describing each node, then start the distributed service with mlx.launch --backend jaccl.

Local automation operations agent

What to do: Deploy a resident local agent that monitors server logs, automatically analyzes anomalies, and executes repair commands.

Why it is worth doing: Operations data such as logs, configuration, and certificates is highly sensitive. Local execution prevents leakage. The agent loop can run for hours over large log contexts without API costs.

How to start: Use OpenCode or a custom Python agent framework, configure tool-calling capabilities such as reading logs and running shell commands, and point it to the local MLX-LM Server.

Private knowledge-base Q&A

What to do: Index internal company documents, wikis, and code repositories, then provide a natural-language query interface through a local agent.

Why it is worth doing: The central pain point of RAG, or retrieval-augmented generation, is that sensitive documents cannot be sent to the cloud. A local MLX model, local vector database, and local agent form a complete closed loop.

How to start: Use mlx-lm to load an embedding model for document vectorization, use a chat model for generation, and let the agent orchestrate retrieval and answering.

Comments

GitHub Issues · utterances