AI server for Mac computers

What you get

• Apple MLX and Metal hardware acceleration: Completely eliminates the memory copying bottleneck between the CPU and GPU by directly utilizing Apple Silicon processors' Unified Memory Architecture (UMA).
• Continuous Batching: Combines a large number of concurrent user and agent prompts into a single computation cycle, increasing server throughput by up to 3x.
• SSD caching and Chunked Prefill: Prevents out-of-memory (OOM) crashes in long context windows by storing the key-value (KV) cache on an NVMe SSD.
• OpenAI-compatible standard API: Thanks to the /v1/chat/completions and /v1/models endpoints, it works with zero configuration with tools like Cursor, Open WebUI, Continue, and LangChain.
• macOS menu bar control: Offers the convenience of starting, stopping, and selecting models for the server, as well as monitoring memory consumption with live charts, all without entering the Terminal.

Installation

brew tap jundot/omlx https://github.com/jundot/omlx
brew install jundot/omlx/omlx

Running it

omlx start

omlx run mlx-community/Llama-3.2-3B-Instruct-4bit

Technical architecture and working principle

• Full utilization of unified memory (UMA): Unlike PCs with discrete graphics cards, up to 128 GB or 192 GB of RAM on Apple Silicon Macs can be directly addressed by the GPU cores. Omlx processes this massive memory pool with zero latency using Metal Shading Language (MSL) kernels.
• Dynamic KV cache management (PagedAttention): Allocates key-value tensors in paged blocks to prevent memory fragmentation across multiple sessions. The used memory is immediately freed when the prompt finishes.
• SSD-overflow cache layer: When the KV cache exceeds RAM in massive context windows like 32K and 128K, Omlx automatically pages out to Apple's high-speed unified SSD storage. This allows the model to continue inference without crashing.

OpenAI-compatible native API integration

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Apple Silicon mimarisinin temel avantajı nedir?"}],
    "temperature": 0.7
  }'

If you don't write code

Frequently asked questions

• What is the main difference between Omlx and Ollama? While Ollama generally uses the C++-based llama.cpp infrastructure, Omlx runs directly on the MLX framework developed by Apple. This enables deeper integration with the Metal and neural engine units of Apple Silicon chips, delivering higher token generation speeds, particularly in continuous batching and long contexts.
• Which models can be run with 16 GB or 24 GB of RAM? 4-bit quantized 8B parameter models (Llama 3, Qwen 2.5, Mistral) take up about 5-6 GB of memory and run extremely smoothly on 16 GB Macs. On devices with 24 GB or 36 GB of unified memory, 14B or 32B models can be loaded comfortably.
• Does SSD caching wear down the Mac's disk lifespan? No. Omlx uses smart buffering algorithms to prevent unnecessary write cycles during caching operations. It only engages when the context memory approaches the RAM limit, keeping disk wear to a minimum.
• Does it run on older Intel-based Macs? No. Omlx is specifically optimized for Apple Silicon (ARM architecture) and the Apple MLX framework. It does not run on Intel-based Macs or Windows/Linux x86 computers.

Related dictionary terms

Links

• GitHub repository →
• Read in Turkish →