LLM & Inference

Flash weight streaming for MLX: run massive models larger than your RAM on Apple Silicon.

User friendly GUI for configuring and launching llama.cpp

An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU (DAC ‘26)

Local AI for Omarchy: the model validated for your GPU, one button on the bar. Start serves it, any coding agent opens on it, one click shares it on your tailnet.

[ICML 2026] A neuro-symbolic framework for abductive reasoning that grounds multi-agent collaboration in structured belief states using a causal graph and state machine.

Qwen3.8-Flash-Next on 2× RTX 3090: with 128 GB RAM up to 4,191 tok/s prefill · 111.5 tok/s decode (131K prompt), full 256K window at 2,865 tok/s; with 64 GB RAM 3,410 tok/s prefill · 84 tok/s decode. New: opt-in uncensored mode (runtime abliteration, no new weights).

An offline coding agent for local models. One command sets up the best model your GPU can run, and on 24 GB+ RTX 30/40/50-series cards Qwen3.8-27B runs up to 1.9x faster than llama.cpp.

[COLMW'26 Spotlight] AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

Local AI desktop app: a Chromium browser with a GPU-aware llama.cpp runtime, OpenAI-compatible API, AI chat and agents, Stable Diffusion image/video, TTS and music, plus an MCP server so Claude Code, Codex and Cline can drive your browser. Windows, macOS, Linux.

[NeurIPS 2026] DMax: Aggressive Parallel Decoding for dLLMs

Easy fine-tuning for Qwen3-TTS: Fast voice cloning and high-quality multilingual speech synthesis.

llama.cpp optimizations for IBM POWER8: vec_perm non-bijunctive collapse, PSE hardware entropy, DCBT prefetch. Sovereign inference. Part of the Proof of Physical AI stack.

Simulate AI Personas in Social Feeds with Real-Time Chat Bots 2026

[Official] Prima.cpp: Scale Your Local AI Beyond One Device.

[ICML 2026] When Drafts Evolve: Speculative Decoding Meets Online Learning

Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

Can AI Agents Build Bespoke Systems?

A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching

Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.

Kindi 0.4: local LLM inference on low-cost NVIDIA GPUs, Radeon 780M and CPUs. Bonsai 2B-27B, AH-Nanbeige4.3-33B and experimental Nemotron, with an OpenAI-compatible server.

动手学大模型全栈：CS336 中文精讲 · PyTorch 手搓 Transformer · 单卡复现 Pretrain/SFT/LoRA/DPO/GRPO/RLVR · PEFT/Agent/RAG 落地（含真实实验、曲线与复现脚本）

FlowBlock: Wavefront-Parallel Decoding Framework for Self-Correcting Diffusion Language Models

[ICLR 2026] GRAPE: Group Representational Position Encoding (https://arxiv.org/abs/2512.07805)

Swift + Metal MoE inference for Apple Silicon: Qwen 3.6 35B at 23.5–29.3 tok/s decode with 2.20× faster long-prompt prefill on a 24 GB M5; Gemma 4 26B in ~2 GB, DeepSeek-V4-Flash 284B, Inkling-Small 276B, native Mac app, CLI, and OpenAI-compatible server.

LoRA fine-tune Gemma 4 31B to speak caveman-mode natively. Style: github.com/JuliusBrussee/caveman

Fine-tuning toolkit for Chatterbox TTS & Chatterbox TURBO & Chatterbox NANO models. Supports 23 languages with smart vocabulary extension. Features offline preprocessing, automatic VAD trimming, and voice cloning capabilities. Train custom TTS models with your own dataset in LJSpeech and file-based format.

Granite Switch — Build AI models like you build software

Open-source control panel for the real llama.cpp: see where a model lands in your VRAM before loading, every llama-server flag per model, one-click llama.cpp updates, several models at once, and Embers (local agents whose quotes are checked by code). Windows · Linux · macOS.

Source code of paper "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation"

152 open-source tools to run LLMs 100% locally – no cloud, no API keys, no censorship

German-OCR is specifically trained to extract text from German documents including invoices, receipts, forms, and other business documents.

A continuously-verified dataset of free-tier & trial-credit LLM APIs for developers. Every entry dated, sourced to the provider's own docs, and machine-readable. No hype, no dead links.

Source code of Local Code, a private AI agent for Mac, iPad and iPhone. Runs DeepSeek 4.1 on device, with Python, Node.js and a shell built into the app.

CUDA编程练习项目-Hands-on CUDA kernels and performance optimization, covering GEMM, FlashAttention, Tensor Cores, CUTLASS, quantization, KV cache, NCCL, and profiling.

Distributed 56M-parameter LLM inference across 3 ESP32-S3 boards via ESP-NOW , Split-PLE + KV cache, fully offline.

A curated list of strategies, tools, papers, and resources for reducing LLM token costs and improving efficiency in production.

[EMNLP 2026 Main] Speaking While Listening: a survey and empirical audit of full-duplex spoken dialogue systems — L0–L3 architectural hierarchy, T×I×R interaction ontology, and a five-state decision machine, with a curated list of models, datasets, and benchmarks.

LabVLA: Grounding Vision–Language–Action Models in Scientific Laboratories

One-command vLLM installation for NVIDIA DGX Spark with Blackwell GB10 GPUs (sm_121 architecture)

LLM agents: curated papers on planning, memory, tool use, multi-agent systems, evaluation, and safety.

PasLLM - LLM inference engine in Object Pascal (synced from my private work repository)

A comprehensive GraphRAG (Graph Retrieval-Augmented Generation) system designed for biomedical research

High-throughput synthetic and pretraining dataset sifter for TypeSafe Jev. Rust streaming core, Parquet and JSONL I/O, typed Choice/Score/Noul judgments, speculative fan-out, 24.0 rows/sec measured.

PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.

Official Project Page for HLA: Higher-order Linear Attention (https://arxiv.org/abs/2510.27258)

The first Task-Aware MCP server and automated VRAM calculator for LLM fine-tuning. Instantly snipe the cheapest, fastest GPUs across 10+ cloud providers.

Nano vLLM with vLLM v1's request scheduling strategy and chunked prefill

A local app that rents a GPU in your own Colab account and serves Qwen3.8 on it: 27B on an A100-40G, or Flash-Next (125B-A6B MoE) on an A100-80G. The cost is shown before it starts; chat with images; Codex or any OpenAI client uses a fixed local /v1. Auto-stop, CU ledger, live prefill/decode. Windows, macOS, Linux.

Dory treats hosted LLM interfaces as disposable tools by offloading state to your local environment and enforcing atomic sessions.