Welcome to llmc’s documentation!

llmc is a tool for large model compression, supporting a variety of models and a variety of compression algorithms.

github: ModelTC/llmc

arxiv: https://arxiv.org/abs/2405.06001

Quick Start

• Installing LLMC
• Preparing the Model
• Downloading the Dataset
• Setting Configuration Files
• Running LLMC
• Quantization Inference
• FAQ

Configs

• Configs’ brief description
• Configs’ detailed description
  • base
  • model
  • calib
  • eval
  • quant
  • sparse
  • save

Advanced

• Model accuracy test V1
  • Accuracy test pipeline
  • Use of the lightLLM inference engine
  • Use of the opencompass evaluation tool
  • Use of the lm-evaluation-harness evaluation tool
  • FAQ
• Model accuracy test V2
  • Multi-GPU parallel test
• Custom calibration datasets
• Vit quant and img datatsets
  • Vit quant
  • img datatsets
• VLM Quantization and custom_mm Datasets
  • VLM Quantization
  • custom_mm datatsets
  • VLM Evaluation
• Layerwise mixed bits quantization
• Model Sparsification
• Token Reduction
  • Combining Quantization

Best Practice

• AWQ
  • 1.1 Weight-only Quantization
  • 1.2 Weight-Activation Quantization
• AWQ + OmniQuant
  • 1.1 Weight-only Quantization
  • 1.2 Weight-Activation Quantization
• QuaRot + GPTQ
  • 1.1 Weight-Activation Quantization

Backbend

• VLLM Quantized Inference
  • 1.1 Environment Setup
  • 1.2 Quantization Formats
  • 1.3 Using LLMC for Model Quantization
  • 1.4 Using VLLM for Inference
• SGLang Quantized Inference
  • 1.1 Environment Setup
  • 1.2 Quantization Format
  • 1.3 Using LLMC for Model Quantization
  • 1.4 Using Sglang for Inference
• AutoAWQ Quantized Inference
  • 1.1 Environment Setup
  • 1.2 Quantization Formats
  • 1.3 Using LLMC for Model Quantization
  • 1.4 Using AutoAWQ for Inference
• MLC LLM Quantized Inference
  • 1.1 Environment Setup
  • 1.2 Quantization Format
  • 1.3 Using LLMC for Model Quantization
  • 1.4 Using MLC LLM for Inference