View source for Llama.cpp

{{Short description|Software library for LLM inference}}

{{Infobox software
| name = llama.cpp 
| logo = File:Llama1-logo.png
| author = Georgi Gerganov
| developer = Georgi Gerganov and community
| released = {{Start date and age|2023|3|10}}<ref name="githubrelease"/>
| programming language =  [[C++]], [[C (programming language)|C]]
| genre = [[Library (computing)|Library]] for [[Large language model|large language models]]
| license = [[MIT License]]<ref name="license"/>
| repo = {{URL|github.com/ggerganov/llama.cpp}}
}}

{{Lowercase title}}'''llama.cpp''' is an [[open source]] [[software library]] mostly written in C++ that performs [[Inference engine|inference]] on various [[Large language model|Large Language Models]] such as [[Llama (language model)|Llama]].<ref name="register-llamafile"/> A [[Command-line interface|CLI]] and web server is included with the library.<ref name="lwn" /> llama.cpp is co-developed alongside the GGML project, a general-purpose tensor library.<ref name="ggml"/> 

== Background ==
Towards the end of September 2022, Georgi Gerganov started work on the GGML library, a C library implementing [[tensor algebra]]. Gerganov developed the library with the intention of the strict memory management and multi-threading. The creation of GGML was inspired by [[Fabrice Bellard]]'s work on LibNC.<ref name="changelog-podcast-mar-2023" />

Before llama.cpp, Gerganov worked on a similar library called whisper.cpp which implemented [[Whisper (speech recognition system)|Whisper]], a speech to text model by [[OpenAI]].<ref name="whisper"/>

Gerganov has a background in [[medical physics]], and was part of the Faculty of Physics in [[Sofia University]].<ref name="Medical Physics 21 June 2012" /> In 2006 he won a silver medal in the [[International Physics Olympiad]].<ref name="Tichy-Rács" /><ref name="sega bg July 21, 2006" /> In 2008 he won a programming competition organized by the Bulgarian Association of Software Companies, [[PC Magazine]] and Musala Soft, a Bulgarian software services company.<ref name="karieri" />

== History ==
llama.cpp began development in March 2023 by Georgi Gerganov as an implementation of the [[Llama (language model)|Llama]] inference code in pure C/C++ with no dependencies. This bettered performance on computers without [[GPU]] or other dedicated hardware, which was a goal of the project.<ref name="register-llamafile" /><ref name="arstechnica"/><ref name="Wiest" /> As of September 2024 it has 65,000 stars on GitHub.<ref name="llama.cpprepo"/> llama.cpp gained traction with users who lacked specialized hardware as it could run on just a [[CPU]] including on [[Android (operating system)|Android]] devices.<ref name="arstechnica"/><ref name="mozilla-introducing-llamafile">{{cite web |last1=Hood |first1=Stephen |title=llamafile: bringing LLMs to the people, and to your own computer |url=https://future.mozilla.org/builders/news_insights/introducing-llamafile/ |website=Mozilla Innovations |access-date=28 July 2024 |language=en}}</ref><ref>{{cite web |title=Democratizing AI with open-source language models |url=https://lwn.net/Articles/931853/ |website=lwn.net |access-date=28 July 2024}}</ref> Initially designed for CPUs, GPU inference support was later added.<ref name="Rajput" />

In March 2024 [[Justine Tunney]] introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for [[FP16]] and 8-bit quantized data types.<ref name="llamafileregister" /> These improvements were committed upstream to llama.cpp.<ref name="llamafileregister" /> Tunney also created a tool called llamafile that bundles models and llama.cpp into a single file that runs on multiple operating systems via the Cosmopolitan Libc library also created by Tunney which allows C/C++ to be more portable across operating systems.<ref name="llamafileregister" />

== Architecture ==

llama.cpp supports multiple hardware targets including x86, ARM, CUDA and Metal.<ref name="Kluska" /> These back-ends make up the GGML tensor library which is used by the front-end model-specific llama.cpp code.<ref name="tomshardware"/> llama.cpp supports ahead of time model [[Quantization (signal processing)|quantization]] as opposed to on-the-fly quantization.<ref name="Walkowiak"/> llama.cpp makes use of several CPU extensions for optimization: [[Advanced Vector Extensions|AVX]], [[AVX2]] and [[AVX-512]] for [[X86-64]], and [[ARM_architecture_family#Advanced_SIMD_(Neon)|Neon]] on ARM. [[Apple silicon]] is an important target for the project.<ref name="llama.cpprepo"/><ref name="phoronix-llamafile"/> It supports grammar-based output formatting as [[JSON]].<ref name="Wiest" />

== GGUF file format ==
{{Infobox file format
| name = GGUF
| icon = GGML_logo.svg
| _noextcode = on
| extension = {{code|.gguf}}
| magic = {{code|0x47}} {{code|0x47}} {{code|0x55}} {{code|0x46}}
| developer = Georgi Gerganov and community
| released = {{Start date and age|2023|8|22}}<ref name="githubgguf"/>
| latest_release_version = v3<ref name="ggufdoc"/>
| type = [[Machine-learning]] [[Tensor (machine learning)|tensors]]
}}

The GGUF (GPT-Generated Unified Format) file format is a binary format used by llama.cpp that stores both tensors and metadata in a single file.<ref name="huggingface"/> It was created to better maintain backwards compatibility as llama.cpp expanded its support for other model architectures.<ref name="Rajput" /><ref name="ibm-gguf-vs-ggml" /> It was a successor format for GGML (GPT-Generated Model Language).

GGUF files are typically created by converting models developed with a different machine learning library such as [[PyTorch]], although [[Fine-tuning (deep learning)|fine-tuning]] is supported natively.<ref name="tech-blog-should-replace" />

=== Design ===
The format focuses on quantization, the act of reducing precision in the model weights. This can lead to reduced memory usage, and increased speed at the expense of lower model accuracy.<ref name="towardsdatascience" /><ref name="ibm-gguf-vs-ggml" />

GGUF supports 2-bit to 8-bit quantized integer types;<ref name="Cabezas" /> common floating-point data formats such as [[float32]], [[float16]], and [[bfloat16]]; and 1.56 bit quantization.<ref name="theregister 14 Jul 2024" />

This file format contains information necessary for running a GPT-like language model such as the tokenizer vocabulary, context length, tensor info and other attributes.<ref name="Accelerating GGUF Models with Transformers" />

== Supported models ==
{{div col}}
* [[Llama_(language_model)#LLaMA|LLaMA]]
* [[Llama 2]]
* [[Llama 3]]
* [[Mistral_AI#Mistral_7B|Mistral 7B]]
* [[Mistral_AI#Mixtral_8x7B|Mixtral 8x7B]]
* [[Mistral_AI#Mixtral_8x22B|Mixtral 8x22B]]
* [[DBRX]]
* [[GPT-2]]
* [[BLOOM_(language_model)|BLOOM]]
* [[Gemini_(language_model)|Gemma]]
* [[Grok_(chatbot)|Grok-1]]
* [[Mamba_(deep_learning_architecture)|Mamba]]
{{div col end}}

== References ==

{{Reflist|refs=

<ref name="githubrelease">{{cite web |title=Initial release · ggerganov/llama.cpp@26c0846 |url=https://github.com/ggerganov/llama.cpp/commit/26c084662903ddaca19bef982831bfb0856e8257 |website=GitHub |access-date=15 May 2024 |language=en}}</ref>

<ref name="license">{{cite web |title=llama.cpp/LICENSE at master · ggerganov/llama.cpp |url=https://github.com/ggerganov/llama.cpp/blob/master/LICENSE |website=GitHub |language=en}}</ref>

<ref name="ggml">{{cite web |last1=Gerganov |first1=Georgi |title=ggerganov/ggml |website=[[GitHub]] |url=https://github.com/ggerganov/ggml |date=17 May 2024}}</ref>

<ref name="register-llamafile">{{cite web |last1=Connatser |first1=Matthew |title=How this open source LLM chatbot runner hit the gas on x86, Arm CPUs |url=https://www.theregister.com/2024/04/03/llamafile_performance_gains/ |website=theregister.com |access-date=15 April 2024}}</ref>

<ref name="llama.cpprepo">{{cite web |title=ggerganov/llama.cpp |website=[[GitHub]] |url=https://github.com/ggerganov/llama.cpp}}</ref> 

<ref name="whisper">{{cite web |title=ggerganov/whisper.cpp |website=[[GitHub]] |url=https://github.com/ggerganov/whisper.cpp}}</ref>

<ref name="arstechnica">{{cite web |last1=Edwards |first1=Benj |title=You can now run a GPT-3-level AI model on your laptop, phone, and Raspberry Pi |url=https://arstechnica.com/information-technology/2023/03/you-can-now-run-a-gpt-3-level-ai-model-on-your-laptop-phone-and-raspberry-pi/ |website=arstechnica.com |date=13 March 2023 |access-date=15 April 2024}}</ref>

<ref name="tomshardware">{{cite web |last1=Pounder |first1=Les |title=How To Create Your Own AI Chatbot Server With Raspberry Pi 4 |url=https://www.tomshardware.com/how-to/create-ai-chatbot-server-on-raspberry-pi |website=tomshardware.com |date=25 March 2023 |access-date=16 April 2024}}</ref>

<ref name="Walkowiak">{{cite journal |last1=Walkowiak |first1=Bartosz |last2=Walkowiak |first2=Tomasz |journal=International Journal of Electronics and Telecommunications|date=2024 |volume=70 |issue=1 |pages=153–159 |doi=10.24425/ijet.2024.149525 |url=https://journals.pan.pl/Content/130704/18_4466_Walkowiak_L_sk.pdf |access-date=8 May 2024| title=Implementation of language models within an infrastructure designed for Natural Language Processing}}</ref>

<ref name="huggingface">{{cite web |title=GGUF |url=https://huggingface.co/docs/hub/gguf |website=huggingface.co |access-date=9 May 2024}}</ref>

<ref name="ggufdoc">{{cite web |title=ggml/docs/gguf.md at master · ggerganov/ggml |url=https://github.com/ggerganov/ggml/blob/master/docs/gguf.md |website=GitHub |language=en}}</ref>

<ref name="towardsdatascience">{{cite web |last1=Labonne |first1=Maxime |title=Quantize Llama models with GGUF and llama.cpp |url=https://towardsdatascience.com/quantize-llama-models-with-ggml-and-llama-cpp-3612dfbcc172 |website=Medium |publisher=Towards Data Science |access-date=9 May 2024 |language=en |date=29 November 2023}}</ref>

<ref name="githubgguf">{{cite web |title=GGUF by ggerganov · Pull Request #2398 · ggerganov/llama.cpp |url=https://github.com/ggerganov/llama.cpp/pull/2398 |website=GitHub |language=en}}</ref>

<ref name="Medical Physics 21 June 2012">{{cite journal |last1=Mitev |first1=Krasimir |last2=Gerganov |first2=Georgi |last3=Kirov |first3=Assen S. |last4=Schmidtlein |first4=C. Ross |last5=Madzhunkov |first5=Yordan |last6=Kawrakow |first6=Iwan |title=Influence of photon energy cuts on PET Monte Carlo simulation results: Influence of photon energy cuts |journal=Medical Physics |date=21 June 2012 |volume=39 |issue=7Part1 |pages=4175–4186 |doi=10.1118/1.4725168 |url=https://aapm.onlinelibrary.wiley.com/doi/10.1118/1.4725168 |language=en}}</ref>

<ref name="Tichy-Rács">{{cite book |last1=Tichy-Rács |first1=Ádám |title=LIST OF WINNERS IN 1ST – 45TH INTERNATIONAL PHYSICS OLYMPIADS |date=2015 |publisher=BME OMIKK |isbn=978-963-593-500-0 |pages=246 |url=https://archive.org/details/41a-282b-0-3753-4b-06-bf-25-6fce-29290f-61/page/246/mode/2up}}</ref>

<ref name="Cabezas">{{cite book |last1=Cabezas |first1=Darío |last2=Fonseca-Delgado |first2=Rigoberto |last3=Reyes-Chacón |first3=Iván |last4=Vizcaino-Imacaña |first4=Paulina |last5=Morocho-Cayamcela |first5=Manuel |title=Proceedings of the 19th International Conference on Software Technologies |chapter=Integrating a LLaMa-based Chatbot with Augmented Retrieval Generation as a Complementary Educational Tool for High School and College Students |date=2024 |pages=395–402 |doi=10.5220/0012763000003753|isbn=978-989-758-706-1 }}</ref>

<ref name="Kluska">{{cite journal |last1=Kluska |first1=Piotr |last2=Castell´o |first2=Adri´an |last3=Scheidegger |first3=Florian |last4=I. Malossi |first4=A. Cristiano |last5=Quintana-Ort´ı |first5=Enrique |title=QAttn: Efficient GPU Kernels for mixed-precision Vision Transformers |journal=Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops |date=June 2024 |url=https://openaccess.thecvf.com/content/CVPR2024W/eLVM/papers/Kluska_QAttn_Efficient_GPU_Kernels_for_Mixed-precision_Vision_Transformers_CVPRW_2024_paper.pdf}}</ref>

<ref name="theregister 14 Jul 2024">{{cite web |last1=Mann |first1=Tobias |title=Honey, I shrunk the LLM! A beginner's guide to quantization – and testing it |url=https://www.theregister.com/2024/07/14/quantization_llm_feature/ |website=theregister |date=14 Jul 2024}}</ref>

<ref name="Rajput">{{cite book |last1=Rajput |first1=Saurabhsingh |last2=Sharma |first2=Tushar |chapter=Benchmarking Emerging Deep Learning Quantization Methods for Energy Efficiency |title=2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C) |date=4 June 2024 |pages=238–242 |doi=10.1109/ICSA-C63560.2024.00049|isbn=979-8-3503-6625-9 }}</ref>

<ref name="ibm-gguf-vs-ggml">{{cite web |last1=Mucci |first1=Tim |title=GGUF versus GGML |url=https://www.ibm.com/think/topics/gguf-versus-ggml |website=www.ibm.com |access-date=26 July 2024 |language=en-us |date=3 July 2024}}</ref>

<ref name="tech-blog-should-replace">{{cite web |last1=Boykis |first1=Vicki |title=GGUF, the long way around |url=https://vickiboykis.com/2024/02/28/gguf-the-long-way-around |website=Vicki Boykis |access-date=26 July 2024 |date=28 February 2024}}</ref>

<ref name="Wiest">{{cite journal |last1=Wiest |first1=Isabella Catharina |last2=Ferber |first2=Dyke |last3=Zhu |first3=Jiefu |last4=van Treeck |first4=Marko |last5=Meyer |first5=Meyer, Sonja K. |last6=Juglan |first6=Radhika |last7=Carrero |first7=Zunamys I. |last8=Paech |first8=Daniel |last9=Kleesiek |first9=Jens |last10=Ebert |first10=Matthias P. |last11=Truhn |first11=Daniel |last12=Kather |first12=Jakob Nikolas |title=Privacy-preserving large language models for structured medical information retrieval |journal=npj Digital Medicine |date=2024 |volume=7 |issue=257 |doi=10.1038/s41746-024-01233-2|doi-broken-date=2024-09-21 }}</ref>

<ref name="Accelerating GGUF Models with Transformers">{{cite magazine |last1=Dong |first1=Bo |last2=Lin |first2=Jun |last3=Yu |first3=Zhentao |last4=Xu |first4=Zhenzhong |last5=Luo |first5=Yu |last6=Chang |first6=Hanwen |last7=Shen |first7=Haihao |title=Accelerating GGUF Models with Transformers |journal=The Parallel Universe |date=July 2024 |issue=57 |pages=28-33 |url=https://www.intel.com/content/www/us/en/developer/articles/technical/accelerate-gguf-models-with-transformers.html |language=en |publisher=Intel}}</ref>

<ref name="karieri">{{cite web |last1=Станева |first1=Ирина |title=Студенти от СУ спечелиха конкурса по програмиране на БАСКОМ |url=https://www.karieri.bg/news/24140_studenti_ot_su_specheliha_konkursa_po_programirane_na |website=karieri |date=January 30, 2008}}</ref>

<ref name="changelog-podcast-mar-2023">{{cite web |title=Bringing Whisper and LLaMA to the masses with Georgi Gerganov (Changelog Interviews #532) |url=https://changelog.com/podcast/532 |website=Changelog |access-date=28 July 2024 |language=en |date=22 March 2023}}</ref> 

<ref name="sega bg July 21, 2006">{{cite web |last1=Захариев |first1=Боян |title=България с 11 медала от международни олимпиади |url=https://old.segabg.com/article.php?id=279681 |website=sega bg |date=July 21, 2006}}</ref>

<ref name="llamafileregister">{{cite web |last1=Connatser |first1=Matthew |title=Llamafile LLM driver project boosts performance on CPU cores |url=https://www.theregister.com/2024/04/03/llamafile_performance_gains/ |website=www.theregister.com |access-date=10 May 2024 |language=en}}</ref>

<ref name="phoronix-llamafile">{{cite web |last1=Larabel |first1=Michael |title=Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times For AMD Zen 4 |url=https://www.phoronix.com/news/Llamafile-0.7 |website=www.phoronix.com |language=en}}</ref>

<ref name="lwn">{{cite web |last1=Alden |first1=Daroc |title=Portable LLMs with llamafile [LWN.net] |url=https://lwn.net/Articles/971195/ |website=lwn.net |access-date=30 July 2024}}</ref>

}}

[[Category:Large language models]]
[[Category:Open-source artificial intelligence]]
[[Category:Free and open-source software]]