| @@ -13,21 +13,29 @@ both Windows and Linux and does NOT require compiling llama.cpp yourself. | |||||
| ## Installation | ## Installation | ||||
| Just search `LLamaSharp` or `LLamaSharp.Cpu` in nuget package manager and install it! | |||||
| Firstly, search `LLamaSharp` in nuget package manager and install it. | |||||
| ``` | ``` | ||||
| PM> Install-Package LLamaSharp | PM> Install-Package LLamaSharp | ||||
| ``` | ``` | ||||
| If you want to run it only on CPUs, please install `LLamaSharp.Cpu`. The only difference between them is native library. | |||||
| Then, search and install one of the following backends: | |||||
| If you have GPU on windows but fail to run with the error `failed to load libllama.dll or one of its dependencies`, please try downloading the cublas support dlls [here](https://github.com/ggerganov/llama.cpp/releases). For example, [cudart-llama-bin-win-cu11.7.1-x64.zip | |||||
| ](https://github.com/ggerganov/llama.cpp/releases/download/master-773ee24/cudart-llama-bin-win-cu11.7.1-x64.zip) is the one for CUDA 11.7. | |||||
| ``` | |||||
| LLamaSharp.Backend.Cpu | |||||
| LLamaSharp.Backend.Cuda11 | |||||
| LLamaSharp.Backend.Cuda12 | |||||
| ``` | |||||
| Note that version v0.2.1 has a package named `LLamaSharp.Cpu`. After v0.2.2 it will be dropped. | |||||
| We publish the backend with cpu, cuda11 and cuda12 because they are the most popular ones. If none of them matches, please compile the [llama.cpp](https://github.com/ggerganov/llama.cpp) | |||||
| from source and put the `libllama` under your project's output path. When building from source, please add `-DBUILD_SHARED_LIBS=ON` to enable the library generation. | |||||
| ## Simple Benchmark | ## Simple Benchmark | ||||
| Currently it's only a simple benchmark to indicate that the performance of `LLamaSharp` is close to `llama.cpp`. Experiments run on a computer | Currently it's only a simple benchmark to indicate that the performance of `LLamaSharp` is close to `llama.cpp`. Experiments run on a computer | ||||
| with Intel i7-12700, 3060Ti with 7B model. Note that the benchmark uses `LLamaModel` instead of `LLamaModelV1`. | |||||
| with Intel i7-12700, 3060Ti with 7B model. Note that the benchmark uses `LLamaModel` instead of `LLamaModelV1`. | |||||
| #### Windows | #### Windows | ||||
| @@ -82,6 +90,8 @@ else | |||||
| } | } | ||||
| ``` | ``` | ||||
| For more usages, please refer to [Examples](./LLama.Examples). | |||||
| #### Web API | #### Web API | ||||
| We provide the integration of ASP.NET core [here](./LLama.WebAPI). Since currently the API is not stable, please clone the repo and use it. In the future we'll publish it on NuGet. | We provide the integration of ASP.NET core [here](./LLama.WebAPI). Since currently the API is not stable, please clone the repo and use it. In the future we'll publish it on NuGet. | ||||