- made `llama_backend_init` private. This is automatically called, there is no way it can correctly be used externally.
 - made `llama_token_to_piece` safe (Span instead of pointer)

Removed some unnecessary uses of `unsafe`

- Wrapped some native functions which take (pointer, length) in function which take a `span` instead.

Code cleanup driven by R# suggestions

- Made `NativeApi` into a `static class` (it's not intended to be instantiated)
 - Moved `LLamaTokenType` enum out into a separate file
 - Made `LLamaSeqId` and `LLamaPos` into `record struct`, convenient to have equality etc

build(deps): bump Microsoft.KernelMemory.Core and Microsoft.KernelMemory.Abstractions

build(deps): bump Microsoft.KernelMemory.Abstractions from 0.18.231209.1-preview to 0.24.231228.5

Bumps [Microsoft.KernelMemory.Abstractions](https://github.com/microsoft/kernel-memory) from 0.18.231209.1-preview to 0.24.231228.5.
- [Release notes](https://github.com/microsoft/kernel-memory/releases)
- [Commits](https://github.com/microsoft/kernel-memory/compare/dotnet-0.18.231209.1-preview...0.24.231228.5)

---
updated-dependencies:
- dependency-name: Microsoft.KernelMemory.Abstractions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Bumps [Microsoft.KernelMemory.Core](https://github.com/microsoft/kernel-memory) and [Microsoft.KernelMemory.Abstractions](https://github.com/microsoft/kernel-memory). These dependencies needed to be updated together.

Updates `Microsoft.KernelMemory.Core` from 0.18.231209.1-preview to 0.24.231228.5
- [Release notes](https://github.com/microsoft/kernel-memory/releases)
- [Commits](https://github.com/microsoft/kernel-memory/compare/dotnet-0.18.231209.1-preview...0.24.231228.5)

Updates `Microsoft.KernelMemory.Abstractions` from 0.18.231209.1-preview to 0.24.231228.5
- [Release notes](https://github.com/microsoft/kernel-memory/releases)
- [Commits](https://github.com/microsoft/kernel-memory/compare/dotnet-0.18.231209.1-preview...0.24.231228.5)

---
updated-dependencies:
- dependency-name: Microsoft.KernelMemory.Core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: Microsoft.KernelMemory.Abstractions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump Swashbuckle.AspNetCore from 6.4.0 to 6.5.0

build(deps): bump xunit.runner.visualstudio from 2.5.5 to 2.5.6

Bumps [xunit.runner.visualstudio](https://github.com/xunit/visualstudio.xunit) from 2.5.5 to 2.5.6.
- [Release notes](https://github.com/xunit/visualstudio.xunit/releases)
- [Commits](https://github.com/xunit/visualstudio.xunit/compare/2.5.5...2.5.6)

---
updated-dependencies:
- dependency-name: xunit.runner.visualstudio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

build(deps): bump xunit from 2.6.3 to 2.6.4

Bumps [xunit](https://github.com/xunit/xunit) from 2.6.3 to 2.6.4.
- [Commits](https://github.com/xunit/xunit/compare/2.6.3...2.6.4)

---
updated-dependencies:
- dependency-name: xunit
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Bumps [Swashbuckle.AspNetCore](https://github.com/domaindrivendev/Swashbuckle.AspNetCore) from 6.4.0 to 6.5.0.
- [Release notes](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/releases)
- [Commits](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/compare/v6.4.0...v6.5.0)

---
updated-dependencies:
- dependency-name: Swashbuckle.AspNetCore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Metadata Fixes

Update compile.yml to fix not building for windows

This fixes the build step: Compile (Windows) (avx512, -DLLAMA_AVX512=ON -LLAMA_AVX512_VBMI=ON -DLLAMA_AVX512_VNNI=ON)

There is a typo in the cmake command: `cmake .. -DLLAMA_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_SERVER=OFF -DBUILD_SHARED_LIBS=ON -DLLAMA_AVX512=ON -LLAMA_AVX512_VBMI=ON -DLLAMA_AVX512_VNNI=ON
  cmake --build . --config Release -j ${env:NUMBER_OF_PROCESSORS}`

-LLAMA_AVX512_VBMI
Should be
-DLLAMA_AVX512_VBMI

Fixed loading of very large metadata values

bump sk to 1.0.1 & km to 0.18

Updated & Fixed WebAPI

Added `LLamaWeights.Metadata` property

Upgraded to .NET8.0
Fixed logging (removed Console replaced with Ilogger)
Fixed warnings (Null Strings, etc).

The application now returns data from the back-end chat services.

Fix broken references in docs

Improved support for AVX512

Reset Custom Sampling Pipeline

- Enabled more features in build process (VBMI and VNNI)
 - Added runtime checking for this features
 - Improved runtime checking to no longer require dotnet8.0

Context Set Seed

Renamed `llama_sample_temperature` to `llama_sample_temp`

Clone Grammar

Added missing field to LLamaModelQuantizeParams