4 Commits (e50f30d740c9245e57ffb392894e2a8dc371317c)

Author SHA1 Message Date
  Martin Evans 91a7967869
`ReadOnlySpan<float>` in ISamplingPipeline (#538) 1 year ago
  Martin Evans b0acecf080 Created a new `BatchedExecutor` which processes multiple "Conversations" in one single inference batch. This is faster, even when the conversations are unrelated, and is much faster if the conversations share some overlap (e.g. a common system prompt prefix). 1 year ago
  Martin Evans 42be9b136d Switched form using raw integers, to a `LLamaToken` struct 1 year ago
  Martin Evans 835958398c - Removed the object wrappers and configurable pipeline, they can be better written in code. 1 year ago