7 Commits (ccc49eb1e0a245b43b8866c5831d76464f0c405b)

Author SHA1 Message Date
  Martin Evans ccc49eb1e0
BatchedExecutor Save/Load (#681) 1 year ago
  Martin Evans 268f3a6b07
BatchedExecutor Fixed Forking (#621) 1 year ago
  Martin Evans ad682fbebd
`BatchedExecutor.Create()` method (#613) 1 year ago
  Martin Evans c7d0dc915a Assorted small changes to clean up some code warnings 1 year ago
  Martin Evans 1cc463b9b7 Added a finalizer to `BatchedExecutor` 1 year ago
  Martin Evans 949861a581 - Added a `Modify` method to `Conversation`. This grants **temporary** access to directly modify the KV cache. 1 year ago
  Martin Evans b0acecf080 Created a new `BatchedExecutor` which processes multiple "Conversations" in one single inference batch. This is faster, even when the conversations are unrelated, and is much faster if the conversations share some overlap (e.g. a common system prompt prefix). 1 year ago