8 Commits (master)

Author SHA1 Message Date
  Martin Evans 3ba49754b1 Removed (marked as obsolete) prompting with a string for `Conversation`. Tokenization requires extra parameters (e.g. addBos, special) which require special considersation. For now it's better to tokenize using other tools and pass the tokens directly. 1 year ago
  Martin Evans ccc49eb1e0
BatchedExecutor Save/Load (#681) 1 year ago
  Martin Evans 268f3a6b07
BatchedExecutor Fixed Forking (#621) 1 year ago
  Martin Evans ad682fbebd
`BatchedExecutor.Create()` method (#613) 1 year ago
  Martin Evans c7d0dc915a Assorted small changes to clean up some code warnings 1 year ago
  Martin Evans 1cc463b9b7 Added a finalizer to `BatchedExecutor` 1 year ago
  Martin Evans 949861a581 - Added a `Modify` method to `Conversation`. This grants **temporary** access to directly modify the KV cache. 1 year ago
  Martin Evans b0acecf080 Created a new `BatchedExecutor` which processes multiple "Conversations" in one single inference batch. This is faster, even when the conversations are unrelated, and is much faster if the conversations share some overlap (e.g. a common system prompt prefix). 1 year ago