5 Commits (18b15184ea6dceba07b9e6a6a73594e823714d40)

Author SHA1 Message Date
  Martin Evans 1f8c94e386 Added in the `special` parameter to the tokenizer (introduced in https://github.com/ggerganov/llama.cpp/pull/3538) 2 years ago
  Martin Evans fe54f6764f - Added unit tests for extension methods 2 years ago
  Martin Evans d0e57a8c92 sealed test class 2 years ago
  Martin Evans 3f082c6f2c Fixed naming in tests 2 years ago
  Martin Evans 614ba40948 - Added a `TokensEndsWithAnyString` extension to `IReadOnlyList<int>` which efficiently checks if a set of tokens ends with one of a set of strings. 2 years ago