5 Commits (11ffc64501a9cc5aa0464ea356d2383f33dfbde0)

Author SHA1 Message Date
  Niklas Gustafsson 11ffc64501 Revert "Revert "Generalized Tokenizer argument types."" 4 years ago
  Niklas Gustafsson a468cd4c0c Revert "Generalized Tokenizer argument types." 4 years ago
  Niklas Gustafsson d9465d8ab0 Generalized Tokenizer argument types. 4 years ago
  Niklas Gustafsson 0413e5197a Adding more implementations to Tokenizer -- seq->text, pre-tokenized texts, etc. 4 years ago
  Niklas Gustafsson 606dba816e Started implementation of Tokenizer(), as well as pad_sequences(). 4 years ago