skeletoken

What's Changed Fix merges when decasing by @stephantul in https://github.com/stephantul/skeletoken/pull/37 refactor decasing to be more robust by @stephantul in https://github.com/stephantul/skeletoken/pull/38 fix: no longer use casefold by @stephantul in https://github.com/stephantul/skeletoken/pull/39 feat: add pylate and sentence-transformers integrations by @stephantul in https://github.com/stephantul/skeletoken/pull/40 feat: add unk and pad token id by @stephantul in https://github.com/stephantul/skeletoken/pull/41 feat: add original class when initializing from a transformers tokenizer by @stephantul in https://github.com/stephantul/skeletoken/pull/42 add tutorials by @stephantul in https://github.com/stephantul/skeletoken/pull/43 feat: fix padding token post init by @stephantul in https://github.com/stephantul/skeletoken/pull/44 add additional tutorial by @stephantul in https://github.com/stephantul/skeletoken/pull/45 add README example by @stephantul in https://github.com/stephantul/skeletoken/pull/46 Type check by @stephantul in https://github.com/stephantul/skeletoken/pull/47 feat: add make commands and ty by @stephantul in https://github.com/stephantul/skeletoken/pull/48 Add docs for conversion by @stephantul in https://github.com/stephantul/skeletoken/pull/49 make everything functional by @stephantul in https://github.com/stephantul/skeletoken/pull/50 add in-place functions for normalization by @stephantul in https://github.com/stephantul/skeletoken/pull/51 fix logging message by @stephantul in https://github.com/stephantul/skeletoken/pull/52 Typing fixes by @stephantul in https://github.com/stephantul/skeletoken/pull/53 feat: Add token prep by @stephantul in https://github.com/stephantul/skeletoken/pull/54 add id for bos/eos by @stephantul in https://github.com/stephantul/skeletoken/pull/55 feat: add addedtokens helper function by @stephantul in https://github.com/stephantul/skeletoken/pull/56 fix: addedtokens bug in decasing by @stephantul in https://github.com/stephantul/skeletoken/pull/57 feat: add prefix space property by @stephantul in https://github.com/stephantul/skeletoken/pull/58 fix: crash when tokenizer is empty by @stephantul in https://github.com/stephantul/skeletoken/pull/59 Full Changelog: https://github.com/stephantul/skeletoken/compare/0.2.2...0.3.0