Show HN: FlashTokenizer – 10x faster C++ tokenizer for Python https://bit.ly/3Rvb2NM

Show HN: FlashTokenizer – 10x faster C++ tokenizer for Python I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions! https://bit.ly/4cehmCW April 3, 2025 at 07:46AM