Tag: Tokenizers
-
A beautiful gem of code from the Python docs.
So once again I found myself looking at the documentation for the Python re module, trying to remember the difference between matches and matching groups and I saw a code snippet that was so beautiful that I wanted to break it down and share it. It’s a fully-functional mini-language tokenizer in only 41 lines of code. I’ll go through…