I wonder how you parse markup languages like markdown which allow just random text anywhere. Like... I know how to handle normal programming languages(TM) -- you tokenize/lex first, transforming text into a pile on "things" (tokens), then you match a couple of them at a time, building structure as you go... But what do you do with markdown? Does tokenization still make sense? What are the tokens? How you handle when things are just text (like [ which is not part of a [link])? How do you properly specify it? [...]
This should not be so confusing. Yet it does confuse me. I should learn more...