Definitions

1. noun

A computer program that translates source code into a format that can be understood by a parser.

“The compiler uses a lexer to break down the source code into individual tokens.”

2. verb

To process source code and divide it into individual tokens.

“The lexer is used to tokenize the source code before it’s compiled.”