vhdl-style-guide/docs/theory_of_operation/tokenizing.rst

36 lines
958 B
ReStructuredText

Tokenizing
----------
The tokenizing process is responsible for taking the VHDL text file and grouping characters into tokens.
This process is performed on a line by line basis.
Each line is processed a single character at a time.
Characters are combined to form tokens based on several rules.
* keywords and identifiers
* multiple character sequences (<=, :=, etc...)
* whitespace
* comments
The result will be a list of lists.
The outer list is the entire file and each inner list is a line in the file.
For example, the following code...
.. code-block:: vhdl
architecture rtl of fifo is
begin
a <= b; -- Assign output
end architecture rtl;
would be tokenized into this:
.. code-block:: python
tokens = [
['architecture', ' ', 'rtl', ' ', 'of', ' ', 'fifo', ' ', 'is'],
['begin'],
[' ', 'a', ' ', '<=', ' ', 'b', ';', ' ', '-- Assign output'],
['end', ' ', 'architecture', ' ', 'rtl', ';']
]