Introduce input ports.

Ports are the lisp abstraction of files and streams. Actinide ports additionally guarantee a peek operation.

This makes ``tokenize`` (now ``read_token``) callable as a lisp function, as it
takes a port and reads one token from it. This is a substantial refactoring.

As most of the state is now captured by closures, it's no longer practical to
test individual states as readily. However, the top-level tokenizer tests
exercise the full state space.

0 files changed, 0 insertions, 0 deletions