2 Commits

Author	SHA1	Message	Date
kenkeiras	79034f85a9	Move to a chaining model for tokenization. ... This model also explores more tokenization possibilities. With this, the tokenization tests are passed.	2018-04-15 20:06:23 +02:00
kenkeiras	fc37450565	Add (non-passing) tokenization.	2018-04-15 16:41:07 +02:00