impl<'a> Iterator for Tokenizer<'a> {
type Item = ();
fn next(&mut self) -> Option<Self::Item> {
let c = self.iter.next()?;
self.tokenize(c);
Some(())
}
}That's how we advance. Think of it.
We reached another turning point, where the complexity of our scanner increases once again to solve more complicated challenges. In the next stage, we want to capture multi-character lexemes such as == or !=.
Clearly, the structure we created so far is not good enough. The current implementation uses the simplistic strategy of looking at each character in a loop, but when it comes to multi-character lexemes this approach falls short. If we look at a single character the lexeme == will be mistaken for two tokens EQUAL instead of being recognised as a more complicated piece of syntax.
We need to get rid of the rigid structure of the character-by-character loop. We can do it by storing the index of the current character in the structure Tokenizer and creating a function that allows us to move the index. This by the way is what Robert Nystrom does in his book with the function advance (see 4.5 Recognizing Lexemes).
In compilers and interpreters, both scanners (or lexers) and parsers might need to be able to look past the current character or token. This feature is called "lookahead".
The current stage of the challenge is a good example of why a scanner might need to look at more than one character. Additionally, scanners are typically greedy, which means that they match the longest possible set of characters. For example, the characters === would be interpreted as == and = and not as = followed by ==. We will shortly see other examples in our project.
Outside the present challenge, where the author of Lox decided to implement everything from scratch, regular expressions are commonly used to extract tokens.
There is a problem, however. If we don't want to use for_each we need to decide how to manage the input characters. We have two options:
Iterators are extremely convenient when it comes to memory management, as they don't require storing the full data in memory. In the present challenge, we will hardly have that problem, as we will never parse millions of lines, but it's nevertheless a good idea to keep the iterator paradigm. This will give us an excuse to learn how iterators are created in Rust.
The plan for this stage is the following:
Tokenizer into an iterator and discuss the iterator protocol.Tokenizer has a way to keep track of the current character.next to update the current character stored in Tokenizer.==.An iterator, in Rust, is a structure that implements the trait Iterator [docs]. The trait is rather simple to implement, despite the fact that it provides a huge amount of functionality. As you can see in the documentation we need to define the type of the Item and the method next.
src/tokenizer.rs impl<'a> Iterator for Tokenizer<'a> {
type Item = ();
fn next(&mut self) -> Option<Self::Item> {
let c = self.iter.next()?;
self.tokenize(c);
Some(())
}
}As you might have noticed, this is not a great iterator. The item type is the unit (), so the method next will return either Some(()) or None, which are not too helpful. At the moment, however, we are still printing tokens on the standard output, so this is an acceptable compromise for the first step.
Every time we call next we want our iterator to process the current character and print the token. The process must stop when there are no more characters. The syntax
let c = self.iter.next()?;is extremely compact and powerful. First, we call self.iter.next(), which can return either Some(char) (as self.iter is Chars) or None. If the result is None the operator ? terminates the function and returns None, otherwise it extracts the content of Some and stores it in the variable c. Basically, this is the equivalent of
let c = match self.iter.next() {
Some(c) => c,
None => { return None; },
};Once this is done, we can process c using self.tokenize and return its value (which at the moment is always the unit ()).
The operator ? [docs] is a very elegant way to manage error or termination conditions using Result and Option. To use it, you need to ensure compatibility between the return type of the function you are calling and the current function.
The best way to do this when complex types are involved is to implement automatic type conversion through the trait From [docs].
Once Iterator has been implemented, we need to adjust tokenize so that it doesn't iterate on self.iter any more. The new version receives the current character as an argument.
src/tokenizer.rs impl<'a> Tokenizer<'a> {
// Process each character in the iterator
// and print a token in the form
// `TYPE LEXEME null` if the token is valid or
// an error message if not.
pub fn tokenize(&mut self) {
self.iter.clone().for_each(|c| match c {
fn tokenize(&mut self, c: char) {
match c {
',' => println!("COMMA , null"),
'.' => println!("DOT . null"),
'-' => println!("MINUS - null"),
'+' => println!("PLUS + null"),
';' => println!("SEMICOLON ; null"),
'*' => println!("STAR * null"),
'(' => println!("LEFT_PAREN ( null"),
')' => println!("RIGHT_PAREN ) null"),
'{' => println!("LEFT_BRACE {{ null"),
'}' => println!("RIGHT_BRACE }} null"),
other => {
self.errors = true;
eprintln!("[line 1] Error: Unexpected character: {}", other);
}
});
}Last, we need to use Tokenizer as an iterator in main.
src/main.rs fn main() {
match command.as_str() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars(),
errors: false,
};
// Call the tokenizer.
t.tokenize();
// Iterate on the tokenizer.
for _item in &mut t {
continue;
}We need to use &mut t when we iterate because the for loop takes ownership of the iterator and this would make it impossible to access t.errors later.
Technically speaking, the ownership in the for loop is taken by an implicit call to into_iter.
It is fundamental for any Rust developer to learn what happens behind the scenes of loops, and in particular to make sure of the distinction between the methods iter, into_iter, and iter_mut.
You can read more about these topics in Rust By Example and in the Rust Book.
When parsing more than one character at a time, it is convenient to store the current character in an accessible field, and to have a method to progress the iterator that frees us from the rigidity of a for loop.
The method advance moves the iterator forward, stores the current character, and returns it.
src/tokenizer.rs pub struct Tokenizer<'a> {
pub iter: Chars<'a>,
pub current_character: Option<char>,
pub errors: bool,
}
impl<'a> Tokenizer<'a> {
// Advance the iterator and
// return the current character.
fn advance(&mut self) -> Option<char> {
// Advance the iterator and store
// the current character for future
// retrieval.
self.current_character = self.iter.next();
// Return the current character.
self.current_character
}The addition of a new field must be reflected where the structure is initialised.
src/main.rs fn main() {
match command.as_str() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars(),
current_character: None,
errors: false,
};The choice of Option<char> for the field current_character allows us to initialise it with None, but also makes the code that uses the field a little more complicated, as the value of the Option has to be checked every time. The question mark operator in Rust makes it easier to perform such a check, which would otherwise make the code more complicated and difficult to read, but the presence of an Option is at any rate impacting the code that uses the field.
A different option would be to just store a char and directly initialise Tokenizer with the first character, and this would have a clear impact on the design of all the other functions. At the end of the day, as happens many times, the final structure is mostly a matter of personal preference.
We can now use the field we added to update and read the current character instead of passing it as an argument to the method tokenize. Moreover, an Iterator in Rust returns an Option, so it is convenient to stick to that convention as much as possible in our internal functions.
src/tokenizer.rs impl<'a> Tokenizer<'a> {
// Process each character in the iterator
// and print a token in the form
// `TYPE LEXEME null` if the token is valid or
// an error message if not.
fn tokenize(&mut self, c: char) {
fn tokenize(&mut self) -> Option<()> {
// Extract the current character from
// the Option and return if it is None.
let c = self.current_character?;
match c {
',' => println!("COMMA , null"),
'.' => println!("DOT . null"),
'-' => println!("MINUS - null"),
'+' => println!("PLUS + null"),
';' => println!("SEMICOLON ; null"),
'*' => println!("STAR * null"),
'(' => println!("LEFT_PAREN ( null"),
')' => println!("RIGHT_PAREN ) null"),
'{' => println!("LEFT_BRACE {{ null"),
'}' => println!("RIGHT_BRACE }} null"),
other => {
self.errors = true;
eprintln!("[line 1] Error: Unexpected character: {}", other);
}
}
};
Some(())
}
impl<'a> Iterator for Tokenizer<'a> {
fn next(&mut self) -> Option<Self::Item> {
let c = self.iter.next()?;
// Advance the current token and
// return if we reached the end
// of the input file.
self.advance()?;
self.tokenize(c);
Some(())
// Run the tokenizer on the
// current character.
self.tokenize()
}The method advance is the first function that doesn't just print on the standard output but returns an actual value. This means that we can easily write a test for it.
src/tokenizer.rs #[cfg(test)]
mod tests {
use super::*;
#[test]
// Test that the iterator can be
// correctly advanced to the next
// character.
fn test_advance() {
let mut t = Tokenizer {
iter: "some string".chars(),
current_character: None,
errors: false,
};
// This advances the iterator to the initial character
t.advance();
// This advances the iterator to 'o'
let character = t.advance();
assert_eq!(t.current_character, Some('o'));
assert_eq!(character, Some('o'));
}
}A normal iterator in Rust can only see the current element but scanners often need to look ahead, reading the next element in advance. This allows the programming language to use richer syntax constructs like == that would be harder to process with a scanner that cannot look into the future.
A good solution from the standard library is Peekable [docs], which can be created directly from an iterator through the method peekable [docs].
src/tokenizer.rs use std::iter::Peekable;
use std::str::Chars;
pub struct Tokenizer<'a> {
pub iter: Chars<'a>,
pub iter: Peekable<Chars<'a>>,
pub current_character: Option<char>,
pub errors: bool,
}
mod tests {
fn test_advance() {
let mut t = Tokenizer {
iter: "some string".chars(),
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
};At this point, we clearly need to adjust the initialisation of Tokenizer in main.
src/main.rs fn main() {
match command.as_str() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars(),
iter: file_contents.chars().peekable(),
current_character: None,
errors: false,
};So far, we printed a string on the standard output every time we found a token. It is time to implement a proper structure for tokens, and to make sure our iterator returns those, so that they can be processed as we see fit in main.
It is also time to structure lexemes in a better way. The match construct in tokenize starts to become too long and we should find a better way to list the lexemes that the language supports and their names.
Token types and names can be implemented through an enum and the trait Display. Let's create both in a new file.
src/token.rs use std::fmt;
#[derive(Clone)]
pub enum TokenType {
Comma,
Dot,
LBrace,
LParen,
Minus,
Plus,
RBrace,
RParen,
Semicolon,
Star,
}
impl fmt::Display for TokenType {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let name = match self {
Self::Comma => "COMMA",
Self::Dot => "DOT",
Self::LBrace => "LEFT_BRACE",
Self::LParen => "LEFT_PAREN",
Self::Minus => "MINUS",
Self::Plus => "PLUS",
Self::RBrace => "RIGHT_BRACE",
Self::RParen => "RIGHT_PAREN",
Self::Semicolon => "SEMICOLON",
Self::Star => "STAR",
};
write!(f, "{}", name)
}
}This is a simple and idiomatic solution in Rust.
Next, we need a structure to represent a token. Since the scanning process can run into characters that are not part of the language, it's also useful to define an error type and since both the token and the error will be eventually represented as strings, we should implement Display for them as well.
src/token.rs pub struct Token {
pub ttype: TokenType,
pub lexeme: String,
pub literal: String,
}
impl fmt::Display for Token {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "{} {} {}", self.ttype, self.lexeme, self.literal)
}
}
pub struct TokenError {
pub message: String,
}
impl fmt::Display for TokenError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "[line 1] Error: {}", self.message)
}
}Last, we need to keep a list of the actual lexemes paired with the corresponding enum variant. We can do it by declaring a const like the following
src/token.rs pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
(",", TokenType::Comma),
(".", TokenType::Dot),
("{", TokenType::LBrace),
("(", TokenType::LParen),
("-", TokenType::Minus),
("+", TokenType::Plus),
("}", TokenType::RBrace),
(")", TokenType::RParen),
(";", TokenType::Semicolon),
("*", TokenType::Star),
];The keyword const [docs] declares a constant in Rust, that is a named value computed at compile time and inlined every time it is used. Unlike variables, that are created with let, constants must be annotated with an explicit type and cannot be shadowed or reassigned. By convention, their names use SCREAMING_SNAKE_CASE (see RFC 430).
Constants should not be confused with static items [docs]. A static occupies a fixed location in memory and lives for the entire duration of the program, while a const has no address of its own: the compiler is free to duplicate its value wherever it is used. For references, this is the reason why the 'static lifetime is implicit in constants.
The value assigned to a const must be a constant expression [docs], which means it can only invoke functions marked as const fn. This is a strong restriction, and rules out any operation that requires allocation on the heap or non-trivial runtime setup.
You can read a more complete discussion in the Rust Book and in the Rust Reference.
During the scanning process we need to connect known characters to lexemes, and the best structure for that job is a hash map. We can create it from the list above with a function
src/token.rs use std::collections::HashMap;
pub fn build_lexemes_map(list: &[(&str, TokenType)]) -> HashMap<String, TokenType> {
HashMap::from_iter(list.iter().map(|n| (n.0.to_owned(), n.1.clone())))
}In this case we cannot create a const HashMap directly, as inserting entries requires heap allocation, which the compiler cannot perform at compile time. See the HashMap docs and the chapter on Constant Evaluation.
There are ways to implement that, however. For compile-time populated maps use the phf crate, and for lazily initialised static maps, use std::sync::LazyLock [docs].
In this book we will go for a simple vanilla solution.
Now we can add a field to Tokenizer that contains the lexemes hash map.
src/tokenizer.rs use std::collections::HashMap;
use std::iter::Peekable;
use std::str::Chars;
use crate::token::TokenType;
pub struct Tokenizer<'a> {
pub iter: Peekable<Chars<'a>>,
pub current_character: Option<char>,
pub errors: bool,
// A map of single-character lexemes.
pub sc_lexemes: HashMap<String, TokenType>,
}
impl<'a> Tokenizer<'a> {
// Process each character in the iterator
// and print a token in the form
// `TYPE LEXEME null` if the token is valid or
// an error message if not.
fn tokenize(&mut self) -> Option<()> {
// Extract the current character from
// the Option and return if it is None.
let c = self.current_character?;
match c {
',' => println!("COMMA , null"),
'.' => println!("DOT . null"),
'-' => println!("MINUS - null"),
'+' => println!("PLUS + null"),
';' => println!("SEMICOLON ; null"),
'*' => println!("STAR * null"),
'(' => println!("LEFT_PAREN ( null"),
')' => println!("RIGHT_PAREN ) null"),
'{' => println!("LEFT_BRACE {{ null"),
'}' => println!("RIGHT_BRACE }} null"),
other => {
// Look for the current character in
// the map of single-character lexemes.
match self.sc_lexemes.get(&String::from(c)) {
Some(ttype) => println!("{} {} null", ttype, c),
None => {
self.errors = true;
eprintln!("[line 1] Error: Unexpected character: {}", other);
eprintln!("[line 1] Error: Unexpected character: {}", c);
}
};
Some(())
}We must also adjust the existing tests and create a new one for the new error condition.
src/tokenizer.rs mod tests {
#[test]
// Test that the iterator can be
// correctly advanced to the next
// character.
fn test_advance() {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
};
// This advances the iterator to the initial character
t.advance();
// This advances the iterator to 'o'
let character = t.advance();
assert_eq!(t.current_character, Some('o'));
assert_eq!(character, Some('o'));
}
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is not
// initialised and there is no current
// character.
fn test_tokenize_no_current_character() {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
};
assert_eq!(t.tokenize(), None);
}Last, we have to update the code in main that initialises Tokenizer.
src/main.rs use std::env;
use std::fs;
use std::process;
use crate::token::{build_lexemes_map, SINGLE_CHARACTER_LEXEMES};
use crate::tokenizer::Tokenizer;
mod token;
mod tokenizer;
fn main() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(SINGLE_CHARACTER_LEXEMES),
};The last step of this refactoring is to define and use a TokenResult as the Item type of the iterator. So far, the iterator returns units and the function tokenize prints results directly on the standard output. It is arguably better to separate the concerns and let the iterator return tokens (or errors) and the logic that loops on it decide what to do with them. It is also high time we introduced tests for the scanning logic, and separation of concerns is at the core of any successful testing strategy.
The definition of TokenResult is straightforward.
src/token.rs pub type TokenResult = Result<Token, TokenError>;With that, we can change the implementation of tokenize so that it returns an Option<TokenResult> instead of an Option<()>.
src/tokenizer.rs use std::collections::HashMap;
use std::iter::Peekable;
use std::str::Chars;
use crate::token::TokenType;
use crate::token::{Token, TokenError, TokenResult, TokenType};
pub struct Tokenizer<'a> {
pub iter: Peekable<Chars<'a>>,
pub current_character: Option<char>,
pub errors: bool,
// A map of single-character lexemes.
pub sc_lexemes: HashMap<String, TokenType>,
}
impl<'a> Tokenizer<'a> {
// Advance the iterator and
// return the current character.
fn advance(&mut self) -> Option<char> {
// Advance the iterator and store
// the current character for future
// retrieval.
self.current_character = self.iter.next();
// Return the current character.
self.current_character
}
// Process each character in the iterator
// and print a token in the form
// `TYPE LEXEME null` if the token is valid or
// an error message if not.
fn tokenize(&mut self) -> Option<()> {
// and return a token result.
fn tokenize(&self) -> Option<TokenResult> {
// Extract the current character from
// the Option and return if it is None.
let c = self.current_character?;
// Look for the current character in
// the map of single-character lexemes.
match self.sc_lexemes.get(&String::from(c)) {
Some(ttype) => println!("{} {} null", ttype, c),
None => {
self.errors = true;
eprintln!("[line 1] Error: Unexpected character: {}", c);
}
};
Some(())
// Return None if no character can be found.
let ttype = self.sc_lexemes.get(&String::from(c))?.clone();
Some(Ok(Token {
ttype,
lexeme: String::from(c),
literal: String::from("null"),
}))
}
}
impl<'a> Iterator for Tokenizer<'a> {
type Item = ();
type Item = TokenResult;
fn next(&mut self) -> Option<Self::Item> {
// Advance the current token and
// return if we reached the end
// of the input file.
self.advance()?;
// Run the tokenizer on the
// current character.
self.tokenize()
let current = self.advance()?;
// Run the tokenizer on the current character
// and return the result if not None.
if let Some(t) = self.tokenize() {
return Some(t);
}
// If we reach this point we couldn't
// match the current character with
// any lexeme, so this is an error.
self.errors = true;
Some(Err(TokenError {
message: format!("Unexpected character: {}", current),
}))
}
}With this implementation, we can write a good amount of tests. We can check that the function tokenize correctly detects valid and invalid characters.
src/tokenizer.rs mod tests {
use super::*;
use crate::token::build_lexemes_map;
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is not
// initialised and there is no current
// character.
fn test_tokenize_no_current_character() {
let mut t = Tokenizer {
let t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
};
assert_eq!(t.tokenize(), None);
}
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is
// initialised and there is a current
// character.
fn test_tokenize_valid_character() {
let mut t = Tokenizer {
iter: ",".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[(",", TokenType::Comma)]),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.tokenize();
assert_eq!(
token,
Some(Ok(Token {
ttype: TokenType::Comma,
lexeme: String::from(","),
literal: String::from("null"),
}))
);
}
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is
// initialised and there is a current
// character, but the character is
// not a valid lexeme.
fn test_tokenize_invalid_character() {
let mut t = Tokenizer {
iter: "@".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[(",", TokenType::Comma)]),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.tokenize();
assert_eq!(token, None);
}All the tests we have written so far are very granular, and it's time we started introducing some tests for the iterator as a whole. These can still be considered unit tests, in particular because they don't require setting up any external system, but it would be useful to keep them separated to highlight the different approach they entail.
Fortunately, Rust allows us to create as many test modules as we want, so separating tests is just a matter of creating a new mod.
src/tokenizer.rs #[cfg(test)]
mod iterator_tests {
use super::*;
use crate::token::build_lexemes_map;
#[test]
// Test the iterator end-to-end on
// a string of valid characters.
fn test_valid_characters() {
const LEXEMES: &[(&str, TokenType)] = &[
(",", TokenType::Comma),
("(", TokenType::LParen),
("+", TokenType::Plus),
];
let t = Tokenizer {
iter: ",(+".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(LEXEMES),
};
let result: Vec<TokenResult> = t.collect();
assert_eq!(
result,
[
Ok(Token {
ttype: TokenType::Comma,
lexeme: String::from(","),
literal: String::from("null")
}),
Ok(Token {
ttype: TokenType::LParen,
lexeme: String::from("("),
literal: String::from("null")
}),
Ok(Token {
ttype: TokenType::Plus,
lexeme: String::from("+"),
literal: String::from("null")
}),
]
);
}
#[test]
// Test the iterator end-to-end on
// a string that contains an invalid character.
fn test_invalid_characters() {
const LEXEMES: &[(&str, TokenType)] = &[
(",", TokenType::Comma),
("(", TokenType::LParen),
("+", TokenType::Plus),
];
let mut t = Tokenizer {
iter: ",@+".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(LEXEMES),
};
let mut result = Vec::<TokenResult>::new();
for token in &mut t {
result.push(token);
}
assert_eq!(
result,
[
Ok(Token {
ttype: TokenType::Comma,
lexeme: String::from(","),
literal: String::from("null")
}),
Err(TokenError {
message: String::from("Unexpected character: @")
}),
Ok(Token {
ttype: TokenType::Plus,
lexeme: String::from("+"),
literal: String::from("null")
}),
]
);
assert!(t.errors);
}
#[test]
// Test the iterator end-to-end on
// an empty string.
fn test_empty_input() {
let t = Tokenizer {
iter: "".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[]),
};
let result: Vec<TokenResult> = t.collect();
assert_eq!(result, []);
}
}To make these tests work we need to derive Debug and PartialEq for TokenType, Token, and TokenError.
src/token.rs #[derive(Clone)]
#[derive(Clone, Debug, PartialEq)]
pub enum TokenType {
Comma,
Dot,
LBrace,
LParen,
Minus,
Plus,
RBrace,
RParen,
Semicolon,
Star,
}
#[derive(Debug, PartialEq)]
pub struct Token {
pub ttype: TokenType,
pub lexeme: String,
pub literal: String,
}
#[derive(Debug, PartialEq)]
pub struct TokenError {
pub message: String,
}Last, since we delegated the processing of token results to the function that calls the iterator, we need to update main.
src/main.rs fn main() {
match command.as_str() {
// Iterate on the tokenizer.
for _item in &mut t {
continue;
// Iterate on the tokenizer
// and process each result.
for token in &mut t {
match token {
Ok(token) => {
println!("{}", token);
}
Err(e) => {
eprintln!("{}", e);
}
}
}At this point, Clippy highlights the fact that TokenType and TokenError are deriving PartialEq and can derive Eq as well. The official documentation of the warning derive_partial_eq_without_eq states: "if a type T derives PartialEq and all of its members implement Eq, then T can always implement Eq. Implementing Eq allows T to be used in APIs that require Eq types. It also allows structs containing T to derive Eq themselves."
So, clearly in this case it's a matter of future-proofing the types more than an immediate need. Once TokenType becomes Eq, Clippy will highlight that Token can become Eq as well.
src/token.rs #[derive(Clone, Debug, PartialEq)]
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum TokenType {
Comma,
Dot,
LBrace,
LParen,
Minus,
Plus,
RBrace,
RParen,
Semicolon,
Star,
}
#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Eq)]
pub struct Token {
pub ttype: TokenType,
pub lexeme: String,
pub literal: String,
}
#[derive(Debug, PartialEq)]
#[derive(Debug, PartialEq, Eq)]
pub struct TokenError {
pub message: String,
}PartialEq vs EqWhat is the difference between PartialEq and Eq, and why do we need both?
The official documentation says that PartialEq is a "trait for comparisons using the equality operator", while Eq is "for comparisons corresponding to equivalence relations". The mathematics behind equivalence relations is interesting, but in this instance we can keep things simple.
PartialEq is a trait that requires the implementation of the actual method eq, that is compiled code that compares an instance of a type with another instance of the same type. Clearly, the code might or might not actually perform the check, but with PartialEq we provide a method that will be called upon comparison. At compile time, the code x == y becomes x.eq(y).
Eq is a trait that promises reflexivity, but that doesn't implement anything. It just states that the compiler can rely on the fact that x == x. The classic (and somewhat niche) examples of non-reflexive entities are "not found" errors or values like NaN (not a number) that you get when you perform illegal mathematical operations. An error like NaN cannot be equal to itself because there is nothing to compare, and practically speaking this means that an operation that returns NaN cannot pass an equality test with another NaN. This prevents tests from passing when one compares the results of two operations that can fail, for different reasons, with the same error.
After all this work, we are finally ready to add multi-character lexemes to the scanner. In this stage we need to identify == as a single token, rather than two tokens corresponding to adjacent characters =. The strategy is simple: we will collect the current character and the next (visible thanks to std::iter::Peekable) and try to find this lexeme in a list similar to SINGLE_CHARACTER_LEXEMES. If a match is found, we will return the token, otherwise we will try the single-character algorithm.
Let's start creating the lexemes.
src/token.rs #[derive(Clone, Debug, PartialEq, Eq)]
pub enum TokenType {
Comma,
Dot,
Equal,
EqualEqual,
LBrace,
LParen,
Minus,
Plus,
RBrace,
RParen,
Semicolon,
Star,
}
impl fmt::Display for TokenType {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let name = match self {
Self::Comma => "COMMA",
Self::Dot => "DOT",
Self::Equal => "EQUAL",
Self::EqualEqual => "EQUAL_EQUAL",
Self::LBrace => "LEFT_BRACE",
Self::LParen => "LEFT_PAREN",
Self::Minus => "MINUS",
Self::Plus => "PLUS",
Self::RBrace => "RIGHT_BRACE",
Self::RParen => "RIGHT_PAREN",
Self::Semicolon => "SEMICOLON",
Self::Star => "STAR",
};
write!(f, "{}", name)
}
}
pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
(",", TokenType::Comma),
(".", TokenType::Dot),
("=", TokenType::Equal),
("{", TokenType::LBrace),
("(", TokenType::LParen),
("-", TokenType::Minus),
("+", TokenType::Plus),
("}", TokenType::RBrace),
(")", TokenType::RParen),
(";", TokenType::Semicolon),
("*", TokenType::Star),
];
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[("==", TokenType::EqualEqual)];Nothing new here. The variants of TokenType correspond to the single and the double equal, and each has a string representation. The lexeme for the single equal goes in the list of single-character lexemes, while for the double equal we create a new list.
The struct Tokenizer needs a new field to host a list of multi-character lexemes.
src/tokenizer.rs pub struct Tokenizer<'a> {
pub iter: Peekable<Chars<'a>>,
pub current_character: Option<char>,
pub errors: bool,
// A map of single-character lexemes.
pub sc_lexemes: HashMap<String, TokenType>,
// A map of multi-character lexemes.
pub mc_lexemes: HashMap<String, TokenType>,
}After which we can define the method Tokenizer::multi_character_token.
src/tokenizer.rs impl<'a> Tokenizer<'a> {
// Test the current and next characters and
// try to match one of the multi-character lexemes
fn multi_character_token(&mut self) -> Option<TokenResult> {
// If the current character is None return None.
let c = self.current_character?;
// If there is nothing after this character just return None.
let &n = self.iter.peek()?;
// Build the full token.
let full = [c, n].iter().collect::<String>();
// If the full token is not in the multi-character
// tokens just return None.
let ttype = self.mc_lexemes.get(&String::from(&full))?.clone();
// Since we found a match, we must
// skip the second character.
self.advance();
Some(Ok(Token {
ttype,
lexeme: full,
literal: String::from("null"),
}))
}Here, we follow a strategy similar to the one implemented in tokenize. First of all we get the current character and then we peek at the next one. Remember that since we are inside an iterator the item is an Option and not a character itself. Strictly speaking, it is unnecessary to check the current character, as we already did it at the beginning of the method next, but the syntax self.current_character? is a very elegant way to deconstruct the option.
The result of peek instead must be checked because we might be at the end of the file and there might not be a next character, in which case the scan for a multi-character lexeme should fail. Once we have created a string from the current and the next characters we can look it up in the hash map self.mc_lexemes and we can once again leverage the operator ? for an elegant error management solution. Finally, if we find a match we move forward, consuming the second character, and we return the token.
At this point, it makes sense to rename tokenize to single_character_token to keep naming consistent.
src/tokenizer.rs impl<'a> Tokenizer<'a> {
// Process each character in the iterator
// and return a token result.
fn tokenize(&self) -> Option<TokenResult> {
// Test the current character and
// try to match one of the single-character lexemes
fn single_character_token(&self) -> Option<TokenResult> {
// Extract the current character from
// the Option and return if it is None.
let c = self.current_character?;
// Look for the current character in
// the map of single-character lexemes.
// Return None if no character can be found.
let ttype = self.sc_lexemes.get(&String::from(c))?.clone();
Some(Ok(Token {
ttype,
lexeme: String::from(c),
literal: String::from("null"),
}))
}And both functions must be called in Tokenizer::next.
src/tokenizer.rs impl<'a> Iterator for Tokenizer<'a> {
fn next(&mut self) -> Option<Self::Item> {
// Advance the current token and
// return if we reached the end
// of the input file.
let current = self.advance()?;
// Try to match a multi-character token.
if let Some(t) = self.multi_character_token() {
return Some(t);
}
// Run the tokenizer on the current character
// and return the result if not None.
if let Some(t) = self.tokenize() {
return Some(t);
}
// Try to match a single-character token.
if let Some(t) = self.single_character_token() {
return Some(t);
}
// If we reach this point we couldn't
// match the current character with
// any lexeme, so this is an error.
self.errors = true;
Some(Err(TokenError {
message: format!("Unexpected character: {}", current),
}))
}There is an important concept hidden in plain sight here. Scanners, just like parsers later, need to establish precedence between matching algorithms. As mentioned before, the example at hand is that the syntax === should be interpreted as == followed by =, and not the opposite. When it comes to parsers, another classic example is the precedence between multiplication and addition in mathematical expressions.
As we see in Tokenizer::next, the simplest way to establish precedence is to call a matching function before another. If we called single_character_token before multi_character_token we would change the precedence. As we will see later with strings and numbers, sometimes the order of functions is not important because tokens are unambiguous, but in general we need to be aware that call order is relevant.
The existing tests need to be updated to reflect the changes in function names and structure fields.
src/tokenizer.rs mod tests {
fn test_advance() {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: HashMap::new(),
};
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is not
// Test that the method single_character_token
// behaves correctly when the tokeniser is not
// initialised and there is no current
// character.
fn test_tokenize_no_current_character() {
let t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: HashMap::new(),
};
assert_eq!(t.tokenize(), None);
assert_eq!(t.single_character_token(), None);
}
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is
// Test that the method single_character_token
// behaves correctly when the tokeniser is
// initialised and there is a current
// character.
fn test_tokenize_valid_character() {
let mut t = Tokenizer {
iter: ",".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[(",", TokenType::Comma)]),
mc_lexemes: HashMap::new(),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.tokenize();
let token = t.single_character_token();
assert_eq!(
token,
Some(Ok(Token {
ttype: TokenType::Comma,
lexeme: String::from(","),
literal: String::from("null"),
}))
);
}
#[test]
// Test that the method tokenize behaves
// correctly when the tokeniser is
// Test that the method single_character_token
// behaves correctly when the tokeniser is
// initialised and there is a current
// character, but the character is
// not a valid lexeme.
fn test_tokenize_invalid_character() {
let mut t = Tokenizer {
iter: "@".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[(",", TokenType::Comma)]),
mc_lexemes: HashMap::new(),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.tokenize();
let token = t.single_character_token();
assert_eq!(token, None);
}
mod iterator_tests {
fn test_valid_characters() {
let t = Tokenizer {
iter: ",(+".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(LEXEMES),
mc_lexemes: HashMap::new(),
};
fn test_invalid_characters() {
let mut t = Tokenizer {
iter: ",@+".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(LEXEMES),
mc_lexemes: HashMap::new(),
};
fn test_empty_input() {
let t = Tokenizer {
iter: "".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(&[]),
mc_lexemes: HashMap::new(),
};We can also add three new tests to check the method multi_character_token with valid, invalid, and no characters.
src/tokenizer.rs mod tests {
#[test]
// Test that the method multi_character_token
// behaves correctly when the tokeniser is not
// initialised and there is no current
// character.
fn test_multi_character_no_current_character() {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: HashMap::new(),
};
assert_eq!(t.multi_character_token(), None);
}
#[test]
// Test that the method multi_character_token
// behaves correctly when the tokeniser is
// initialised and there is a current
// character.
fn test_multi_character_valid_character() {
let mut t = Tokenizer {
iter: "==".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: build_lexemes_map(&[("==", TokenType::EqualEqual)]),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.multi_character_token();
assert_eq!(
token,
Some(Ok(Token {
ttype: TokenType::EqualEqual,
lexeme: String::from("=="),
literal: String::from("null"),
}))
);
}
#[test]
// Test that the method multi_character_token
// behaves correctly when the tokeniser is
// initialised and there is a current
// character, but the character is
// not a valid lexeme.
fn test_multi_character_invalid_character() {
let mut t = Tokenizer {
iter: "@".chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: build_lexemes_map(&[("==", TokenType::EqualEqual)]),
};
// This advances the iterator to the initial character
t.advance();
// Run the tokenizer on the initial character
let token = t.multi_character_token();
assert_eq!(token, None);
}Last, we need to build the hash map that contains multi-character lexemes in main when we instantiate Tokenizer.
src/main.rs use std::env;
use std::fs;
use std::process;
use crate::token::{build_lexemes_map, SINGLE_CHARACTER_LEXEMES};
use crate::token::{build_lexemes_map, MULTI_CHARACTER_LEXEMES, SINGLE_CHARACTER_LEXEMES};
use crate::tokenizer::Tokenizer;
mod token;
mod tokenizer;
fn main() {
match command.as_str() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars().peekable(),
current_character: None,
errors: false,
sc_lexemes: build_lexemes_map(SINGLE_CHARACTER_LEXEMES),
mc_lexemes: build_lexemes_map(MULTI_CHARACTER_LEXEMES),
};Stage 6 - Scanning: Assignment & equality Operators
This version of the code passes Stage 6 of the base CodeCrafters challenge.
This and the next stage will be trivial thanks to the structure that we built. In this stage we need to add support for the single character negation ! and for the inequality !=.
src/token.rs pub enum TokenType {
Bang,
BangEqual,
Comma,
Dot,
impl fmt::Display for TokenType {
Self::Bang => "BANG",
Self::BangEqual => "BANG_EQUAL",
Self::Comma => "COMMA",
Self::Dot => "DOT",
pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
("!", TokenType::Bang),
(",", TokenType::Comma),
(".", TokenType::Dot),
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[("==", TokenType::EqualEqual)];
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] =
&[("!=", TokenType::BangEqual), ("==", TokenType::EqualEqual)];Stage 7 - Scanning: Negation & inequality operators
This version of the code passes Stage 7 of the base CodeCrafters challenge.
In this stage, we will apply a set of changes that mirrors what we did in the previous one, targeting the two single characters < and > and the corresponding multi-character versions <= and >=.
src/token.rs pub enum TokenType {
Equal,
EqualEqual,
Greater,
GreaterEqual,
LBrace,
LParen,
Less,
LessEqual,
Minus,
impl fmt::Display for TokenType {
Self::Equal => "EQUAL",
Self::EqualEqual => "EQUAL_EQUAL",
Self::Greater => "GREATER",
Self::GreaterEqual => "GREATER_EQUAL",
Self::LBrace => "LEFT_BRACE",
Self::LParen => "LEFT_PAREN",
Self::Less => "LESS",
Self::LessEqual => "LESS_EQUAL",
Self::Minus => "MINUS",
pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
("=", TokenType::Equal),
(">", TokenType::Greater),
("{", TokenType::LBrace),
("(", TokenType::LParen),
("<", TokenType::Less),
("-", TokenType::Minus),
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] =
&[("!=", TokenType::BangEqual), ("==", TokenType::EqualEqual)];
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
("!=", TokenType::BangEqual),
("==", TokenType::EqualEqual),
(">=", TokenType::GreaterEqual),
("<=", TokenType::LessEqual),
];Stage 8 - Scanning: Relational operators
This version of the code passes Stage 8 of the base CodeCrafters challenge.
This stage presents a new challenge. Adding the division operator / is not a big issue, and we can do it exactly as we did for characters like ! or {.
However, the double slash // has a peculiarity: anything that follows that lexeme on the same line has to be ignored, being a comment. The implementation of this feature is not complicated, but it is something different from what we have done so far.
We can start by defining the lexemes as we did in the previous steps.
src/token.rs pub enum TokenType {
Comma,
Comment,
Dot,
...
Semicolon,
Slash,
Star,
impl fmt::Display for TokenType {
Self::Comma => "COMMA",
Self::Comment => "COMMENT",
Self::Dot => "DOT",
...
Self::Semicolon => "SEMICOLON",
Self::Slash => "SLASH",
Self::Star => "STAR",
pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
(";", TokenType::Semicolon),
("/", TokenType::Slash),
("*", TokenType::Star),
pub const MULTI_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
("!=", TokenType::BangEqual),
("//", TokenType::Comment),
("==", TokenType::EqualEqual),This is sufficient to process a single / but still doesn't process comments correctly. It's the first time we see a multi-line input file, so we need to introduce a field of Tokenizer that keeps track of the current line.
src/tokenizer.rs pub struct Tokenizer<'a> {
pub iter: Peekable<Chars<'a>>,
pub current_character: Option<char>,
pub current_line: u16,
pub errors: bool,
// A map of single-character lexemes.
pub sc_lexemes: HashMap<String, TokenType>,
// A map of multi-character lexemes.
pub mc_lexemes: HashMap<String, TokenType>,
}
impl<'a> Tokenizer<'a> {
fn multi_character_token(&mut self) -> Option<TokenResult> {
// Since we found a match, we must
// skip the second character.
self.advance();
// If we found a comment we need to
// skip the whole line.
if ttype == TokenType::Comment {
// Advance the iterator until we
// reach the end of line
while let Some(v) = self.advance() {
if v == '\n' {
break;
}
}
// We are skipping this line,
// increase the line number.
self.current_line += 1;
} else {
// Since we found a match, we must
// skip the second character.
self.advance();
}If we just identified a comment token, the while loop advances the current character until we reach the end of the line (\n) or the end of the file (None). In any case, the current line has been fully processed, and we need to increase the line number. However, at this point we can't advance again as we did before, so the code self.advance() has to become part of the else branch.
Having added a new field to the structure, we need to fix all tests. This is a rather straightforward task, so the code below shows only one of the required changes. We also need to add a new test to check that comments are treated correctly.
src/tokenizer.rs mod tests {
#[test]
// Test that the iterator can be
// correctly advanced to the next
// character.
fn test_advance() {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
current_line: 1,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: HashMap::new(),
};
// This advances the iterator to the initial character
t.advance();
// This advances the iterator to 'o'
let character = t.advance();
assert_eq!(t.current_character, Some('o'));
assert_eq!(character, Some('o'));
}
// FIX ALL REMAINING TESTS
mod iterator_tests {
#[test]
// Test the iterator end-to-end on
// a commented line.
fn test_comment() {
const SINGLE_LEXEMES: &[(&str, TokenType)] = &[("/", TokenType::Slash)];
const MULTI_LEXEMES: &[(&str, TokenType)] = &[("//", TokenType::Comment)];
let mut t = Tokenizer {
iter: "// Comment".chars().peekable(),
current_character: None,
current_line: 1,
errors: false,
sc_lexemes: build_lexemes_map(SINGLE_LEXEMES),
mc_lexemes: build_lexemes_map(MULTI_LEXEMES),
};
let mut result = Vec::<TokenResult>::new();
for token in &mut t {
result.push(token);
}
assert_eq!(
result,
[Ok(Token {
ttype: TokenType::Comment,
lexeme: String::from("//"),
literal: String::from("null"),
})]
);
assert_eq!(t.current_line, 2);
}At this point, we have everything we need to skip comment tokens in main:
src/main.rs use std::env;
use std::fs;
use std::process;
use crate::token::TokenType;
use crate::token::{build_lexemes_map, MULTI_CHARACTER_LEXEMES, SINGLE_CHARACTER_LEXEMES};
use crate::tokenizer::Tokenizer;
mod token;
mod tokenizer;
fn main() {
match command.as_str() {
// Create a new tokenizer and
// instantiate it with the content
// of the input file.
let mut t = Tokenizer {
iter: file_contents.chars().peekable(),
current_character: None,
current_line: 1,
errors: false,
sc_lexemes: build_lexemes_map(SINGLE_CHARACTER_LEXEMES),
mc_lexemes: build_lexemes_map(MULTI_CHARACTER_LEXEMES),
};
// Iterate on the tokenizer
// and process each result.
for token in &mut t {
match token {
Ok(token) => {
println!("{}", token);
}
Ok(token) => match token.ttype {
TokenType::Comment => continue,
_ => println!("{}", token),
},
Err(e) => {
eprintln!("{}", e);
}
}
}Stage 9 - Scanning: Division operator & comments
This version of the code passes Stage 9 of the base CodeCrafters challenge.
This stage of the challenge introduces whitespace. On the one hand, whitespace (both the space and the tab) is extremely important to separate lexemes in order to disambiguate the syntax of our language, but on the other hand once we identify it we can safely ignore it. The stage silently introduces newlines outside comments, so we will have to deal with those as well.
As we did for comments, we can fully implement the scanning mechanism and defer ignoring the produced tokens to the very end of the process.
src/token.rs pub enum TokenType {
Minus,
Newline,
Plus,
RBrace,
RParen,
Semicolon,
Slash,
Star,
Whitespace,
impl fmt::Display for TokenType {
Self::Minus => "MINUS",
Self::Newline => "NEWLINE",
Self::Plus => "PLUS",
Self::RBrace => "RIGHT_BRACE",
Self::RParen => "RIGHT_PAREN",
Self::Semicolon => "SEMICOLON",
Self::Slash => "SLASH",
Self::Star => "STAR",
Self::Whitespace => "WHITESPACE",
pub const SINGLE_CHARACTER_LEXEMES: &[(&str, TokenType)] = &[
("-", TokenType::Minus),
("\n", TokenType::Newline),
("+", TokenType::Plus),
("}", TokenType::RBrace),
(")", TokenType::RParen),
(";", TokenType::Semicolon),
("/", TokenType::Slash),
("*", TokenType::Star),
(" ", TokenType::Whitespace),
("\t", TokenType::Whitespace),For the first time, we have multiple lexemes (space and tab) that correspond to the same token type.
We have to update single_character_token to deal with newlines.
src/tokenizer.rs impl<'a> Tokenizer<'a> {
// Test the current character and
// try to match one of the single-character lexemes
fn single_character_token(&self) -> Option<TokenResult> {
fn single_character_token(&mut self) -> Option<TokenResult> {
// Extract the current character from
// the Option and return if it is None.
let c = self.current_character?;
// Look for the current character in
// the map of single-character lexemes.
// Return None if no character can be found.
let ttype = self.sc_lexemes.get(&String::from(c))?.clone();
// If the token we read is a newline we need to
// update the current line.
if ttype == TokenType::Newline {
self.current_line += 1;
}
Some(Ok(Token {
ttype,
lexeme: String::from(c),
literal: String::from("null"),
}))
}Given the change to the signature we need to adjust one existing test, and to check the additional logic we can add a new test.
src/tokenizer.rs mod tests {
#[test]
// Test that the method single_character_token
// behaves correctly when the tokeniser is not
// initialised and there is no current
// character.
fn test_tokenize_no_current_character() {
let t = Tokenizer {
let mut t = Tokenizer {
iter: "some string".chars().peekable(),
current_character: None,
current_line: 1,
errors: false,
sc_lexemes: HashMap::new(),
mc_lexemes: HashMap::new(),
};
assert_eq!(t.single_character_token(), None);
}
mod iterator_tests {
#[test]
// Test the iterator end-to-end on
// a single new line character.
fn test_newline() {
const SINGLE_LEXEMES: &[(&str, TokenType)] = &[("\n", TokenType::Newline)];
let mut t = Tokenizer {
iter: "\n".chars().peekable(),
current_character: None,
current_line: 1,
errors: false,
sc_lexemes: build_lexemes_map(SINGLE_LEXEMES),
mc_lexemes: HashMap::new(),
};
let mut result = Vec::<TokenResult>::new();
for token in &mut t {
result.push(token);
}
assert_eq!(
result,
[Ok(Token {
ttype: TokenType::Newline,
lexeme: String::from("\n"),
literal: String::from("null"),
})]
);
assert_eq!(t.current_line, 2);
}As mentioned, the scanner is going to identify whitespace tokens, but the calling logic is in charge of removing them. To do this, we need to update the match construct in main.
src/main.rs fn main() {
match command.as_str() {
for token in &mut t {
match token {
Ok(token) => match token.ttype {
TokenType::Comment => continue,
TokenType::Whitespace => continue,
TokenType::Newline => continue,
_ => println!("{}", token),
},
Err(e) => {
eprintln!("{}", e);
}
}
}Stage 10 - Scanning: Whitespace
This version of the code passes Stage 10 of the base CodeCrafters challenge.
This stage requires our code to print out errors with the correct line number, as so far tests contained only one line of Lox code. To implement this feature, we need to add a field to TokenError that hosts the line number. We also need to change the implementation of fmt::Display so that the line is shown correctly before the error message.
src/token.rs #[derive(Debug, PartialEq, Eq)]
pub struct TokenError {
pub message: String,
pub line: u16,
}
impl fmt::Display for TokenError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "[line 1] Error: {}", self.message)
write!(f, "[line {}] Error: {}", self.line, self.message)
}
}Clearly, we need to initialise the new field when we create a TokenError.
src/tokenizer.rs impl<'a> Iterator for Tokenizer<'a> {
fn next(&mut self) -> Option<Self::Item> {
// If we reach this point we couldn't
// match the current character with
// any lexeme, so this is an error.
self.errors = true;
Some(Err(TokenError {
message: format!("Unexpected character: {}", current),
line: self.current_line,
}))
mod iterator_tests {
fn test_invalid_characters() {
assert_eq!(
result,
[
Ok(Token {
ttype: TokenType::Comma,
lexeme: String::from(","),
literal: String::from("null")
}),
Err(TokenError {
message: String::from("Unexpected character: @")
message: String::from("Unexpected character: @"),
line: 1,
}),
Ok(Token {
ttype: TokenType::Plus,
lexeme: String::from("+"),
literal: String::from("null")
}),
]
);Stage 11 - Scanning: Multi-line errors
This version of the code passes Stage 11 of the base CodeCrafters challenge.