use crate::expression::Expression;
#[derive(Debug, PartialEq)]
pub enum Statement {
Print(Expression),
}If I may, I'd like to begin by making a statement for the record.
So far, our interpreter has been processing single expressions and returning their value. This is enough to build a calculator, but as we know a programming language is much more than that. Programs are sequences of instructions that do something, like defining variables, printing values on the screen, or calling functions, and all these are grouped under the name statements. The main difference between expressions and statements is that the former produce a value, while the latter produce an effect.
In this chapter, we will tackle the extension "Statements & state" of the CodeCrafters challenge. We will start from the simple print and move on to expression statements, variable declarations, assignments, and code blocks. To do that, we will also introduce a place to store the values of variables, an environment.
The amount of code added in this chapter is significant, and we will soon spot some duplication in both the parser and the execution logic, which will require refactoring steps to make the codebase simpler to maintain.
However, we will not complete the extension in this chapter. Code blocks in their simplest form are the last feature we will implement here, as their full form requires an interaction between environments which is not trivial to implement in Rust. This will give us a lot to discuss in terms of the Rust language, so it deserves a chapter on its own.
In this stage, we need the interpreter to do two new things:
print EXPRESSION;.run.Both are simple, but the introduction of statements will highlight a code pattern that can benefit from a helper function, so we will implement the stage in 4 steps.
As we know, in a programming language there are different types of statements, so the best way to establish a good foundation for statement processing is to create a new enum:
src/statement.rs use crate::expression::Expression;
#[derive(Debug, PartialEq)]
pub enum Statement {
Print(Expression),
}The module needs to be added to the system:
src/main.rs mod evaluate;
mod expression;
mod parser;
mod statement;
mod token;
mod tokenizer;Parsing statements in the form print EXPRESSION; is not complicated, and we can do that with a new method Parser::statement:
src/parser.rs use crate::expression::{Expression, Literal};
use crate::statement::Statement;
use crate::token::{Keyword, Token, TokenLiteral, TokenType};
impl Parser {
// The main entry point of the parser.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Enforce the presence of the keyword
// `print`. If not present, the statement
// is invalid.
if self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_none()
{
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}
// Parse the expression contained
// in the print statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
if self.match_current(&[TokenType::Semicolon]).is_none() {
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}
Ok(Statement::Print(expression))
}
// The main entry point of the parser.
// Parse the next tokens as an expression.
pub fn expression(&mut self) -> ParserResult<Expression> {
self.equality()
}
...Currently, there is only one type of statement, so the method statement is very straightforward: it first enforces the presence of the keyword print, then parses the expression and enforces the presence of the semicolon. Later, we will need to support different types of statements, so the code will become more complicated.
We can add several tests to check the behaviour of the code:
src/parser/tests/statement.rs use super::*;
use crate::expression::{Expression, Literal};
#[test]
// Test that the method `statement` works
// correctly on a statement in the form
// `print EXPRESSION;`.
fn test_print_valid_statement() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Ok(Statement::Print(Expression::Literal(Literal::Boolean(
true
))))
);
assert_eq!(parser.index, 3);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a print
// statement that contains an invalid expression.
fn test_print_invalid_expression() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Equal),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid primary token"),
token: dummy_token(TokenType::Equal)
})
);
assert_eq!(parser.index, 1);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a
// statement that does not start with `print`.
fn test_print_missing_keyword() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::True)),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::Reserved(Keyword::True)),
})
);
assert_eq!(parser.index, 0);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a
// statement that does not end with a semicolon.
fn test_print_missing_semicolon() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::True)),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::Eof)
})
);
assert_eq!(parser.index, 2);
}The new method Parser::statement contains a repeated pattern that we will probably see many times in the future:
src/parser.rs impl Parser {
pub fn statement(&mut self) -> ParserResult<Statement> {
// Enforce the presence of the keyword
// `print`. If not present, the statement
// is invalid.
if self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_none()
{
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}
...
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
if self.match_current(&[TokenType::Semicolon]).is_none() {
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}Here, we need to make sure the code contains a specific token, in this case the keyword print and the semicolon. If the token is not present, we need to return an error with a given message and the current token for debug purposes. This behaviour can be captured by a helper method:
src/parser.rs impl Parser {
// Consume the current token if it matches
// the given type, otherwise return an
// error with the given message.
fn expect(&mut self, token_type: &TokenType, message: &str) -> ParserResult<&Token> {
if !self.check_current(token_type) {
return Err(ParserError {
message: String::from(message),
token: self.current_token().clone(),
});
}
Ok(self.consume())
}Importantly, the helper checks the current token and consumes it only if it matches the requested token type. The function must be tested:
src/parser/tests/utils.rs #[test]
// Test that the method `expect` consumes
// and returns the current token when its
// type matches the expected one.
fn test_expect_match() {
let tokens = vec![
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Identifier),
];
let mut parser = Parser::new(tokens);
let result = parser.expect(&TokenType::Semicolon, "Expected ';'");
assert_eq!(result, Ok(&dummy_token(TokenType::Semicolon)));
assert_eq!(parser.index, 1);
}
#[test]
// Test that the method `expect` returns
// an error with the given message and the
// current token when the types don't match.
fn test_expect_no_match() {
let tokens = vec![dummy_token(TokenType::Identifier), Token::eof()];
let mut parser = Parser::new(tokens);
let result = parser.expect(&TokenType::Semicolon, "Expected ';'");
assert_eq!(
result,
Err(ParserError {
message: String::from("Expected ';'"),
token: dummy_token(TokenType::Identifier)
})
);
assert_eq!(parser.index, 0);
}
#[test]
// Test that the method `expect` returns
// an error when the parser reached `Eof`,
// even if `Eof` is the expected type.
fn test_expect_end() {
let tokens = vec![Token::eof()];
let mut parser = Parser::new(tokens);
let result = parser.expect(&TokenType::Eof, "Expected end");
assert_eq!(
result,
Err(ParserError {
message: String::from("Expected end"),
token: Token::eof()
})
);
assert_eq!(parser.index, 0);
}We can then use it directly in Parser::statement:
src/parser.rs impl Parser {
// The main entry point of the parser.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Enforce the presence of the keyword
// `print`. If not present, the statement
// is invalid.
if self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_none()
{
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}
self.expect(&TokenType::Reserved(Keyword::Print), "Invalid statement")?;
// Parse the expression contained
// in the print statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
if self.match_current(&[TokenType::Semicolon]).is_none() {
return Err(ParserError {
message: String::from("Invalid statement"),
token: self.current_token().clone(),
});
}
self.expect(&TokenType::Semicolon, "Invalid statement")?;
Ok(Statement::Print(expression))
}There is another place in the code where we use the same code pattern: when the method Parser::primary parses a group it has to enforce the presence of the closing round bracket ). We cannot however directly replace the code with the helper Parser::expect because that contains a call to self.consume() and the whole function relies on a final common self.consume() that cannot be avoided.
Therefore, we can isolate the check for a group and move it to the beginning of the method:
src/parser.rs impl Parser {
fn primary(&mut self) -> ParserResult<Expression> {
// Check if we are dealing with a group.
// A group is closed by a specific token
// (RParen), which needs to be matched
// after the expression is parsed.
if self.match_current(&[TokenType::LParen]).is_some() {
// Parse the expression between brackets.
let p = self.expression()?;
// Enforce the presence of a
// closing round bracket `)`.
self.expect(&TokenType::RParen, "Missing closing round bracket ')'")?;
return Ok(Expression::Group(Box::new(p)));
}
// Get the current token.
let c: &Token = self.current_token();
// Check the type of the current token.
let result = match (&c.ttype, &c.literal) {
(TokenType::Number, TokenLiteral::Number(n)) => {
Expression::Literal(Literal::Number(*n))
}
(TokenType::Reserved(s), _) => match s {
Keyword::True => Expression::Literal(Literal::Boolean(true)),
Keyword::False => Expression::Literal(Literal::Boolean(false)),
Keyword::Nil => Expression::Literal(Literal::Nil),
_ => {
return Err(ParserError {
message: String::from("Invalid primary token"),
token: c.clone(),
})
}
},
(TokenType::String, TokenLiteral::String(s)) => {
Expression::Literal(Literal::String(s.to_string()))
}
(TokenType::LParen, _) => {
[...]
}
_ => {
return Err(ParserError {
message: String::from("Invalid primary token"),
token: c.clone(),
});
}
};
// If we reach this point, the token we
// found was valid. Advance the index
// consuming the current token.
self.consume();
// Return the token as a Result.
Ok(result)
}
}The last part of this stage is the addition of the new CLI command run. Thanks to the refactoring we went through in the previous chapter this is a simple change:
src/main.rs use crate::evaluate::evaluate;
use crate::parser::Parser;
use crate::statement::Statement;
use crate::token::{Token, TokenError, TokenResult};
use crate::tokenizer::tokenize;
#[derive(CSubcommand)]
enum Commands {
Evaluate { filename: PathBuf },
Parse { filename: PathBuf },
Run { filename: PathBuf },
Tokenize { filename: PathBuf },
}
fn main() {
match args.command {
Commands::Run { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
let statement = p.statement().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
match statement {
Statement::Print(e) => {
let result = evaluate(&e);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
}
}The command is very similar to Commands::Evaluate but focuses on statements instead of expressions.
Stage 1 - Print: Generate output
This version of the code passes Stage 1 of the extension "Statements & state".
Supporting multiple statements requires the introduction of an explicit loop that parses a list of statements until the end of the file is reached. To check this condition we can first of all create a helper:
src/parser.rs impl Parser {
// Check if the current token is
// the End of File.
fn check_eof(&self) -> bool {
self.current_token().ttype == TokenType::Eof
}The logic requires new tests:
src/parser/tests/utils.rs #[test]
// Test that the method `check_eof`
// returns false when the current
// token is not `Eof`.
fn test_check_eof_false() {
let tokens = vec![dummy_token(TokenType::Plus), Token::eof()];
let parser = Parser::new(tokens);
assert!(!parser.check_eof());
assert_eq!(parser.index, 0);
}
#[test]
// Test that the method `check_eof`
// returns true when the current
// token is `Eof`.
fn test_check_eof_true() {
let tokens = vec![Token::eof()];
let parser = Parser::new(tokens);
assert!(parser.check_eof());
assert_eq!(parser.index, 0);
}
#[test]
// Test that the method `check_eof`
// returns true when the list of
// tokens is empty, as `new` adds
// an `Eof` token.
fn test_check_eof_empty() {
let parser = Parser::new(Vec::new());
assert!(parser.check_eof());
assert_eq!(parser.index, 0);
}To keep the logic of parsing multiple statements neatly separated, we can implement it in a new method called Parser::parse:
src/parser.rs impl Parser {
// The main entry point of the parser.
pub fn parse(&mut self) -> ParserResult<Vec<Statement>> {
// Create a vector to host
// the parsed statements.
let mut statements = Vec::<Statement>::new();
// Parse and collect statements
// until we reach the end of the file.
while !self.check_eof() {
statements.push(self.statement()?);
}
Ok(statements)
}
// The main entry point of the parser.
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
...We can add tests for this new method:
src/parser/tests/parser.rs use crate::expression::{Expression, Literal};
#[test]
// Test that the method `parse` works
// correctly on a list of valid statements.
fn test_parse_valid_statements() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::False)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.parse();
assert_eq!(
result,
Ok(vec![
Statement::Print(Expression::Literal(Literal::Boolean(true))),
Statement::Print(Expression::Literal(Literal::Boolean(false))),
])
);
assert_eq!(parser.index, 6);
}
#[test]
// Test that the method `parse` works
// correctly on a list that contains
// an invalid statement.
fn test_parse_invalid_statement() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Reserved(Keyword::Print)),
dummy_token(TokenType::Reserved(Keyword::False)),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.parse();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::Eof)
})
);
assert_eq!(parser.index, 5);
}Finally, the method Parser::parse must replace the method Parser::statement in the main function:
src/main.rs fn main() {
match args.command {
Commands::Run { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
let statement = p.statement().unwrap_or_else(|e| {
let statements = p.parse().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
for statement in statements {
match statement {
Statement::Print(e) => {
let result = evaluate(&e);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
}
}
}Stage 2 - Print: Multiple statements
This version of the code passes Stage 2 of the extension "Statements & state".
The next stage of the challenge introduces expression statements in the form EXPRESSION;. The first change is the addition of a new variant of the enum Statement:
src/statement.rs #[derive(Debug, PartialEq)]
pub enum Statement {
Expression(Expression),
Print(Expression),
}The logic to detect expression statements can be added to the method Parser::statement:
src/parser.rs impl Parser {
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Enforce the presence of the keyword
// `print`. If not present, the statement
// is invalid.
self.expect(&TokenType::Reserved(Keyword::Print), "Invalid statement")?;
// Check the presence of the keyword `print`.
let print = self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_some();
// Parse the expression contained
// in the print statement.
// in the statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Semicolon, "Invalid statement")?;
Ok(Statement::Print(expression))
// If this is a print statement
// output the correct variant.
if print {
return Ok(Statement::Print(expression));
}
Ok(Statement::Expression(expression))
}The complexity of the function increases but is still manageable. We first detect if the keyword print is present or not, and we use the result at the end to decide which variant of Statement to use.
The tests must be changed more radically, however: the test test_print_missing_keyword is not useful any more, as the missing keyword introduces a different type of statement.
src/parser/tests/statement.rs #[test]
// Test that the method `statement` returns
// the correct error when it parses a
// statement that does not start with `print`.
fn test_print_missing_keyword() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::True)),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::Reserved(Keyword::True)),
})
);
assert_eq!(parser.index, 0);
}
#[test]
// Test that the method `statement` works
// correctly on a statement in the form
// `EXPRESSION;`.
fn test_expression_valid_statement() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Ok(Statement::Expression(Expression::Literal(
Literal::Boolean(true)
)))
);
assert_eq!(parser.index, 2);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses an expression
// statement that contains an invalid expression.
fn test_expression_invalid_expression() {
let tokens = vec![
dummy_token(TokenType::Equal),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid primary token"),
token: dummy_token(TokenType::Equal)
})
);
}Last, we need to cover the new variant of Statement in the main function:
src/main.rs fn main() {
match args.command {
Commands::Run { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
let statements = p.parse().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
for statement in statements {
match statement {
Statement::Expression(e) => {
let result = evaluate(&e);
match result {
Ok(_) => {}
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
Statement::Print(e) => {
let result = evaluate(&e);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
}
}
}The difference between the code for Statement::Expression and the code for Statement::Print is only that the second prints the output on the terminal, and there is clearly a high degree of duplication. We are going to simplify it later in the chapter.
Stage 3 - Expression statements
This version of the code passes Stage 3 of the extension "Statements & state".
The CodeCrafters challenge, at this point, asks us to implement variable declarations in the form var IDENTIFIER = EXPRESSION;. Unfortunately, the provided tests don't check that syntax only, but also check that print statements support variables in the form print IDENTIFIER;. This means that it will take us several steps to complete this single stage. The plan is to go through the following steps:
Before these steps, we will briefly change the structure of the method Parser::statement, as the addition of the new logic might make it too complicated to read.
The current structure of the method Parser::statement makes it difficult to maintain several types of statements, as the logic to process each one of them would be in the same function. The method performs something that is pretty typical in an interpreter, which is to try different parsing options until one succeeds, which is represented by the following metacode:
src/parser.rs impl Parser {
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
TRY self.statement_print() OR CONTINUE;
TRY self.statement_var() OR CONTINUE;
...
return self.statement_expression();
}The problem is that self.statement_print(), self.statement_var(), and any other statement method can fail in two ways. If they fail because the statement is not the type of statement they parse the function must continue, but if they fail because of a syntax error the function must stop.
A good way to implement this logic in Rust is to use Result<Option<T>>. Basically, if the functions self.statement_TYPE() return Err() we know that something wrong happened and we will stop the execution, but if they succeed with Ok() we still need to check if they returned Some(), in which case they actually parsed the code, or None, when they could not parse it.
Let's see an implementation of this idea:
src/parser.rs impl Parser {
// Parse the next tokens as a print statement.
pub fn statement_print(&mut self) -> ParserResult<Option<Statement>> {
// Check the presence of the keyword `print`.
if self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_none()
{
return Ok(None);
}
// Parse the expression contained
// in the statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Semicolon, "Invalid statement")?;
Ok(Some(Statement::Print(expression)))
}Here, the function checks for print and if that is not present returns Ok(None). According to the explanation given above, this means that the function cannot parse the code, but that it cannot call the syntax incorrect either. It simply doesn't have anything to say about it. If the parsing is successful, instead, it returns Ok(Some()). The unsuccessful return is represented by the question mark operator in self.expression()?: if the parsing of the expression fails the whole function must return the Err().
The implementation of statement_expression is simpler as it is the terminal operation. Please note the different return type.
src/parser.rs impl Parser {
// Parse the next tokens as an expression statement.
pub fn statement_expression(&mut self) -> ParserResult<Statement> {
// Parse the expression contained
// in the statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Semicolon, "Invalid statement")?;
Ok(Statement::Expression(expression))
}The code of Parser::statement changes drastically:
src/parser.rs impl Parser {
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Check the presence of the keyword `print`.
let print = self
.match_current(&[TokenType::Reserved(Keyword::Print)])
.is_some();
// Try to parse the next tokens
// as a print statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_print()? {
return Ok(s);
}
// Parse the expression contained
// in the statement.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Semicolon, "Invalid statement")?;
// If this is a print statement
// output the correct variant.
if print {
return Ok(Statement::Print(expression));
}
Ok(Statement::Expression(expression))
// If we are here, the statement
// contains only an expression.
self.statement_expression()
}It is clear that the bottom part of the function will remain the same in the future, and that we only need to add checks for other statement types in the top part, making the code much easier to read.
We might decide to rewrite tests to target the single statement_TYPE functions, but it doesn't seem incorrect to keep them targeting Parser::statement. This way we check both the parsing and the delegation logic at the same time at the cost of a small deviation from the concept of unit testing.
The next type of statement that we want to parse is a variable declaration in the form var IDENTIFIER = EXPRESSION;. First of all, we need a new variant of Statement to represent it:
src/statement.rs #[derive(Debug, PartialEq)]
pub enum Statement {
Expression(Expression),
Print(Expression),
Var(String, Expression),
}Then we can isolate the logic in the method Parser::statement_var thanks to the refactoring done in the previous section:
src/parser.rs impl Parser {
// Parse the next tokens as a var statement.
pub fn statement_var(&mut self) -> ParserResult<Option<Statement>> {
// Check the presence of the keyword `var`.
if self
.match_current(&[TokenType::Reserved(Keyword::Var)])
.is_none()
{
return Ok(None);
}
// Enforce the presence of an identifier.
// If not present, the statement
// is invalid.
let identifier = self
.expect(&TokenType::Identifier, "Invalid statement")?
.clone();
// Enforce the presence of an equal `=`.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Equal, "Invalid statement")?;
// Parse the remaining tokens
// as an expression.
let expression = self.expression()?;
// Enforce the presence of a semicolon.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Semicolon, "Invalid statement")?;
Ok(Some(Statement::Var(identifier.lexeme, expression)))
}And we can add the new statement type to Parser::statement:
src/parser.rs impl Parser {
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Try to parse the next tokens
// as a print statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_print()? {
return Ok(s);
}
// Try to parse the next tokens
// as a var statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_var()? {
return Ok(s);
}
// If we are here, the statement
// contains only an expression.
self.statement_expression()
}As we see, the refactoring gave us a much more readable structure. Thanks to this, the new function can be tested through Parser::statement as we are doing for the others:
src/parser/tests/statement.rs #[test]
// Test that the method `statement` works
// correctly on a statement in the form
// `var IDENTIFIER = EXPRESSION;`.
fn test_var_valid_statement_with_expression() {
let varname = String::from("somevar");
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Var)),
Token {
ttype: TokenType::Identifier,
lexeme: varname.clone(),
literal: TokenLiteral::Null,
line: 1,
},
dummy_token(TokenType::Equal),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Ok(Statement::Var(
varname,
Expression::Literal(Literal::Boolean(true))
))
);
assert_eq!(parser.index, 5);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a var
// statement without an identifier.
fn test_var_missing_identifier() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Var)),
dummy_token(TokenType::Equal),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::Equal)
})
);
assert_eq!(parser.index, 1);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a var
// statement with an invalid expression.
fn test_var_invalid_expression() {
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Var)),
Token {
ttype: TokenType::Identifier,
lexeme: String::from("varname"),
literal: TokenLiteral::Null,
line: 1,
},
dummy_token(TokenType::Equal),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid primary token"),
token: dummy_token(TokenType::Semicolon)
})
);
assert_eq!(parser.index, 3);
}The last change is to add the variant to the main function:
src/main.rs fn main() {
match args.command {
Commands::Run { filename } => {
Statement::Print(e) => {
let result = evaluate(&e);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
Statement::Var(_s, e) => {
let _value = evaluate(&e);
}Clearly, we still don't have any way to actually assign the evaluated value to the given identifier, so this piece of code is incomplete.
At this point, we need to be able to parse identifier names. Since there is still no place to keep the values of each variable, for the time being we just need to be able to parse the syntax, and in the next step we will close the gap between identifiers and values through environments.
The first thing we need to implement is a variant of Expression that contains a variable:
src/expression.rs #[derive(Debug, PartialEq)]
pub enum Expression {
Binary {
left: Box<Self>,
operator: Token,
right: Box<Self>,
},
Group(Box<Self>),
Literal(Literal),
Unary {
operator: Token,
right: Box<Self>,
},
Variable(Token),
}
impl fmt::Display for Expression {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let name: String = match self {
Self::Binary {
left,
operator,
right,
} => format!("({} {} {})", operator.lexeme, left, right),
Self::Group(p) => format!("(group {})", p),
Self::Literal(s) => s.to_string(),
Self::Unary { operator, right } => format!("({} {})", operator.lexeme, right),
Self::Variable(t) => t.lexeme.clone(),
};
write!(f, "{}", name)
}
}The choice of storing a token inside the variant instead of a pure value is the same behind the nature of operators in Expression::Binary and Expression::Unary: error messages. When the parser finds a variable, the identifier might not correspond to an actual value (undefined variable), so we might need to give a proper error report, and since tokens contains the line number they should be preferred over plain values.
An identifier is a primary element of the grammar, so we add the TokenType variant to Parser::primary:
src/parser.rs impl Parser {
// Parse the current token and try to
// identify a primary element.
//
// primary -> NUMBER | STRING | "true" | "false" | "nil" | "(" expression ")"
// primary -> NUMBER | STRING | "true" | "false" | "nil" | "(" expression ")" | IDENTIFIER
fn primary(&mut self) -> ParserResult<Expression> {
let result = match (&c.ttype, &c.literal) {
(TokenType::String, TokenLiteral::String(s)) => {
Expression::Literal(Literal::String(s.to_string()))
}
(TokenType::Identifier, _) => Expression::Variable(c.clone()),
_ => {
return Err(ParserError {
message: String::from("Invalid primary token"),
token: c.clone(),
});
}And this new piece of logic can be tested:
src/parser/tests/primary.rs #[test]
// Test that the method `primary` works
// correctly on a list of tokens that
// contains an identifier.
fn test_identifier() {
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: String::from("somevar"),
literal: TokenLiteral::Null,
line: 1,
};
let tokens = vec![identifier_token.clone(), Token::eof()];
let mut parser = Parser::new(tokens);
let result = parser.primary();
assert_eq!(result, Ok(Expression::Variable(identifier_token)));
assert_eq!(parser.index, 1);
}Finally, we must add the variant to evaluate, even though we need to skip the implementation as there is at the moment no way to store and retrieve the value of a variable.
src/evaluate.rs pub fn evaluate(expression: &Expression) -> EvaluationResult<Value> {
match expression {
Expression::Binary {
left,
operator,
right,
} => evaluate_binary(operator, left, right),
Expression::Group(g) => evaluate(g),
Expression::Literal(Literal::Boolean(b)) => Ok(Value::Boolean(*b)),
Expression::Literal(Literal::Nil) => Ok(Value::Nil),
Expression::Literal(Literal::Number(n)) => Ok(Value::Number(*n)),
Expression::Literal(Literal::String(s)) => Ok(Value::String(s.clone())),
Expression::Unary { operator, right } => evaluate_unary(operator, right),
Expression::Variable(_t) => todo!(),
}
}It's time to connect all the components we added to the system providing a way to store variable values. Following what is done in "Crafting Interpreters", we will call the entity that maps names to values an environment. Let's start by creating the type with what seems to be the best data structure for this job, a HashMap [docs]:
src/environment.rs use std::collections::HashMap;
use std::fmt;
use crate::evaluate::Value;
#[derive(Debug)]
pub struct Environment {
values: HashMap<String, Value>,
}Storing and retrieving values associated with variables will have a huge amount of potential sources of errors, so we also need a type to host results related to environments:
src/environment.rs #[derive(Debug, PartialEq, Eq)]
pub struct EnvironmentError {
pub message: String,
}
impl fmt::Display for EnvironmentError {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
write!(f, "{}", self.message)
}
}
pub type EnvironmentResult<T> = Result<T, EnvironmentError>;Finally, we want to provide convenience methods to simplify the usage of the structure:
src/environment.rs impl Environment {
pub fn new() -> Self {
Self {
values: HashMap::new(),
}
}
// Retrieve the value of a variable
// from the environment. Return an
// error if the variable is not
// available (undefined).
pub fn get(&self, name: &str) -> EnvironmentResult<Value> {
self.values
.get(name)
.cloned()
.ok_or_else(|| EnvironmentError {
message: format!("Undefined variable {}.", name),
})
}
// Create a variable with the given
// value in the environment.
pub fn define(&mut self, name: String, value: Value) {
self.values.insert(name, value);
}
}The implementation of Environment::get deserves an explanation. The output of the method is an owned value, but HashMap::get [docs] returns in this case an Option<&Value> and it is not possible to move a value out of a reference. The method Option::cloned [docs] transforms Option<&T> into Option<T> simply calling T::clone. It is important to highlight that returning an owned value EnvironmentResult<Value> instead of a borrow EnvironmentResult<&Value> is a design choice, and nothing prevents us from going down the borrow route, adapting the caller (in this case the function evaluate).
Please also note that the error message has been created following what Robert Nystrom does in his book.
The current implementation of Value doesn't allow cloning, so we need to add it to the type:
src/evaluate.rs #[derive(Debug, PartialEq)]
#[derive(Clone, Debug, PartialEq)]
pub enum Value {
Boolean(bool),
Nil,
Number(f64),
String(String),
}Of course, the logic of the new methods can be tested:
src/environment.rs #[cfg(test)]
mod tests {
use super::*;
#[test]
// Test that a newly created environment
// is configured correctly.
fn test_new() {
let environment = Environment::new();
assert_eq!(environment.values.len(), 0)
}
#[test]
// Test that the method `get` returns
// the correct error when the variable
// doesn't exist.
fn test_get_error() {
let environment = Environment::new();
assert_eq!(
environment.get("somevar"),
Err(EnvironmentError {
message: String::from("Undefined variable somevar."),
})
);
}
#[test]
// Test that the method `get` works
// correctly when used to retrieve the
// value of an existing variable.
fn test_get_variable() {
let mut environment = Environment::new();
// Define the variable without using a
// method of Environment to avoid cyclic
// test definitions.
environment
.values
.insert(String::from("somevar"), Value::Boolean(true));
assert_eq!(environment.get("somevar"), Ok(Value::Boolean(true)));
}
#[test]
// Test that the method `define` works
// correctly when used to create a
// variable that doesn't exist yet.
fn test_define_non_existing_variable() {
let mut environment = Environment::new();
environment.define(String::from("somevar"), Value::Boolean(true));
// Check that the variable is present
// without using a method of Environment
// to avoid cyclic test definitions.
assert_eq!(
environment.values.get("somevar"),
Some(&Value::Boolean(true))
);
}
#[test]
// Test that the method `define` works
// correctly when used to create a
// variable that already exists.
fn test_define_existing_variable() {
let mut environment = Environment::new();
// Define the variable without using a
// method of Environment to avoid cyclic
// test definitions.
environment
.values
.insert(String::from("somevar"), Value::Boolean(false));
environment.define(String::from("somevar"), Value::Boolean(true));
// Check that the variable is present
// without using a method of Environment
// to avoid cyclic test definitions.
assert_eq!(
environment.values.get("somevar"),
Some(&Value::Boolean(true))
);
}
}With this code in place, we can introduce environments in the evaluation mechanism:
src/evaluate.rs use crate::environment::Environment;
use crate::expression::{Expression, Literal};
use crate::token::{Token, TokenType};
pub fn evaluate(expression: &Expression) -> EvaluationResult<Value> {
pub fn evaluate(expression: &Expression, env: &mut Environment) -> EvaluationResult<Value> {
match expression {
Expression::Binary {
left,
operator,
right,
} => evaluate_binary(operator, left, right),
Expression::Group(g) => evaluate(g),
} => evaluate_binary(operator, left, right, env),
Expression::Group(g) => evaluate(g, env),
Expression::Literal(Literal::Boolean(b)) => Ok(Value::Boolean(*b)),
Expression::Literal(Literal::Nil) => Ok(Value::Nil),
Expression::Literal(Literal::Number(n)) => Ok(Value::Number(*n)),
Expression::Literal(Literal::String(s)) => Ok(Value::String(s.clone())),
Expression::Unary { operator, right } => evaluate_unary(operator, right),
Expression::Variable(_t) => todo!(),
Expression::Unary { operator, right } => evaluate_unary(operator, right, env),
Expression::Variable(_t) => todo!(),
}
}
fn evaluate_unary(operator: &Token, right: &Expression) -> EvaluationResult<Value> {
let value = evaluate(right)?;
fn evaluate_unary(
operator: &Token,
right: &Expression,
env: &mut Environment,
) -> EvaluationResult<Value> {
let value = evaluate(right, env)?;
...
}
fn evaluate_binary(
operator: &Token,
left: &Expression,
right: &Expression,
env: &mut Environment,
) -> EvaluationResult<Value> {
let lvalue = evaluate(left)?;
let rvalue = evaluate(right)?;
let lvalue = evaluate(left, env)?;
let rvalue = evaluate(right, env)?;
...
}Unsurprisingly, a change in the signature of a function affects several parts of the code, in particular tests. In all tests that call evaluate or one of its sibling functions we need to create an environment and pass it in the call.
src/evaluate.rs mod tests {
#[test]
// Test that `Star` works correctly
// with numbers that have no decimals.
// A float with no decimal part is
// the way we represent an integer.
fn test_star_integer() {
let mut env = Environment::new();
let expression = dummy_binary_expression(
Expression::Literal(Literal::Number(6.0)),
TokenType::Star,
Expression::Literal(Literal::Number(7.0)),
);
assert_eq!(evaluate(&expression), Ok(Value::Number(42.0)));
assert_eq!(evaluate(&expression, &mut env), Ok(Value::Number(42.0)));
}
// FIX ALL REMAINING TESTSNow that the environment is available, the evaluation of a variable inside an expression can finally take place:
src/evaluate.rs pub fn evaluate(expression: &Expression, env: &mut Environment) -> EvaluationResult<Value> {
match expression {
Expression::Binary {
left,
operator,
right,
} => evaluate_binary(operator, left, right, env),
Expression::Group(g) => evaluate(g, env),
Expression::Literal(Literal::Boolean(b)) => Ok(Value::Boolean(*b)),
Expression::Literal(Literal::Nil) => Ok(Value::Nil),
Expression::Literal(Literal::Number(n)) => Ok(Value::Number(*n)),
Expression::Literal(Literal::String(s)) => Ok(Value::String(s.clone())),
Expression::Unary { operator, right } => evaluate_unary(operator, right, env),
Expression::Variable(_t) => todo!(),
Expression::Variable(t) => env
.get(&t.lexeme)
.map_err(|e| EvaluationError::new(t.line, e.message)),
}
}To test the logic we can add the tests in a new module:
src/evaluate/tests/variable.rs use super::*;
use crate::environment::Environment;
#[test]
// Test that a defined variable
// returns its value.
fn test_variable_defined() {
let mut env = Environment::new();
env.define(String::from("somevar"), Value::Number(42.0));
let expression = dummy_variable_expression("somevar", 1);
assert_eq!(evaluate(&expression, &mut env), Ok(Value::Number(42.0)));
}
#[test]
// Test that an undefined variable
// returns an error with the message
// of the environment and the line
// of the variable token.
fn test_variable_undefined() {
let mut env = Environment::new();
let expression = dummy_variable_expression("somevar", 42);
assert_eq!(
evaluate(&expression, &mut env),
Err(EvaluationError {
message: String::from("Undefined variable somevar."),
line: 42,
})
);
}These rely on a helper method to create a variable expression:
src/evaluate/tests.rs mod binary_arithmetic;
mod binary_comparisons;
mod binary_equality;
mod binary_strings;
mod literal_group;
mod unary;
mod variable;
// Build a variable expression with
// the given name on the given line.
fn dummy_variable_expression(name: &str, line: u16) -> Expression {
Expression::Variable(Token {
lexeme: String::from(name),
line,
..dummy_token(TokenType::Identifier)
})
}The double dot .. is called the Struct Update Syntax [docs] and is very convenient when we want to change only some fields of an already existing structure.
The last required change is the creation and use of an environment in the interpreter commands that evaluate expressions and run code:
src/main.rs use crate::environment::Environment;
use crate::evaluate::evaluate;
use crate::parser::Parser;
use crate::statement::Statement;
use crate::token::{Token, TokenError, TokenResult};
use crate::tokenizer::tokenize;
mod environment;
mod evaluate;
mod expression;
mod parser;
mod statement;
mod token;
mod tokenizer;
fn main() {
match args.command {
Commands::Evaluate { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
// Create an environment.
let mut env = Environment::new();
let parsed = p.expression().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
let result = evaluate(&parsed);
let result = evaluate(&parsed, &mut env);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
...
Commands::Run { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
// Create an environment.
let mut env = Environment::new();
let statements = p.parse().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
for statement in statements {
match statement {
Statement::Expression(e) => {
let result = evaluate(&e);
let result = evaluate(&e, &mut env);
match result {
Ok(_) => {}
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
Statement::Print(e) => {
let result = evaluate(&e);
let result = evaluate(&e, &mut env);
match result {
Ok(v) => println!("{}", v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
Statement::Var(_s, e) => {
let _value = evaluate(&e);
}
Statement::Var(s, e) => {
let value = evaluate(&e, &mut env);
match value {
Ok(v) => env.define(s.clone(), v),
Err(e) => {
eprintln!("{}", e);
process::exit(70);
}
}
}
}
}
}The three arms of match statement in Commands::Run are remarkably similar, so in a future step, when the code is more stable, we will reduce the duplication with a small refactoring.
Stages 4-5 - Variables: Declare variables / Variables: Runtime Errors
This version of the code passes Stage 4 and Stage 5 of the extension "Statements & state".
In Lox, variables can be initialised with a default value of nil with the syntax var NAME;. The change is rather simple: when we parse a variable statement we need to take into account the case of missing equal sign and return a hardcoded Literal::Nil:
src/parser.rs impl Parser {
pub fn statement_var(&mut self) -> ParserResult<Option<Statement>> {
// Enforce the presence of an identifier.
// If not present, the statement
// is invalid.
let identifier = self
.expect(&TokenType::Identifier, "Invalid statement")?
.clone();
// Enforce the presence of an equal `=`.
// If not present, the statement
// is invalid.
self.expect(&TokenType::Equal, "Invalid statement")?;
// Parse the remaining tokens
// as an expression.
let expression = self.expression()?;
// If the next token is an equal `=`,
// skip it and parse the remaining tokens
// as an expression. Otherwise
// just return `nil`.
let expression = match self.match_current(&[TokenType::Equal]) {
Some(_) => self.expression()?,
None => Expression::Literal(Literal::Nil),
};The new logic requires a test:
src/parser/tests/statement.rs #[test]
// Test that the method `statement` works
// correctly on a statement in the form
// `var IDENTIFIER;`.
fn test_var_valid_statement_without_expression() {
let varname = String::from("somevar");
let tokens = vec![
dummy_token(TokenType::Reserved(Keyword::Var)),
Token {
ttype: TokenType::Identifier,
lexeme: varname.clone(),
literal: TokenLiteral::Null,
line: 1,
},
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Ok(Statement::Var(varname, Expression::Literal(Literal::Nil)))
);
assert_eq!(parser.index, 3);
}Stages 6-7 - Variables: Initialize variables / Variables: Redeclare variables
This version of the code passes Stage 6 and Stage 7 of the extension "Statements & state".
While variables can be defined with the syntax var NAME = VALUE; they cannot be assigned a value with the syntax NAME = VALUE;. The reason is first of all that Environment supports only variable definitions and not assignments, but also that the parser is not able to process an expression in that form.
To implement this feature let's start by providing environment a way to assign values to existing variables:
src/environment.rs impl Environment {
// Assign a new value to an existing
// variable. Return an error if the
// variable is not defined.
pub fn assign(&mut self, name: &str, value: Value) -> EnvironmentResult<()> {
let current = self.values.get_mut(name).ok_or_else(|| EnvironmentError {
message: format!("Undefined variable {}.", name),
})?;
*current = value;
Ok(())
}The code is rather straightforward, even though we have one of the rare occurrences of the explicit dereference operator * in Rust. Here, HashMap::get_mut [docs] returns Option<&mut Value>, which is a reference to the value, so if we want to change it we need to use *current.
The logic can be easily tested:
src/environment.rs mod tests {
#[test]
// Test that the method `assign` works
// correctly when used to assign a
// value to a variable that already exists.
fn test_assign_variable() {
let mut environment = Environment::new();
// Define the variable without using a
// method of Environment to avoid cyclic
// test definitions.
environment
.values
.insert(String::from("somevar"), Value::Boolean(true));
assert_eq!(environment.assign("somevar", Value::Number(42.0)), Ok(()));
// Check the new value without using
// a method of Environment to avoid
// cyclic test definitions.
assert_eq!(
environment.values.get("somevar"),
Some(&Value::Number(42.0))
);
}
#[test]
// Test that the method `assign` returns
// the correct error when used to assign a
// value to a variable that doesn't exist.
fn test_assign_error() {
let mut environment = Environment::new();
assert_eq!(
environment.assign("somevar", Value::Boolean(true)),
Err(EnvironmentError {
message: String::from("Undefined variable somevar."),
})
);
// Check that the failed assignment
// didn't define the variable.
assert_eq!(environment.values.get("somevar"), None);
}Now that the environment is ready, we must change the code of the parser. First of all, we introduce a new variant of Expression:
src/expression.rs #[derive(Debug, PartialEq)]
pub enum Expression {
Assignment {
identifier: Token,
expression: Box<Self>,
},
Binary {
left: Box<Self>,
operator: Token,
right: Box<Self>,
},
Group(Box<Self>),
Literal(Literal),
Unary {
operator: Token,
right: Box<Self>,
},
Variable(Token),
}
impl fmt::Display for Expression {
fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
let name: String = match self {
Self::Assignment {
identifier,
expression,
} => {
format!("({} = {})", identifier.lexeme, expression)
}
Self::Binary {
left,
operator,
right,
} => format!("({} {} {})", operator.lexeme, left, right),
Self::Group(p) => format!("(group {})", p),
Self::Literal(s) => s.to_string(),
Self::Unary { operator, right } => format!("({} {})", operator.lexeme, right),
Self::Variable(t) => t.lexeme.clone(),
};
write!(f, "{}", name)
}
}Currently, the entry point for expressions is the method self.equality so we need to introduce self.assignment before it:
src/parser.rs impl Parser {
// Parse the next tokens as an expression.
pub fn expression(&mut self) -> ParserResult<Expression> {
self.equality()
self.assignment()
}
// Parse the next three tokens and try to
// identify an assignment.
//
// assignment -> IDENTIFIER "=" assignment | equality
fn assignment(&mut self) -> ParserResult<Expression> {
// Parse the left side of the assignment.
let left = self.equality()?;
// Match an equal `=`.
// If not present the expression
// is simply the left side that
// we already parsed.
let Some(t) = self.match_current(&[TokenType::Equal]) else {
return Ok(left);
};
// If we reach this point we found
// an equal `=` in the expression.
// The token is cloned for later use.
// We cannot use the token itself
// later as it comes from a mutable
// borrow of self (self.match_current)
// and we need another mutable borrow
// later (self.assignment).
let equal = t.clone();
// Parse the right side of the assignment.
// This can be another assignment.
let right = self.assignment()?;
// Check the left side of the assignment.
// If it's not a variable return an error.
match left {
Expression::Variable(identifier) => Ok(Expression::Assignment {
identifier,
expression: Box::new(right),
}),
_ => Err(ParserError {
message: String::from("Invalid assignment target"),
token: equal,
}),
}
}As stated in "Crafting Interpreters" the idea is that we won't check the nature of the left side of an assignment until we actually process it. Therefore, the left side can in theory be an equality in the form VALUE1 == VALUE2 but this would fail in the final match in the code above. Please note that we might run into a small issue with the Rust borrow checker: we need to call self.assignment which borrows self mutably, but we cannot do it if we still hold a borrow with the token t and we are therefore forced to clone the latter.
The new logic requires several tests:
src/parser/tests/assignment.rs use super::*;
#[test]
// Test that the method `assignment` works
// correctly with an expression in the form
// var = PRIMARY.
fn test_assignment_valid() {
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: String::from("varname"),
literal: TokenLiteral::Null,
line: 1,
};
let tokens = vec![
identifier_token.clone(),
dummy_token(TokenType::Equal),
Token {
ttype: TokenType::Number,
lexeme: String::new(),
literal: TokenLiteral::Number(42.0),
line: 1,
},
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.assignment();
assert_eq!(
result,
Ok(Expression::Assignment {
identifier: identifier_token,
expression: Box::new(Expression::Literal(Literal::Number(42.0)))
})
);
assert_eq!(parser.index, 3);
}
#[test]
// Test that the method `assignment` works
// correctly with an expression that is
// missing the equal sign.
fn test_assignment_missing_equal() {
let tokens = vec![
Token {
ttype: TokenType::Number,
lexeme: String::new(),
literal: TokenLiteral::Number(42.0),
line: 1,
},
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.assignment();
assert_eq!(result, Ok(Expression::Literal(Literal::Number(42.0))));
assert_eq!(parser.index, 1);
}
#[test]
// Test that the method `assignment` returns
// the correct error when it parses an expression
// whose left side is an invalid assignment target.
fn test_assignment_invalid_target() {
let tokens = vec![
Token {
ttype: TokenType::Number,
lexeme: String::new(),
literal: TokenLiteral::Number(0.0),
line: 1,
},
dummy_token(TokenType::Equal),
Token {
ttype: TokenType::Number,
lexeme: String::new(),
literal: TokenLiteral::Number(42.0),
line: 1,
},
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.assignment();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid assignment target"),
token: dummy_token(TokenType::Equal),
})
);
assert_eq!(parser.index, 3);
}
#[test]
// Test that the method `assignment` works
// correctly with multiple assignments
// chained by equality tokens in the form
// var_a = var_b = PRIMARY.
fn test_assignment_chain() {
let identifier_token_a = Token {
ttype: TokenType::Identifier,
lexeme: String::from("var_a"),
literal: TokenLiteral::Null,
line: 1,
};
let identifier_token_b = Token {
ttype: TokenType::Identifier,
lexeme: String::from("var_b"),
literal: TokenLiteral::Null,
line: 1,
};
let tokens = vec![
identifier_token_a.clone(),
dummy_token(TokenType::Equal),
identifier_token_b.clone(),
dummy_token(TokenType::Equal),
Token {
ttype: TokenType::Number,
lexeme: String::new(),
literal: TokenLiteral::Number(42.0),
line: 1,
},
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.assignment();
assert_eq!(
result,
Ok(Expression::Assignment {
identifier: identifier_token_a,
expression: Box::new(Expression::Assignment {
identifier: identifier_token_b,
expression: Box::new(Expression::Literal(Literal::Number(42.0)))
})
})
);
assert_eq!(parser.index, 5);
}And the new test module must be declared:
src/parser/tests.rs mod assignment;
mod comparison;
mod equality;
mod factor;
mod parser;
mod primary;
mod statement;
mod term;
mod unary;
mod utils;Finally, the new variant of Expression must be processed in evaluate:
src/evaluate.rs pub fn evaluate(expression: &Expression, env: &mut Environment) -> EvaluationResult<Value> {
match expression {
Expression::Assignment {
identifier,
expression,
} => {
let value = evaluate(expression, env)?;
env.assign(&identifier.lexeme, value.clone())
.map_err(|e| EvaluationError::new(identifier.line, e.message))?;
Ok(value)
}
Expression::Binary {
left,
operator,
right,
} => evaluate_binary(operator, left, right, env),Stage 8 - Assignment operation
This version of the code passes Stage 8 of the extension "Statements & state".
As mentioned previously, the code for Commands::Run in src/main.rs contains a relevant amount of duplication. Since we are going to add at least one more variant to Statement when we implement blocks in the next stage, it is worth going through a small refactoring to make sure the code is in a more manageable state.
We can isolate the statement execution logic in a function:
src/statement.rs use crate::environment::Environment;
use crate::evaluate::{evaluate, EvaluationResult, Value};
use crate::expression::Expression;
pub fn execute(
statement: &Statement,
env: &mut Environment,
output: &mut impl FnMut(Value),
) -> EvaluationResult<()> {
match statement {
Statement::Expression(expression) => {
evaluate(expression, env)?;
}
Statement::Print(expression) => output(evaluate(expression, env)?),
Statement::Var(name, expression) => {
let value = evaluate(expression, env)?;
env.define(name.clone(), value);
}
}
Ok(())
}The logic implemented in each arm is the same that was implemented in main, only rearranged to match the new structure. In particular, the function accepts the parameter output that implements FnMut(Value) and this will be used to provide a closure that prints on the standard output. We might run evaluate only once before the match operation, but we are not yet sure that all the cases will need it, so we can delay that optimisation.
This code can therefore replace most of what we have in main under Commands:Run:
src/main.rs use crate::environment::Environment;
use crate::evaluate::evaluate;
use crate::evaluate::{evaluate, Value};
use crate::parser::Parser;
use crate::statement::Statement;
use crate::statement::execute;
use crate::token::{Token, TokenError, TokenResult};
use crate::tokenizer::tokenize;
fn main() {
match args.command {
Commands::Run { filename } => {
// Process the source and
// extract a parser.
let mut p = parse_source(&filename);
// Create an environment.
let mut env = Environment::new();
let statements = p.parse().unwrap_or_else(|e| {
eprintln!("{}", e);
process::exit(65);
});
for statement in statements {
...
}
for statement in &statements {
let mut output = |v: Value| println!("{}", v);
if let Err(error) = execute(statement, &mut env, &mut output) {
eprintln!("{}", error);
process::exit(70);
}
}
}The function execute has a nice feature: it accepts a function output: &mut impl FnMut(Value) that will receive the outcome of the execution. This allows us to write tests for it, for example:
src/statement.rs #[test]
// Test that the function `execute` works
// correctly when the statement contains
// a variable declaration.
fn test_variable_declaration() {
// A vector to host the output.
let mut output = Vec::new();
// The parent environment.
let mut env = Environment::new();
// The statement we are going to process.
let statement = Statement::Var(
String::from("somevar"),
Expression::Literal(Literal::Number(42.0)),
);
// Execute the statement.
let result = execute(&statement, &mut env, &mut |v| output.push(v));
assert_eq!(result, Ok(()));
assert!(output.is_empty());
assert_eq!(env.get("somevar").unwrap(), Value::Number(42.0));
}It's pretty clear that basically all tests of execute will follow the template above, so it's worth capturing part of it in a helper function:
src/statement.rs #[cfg(test)]
mod tests {
use super::*;
use crate::expression::Literal;
// A helper to simplify tests
// for the function `execute`.
fn run_execute(
statement: &Statement,
env: &mut Environment,
) -> (EvaluationResult<()>, Vec<Value>) {
// A vector to host the output.
let mut output = Vec::new();
// Execute the statement.
let result = execute(statement, env, &mut |v| output.push(v));
(result, output)
}
}With this in place, adding tests becomes simpler:
src/statement.rs mod tests {
use crate::environment::EnvironmentError;
use crate::evaluate::EvaluationError;
use crate::expression::Literal;
use crate::token::{Token, TokenLiteral, TokenType};
#[test]
// Test that the function `execute` works
// correctly when the statement is an
// expression statement with a correct
// expression: an assignment to an
// existing variable.
fn test_expression_correct() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value of the variable
// we are going to use.
let var_value = 42.0;
// Define a variable in the environment.
env.define(var_name.clone(), Value::Boolean(true));
// A token for the assigned variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name.clone(),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// contains an assignment expression.
let statement = Statement::Expression(Expression::Assignment {
identifier: identifier_token,
expression: Box::new(Expression::Literal(Literal::Number(var_value))),
});
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert!(output.is_empty());
assert_eq!(env.get(&var_name).unwrap(), Value::Number(var_value));
}
#[test]
// Test that the function `execute` works
// correctly when the statement is an
// expression statement with an incorrect
// expression: an assignment to an
// undefined variable.
fn test_expression_incorrect() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value of the variable
// we are going to use.
let var_value = 42.0;
// A token for the assigned variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name.clone(),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// contains an assignment expression.
let statement = Statement::Expression(Expression::Assignment {
identifier: identifier_token,
expression: Box::new(Expression::Literal(Literal::Number(var_value))),
});
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert!(matches!(result, Err(EvaluationError { .. })));
assert!(output.is_empty());
assert!(matches!(env.get(&var_name), Err(EnvironmentError { .. })));
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// print statement with a correct
// expression: the name of an existing
// variable.
fn test_print_correct() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value of the variable
// we are going to use.
let var_value = Value::Number(42.0);
// Define a variable in the environment.
env.define(var_name.clone(), var_value.clone());
// A token for the printed variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name.clone(),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// prints the variable.
let statement = Statement::Print(Expression::Variable(identifier_token));
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert_eq!(output, vec![var_value.clone()]);
assert_eq!(env.get(&var_name).unwrap(), var_value);
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// print statement with an incorrect
// expression: the name of an undefined
// variable.
fn test_print_incorrect() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// A token for the printed variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name.clone(),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// prints the variable.
let statement = Statement::Print(Expression::Variable(identifier_token));
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert!(matches!(result, Err(EvaluationError { .. })));
assert!(output.is_empty());
assert!(matches!(env.get(&var_name), Err(EnvironmentError { .. })));
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// variable definition statement with
// a correct expression: a number.
fn test_variable_definition_correct() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value we are going to use.
let value = 42.0;
// The statement we are going to process.
let statement = Statement::Var(
var_name.clone(),
Expression::Literal(Literal::Number(value)),
);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert!(output.is_empty());
assert_eq!(env.get(&var_name).unwrap(), Value::Number(value));
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// variable definition statement with
// an incorrect expression: the name of
// an undefined variable.
fn test_variable_definition_incorrect() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// A token for the undefined variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: String::from("undefined"),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process.
let statement = Statement::Var(var_name.clone(), Expression::Variable(identifier_token));
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert!(matches!(result, Err(EvaluationError { .. })));
assert!(output.is_empty());
assert!(matches!(env.get(&var_name), Err(EnvironmentError { .. })));
}The implementation of the block syntax is not complicated, but it opens the door to a relevant amount of issues related to scope. The management of nested scopes is a difficult topic in the realm of programming languages, and we are going to dedicate the whole next chapter to start scratching its surface.
For the time being, however, we are not required to manage any nesting. Each block has its own independent environment which disappears when the block ends.
Given that blocks define their own environment, and later will be able to access the parent environment, the amount of tests for the new logic will be huge. Therefore, we can first refactor tests as we did for parsing and evaluation, in order to keep them tidy.
As we did in previous chapters, the refactoring is just a matter of moving code and renaming functions. Let's create the directories src/statement and src/statement/tests to host the new module. The changes to src/statement.rs are:
src/statement.rs #[cfg(test)]
mod tests {
...
}
mod tests;The helper and the test imports are moved into src/statement/tests.rs:
src/statement/tests.rs use super::*;
use crate::environment::EnvironmentError;
use crate::evaluate::EvaluationError;
use crate::expression::Literal;
use crate::token::{Token, TokenLiteral, TokenType};
mod expression;
mod print;
mod var;
// A helper to simplify tests
// for the function `execute`.
fn run_execute(statement: &Statement, env: &mut Environment) -> (EvaluationResult<()>, Vec<Value>) {
// A vector to host the output.
let mut output = Vec::new();
// Execute the statement.
let result = execute(statement, env, &mut |v| output.push(v));
(result, output)
}And the single tests are moved into files that correspond to the variants of Statement. Since the code is copied verbatim, and only the function names are changed, I will not show the full code of the tests in the snippets. First let's tackle Statement::Expression:
src/statement/tests/expression.rs use super::*;
#[test]
// Test that the function `execute` works
// correctly when the statement is an
// expression statement with a correct
// expression: an assignment to an
// existing variable.
fn test_correct() {
...
}
#[test]
// Test that the function `execute` works
// correctly when the statement is an
// expression statement with an incorrect
// expression: an assignment to an
// undefined variable.
fn test_incorrect() {
...
}Then Statement::Print:
src/statement/tests/print.rs use super::*;
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// print statement with a correct
// expression: the name of an existing
// variable.
fn test_correct() {
...
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// print statement with an incorrect
// expression: the name of an undefined
// variable.
fn test_incorrect() {
...
}and last Statement::Var:
src/statement/tests/var.rs use super::*;
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// variable definition statement with
// a correct expression: a number.
fn test_correct() {
...
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// variable definition statement with
// an incorrect expression: the name of
// an undefined variable.
fn test_incorrect() {
...
}It's time to implement the actual code for blocks. Let's start by creating a new variant of Statement:
src/statement.rs #[derive(Debug, PartialEq)]
pub enum Statement {
Block(Vec<Self>),
Expression(Expression),
Print(Expression),
Var(String, Expression),
}We can also directly implement the execution logic for it. The idea is simple: when we evaluate a block we first create a new environment that is given to all statements in the block.
src/statement.rs pub fn execute(
statement: &Statement,
env: &mut Environment,
output: &mut impl FnMut(Value),
) -> EvaluationResult<()> {
match statement {
Statement::Block(v) => {
let mut local_env = Environment::new();
for statement in v.iter() {
execute(statement, &mut local_env, output)?;
}
}
Statement::Expression(expression) => {
evaluate(expression, env)?;
}
Statement::Print(expression) => output(evaluate(expression, env)?),
Statement::Var(name, expression) => {
let value = evaluate(expression, env)?;
env.define(name.clone(), value);
}
}
Ok(())
}While the idea is simple, there are several tests that we can write for the new logic:
src/statement/tests/block.rs use super::*;
#[test]
// Test that the function `execute` works
// correctly when the statement is an
// empty block statement.
fn test_empty() {
// The parent environment.
let mut env = Environment::new();
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert!(output.is_empty());
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains correct
// statements.
fn test_correct() {
// The parent environment.
let mut env = Environment::new();
// The value we are going to use.
let value = 42.0;
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![Statement::Print(Expression::Literal(
Literal::Number(value),
))]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert_eq!(output, vec![Value::Number(value)]);
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains incorrect
// statements.
fn test_incorrect() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// A token for the printed variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name,
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![Statement::Print(Expression::Variable(
identifier_token,
))]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert!(matches!(result, Err(EvaluationError { .. })));
assert!(output.is_empty());
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains correct
// statements, returning the output
// in the correct order.
fn test_output_order() {
// The parent environment.
let mut env = Environment::new();
// Three values we are going to use.
let value1 = 41.0;
let value2 = 42.0;
let value3 = 43.0;
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![
Statement::Print(Expression::Literal(Literal::Number(value1))),
Statement::Print(Expression::Literal(Literal::Number(value2))),
Statement::Print(Expression::Literal(Literal::Number(value3))),
]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert_eq!(
output,
vec![
Value::Number(value1),
Value::Number(value2),
Value::Number(value3),
]
);
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains correct
// and incorrect statements, stopping
// at the first incorrect one.
fn test_error_stops_execution() {
// The parent environment.
let mut env = Environment::new();
// A token for the undefined variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: String::from("undefined"),
literal: TokenLiteral::Null,
line: 1,
};
// Three values we are going to use.
let value1 = 41.0;
let value2 = 42.0;
let value3 = 43.0;
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![
Statement::Print(Expression::Literal(Literal::Number(value1))),
Statement::Print(Expression::Literal(Literal::Number(value2))),
Statement::Print(Expression::Variable(identifier_token)),
Statement::Print(Expression::Literal(Literal::Number(value3))),
]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert!(matches!(result, Err(EvaluationError { .. })));
assert_eq!(output, vec![Value::Number(value1), Value::Number(value2)]);
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains correct
// statements and that statements share
// the same local environment.
fn test_share_environment() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value we are going to use.
let var_value = 42.0;
// A token for the printed variable name.
let identifier_token = Token {
ttype: TokenType::Identifier,
lexeme: var_name.clone(),
literal: TokenLiteral::Null,
line: 1,
};
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![
Statement::Var(var_name, Expression::Literal(Literal::Number(var_value))),
Statement::Print(Expression::Variable(identifier_token)),
]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert_eq!(output, vec![Value::Number(var_value)]);
}
#[test]
// Test that the function `execute` works
// correctly when the statement is a
// block statement that contains correct
// statements and that defining a variable
// in the local environment does not affect the
// parent one.
fn test_define_variable_is_local() {
// The parent environment.
let mut env = Environment::new();
// The name of the variable
// we are going to use.
let var_name = String::from("somevar");
// The value we are going to use.
let var_value = 42.0;
// The statement we are going to process
// contains a block.
let statement = Statement::Block(vec![Statement::Var(
var_name.clone(),
Expression::Literal(Literal::Number(var_value)),
)]);
// Execute the statement.
let (result, output) = run_execute(&statement, &mut env);
assert_eq!(result, Ok(()));
assert!(output.is_empty());
assert!(matches!(env.get(&var_name), Err(EnvironmentError { .. })));
}We need to declare this submodule in the main module:
src/statement/tests.rs mod block;
mod expression;
mod print;
mod var;Detecting statement blocks is not more complicated than that. The new function Parser::statement_block has the same signature of Parser::statement_var:
src/parser.rs impl Parser {
pub fn statement_block(&mut self) -> ParserResult<Option<Statement>> {
// Check the presence of the
// opening curly brace `{`.
if self.match_current(&[TokenType::LBrace]).is_none() {
return Ok(None);
}
// Prepare an empty vector to host
// the statements contained in the block.
let mut statements: Vec<Statement> = Vec::new();
// Parse statements until we find the
// closing curly brace `}` or the end
// of the file.
// Add each statement to the vector.
while !self.check_current(&TokenType::RBrace) && !self.check_eof() {
statements.push(self.statement()?);
}
// Enforce the presence of a
// closing curly brace `}`.
// If not present, the statement
// is invalid.
self.expect(&TokenType::RBrace, "Invalid statement")?;
Ok(Some(Statement::Block(statements)))
}Inside the function we reuse techniques we have already seen in other functions to check the presence of the opening curly brace { and to enforce the presence of the closing one }. In between we parse all possible statements and add them to a simple vector.
The tests for this new piece of logic are:
src/parser/tests/statement.rs #[test]
// Test that the method `statement` works
// correctly on a block of statements in the form
// `{ STATEMENTS }`.
fn test_block_valid_statements() {
let tokens = vec![
dummy_token(TokenType::LBrace),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Reserved(Keyword::False)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::RBrace),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Ok(Statement::Block(vec![
Statement::Expression(Expression::Literal(Literal::Boolean(true))),
Statement::Expression(Expression::Literal(Literal::Boolean(false)))
]))
);
assert_eq!(parser.index, 6);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a block
// statement that contains an invalid statement.
fn test_block_invalid_statements() {
let tokens = vec![
dummy_token(TokenType::LBrace),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Reserved(Keyword::False)),
dummy_token(TokenType::RBrace),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: dummy_token(TokenType::RBrace)
})
);
assert_eq!(parser.index, 4);
}
#[test]
// Test that the method `statement` returns
// the correct error when it parses a block
// statement that is missing the closing
// curly brace `}`.
fn test_block_missing_closing_bracket() {
let tokens = vec![
dummy_token(TokenType::LBrace),
dummy_token(TokenType::Reserved(Keyword::True)),
dummy_token(TokenType::Semicolon),
dummy_token(TokenType::Reserved(Keyword::False)),
dummy_token(TokenType::Semicolon),
Token::eof(),
];
let mut parser = Parser::new(tokens);
let result = parser.statement();
assert_eq!(
result,
Err(ParserError {
message: String::from("Invalid statement"),
token: Token::eof()
})
);
assert_eq!(parser.index, 5);
}The function Parser::statement_block must be called as part of the possible options when parsing statements:
src/parser.rs impl Parser {
// Parse the next tokens as a statement.
pub fn statement(&mut self) -> ParserResult<Statement> {
// Try to parse the next tokens
// as a print statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_print()? {
return Ok(s);
}
// Try to parse the next tokens
// as a var statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_var()? {
return Ok(s);
}
// Try to parse the next tokens
// as a block statement. If successful,
// return the parsed statement.
if let Some(s) = self.statement_block()? {
return Ok(s);
}
// If we are here, the statement
// contains only an expression.
self.statement_expression()
}Stage 9 - Block syntax
This version of the code passes Stage 9 of the extension "Statements & state".