Skip to content

Changed Parsing Method of Identifiers #138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hexofyore wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Changed Parsing Method of Identifiers #138

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e0c91f0
Changed Parsing Method of Identifiers
zeroishero Jun 5, 2023
8431650
Changed,removed tests with illegal identifiers.
zeroishero Jun 5, 2023
7911d4e
Merge branch 'main' into feature_literal_errors
ISibboI Jun 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/error/display.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,9 @@ impl fmt::Display for EvalexprError {
write!(f, "This context does not allow disabling builtin functions")
},
IllegalEscapeSequence(string) => write!(f, "Illegal escape sequence: {}", string),
IllegalIdentifierSequence(string) => {
write!(f, "Illegal Identifier Sequence: {}", string)
},
CustomMessage(message) => write!(f, "Error: {}", message),
}
}
Expand Down
3 changes: 3 additions & 0 deletions src/error/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,6 +209,9 @@ pub enum EvalexprError {
/// This context does not allow disabling builtin functions.
BuiltinFunctionsCannotBeDisabled,

/// Error parsing identifier
IllegalIdentifierSequence(String),

/// A custom error explained by its message.
CustomMessage(String),
}
Expand Down
58 changes: 53 additions & 5 deletions src/token/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,12 +348,26 @@ fn partial_tokens_to_tokens(mut tokens: &[PartialToken]) -> EvalexprResult<Vec<T
},
PartialToken::Literal(literal) => {
cutoff = 1;
if let Ok(number) = parse_dec_or_hex(&literal) {
let starts_with_alphabet_or_underscore =
literal.starts_with(|x: char| x.is_alphabetic() || x == '_');
// Alphanumeric || Underscore || Colon
let contains_only_valid_chars = literal
.chars()
.all(|x| x.is_alphanumeric() || x == '_' || x == ':');
let is_not_underscore = literal != "_";
let contains_alphabet = literal.contains(|x: char| x.is_alphabetic());
if let Ok(boolean) = literal.parse::<bool>() {
Some(Token::Boolean(boolean))
} else if starts_with_alphabet_or_underscore
Copy link
Owner

@ISibboI ISibboI Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The decomposition of the conditions into separate variables is great!
Here we should first check only starts_with_alphabet_or_underscore, and if it does but any of the other three are false, then return a the error variant IllegalIdentifierSequence.

Copy link
Contributor Author

@hexofyore hexofyore Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did boolean first because true and false would both be interpreted as identifiers.

Copy link
Owner

@ISibboI ISibboI Jun 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean that the branch after boolean should only check for starts_with_alphabet_or_underscore, and then should have an inner branch that checks the rest.

&& contains_only_valid_chars
&& is_not_underscore
&& contains_alphabet
{
Some(Token::Identifier(literal))
} else if let Ok(number) = parse_dec_or_hex(&literal) {
Some(Token::Int(number))
} else if let Ok(number) = literal.parse::<FloatType>() {
Some(Token::Float(number))
} else if let Ok(boolean) = literal.parse::<bool>() {
Some(Token::Boolean(boolean))
} else {
// If there are two tokens following this one, check if the next one is
// a plus or a minus. If so, then attempt to parse all three tokens as a
Expand All @@ -370,10 +384,16 @@ fn partial_tokens_to_tokens(mut tokens: &[PartialToken]) -> EvalexprResult<Vec<T
cutoff = 3;
Some(Token::Float(number))
} else {
Some(Token::Identifier(literal.to_string()))
return Err(EvalexprError::IllegalIdentifierSequence(
Copy link
Owner

@ISibboI ISibboI Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are parsing the literal as a number, so this should be a new error variant like IllegalNumericLiteral.

Copy link
Contributor Author

@hexofyore hexofyore Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I will change this

literal.to_string(),
));
}
},
_ => Some(Token::Identifier(literal.to_string())),
_ => {
return Err(EvalexprError::IllegalIdentifierSequence(
Copy link
Owner

@ISibboI ISibboI Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are parsing the literal as a number, so this should be a new error variant like IllegalNumericLiteral.

Copy link
Contributor Author

@hexofyore hexofyore Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok

literal.to_string(),
))
},
}
}
},
Expand Down Expand Up @@ -495,4 +515,32 @@ mod tests {
]
);
}

#[test]
fn wrong_identifier_sequence() {
assert_eq!(
tokenize("1b + 1"),
Err(crate::EvalexprError::IllegalIdentifierSequence(
String::from("1b")
))
);
assert_eq!(
tokenize("_b + 1"),
Ok(vec![
Token::Identifier(String::from("_b")),
Token::Plus,
Token::Int(1)
])
);
assert_eq!(
tokenize("1.0 1e+5 _1a2 1"),
Ok(vec![
Token::Float(1.0),
Token::Float(100000.0),
Token::Identifier(String::from("_1a2")),
Token::Int(1)
])
);
// assert_eq!(tokenize("string"))
}
}
69 changes: 23 additions & 46 deletions tests/integration.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_string("3..3"),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3..3".to_owned()))
);
assert_eq!(
eval_string_with_context("string", &context),
Expand All @@ -629,7 +629,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_string_with_context("3..3", &context),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3..3".to_owned()))
);
assert_eq!(
eval_string_with_context_mut("string", &mut context),
Expand All @@ -643,7 +643,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_string_with_context_mut("3..3", &mut context),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3..3".to_owned()))
);

assert_eq!(eval_float("3.3"), Ok(3.3));
Expand Down Expand Up @@ -689,7 +689,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_int("(,);."),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence(".".to_owned()))
);
assert_eq!(eval_int_with_context("3", &context), Ok(3));
assert_eq!(
Expand All @@ -700,7 +700,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_int_with_context("(,);.", &context),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence(".".to_owned()))
);
assert_eq!(eval_int_with_context_mut("3", &mut context), Ok(3));
assert_eq!(
Expand All @@ -711,7 +711,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_int_with_context_mut("(,);.", &mut context),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence(".".to_owned()))
);

assert_eq!(eval_number("3"), Ok(3.0));
Expand Down Expand Up @@ -802,7 +802,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_tuple("3a3"),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3a3".to_owned()))
);
assert_eq!(
eval_tuple_with_context("3,3", &context),
Expand All @@ -816,7 +816,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_tuple_with_context("3a3", &context),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3a3".to_owned()))
);
assert_eq!(
eval_tuple_with_context_mut("3,3", &mut context),
Expand All @@ -830,7 +830,7 @@ fn test_shortcut_functions() {
);
assert_eq!(
eval_tuple_with_context_mut("3a3", &mut context),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
Err(EvalexprError::IllegalIdentifierSequence("3a3".to_owned()))
);

assert_eq!(eval_empty(""), Ok(EMPTY_VALUE));
Expand Down Expand Up @@ -889,8 +889,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("3..3").unwrap().eval_string(),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
build_operator_tree("3..3"),
Err(EvalexprError::IllegalIdentifierSequence("3..3".to_owned()))
);
assert_eq!(
build_operator_tree("string")
Expand All @@ -907,10 +907,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("3..3")
.unwrap()
.eval_string_with_context(&context),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
build_operator_tree("3..3"),
Err(EvalexprError::IllegalIdentifierSequence("3..3".to_owned()))
);
assert_eq!(
build_operator_tree("string")
Expand All @@ -926,12 +924,6 @@ fn test_shortcut_functions() {
actual: Value::Float(3.3)
})
);
assert_eq!(
build_operator_tree("3..3")
.unwrap()
.eval_string_with_context_mut(&mut context),
Err(EvalexprError::VariableIdentifierNotFound("3..3".to_owned()))
);

assert_eq!(build_operator_tree("3.3").unwrap().eval_float(), Ok(3.3));
assert_eq!(
Expand Down Expand Up @@ -993,8 +985,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("(,);.").unwrap().eval_int(),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
build_operator_tree("(,);."),
Err(EvalexprError::IllegalIdentifierSequence(".".to_owned()))
);
assert_eq!(
build_operator_tree("3")
Expand All @@ -1011,10 +1003,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("(,);.")
.unwrap()
.eval_int_with_context(&context),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
build_operator_tree("(,);."),
Err(EvalexprError::IllegalIdentifierSequence(".".to_owned()))
);
assert_eq!(
build_operator_tree("3")
Expand All @@ -1030,12 +1020,6 @@ fn test_shortcut_functions() {
actual: Value::Float(3.3)
})
);
assert_eq!(
build_operator_tree("(,);.")
.unwrap()
.eval_int_with_context_mut(&mut context),
Err(EvalexprError::VariableIdentifierNotFound(".".to_owned()))
);

assert_eq!(build_operator_tree("3").unwrap().eval_number(), Ok(3.0));
assert_eq!(
Expand Down Expand Up @@ -1161,8 +1145,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("3a3").unwrap().eval_tuple(),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
build_operator_tree("3a3"),
Err(EvalexprError::IllegalIdentifierSequence("3a3".to_owned()))
);
assert_eq!(
build_operator_tree("3,3")
Expand All @@ -1178,12 +1162,7 @@ fn test_shortcut_functions() {
actual: Value::Int(33)
})
);
assert_eq!(
build_operator_tree("3a3")
.unwrap()
.eval_tuple_with_context(&context),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
);

assert_eq!(
build_operator_tree("3,3")
.unwrap()
Expand All @@ -1199,10 +1178,8 @@ fn test_shortcut_functions() {
})
);
assert_eq!(
build_operator_tree("3a3")
.unwrap()
.eval_tuple_with_context_mut(&mut context),
Err(EvalexprError::VariableIdentifierNotFound("3a3".to_owned()))
build_operator_tree("3a3"),
Err(EvalexprError::IllegalIdentifierSequence("3a3".to_owned()))
);

assert_eq!(
Expand Down Expand Up @@ -2301,6 +2278,6 @@ fn test_hex() {
eval("0x"),
// The "VariableIdentifierNotFound" error is what evalexpr currently returns,
// but ideally it would return more specific errors for "illegal" literals.
Err(EvalexprError::VariableIdentifierNotFound("0x".into()))
Err(EvalexprError::IllegalIdentifierSequence("0x".into()))
);
}