diff --git a/README.md b/README.md new file mode 100644 index 0000000..7634a42 --- /dev/null +++ b/README.md @@ -0,0 +1,70 @@ +# SPL Lexer + +A lexer (tokenizer) implementation in Go for parsing custom language/query syntax. + +## Overview + +This project implements a lexical analyzer that converts input strings into tokens for a custom language. The lexer recognizes various token types including symbols, operators, keywords, literals, and punctuation. + +## Features + +- Tokenizes input strings into meaningful components +- Supports various token types: + - Symbols (identifiers) + - Operators (comparison, logical) + - Literals (strings, numbers) + - Punctuation (braces, colons, pipes) + - Comments + - Whitespace + +## Token Types + +The lexer supports the following token types: + +- `SYMBOL`: Identifiers +- `COLON`: `:` +- `EXCLAMATION`: `!` +- `EQUAL`: `=` +- `NOT_EQUAL`: `!=` +- `AND`: `and` +- `NOT`: `not` +- `OR`: `or` +- `MORE`: `>` +- `LESS`: `<` +- `MORE_EQUAL`: `>=` +- `LESS_EQUAL`: `<=` +- `OPEN_BRACE`: `(` +- `CLOSED_BRACE`: `)` +- `COMMENT`: `//` comments +- `PIPE`: `|` +- `NUMBER`: Integer numbers +- `FLOAT_NUMBER`: Floating-point numbers +- `STRING_LITERAL`: String literals in quotes +- `SPACE`: Whitespace + +## Usage + +To use the lexer, you can call the `Parse` function from the `lexer` package: + +```go +package main + +import ( + "github.com/e1lama/spl/lexer" +) + +func main() { + tokens := lexer.Parse("some input string") + // Process tokens +} +``` + +## Project Structure + +- `main.go` - Main entry point (currently empty) +- `lexer/lexer.go` - Core lexer implementation +- `lexer/tokens.go` - Token definitions and types + +## License + +MIT \ No newline at end of file