# `Predicator.Lexer`
[🔗](https://github.com/riddler/predicator-ex/blob/v9.4.1/lib/predicator/lexer.ex#L1)

Lexical analyzer for predicator expressions.

The lexer converts input strings into a stream of tokens with complete
position tracking for detailed error reporting. Each token includes:
- Token type and value
- Line and column position
- Length for precise error highlighting

## Error spans

A lexical error returns `{:error, message, line, column, span}`, where
`span`'s start always equals `{line, column}` and its end is exclusive. How
wide the span is depends on the failure, and the three widths are deliberate:

- **An unexpected character** spans one character. The offending character
  *is* one character, so there is nothing wider to underline.
- **An unterminated string literal** spans the opening quote alone. The
  literal runs to end of source by definition, so underlining its true
  extent would underline the rest of the program; pointing at where the
  literal began is the better diagnostic.
- **A malformed or unterminated date/datetime literal** spans the whole
  literal - the opening `#` through the closing `#`, or through end of input
  when there is none. Here the literal is wrong as a unit, and a caret on
  the opening `#` under-describes the failure.

A date literal containing a raw newline gets a span whose end is on a later
line, the same way a multi-line string token's `end_position` does.

## Example

    iex> Predicator.Lexer.tokenize("limit > 85")
    {:ok, [
      {:identifier, 1, 1, 5, "limit"},
      {:gt, 1, 7, 1, ">"},
      {:integer, 1, 9, 2, 85},
      {:eof, 1, 11, 0, nil}
    ]}

# `lexer_state`

```elixir
@type lexer_state() :: %{
  input: binary(),
  position: non_neg_integer(),
  line: pos_integer(),
  column: pos_integer(),
  tokens: [token()]
}
```

Internal lexer state for position tracking.

# `position`

```elixir
@type position() ::
  {line :: pos_integer(), column :: pos_integer(), length :: pos_integer()}
```

Position information for a token.

Contains:
- `line` - 1-based line number
- `column` - 1-based column number
- `length` - number of characters in the token

# `result`

```elixir
@type result() ::
  {:ok, [token()]}
  | {:error, binary(), pos_integer(), pos_integer(), Predicator.Types.span()}
```

Lexer result - either success with tokens or error with details.

# `token`

```elixir
@type token() ::
  {:identifier, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:integer, pos_integer(), pos_integer(), pos_integer(), integer()}
  | {:float, pos_integer(), pos_integer(), pos_integer(), float()}
  | {:string, pos_integer(), pos_integer(), pos_integer(), binary(),
     :double | :single, {pos_integer(), pos_integer()}}
  | {:boolean, pos_integer(), pos_integer(), pos_integer(), boolean()}
  | {:undefined, pos_integer(), pos_integer(), pos_integer(), :undefined}
  | {:null, pos_integer(), pos_integer(), pos_integer(), nil}
  | {:date, pos_integer(), pos_integer(), pos_integer(), Date.t()}
  | {:datetime, pos_integer(), pos_integer(), pos_integer(), DateTime.t()}
  | {:gt, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:lt, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:gte, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:lte, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:eq, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:ne, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:equal_equal, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:strict_equal, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:strict_ne, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:plus, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:minus, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:multiply, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:divide, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:modulo, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:and_and, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:or_or, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:bang, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:and_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:or_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:not_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:lparen, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:rparen, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:lbracket, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:rbracket, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:lbrace, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:rbrace, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:colon, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:double_colon, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:comma, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:semicolon, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:dot, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:in_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:contains_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:if_kw, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:else_kw, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:while_kw, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:function_name, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:qualified_function_name, pos_integer(), pos_integer(), pos_integer(),
     binary()}
  | {:duration_unit, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:fractional_number, pos_integer(), pos_integer(), pos_integer(),
     {integer(), binary()}}
  | {:ago_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:from_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:now_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:next_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:last_op, pos_integer(), pos_integer(), pos_integer(), binary()}
  | {:eof, pos_integer(), pos_integer(), pos_integer(), nil}
```

A lexical token with position information.

Format: `{type, line, column, length, value}`, except `:string`, which is
`{:string, line, column, length, value, quote_type, end_position}`.

`length` is the token's source character count. `end_position` is the
exclusive end - one past the closing quote - and is `{line, column + length}`
for every literal on a single line. The two disagree only when the literal
contains a raw newline, which is why the string token stores it rather than
letting a consumer compute it.

# `tokenize`

```elixir
@spec tokenize(binary()) :: result()
```

Tokenizes an input string into a list of tokens.

## Parameters

- `input` - The expression string to tokenize

## Returns

- `{:ok, tokens}` - Successfully tokenized input
- `{:error, message, line, column, span}` - Lexical error with position and
  extent; `span`'s start always equals `{line, column}`

## Examples

    iex> Predicator.Lexer.tokenize("limit > 85")
    {:ok, [
      {:identifier, 1, 1, 5, "limit"},
      {:gt, 1, 7, 1, ">"},
      {:integer, 1, 9, 2, 85},
      {:eof, 1, 11, 0, nil}
    ]}

    iex> Predicator.Lexer.tokenize("age >= 18")
    {:ok, [
      {:identifier, 1, 1, 3, "age"},
      {:gte, 1, 5, 2, ">="},
      {:integer, 1, 8, 2, 18},
      {:eof, 1, 10, 0, nil}
    ]}

    iex> Predicator.Lexer.tokenize("name == \"John\"")
    {:ok, [
      {:identifier, 1, 1, 4, "name"},
      {:equal_equal, 1, 6, 2, "=="},
      {:string, 1, 9, 6, "John", :double, {1, 15}},
      {:eof, 1, 15, 0, nil}
    ]}

    iex> Predicator.Lexer.tokenize("limit > 85 AND age >= 18")
    {:ok, [
      {:identifier, 1, 1, 5, "limit"},
      {:gt, 1, 7, 1, ">"},
      {:integer, 1, 9, 2, 85},
      {:and_op, 1, 12, 3, "AND"},
      {:identifier, 1, 16, 3, "age"},
      {:gte, 1, 20, 2, ">="},
      {:integer, 1, 23, 2, 18},
      {:eof, 1, 25, 0, nil}
    ]}

---

*Consult [api-reference.md](api-reference.md) for complete listing*