# `Predicator.Vocabulary`
[🔗](https://github.com/riddler/predicator-ex/blob/v9.4.1/lib/predicator/vocabulary.ex#L1)

The grammar's fixed vocabulary, enumerated for editor tooling.

An expression editor that offers completion needs to know what the language
accepts: which operators exist, which words are reserved, which duration
units follow a number, and which functions are callable. Function names have
always been reachable - `Predicator.Context.new/2` resolves them from
`Predicator.FunctionProvider` modules - but the operator and keyword lexemes
lived only inside `Predicator.Lexer`'s private clause heads, so a consumer
wanting completion had to hard-code a second copy of the grammar and keep it
in sync by hand. This module is the first copy made public instead.

It is a reading surface, not a parsing one. Nothing here participates in
lexing, parsing, compiling, or evaluating; the entries describe the grammar
the lexer already implements, and adding a category or a doc string changes
no program's meaning. `test/predicator/vocabulary_sync_test.exs` binds the
two together: it round-trips every enumerated lexeme through
`Predicator.Lexer.tokenize/1`, and it checks the enumeration against the
lexer's own `t:Predicator.Lexer.token/0` union, its `classify_identifier/1`
clause heads, and its `duration_unit?/1` clause heads, so a token added to
the lexer without a matching entry here turns the suite red rather than
quietly shipping an editor that cannot complete it.

## Entries

Every static entry is a map with at least these five keys:

- `:lexeme` - the exact source text, e.g. `">="`, `"contains"`
- `:token_type` - the `t:Predicator.Lexer.token/0` tag it lexes to
- `:category` - one of `categories/0`
- `:display` - what an editor shows in a completion list, which differs from
  the lexeme only where the bare lexeme reads badly on its own (`"a + b"`
  for `"+"`, `"1d"` for the duration unit `"d"`)
- `:doc` - one line of prose, sentence case, no trailing period

A function entry carries the same five keys plus `:arity`, since a function's
arity is part of what an editor needs to complete a call. Its `:doc` is
`nil`: a provider binds a name to `{arity, atom}` and carries no description,
so there is nothing truthful to put there, and an invented sentence in a
documented field is worse than an absent one.

## Operator entries

An entry in an operator category - the six categories `operators/0` returns -
carries five more keys, because a structured editor needs more about an
operator than a completion list does. A picklist has to name the operator in
a sentence, know how many operands it takes, know which atom the parser will
put in the AST for it, know which kinds of value it is worth offering it for
at all, and - where the same operator is enumerated twice - know which of the
two spellings to write:

- `:label` - a short phrase read in place of the operator, e.g. `"is at
  least"` for `">="`. `:display` is a template and `:doc` is a sentence of
  prose about the semantics; neither is a UI string, which is why this is
  its own key
- `:arity` - how many operands the operator takes. `"-"` takes `[1, 2]`,
  since it is both subtraction and negation
- `:ast_op` - the atom `Predicator.Parser.parse/2` puts in the node it
  builds, which differs from `:token_type` for three operators (`:in_op`
  becomes `:in`, `:contains_op` becomes `:contains`, `:strict_equal`
  becomes `:strict_eq`). `nil` where no node carries the operator as an
  atom of its own
- `:value_kinds` - the `t:value_kind/0`s the operator is worth offering for,
  as the kind of the value on its right. `nil` for an operator that does not
  compare a field against a value at all - the logical, arithmetic, temporal
  and cast categories
- `:canonical` - whether this spelling is the one `Predicator.decompile/2`
  writes for the operator. See "Case" below: a word operator is enumerated in
  both cases, and only one of the two entries is the spelling a caller that
  means to render source should offer

Entries outside an operator category carry none of the five. That is the
shape `t:function_entry/0` already has against `t:entry/0`: an entry carries
the keys its kind of thing has, and no placeholder keys for the ones it does
not.

### What `:value_kinds` admits, and two deliberate exclusions

The admissions follow `Predicator.Evaluator`'s own comparison semantics.
Ordered comparison (`>`, `>=`, `<`, `<=`) is admitted where the evaluator
orders the kind meaningfully: numbers and strings through its `types_match`
guard, dates and datetimes chronologically, durations as the numbers they
reduce to. Equality is admitted for every scalar kind. `in` takes a list on
its right and nothing else; `contains` takes a list on its *left*, so the
value beside it is a scalar of any kind.

Two things the evaluator does answer are still not admitted, because a
picklist offering them would be offering nonsense:

| Excluded | Why |
|---|---|
| Ordered comparison of booleans | `types_match` admits them, so `true > false` evaluates - by Erlang term order, which is not a fact about the author's data |
| Equality against a list | Lists compare with `==`, but the operator an author reaches for beside a list is `in`, and offering both invites writing the one that is almost never meant |

Neither exclusion narrows the grammar. Both are the kind of judgement
`Predicator.Simple` already records for the shapes it leaves out: the
language still accepts them, and the structured surface still does not offer
them.

## Case

The word operators are accepted in two cases - `and` and `AND` both lex to
`:and_op` - and both are enumerated, as separate entries with the same
`:token_type`. An editor offering only one of them would be offering a house
style the grammar does not have. `if`, `else`, `while`, and the temporal
words are lower-case only, and are enumerated only that way.

Enumerating both cases leaves a caller that renders source with a question
the enumeration alone cannot answer: of the two entries for one operator,
which spelling does `Predicator.decompile/2` write? That is a fact about this
vocabulary, so `:canonical` answers it here rather than leaving each caller
to re-derive it. It is `false` on exactly one thing: the lower-case entry of
a word operator enumerated in both cases. Every other operator entry is the
only spelling there is and is therefore its own canonical one - `">="`, and
the lower-case-only temporal words alike. It says nothing about what parses:
both spellings lex to the same token type, and an editor offering completions
wants both.

## Examples

    iex> Predicator.Vocabulary.by_category(:arithmetic) |> Enum.map(& &1.lexeme)
    ["+", "-", "*", "/", "%"]

    iex> Predicator.Vocabulary.tokens() |> Enum.find(&(&1.lexeme == ">=")) |> Map.take([:token_type, :category])
    %{token_type: :gte, category: :comparison}

    iex> Predicator.Vocabulary.functions() |> Enum.any?(&(&1.lexeme == "len"))
    true

# `category`

```elixir
@type category() ::
  :comparison
  | :logical
  | :arithmetic
  | :membership
  | :temporal
  | :control
  | :literal
  | :grouping
  | :punctuation
  | :cast
  | :duration_unit
  | :function
```

The kind of thing an entry is, which is what an editor groups a completion
list by.

`:cast` holds the single `::` postfix operator (ADR-0011); `:grouping` holds
the bracket pairs; `:punctuation` holds the separators that are neither.

# `entry`

```elixir
@type entry() :: %{
  lexeme: binary(),
  token_type: atom(),
  category: category(),
  display: binary(),
  doc: binary()
}
```

A fixed lexeme of the grammar - an operator, keyword, literal word, separator, or duration unit.

# `function_entry`

```elixir
@type function_entry() :: %{
  lexeme: binary(),
  token_type: :function_name | :qualified_function_name,
  category: :function,
  display: binary(),
  doc: nil,
  arity: Predicator.Evaluator.function_arity()
}
```

A callable function, resolved from the providers rather than from the lexer.

`:token_type` is `:qualified_function_name` for a namespaced name
(`"Math.abs"`) and `:function_name` for a bare one (`"len"`), matching what
the lexer produces for a call to it.

# `operator_entry`

```elixir
@type operator_entry() :: %{
  lexeme: binary(),
  token_type: atom(),
  category: category(),
  display: binary(),
  doc: binary(),
  label: binary(),
  arity: 0 | 1 | 2 | [1 | 2, ...],
  ast_op: atom() | nil,
  value_kinds: [value_kind()] | nil,
  canonical: boolean()
}
```

An operator entry: an `t:entry/0` plus what a structured editor needs in
order to offer the operator as a choice. See "Operator entries" above.

# `value_kind`

```elixir
@type value_kind() ::
  :string | :number | :boolean | :date | :datetime | :duration | :list
```

The kind of a value on the right of an operator, as a structured editor
models it.

These are the kinds `Predicator.Simple` admits as a scalar or as a list of
them, and no others.

`:number` covers integer and float literals together. They are two shapes in
`t:Predicator.Simple.scalar/0`, because the AST literal differs and the
round-trip has to preserve which one was written, but they are one kind
here: every operator worth offering beside `19.99` is worth offering beside
`500`, so a `:float` kind would duplicate `:number`'s list exactly and make
every editor branch on a distinction that changes nothing it renders
(px-gv1).

There is no `:relative_date` either, because `3d ago` is a datetime by the
time anything compares it.

# `all`

```elixir
@spec all(keyword()) :: [entry() | operator_entry() | function_entry(), ...]
```

Every entry: `tokens/0` followed by `functions/1` on the same `opts`.

## Examples

    iex> length(Predicator.Vocabulary.all()) == length(Predicator.Vocabulary.tokens()) + length(Predicator.Vocabulary.functions())
    true

# `by_category`

```elixir
@spec by_category(category()) :: [entry() | operator_entry() | function_entry()]
```

The entries in one category.

The argument is guarded against `categories/0`, so a misspelled category
raises `FunctionClauseError` rather than returning an empty list that reads
like a category the grammar happens not to use.

## Examples

    iex> Predicator.Vocabulary.by_category(:cast) |> Enum.map(& &1.lexeme)
    ["::"]

    iex> Predicator.Vocabulary.by_category(:literal) |> Enum.map(& &1.lexeme)
    ["true", "false", "null", "undefined"]

# `categories`

```elixir
@spec categories() :: [category(), ...]
```

Every category an entry can carry, in the order `tokens/0` groups them.

## Examples

    iex> :duration_unit in Predicator.Vocabulary.categories()
    true

# `functions`

```elixir
@spec functions(keyword()) :: [function_entry()]
```

The callable functions, resolved the same way `Predicator.Context.new/2`
resolves them.

`opts` takes `:builtins`, `:providers`, and `:functions`, and is passed
straight to `Predicator.Context.resolve_functions/1`, so the names returned
here are exactly the names a context built with the same options will
accept - including a host's own providers, which is the case an editor
embedded in a host application actually has.

Entries are sorted by name, since a dispatch map has no order and a
completion list needs one.

## Examples

    iex> Predicator.Vocabulary.functions(builtins: false)
    []

    iex> Predicator.Vocabulary.functions() |> Enum.find(&(&1.lexeme == "len")) |> Map.take([:display, :arity, :doc])
    %{display: "len(...)", arity: 1, doc: nil}

    iex> Predicator.Vocabulary.functions() |> Enum.find(&(&1.lexeme == "Math.abs")) |> Map.fetch!(:token_type)
    :qualified_function_name

# `keywords`

```elixir
@spec keywords() :: [entry() | operator_entry(), ...]
```

The word-shaped entries: everything an editor must not offer as a plain
identifier, because the lexer classifies it as something else.

Duration units are excluded. `d` is an ordinary identifier everywhere except
immediately after a number, so treating it as a reserved word would be
wrong; `by_category(:duration_unit)` is where those live.

## Examples

    iex> Predicator.Vocabulary.keywords() |> Enum.map(& &1.lexeme) |> Enum.member?("contains")
    true

    iex> Predicator.Vocabulary.keywords() |> Enum.map(& &1.lexeme) |> Enum.member?("d")
    false

# `operators`

```elixir
@spec operators() :: [operator_entry(), ...]
```

The entries that combine or compare values: the comparison, logical,
arithmetic, membership, temporal, and cast categories.

Every entry here is an `t:operator_entry/0`, carrying `:label`, `:arity`,
`:ast_op`, `:value_kinds`, and `:canonical` on top of what `tokens/0`
carries. See "Operator entries" in the module documentation for what each one
means.

## Examples

    iex> Predicator.Vocabulary.operators() |> Enum.all?(&(&1.category != :literal))
    true

    iex> Predicator.Vocabulary.operators() |> Enum.find(&(&1.lexeme == ">=")) |> Map.take([:label, :arity, :ast_op])
    %{label: "is at least", arity: 2, ast_op: :gte}

    iex> Predicator.Vocabulary.operators() |> Enum.find(&(&1.lexeme == "IN")) |> Map.fetch!(:value_kinds)
    [:list]

    iex> Predicator.Vocabulary.operators() |> Enum.filter(&(&1.token_type == :in_op)) |> Enum.map(&{&1.lexeme, &1.canonical})
    [{"in", false}, {"IN", true}]

# `tokens`

```elixir
@spec tokens() :: [entry() | operator_entry(), ...]
```

Every fixed lexeme of the grammar: operators, keywords, literal words,
brackets, separators, and duration units.

Functions are not here - they depend on which providers a caller resolves,
so they come from `functions/0` and `functions/1` instead. `all/0` is the
two lists together.

## Examples

    iex> Predicator.Vocabulary.tokens() |> Enum.map(& &1.category) |> Enum.uniq() |> Enum.member?(:function)
    false

# `value_kinds`

```elixir
@spec value_kinds() :: [value_kind(), ...]
```

Every kind of value an operator can be offered for.

The vocabulary of `:value_kinds` on an `t:operator_entry/0`, enumerated so a
caller can iterate the kinds rather than hard-code them.

## Examples

    iex> Predicator.Vocabulary.value_kinds()
    [:string, :number, :boolean, :date, :datetime, :duration, :list]

---

*Consult [api-reference.md](api-reference.md) for complete listing*