Conflicts:

	tests/test_parser.py

Note: This refactor opens the door for implementing a ContextualLexer for Earley.
But unlike the existing one for LALR, it will have to be computed at runtime,
rather than ahead of time.

regression-fix for #760

Fixed pyparsing link to point to 

https://github.com/pyparsing/pyparsing

See https://gitter.im/lark-parser/Lobby?at=5fb83369771c185e0eb8c0e2

mypy thinks that Token's __init__ is inherited from __str__(). That's
not wrong -- it's just irrelevant, because Token also implements
__new__(). Token's _effective_ constructor signature is determined by
its __new__() method, so that's what I have used in the type hint.

Not clear if 'value' is supposed to be Any, but that's what I need in
my application. And it works just fine!

The code is crystal clear:

  assert lexer in ('standard', ...) or issubclass(lexer, Lexer)

But the type hint said that lexer must be an _instance_ of Lexer, not
a subclass. This change fixes it to require a subclass of Lexer.

Adding missing Testcase to `__main__`

Fix IndexError

📝 Updated docs

Replaced the notice saying only `common.lark` is available into "`common.lark`, `python.lark`, and `unicode.lark`".

Make `LALR_CustomLexerWrapper` behave correctly + tests

Make `UnexpectedCharacters` print the repr

This is an additional response to #738.

I also often came across this when unexpectedly having a newline, which would previously result in the message being split up across two lines. This PR makes it easier to identify exactly which special character is causing problems.