Lexer

public struct Lexer : IteratorProtocol, Sequence, Sendable

A type that tokenize a source file.

Show on GitHub

  • The Hylo source being tokenized.

    Declaration

    Swift

    public let sourceCode: SourceFile
    Show on GitHub

  • The current position in the source file.

    Declaration

    Swift

    private(set) var index: String.Index { get }
    Show on GitHub

  • Creates a lexer generating tokens from the contents of source.

    Declaration

    Swift

    public init(tokenizing source: SourceFile)
    Show on GitHub

  • The current location of the lexer in sourceCode.

    Declaration

    Swift

    public var location: SourcePosition { get }
    Show on GitHub

  • true iff the last scanned token was .int.

    Declaration

    Swift

    private var previousTokenWasInt: Bool
    Show on GitHub

  • Advances to the next token and returns it, or returns nil if no next token exists.

    Declaration

    Swift

    public mutating func next() -> Token?
    Show on GitHub

  • Discards count characters from the stream.

    Declaration

    Swift

    private mutating func discard(_ count: Int = 1)
    Show on GitHub

  • Returns the next character in the stream without consuming it, if any.

    Declaration

    Swift

    private func peek() -> Character?
    Show on GitHub

  • Returns the current index and consumes character from the stream, or returns nil if the stream starts with a different character.

    Declaration

    Swift

    public mutating func take(_ character: Character) -> String.Index?
    Show on GitHub

  • Returns the current index and consumes prefix from the stream, or returns nil if the stream starts with a different prefix.

    Declaration

    Swift

    private mutating func take<T: Sequence>(prefix: T) -> String.Index?
where T.Element == Character
    Show on GitHub

  • Consumes the longest substring that satisfies the given predicate.

    Declaration

    Swift

    private mutating func take(while predicate: (Character) -> Bool) -> Substring
    Show on GitHub

  • Consumes an integral literal and returns its kind, or returns nil if it fails to scan a valid integer.

    Declaration

    Swift

    private mutating func scanIntegralLiteral(allowingPlus allowPlus: Bool = false) -> Token.Kind?

    Parameters

    allowingPlus

    If set to true, allows the integral literal to begin with a “+” sign. If set to false (the default), only allows “-” or no sign at the beginning.
    Show on GitHub