Package ai.djl.modality.nlp

Interface Vocabulary

All Known Implementing Classes:
DefaultVocabulary
public interface Vocabulary
Vocabulary is a collection of tokens. The primary purpose of a vocabulary is the map a token to an index.

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    contains(String token)
    Check if the vocabulary contains a token.
    long
    getIndex(String token)
    Returns the index of the given token.
    String
    getToken(long index)
    Returns the token corresponding to the given index.
    long
    size()
    Returns the size of the Vocabulary.

  • Method Details

    • getToken

      String getToken(long index)
      Returns the token corresponding to the given index.
      Parameters:
      index - the index
      Returns:
      the token corresponding to the given index

    • contains

      boolean contains(String token)
      Check if the vocabulary contains a token.
      Parameters:
      token - String token to be checked
      Returns:
      whether this vocabulary contains the token

    • getIndex

      long getIndex(String token)
      Returns the index of the given token.
      Parameters:
      token - the token
      Returns:
      the index of the given token.

    • size

      long size()
      Returns the size of the Vocabulary.
      Returns:
      the size of the Vocabulary