Package ai.djl.modality.nlp

Class DefaultVocabulary

java.lang.Object

ai.djl.modality.nlp.DefaultVocabulary

All Implemented Interfaces:

Vocabulary

public class DefaultVocabulary
extends Object
implements Vocabulary

The default implementation of Vocabulary.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	DefaultVocabulary.Builder	Builder class that is used to build the DefaultVocabulary.

Constructor Summary

Constructors

Constructor	Description
DefaultVocabulary(DefaultVocabulary.Builder builder)	Creates a DefaultVocabulary object with a DefaultVocabulary.Builder.
DefaultVocabulary(List<String> tokens)	Creates a DefaultVocabulary object with the given list of tokens.

Method Summary

Modifier and Type	Method	Description
static DefaultVocabulary.Builder	builder()	Creates a new builder to build a DefaultVocabulary.
boolean	contains(String token)	Check if the vocabulary contains a token.
long	getIndex(String token)	Returns the index of the given token.
String	getToken(long index)	Returns the token corresponding to the given index.
long	size()	Returns the size of the Vocabulary.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DefaultVocabulary

public DefaultVocabulary(List<String> tokens)

Creates a DefaultVocabulary object with the given list of tokens.

Parameters:

tokens - the List of tokens to build the vocabulary with

DefaultVocabulary

public DefaultVocabulary(DefaultVocabulary.Builder builder)

Creates a DefaultVocabulary object with a DefaultVocabulary.Builder.

Parameters:

builder - the DefaultVocabulary.Builder to build the vocabulary with

Method Details

contains

public boolean contains(String token)

Check if the vocabulary contains a token.

Specified by:

contains in interface Vocabulary

Parameters:

token - String token to be checked

Returns:

whether this vocabulary contains the token

getToken

public String getToken(long index)

Returns the token corresponding to the given index.

Specified by:

getToken in interface Vocabulary

Parameters:

index - the index

Returns:

the token corresponding to the given index

getIndex

public long getIndex(String token)

Returns the index of the given token.

Specified by:

getIndex in interface Vocabulary

Parameters:

token - the token

Returns:

the index of the given token.

size

public long size()

Returns the size of the Vocabulary.

Specified by:

size in interface Vocabulary

Returns:

the size of the Vocabulary

builder

public static DefaultVocabulary.Builder builder()

Creates a new builder to build a DefaultVocabulary.

Returns:

a new builder