Package ai.djl.modality.nlp.preprocess

Class PunctuationSeparator

java.lang.Object

ai.djl.modality.nlp.preprocess.PunctuationSeparator

All Implemented Interfaces:

TextProcessor

public class PunctuationSeparator
extends Object
implements TextProcessor

PunctuationSeparator separates punctuation into a separate token.

Constructor Summary

Constructors

Constructor	Description
PunctuationSeparator()

Method Summary

Modifier and Type	Method	Description
List<String>	preprocess(List<String> tokens)	Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

PunctuationSeparator

public PunctuationSeparator()

Method Details

preprocess

public List<String> preprocess(List<String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens