Package ai.djl.training.dataset

Interface Dataset

All Known Subinterfaces:

RawDataset<T>

All Known Implementing Classes:

ArrayDataset, RandomAccessDataset

public interface Dataset

An interface to represent a set of sample data/label pairs to train a model.

See Also:

• The guide to datasets
• The guide to implementing a custom dataset

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static enum	Dataset.Usage	An enum that indicates the mode - training, test or validation.

Method Summary

Modifier and Type	Method	Description
Iterable<Batch>	getData(NDManager manager)	Fetches an iterator that can iterate through the Dataset.
default Iterable<Batch>	getData(NDManager manager, ExecutorService executorService)	Fetches an iterator that can iterate through the Dataset with multiple threads.
default TranslatorOptions	matchingTranslatorOptions()	Returns TranslatorOptions that match the pre-processing and post-processing of this dataset.
default void	prepare()	Prepares the dataset for use.
void	prepare(ai.djl.util.Progress progress)	Prepares the dataset for use with tracked progress.

Method Details

getData

Iterable<Batch> getData(NDManager manager)
                 throws IOException,
TranslateException

Fetches an iterator that can iterate through the Dataset.

Parameters:

manager - the dataset to iterate through

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

getData

default Iterable<Batch> getData(NDManager manager,
 ExecutorService executorService)
                         throws IOException,
TranslateException

Fetches an iterator that can iterate through the Dataset with multiple threads.

Parameters:

manager - the dataset to iterate through

executorService - the executorService to use for multi-threading

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

default void prepare()
              throws IOException,
TranslateException

Prepares the dataset for use.

Throws:

IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

void prepare(ai.djl.util.Progress progress)
      throws IOException,
TranslateException

Prepares the dataset for use with tracked progress.

Parameters:

progress - the progress tracker

Throws:

IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

matchingTranslatorOptions

default TranslatorOptions matchingTranslatorOptions()

Returns TranslatorOptions that match the pre-processing and post-processing of this dataset.

Returns:

matching translators or null if none defined