Package ai.djl.training.dataset

Class DataIterable

java.lang.Object

ai.djl.training.dataset.DataIterable

All Implemented Interfaces:

Iterable<Batch>, Iterator<Batch>

Direct Known Subclasses:

BulkDataIterable

public class DataIterable
extends Object
implements Iterable<Batch>, Iterator<Batch>

DataIterable is a data loader that combines Dataset, Batchifier, Pipeline, and Sampler to provide an iterable over the given RandomAccessDataset.

We don't recommended using DataIterable directly. Instead use RandomAccessDataset combined with Trainer to iterate over the RandomAccessDataset}

Field Summary

Fields

Modifier and Type	Field	Description
protected Batchifier	dataBatchifier
protected RandomAccessDataset	dataset
protected Device	device
protected Batchifier	labelBatchifier
protected NDManager	manager
protected Pipeline	pipeline
protected Pipeline	targetPipeline

Constructor Summary

Constructors

Constructor	Description
DataIterable(RandomAccessDataset dataset, NDManager manager, Sampler sampler, Batchifier dataBatchifier, Batchifier labelBatchifier, Pipeline pipeline, Pipeline targetPipeline, ExecutorService executor, int preFetchNumber, Device device)	Creates a new instance of DataIterable with the given parameters.

Method Summary

Modifier and Type	Method	Description
protected Batch	fetch(List<Long> indices, int progress)
boolean	hasNext()
Iterator<Batch>	iterator()
Batch	next()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Field Details

dataset

protected RandomAccessDataset dataset

manager

protected NDManager manager

dataBatchifier

protected Batchifier dataBatchifier

labelBatchifier

protected Batchifier labelBatchifier

pipeline

protected Pipeline pipeline

targetPipeline

protected Pipeline targetPipeline

device

protected Device device

Constructor Details

DataIterable

public DataIterable(RandomAccessDataset dataset,
 NDManager manager,
 Sampler sampler,
 Batchifier dataBatchifier,
 Batchifier labelBatchifier,
 Pipeline pipeline,
 Pipeline targetPipeline,
 ExecutorService executor,
 int preFetchNumber,
 Device device)

Creates a new instance of DataIterable with the given parameters.

Parameters:

dataset - the dataset to iterate on

manager - the manager to create the arrays

sampler - a sampler to sample data with

dataBatchifier - a batchifier for data

labelBatchifier - a batchifier for labels

pipeline - the pipeline of transforms to apply on the data

targetPipeline - the pipeline of transforms to apply on the labels

executor - an ExecutorService

preFetchNumber - the number of samples to prefetch

device - the Device

Method Details

iterator

public Iterator<Batch> iterator()

Specified by:

iterator in interface Iterable<Batch>

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<Batch>

next

public Batch next()

Specified by:

next in interface Iterator<Batch>

fetch

protected Batch fetch(List<Long> indices,
 int progress)
               throws IOException

Throws:

IOException