Package ai.djl.inference.streaming

Interface StreamingTranslator<I,O>

Type Parameters:

I - the input type

O - the output type

All Superinterfaces:

PostProcessor<O>, PreProcessor<I>, Translator<I,O>

All Known Subinterfaces:

ServingTranslator

public interface StreamingTranslator<I,O>
extends Translator<I,O>

An expansion of the Translator with postProcessing for the StreamingBlock (used by Predictor.streamingPredict(Object).

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	StreamingTranslator.StreamOutput<O>	A StreamingTranslator.StreamOutput represents a streamable output type (either iterative or asynchronous).
static enum	StreamingTranslator.Support	What types of StreamingTranslator.StreamOutputs are supported by a StreamingTranslator.

Method Summary

Modifier and Type	Method	Description
StreamingTranslator.Support	getSupport()	Returns what kind of StreamingTranslator.StreamOutput this StreamingTranslator supports.
StreamingTranslator.StreamOutput<O>	processStreamOutput(TranslatorContext ctx, Stream<NDList> list)	Processes the output NDList to the corresponding output object.
default boolean	supportsAsync()	Returns whether the StreamingTranslator supports iterative output.
default boolean	supportsIterative()	Returns whether the StreamingTranslator supports iterative output.

Methods inherited from interface ai.djl.translate.PostProcessor

processOutput

Methods inherited from interface ai.djl.translate.PreProcessor

processInput

Methods inherited from interface ai.djl.translate.Translator

batchProcessInput, batchProcessOutput, getBatchifier, getExpansions, prepare

Method Details

processStreamOutput

StreamingTranslator.StreamOutput<O> processStreamOutput(TranslatorContext ctx,
 Stream<NDList> list)
                                                 throws Exception

Processes the output NDList to the corresponding output object.

Parameters:

ctx - the toolkit used for post-processing

list - the output NDList after inference, usually immutable in engines like PyTorch. @see Issue 1774

Returns:

the output object of expected type

Throws:

Exception - if an error occurs during processing output

getSupport

StreamingTranslator.Support getSupport()

Returns what kind of StreamingTranslator.StreamOutput this StreamingTranslator supports.

Returns:

what kind of StreamingTranslator.StreamOutput this StreamingTranslator supports

supportsIterative

default boolean supportsIterative()

Returns whether the StreamingTranslator supports iterative output.

Returns:

whether the StreamingTranslator supports iterative output

See Also:

• StreamingTranslator.StreamOutput.getIterativeOutput()

supportsAsync

default boolean supportsAsync()

Returns whether the StreamingTranslator supports iterative output.

Returns:

whether the StreamingTranslator supports iterative output

See Also:

• StreamingTranslator.StreamOutput.getAsyncOutput()