Package ai.djl.inference.streaming

Class StreamingTranslator.StreamOutput<O>

java.lang.Object
ai.djl.inference.streaming.StreamingTranslator.StreamOutput<O>
Type Parameters:
O - the output type
Enclosing interface:
StreamingTranslator<I,O>
public abstract static class StreamingTranslator.StreamOutput<O> extends Object
A StreamingTranslator.StreamOutput represents a streamable output type (either iterative or asynchronous).

There are two modes for the StreamingTranslator.StreamOutput. When using it, you must choose one of the modes and can only access it once. Any other usage including trying both modes or trying one mode twice will result in an IllegalStateException.

The first mode is the iterative mode which can be used by calling getIterativeOutput(), it returns a result that has an internal iterate method. When calling the iterating method, it will compute an additional part of the output.

The second mode is asynchronous mode. Here, you can produce a mutable output object by calling getAsyncOutput(). Then, calling computeAsyncOutput() will synchronously compute the results and deposit them into the prepared output. This method works best with manual threading where the worker can return the template result to another thread and then continue to compute it.

  • Constructor Details

    • StreamOutput

      public StreamOutput()

  • Method Details

    • getAsyncOutput

      public final O getAsyncOutput()
      Returns a template object to be used with the async output.

      This should only be an empty data structure until computeAsyncOutput() is called.

      Returns:
      a template object to be used with the async output

    • buildAsyncOutput

      protected abstract O buildAsyncOutput()
      Performs the internal implementation of getAsyncOutput().
      Returns:
      the output for getAsyncOutput().

    • computeAsyncOutput

      public final void computeAsyncOutput()
      Computes the actual value and stores it in the object returned earlier by getAsyncOutput().

    • computeAsyncOutputInternal

      protected abstract void computeAsyncOutputInternal(O output)
      Performs the internal implementation of computeAsyncOutput().
      Parameters:
      output - the output object returned by the earlier call to getAsyncOutput().

    • getIterativeOutput

      public final O getIterativeOutput()
      Returns an iterative streamable output.
      Returns:
      an iterative streamable output

    • getIterativeOutputInternal

      public abstract O getIterativeOutputInternal()
      Performs the internal implementation of getIterativeOutput().
      Returns:
      the output for getIterativeOutput()