Package ai.djl.inference

Class Predictor<I,O>

java.lang.Object
ai.djl.inference.Predictor<I,O>
Type Parameters:
I - the input type
O - the output type
All Implemented Interfaces:
AutoCloseable
public class Predictor<I,O> extends Object implements AutoCloseable
The Predictor interface provides a session for model inference.

You can use a Predictor, with a specified Translator, to perform inference on a Model. The following is example code that uses Predictor: 

 Model model = Model.load(modelDir, modelName);

 // User must implement Translator interface, read Translator for detail.
 Translator<String, String> translator = new MyTranslator();

 try (Predictor<String, String> predictor = model.newPredictor(translator)) {
   String result = predictor.predict("What's up");
 }

See the tutorials on:

See Also:

  • Field Details

    • translator

      protected Translator<I,O> translator

    • timestamp

      protected long timestamp

    • prepared

      protected boolean prepared

    • model

      protected Model model

    • manager

      protected NDManager manager

    • metrics

      protected Metrics metrics

    • block

      protected Block block

    • parameterStore

      protected ParameterStore parameterStore

    • dimension

      protected Dimension dimension

  • Constructor Details

    • Predictor

      public Predictor(Model model, Translator<I,O> translator, Device device, boolean copy)
      Creates a new instance of BasePredictor with the given Model and Translator.
      Parameters:
      model - the model on which the predictions are based
      translator - the translator to be used
      device - the device for prediction
      copy - whether to copy the parameters to the parameter store. If the device changes, it will copy regardless

  • Method Details

    • predict

      public O predict(I input) throws TranslateException
      Predicts an item for inference.
      Parameters:
      input - the input
      Returns:
      the output object defined by the user
      Throws:
      TranslateException - if an error occurs during prediction

    • predictInternal

      protected NDList predictInternal(TranslatorContext ctx, NDList ndList) throws TranslateException
      Predicts an item for inference.
      Parameters:
      ctx - the context for the Predictor.
      ndList - the input NDList
      Returns:
      the output NDList
      Throws:
      TranslateException - if an error occurs during prediction

    • batchPredict

      public List<O> batchPredict(List<I> inputs) throws TranslateException
      Predicts a batch for inference.
      Parameters:
      inputs - a list of inputs
      Returns:
      a list of output objects defined by the user
      Throws:
      TranslateException - if an error occurs during prediction

    • streamingPredict

      public StreamingTranslator.StreamOutput<O> streamingPredict(I input) throws TranslateException
      Predicts an item for inference.
      Parameters:
      input - the input
      Returns:
      the output object defined by the user
      Throws:
      TranslateException - if an error occurs during prediction

    • supportsStreaming

      public boolean supportsStreaming()
      Returns true if streaming is supported by the predictor, block, and translator.
      Returns:
      true if streaming is supported by the predictor, block, and translator

    • setMetrics

      public void setMetrics(Metrics metrics)
      Attaches a Metrics param to use for benchmark.
      Parameters:
      metrics - the Metrics class

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable

    • finalize

      protected void finalize() throws Throwable
      Overrides:
      finalize in class Object
      Throws:
      Throwable