Package ai.djl.nn

Class Activation

java.lang.Object
ai.djl.nn.Activation
public final class Activation extends Object
Utility class that provides activation functions and blocks.

Many networks make use of the Linear block and other similar linear transformations. However, any number of linear transformations that are composed will only result in a different linear transformation (\($f(x) = W_2(W_1x) = (W_2W_1)x = W_{combined}x\)). In order to represent non-linear data, non-linear functions called activation functions are interspersed between the linear transformations. This allows the network to represent non-linear functions of increasing complexity.

See wikipedia for more details.

  • Method Details

    • relu

      public static NDArray relu(NDArray array)
      Applies ReLU activation on the input NDArray.

      ReLU is defined by: \( y = max(0, x) \)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying ReLU activation

    • relu

      public static NDList relu(NDList arrays)
      Applies ReLU activation on the input singleton NDList.

      ReLU is defined by: \( y = max(0, x) \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying ReLU activation

    • relu6

      public static NDArray relu6(NDArray array)
      Applies ReLU6 activation on the input NDArray.

      ReLU6 is defined by: \( y = min(6,max(0, x)) \)

      Parameters:
      array - the input singleton NDArray
      Returns:
      the NDArray after applying ReLU6 activation

    • relu6

      public static NDList relu6(NDList arrays)
      Applies ReLU6 activation on the input singleton NDList.

      ReLU6 is defined by: \( y = min(6,max(0, x)) \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying ReLU6 activation

    • sigmoid

      public static NDArray sigmoid(NDArray array)
      Applies Sigmoid activation on the input NDArray.

      Sigmoid is defined by: \( y = 1 / (1 + e^{-x}) \)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying Sigmoid activation

    • sigmoid

      public static NDList sigmoid(NDList arrays)
      Applies Sigmoid activation on the input singleton NDList.

      Sigmoid is defined by: \( y = 1 / (1 + e^{-x}) \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying Sigmoid activation

    • tanh

      public static NDArray tanh(NDArray array)
      Applies Tanh activation on the input NDArray.

      Tanh is defined by: \( y = (e^x - e^{-x}) / (e^x + e^{-x}) \)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying Tanh activation

    • tanh

      public static NDList tanh(NDList arrays)
      Applies Tanh activation on the input singleton NDList.

      Tanh is defined by: \( y = (e^x - e^{-x}) / (e^x + e^{-x}) \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying tanh activation

    • softPlus

      public static NDArray softPlus(NDArray array)
      Applies softPlus activation on the input NDArray.

      softPlus is defined by: \( y = log(1 + e^x) \)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying soft ReLU activation

    • softPlus

      public static NDList softPlus(NDList arrays)
      Applies softPlus activation on the input singleton NDList.

      softPlus is defined by: \( y = log(1 + e^x) \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying soft ReLU activation

    • softSign

      public static NDArray softSign(NDArray array)
      Applies softSign activation on the input NDArray.

      softPlus is defined by: \( y = x / 1 + |x| \)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying soft ReLU activation

    • softSign

      public static NDList softSign(NDList arrays)
      Applies softPlus activation on the input singleton NDList.

      softPlus is defined by: \( y = x / 1 + |x| \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying soft ReLU activation

    • leakyRelu

      public static NDArray leakyRelu(NDArray array, float alpha)
      Applies Leaky ReLU activation on the input NDArray.

      Leaky ReLU is defined by: \( y = x \gt 0 ? x : alpha * x \)

      Parameters:
      array - the input NDArray
      alpha - the slope for the activation
      Returns:
      the NDArray after applying Leaky ReLU activation

    • leakyRelu

      public static NDList leakyRelu(NDList arrays, float alpha)
      Applies Leaky ReLU activation on the input singleton NDList.

      Leaky ReLU is defined by: \( y = x \gt 0 ? x : alpha * x \)

      Parameters:
      arrays - the input singleton NDList
      alpha - the slope for the activation
      Returns:
      the singleton NDList after applying Leaky ReLU activation

    • elu

      public static NDArray elu(NDArray array, float alpha)
      Applies ELU activation on the input NDArray.

      ELU is defined by: \( y = x \gt 0 ? x : alpha * (e^x - 1) \)

      Parameters:
      array - the input NDArray
      alpha - the slope for the activation
      Returns:
      the NDArray after applying ELU activation

    • elu

      public static NDList elu(NDList arrays, float alpha)
      Applies ELU(Exponential Linear Unit) activation on the input singleton NDList.

      ELU is defined by: \( y = x \gt 0 ? x : alpha * (e^x - 1) \)

      Parameters:
      arrays - the input singleton NDList
      alpha - the slope for the activation
      Returns:
      the singleton NDList after applying ELU activation

    • selu

      public static NDArray selu(NDArray array)
      Applies Scaled ELU activation on the input NDArray.

      Scaled ELU is defined by: \( y = lambda * (x \gt 0 ? x : alpha * (e^x - 1))\) where \(lambda = 1.0507009873554804934193349852946\) and \(alpha = 1.6732632423543772848170429916717\)

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying Scale ELU activation

    • selu

      public static NDList selu(NDList arrays)
      Applies Scaled ELU activation on the input singleton NDList.

      Scaled ELU is defined by: \( y = lambda * (x \gt 0 ? x : alpha * (e^x - 1))\) where \(lambda = 1.0507009873554804934193349852946\) and \(alpha = 1.6732632423543772848170429916717 \)

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying Scaled ELU activation

    • gelu

      public static NDArray gelu(NDArray array)
      Applies GELU(Gaussian Error Linear Unit) activation on the input NDArray.
      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying GELU activation

    • gelu

      public static NDList gelu(NDList arrays)
      Applies GELU(Gaussian Error Linear Unit) activation on the input singleton NDList.
      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying GELU activation

    • swish

      public static NDArray swish(NDArray array, float beta)
      Applies Swish activation on the input NDArray.

      Swish is defined as \(y = x * sigmoid(beta * x)\)

      Parameters:
      array - the input NDArray
      beta - a hyper-parameter
      Returns:
      the NDArray after applying Swish activation

    • swish

      public static NDList swish(NDList arrays, float beta)
      Applies SWish activation on the input singleton NDList.

      Swish is defined as \(y = x * sigmoid(beta * x)\)

      Parameters:
      arrays - the input singleton NDList
      beta - a hyper-parameter
      Returns:
      the singleton NDList after applying Swish activation

    • mish

      public static NDArray mish(NDArray array)
      Applies Mish activation on the input NDArray.

      Mish is defined as \(y = x * tanh(ln(1 + e^x)\) defined by Diganta Misra in his paper Mish: A Self Regularized Non-Monotonic Neural Activation Function

      Parameters:
      array - the input NDArray
      Returns:
      the NDArray after applying Mish activation

    • mish

      public static NDList mish(NDList arrays)
      Applies Mish activation on the input singleton NDList.

      Mish is defined as \(y = x * tanh(ln(1 + e^x)\) defined by Diganta Misra in his paper Mish: A Self Regularized Non-Monotonic Neural Activation Function

      Parameters:
      arrays - the input singleton NDList
      Returns:
      the singleton NDList after applying Mish activation

    • reluBlock

      public static Block reluBlock()
      Creates a LambdaBlock that applies the ReLU activation function in its forward function.
      Returns:
      the LambdaBlock that applies the ReLU activation function

    • relu6Block

      public static Block relu6Block()
      Creates a LambdaBlock that applies the ReLU6 activation function in its forward function.
      Returns:
      the LambdaBlock that applies the ReLU activation function

    • sigmoidBlock

      public static Block sigmoidBlock()
      Creates a LambdaBlock that applies the Sigmoid activation function in its forward function.
      Returns:
      the LambdaBlock that applies the Sigmoid activation function

    • tanhBlock

      public static Block tanhBlock()
      Creates a LambdaBlock that applies the Tanh activation function in its forward function.
      Returns:
      the LambdaBlock that applies the Tanh activation function

    • softPlusBlock

      public static Block softPlusBlock()
      Creates a LambdaBlock that applies the softPlus(NDList) activation function in its forward function.
      Returns:
      the LambdaBlock that applies the softPlus(NDList) activation function

    • softSignBlock

      public static Block softSignBlock()
      Creates a LambdaBlock that applies the softSign(NDList) activation function in its forward function.
      Returns:
      the LambdaBlock that applies the softSign(NDList) activation function

    • leakyReluBlock

      public static Block leakyReluBlock(float alpha)
      Creates a LambdaBlock that applies the LeakyReLU activation function in its forward function.
      Parameters:
      alpha - the slope for the activation
      Returns:
      the LambdaBlock that applies the LeakyReLU activation function

    • eluBlock

      public static Block eluBlock(float alpha)
      Creates a LambdaBlock that applies the ELU activation function in its forward function.
      Parameters:
      alpha - the slope for the activation
      Returns:
      the LambdaBlock that applies the ELU activation function

    • seluBlock

      public static Block seluBlock()
      Creates a LambdaBlock that applies the SELU activation function in its forward function.
      Returns:
      the LambdaBlock that applies the SELU activation function

    • geluBlock

      public static Block geluBlock()
      Creates a LambdaBlock that applies the GELU activation function in its forward function.
      Returns:
      the LambdaBlock that applies the GELU activation function

    • swishBlock

      public static Block swishBlock(float beta)
      Creates a LambdaBlock that applies the Swish activation function in its forward function.
      Parameters:
      beta - a hyper-parameter
      Returns:
      the LambdaBlock that applies the Swish activation function

    • mishBlock

      public static Block mishBlock()
      Creates a LambdaBlock that applies the Mish activation function in its forward function.
      Returns:
      the LambdaBlock that applies the Mish activation function

    • preluBlock

      public static Block preluBlock()
      Returns a Prelu block.
      Returns:
      a Prelu block