Package ai.djl.training.initializer

Class XavierInitializer

java.lang.Object

ai.djl.training.initializer.XavierInitializer

All Implemented Interfaces:

Initializer

public class XavierInitializer
extends Object
implements Initializer

XavierInitializer is an Initializer that performs "Xavier" initialization for parameters. This initializer is designed to keep the scale of gradients roughly the same in all layers. It was originally defined in the paper Understanding the difficulty of training deep feedforward neural networks.

XavierInitializer is specified by the type of random distribution(XavierInitializer.RandomType), the factor type(XavierInitializer.FactorType), and the magnitude of the scale. By default, XavierInitializer.RandomType is UNIFORM and XavierInitializer.FactorType is AVG. The initializer fills the weights with random numbers in the range of \([-c, c]\), where \(c = \sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}\) where \(n_{in}\) is the number of neurons feeding into weights, and \(n_{out}\) is the number of neurons the result is fed to.

If XavierInitializer.RandomType is UNIFORM and XavierInitializer.FactorType is IN, then \(c = \sqrt{\frac{3.}{n_{in}}}\). Similarly when XavierInitializer.FactorType is OUT, then \(c = \sqrt{\frac{3.}{n_{out}}}\).

If XavierInitializer.RandomType is GAUSSIAN and XavierInitializer.FactorType is AVG, the initializer fills the weights with numbers from normal distribution with a standard deviation of \(\sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}\).

Another common setting of the XavierInitializer is defined in the paper Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. These settings better handle non-linearity when preserving the variance across layers in a neural network. It can be initialized with new XavierInitializer(RandomType.GAUSSIAN, FactorType.IN, 2)).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	XavierInitializer.FactorType	Enum for different types of factor type.
static enum	XavierInitializer.RandomType	Enum for different types of random distributions.

Field Summary

Fields inherited from interface ai.djl.training.initializer.Initializer

ONES, ZEROS

Constructor Summary

Constructors

Constructor	Description
XavierInitializer()	Creates a new instance of XavierInitializer.
XavierInitializer(XavierInitializer.RandomType randomType, XavierInitializer.FactorType factorType, float magnitude)	Initializes a Xavier initializer.

Method Summary

Modifier and Type	Method	Description
NDArray	initialize(NDManager manager, Shape shape, DataType dataType)	Initializes a single NDArray.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

XavierInitializer

public XavierInitializer(XavierInitializer.RandomType randomType,
 XavierInitializer.FactorType factorType,
 float magnitude)

Initializes a Xavier initializer.

Parameters:

randomType - the random generator type, can be GAUSSIAN or UNIFORM

factorType - the factor type, can be one of AVG, IN, or OUT

magnitude - the scale of the random number

XavierInitializer

public XavierInitializer()

Creates a new instance of XavierInitializer.

Method Details

initialize

public NDArray initialize(NDManager manager,
 Shape shape,
 DataType dataType)

Initializes a single NDArray.

Specified by:

initialize in interface Initializer

Parameters:

manager - the NDManager to create the new NDArray in

shape - the Shape for the new NDArray

dataType - the DataType for the new NDArray

Returns:

the NDArray initialized with the manager and shape