UNITS

Units are one of the basic building blocks in a neural network. Every unit has, at each point in time, an activation value. In the NICO toolkit the activation is a real number. Some units, the input units, have their activation determined by external data (see the Streams section) or by the environment (see below). The activations of all other units except mutiplication units are computed by the following formula:

       ___
       \
net  = /    w   a  ;  a = f( net )
   i   ---   ji  j     i        i
        j

Where a_j is the activation unit j and w_ji is the connection weight from unit j to unit i. The function f is different for different types of units. The NICO toolkit supports several different unit types:

Linear units -- the identity function (f(x)=x)
Tanhyp units -- f(x) = tanhyp(x)
Arctan units -- f(x) = arctan(x)
Sigmoid units -- f(x) = 1 / (1 - exp(-x))
Exponential units f(x) = exp(x)
Inverter units f(x) = 1/x

Multiplication units compute their activation value by:
__
a = || a
i j
i.e., independent of the connection weights.

This is the syntax for the function that creates units:

USAGE: AddUnit [options] Name Net
       Option                                                      Default
       -u number Add multiple unnamed units. 'Name' is the parent  (off)
       -S File   Add multiple named units, 'Name' is the parent    (off)
       -i        Add input unit                                    (hidden)
       -o        Add output unit                                   (hidden)
       -s        Add tanhyp unit                                   (default)
       -t        Add arctan unit                                   (tanhyp)
       -l        Add linear unit                                   (tanhyp)
       -f        Add filefilter unit                               (tanhyp)
       -e        Add environment unit                              (tanhyp)
       -r float  Bound on random init of bias                      (0.10)

Layers of units

The most common use of AddUnit is to add a set of units as members of a group. This is done with the -u option. For example:

AddUnit -u17 my_group my_net.rtdnn

adds 17 unnamed units to the group "my_group" in the network "my_net.rtdnn".

Units optionally have names in the NICO toolkit. This is useful if a unit have some special function in the network. Then this individual unit can be referred to in, for example, connection commands, or it can be individually selected for output in the Excite command. The following command adds one unit named "Charlie" to the network:

AddUnit Charlie my_net.rtdnn

Special purpose unit types

The activity of an environment unit depends on the system environment variable with the name of the unit. For example, the command:

AddUnit -e MYSWITCH my_net.rtdnn

adds an environment unit that is dependent on the environment variable MYSWITCH. If the value of the environment variable is numerical, it is copied to the unit's activation. Otherwize the activation is 1.0 for defined variables and 0.0 for undefined variables.

A filefilter unit gets the activation 1.0 if the unit's name is a substring of the base name of the current external data file. Oterwize the unit's activation is 0.0. This can be useful if the datafiles have names that indicates special properties of the data. In speech applications for example, the speakers identity is often coded by a few letters or a number in the filename.

All networks have a special-purpose unit with name bias. The bias unit always have activation 1.0. and units are by default connected to this unit when they are created (except input, filefilter and environment units). The -r option controls this connection. The connection strength is a square distributed random variable. The option: -rX, specifies the range of this varible to [-X; +X]. If the option is given with a zero argument (-r0.0) the connetion is not created at all.

Other primitive unit types

Most unit types can be selected by the switches of the AddUnit command. However, a few types can only be selected by the SetType command. The SetType command can also be used to change the error function of output units. The syntax is:

USAGE: SetType [option] Unit Net
       Option                                                        Default
       -o      Change to an output unit                              (off)
       -n      Change to a non-output unit                           (off)
       -i      Change to an input unit                               (off)
       -O efun Change to an output unit with specified error function
               efun = ('0/1' L2, L4, L10, 'abs', 'cross0/1'
               'interact', 'cross' or 'noerr'                        (off)
       -t      Change to a tanhyp unit                               (off)
       -a      Change to an arctan unit                              (off)
       -s      Change to a sigmoid unit                              (off)
       -l      Change to a linear unit                               (off)
       -m      Change to a multiplication unit                       (off)
       -d      Change to a inverter (1/x) unit                       (off)
       -x      Change to a exponential e(x) unit                     (off)
       -e      Change to a environment unit                          (off)
       -f      Change to a filefilter unit                           (off)

Complex unit types

By combining the primitive unit types supported by the tool-kit, many other complex units can be created. For example, a unit that is the square of another unit's activation can easily be constructed. Let's say we have a unit called "x" in the network "my_net.rtdnn". The following commands creates a new unit "x2" that is the square of "x".

AddUnit -r0 -l tmp my_net.rtdnn

Connect -w1.0 x tmp my_net.rtdnn

AddUnit -r0 x2 my_net.rtdnn

SetType -m x2 my_net.rtdnn

Connect -w1.0 x x2 my_net.rtdnn

Connect -w1.0 tmp x2 my_net.rtdnn

In the same spirit, we can construct a group of units with the softmax activation function (this example requires understanding of groups, expalined in the groups section ). Let's say we need a group of five softmax units called softmax. The following commands creates two groups, the first, called input, has five exponential units that should be connected to, and the second group, called output, has five multiplication units that have the softmax activation values.

AddGroup input my_net.rtdnn

AddUnit -u5 input my_net.rtdnn

SetType -x input my_net.rtdnn

AddGroup output my_net.rtdnn

AddUnit -u5 output my_net.rtdnn

SetType -m output my_net.rtdnn

AddUnit softsum my_net.rtdnn

SetType -d softsum my_net.rtdnn

Connect -w1.0 input softsum my_net.rtdnn

Connect -w1.0 softsum output my_net.rtdnn

Pipe -w1.0 input output my_net.rtdnn

The unit "softsum" computes the sum of all units in the input group and inverts (1/x) this sum. The multiplication units in the group "output" then normalizes the activations by multiplying with the activity of "softsum". Using concepts explained in the groups section we finish our softmax example by encapsulating it in a supergroup called softmax.

AddGroup softmax my_net.rtdnn

AddGroup internal my_net.rtdnn

Move internal network softmax my_net.rtdnn

Move input network softmax my_net.rtdnn

Move output network softmax my_net.rtdnn

Move softsum network softmax.internal my_net.rtdnn

Protect -P softmax.internal my_net.rtdnn

Protect -R softmax.input my_net.rtdnn

Protect -S softmax.output my_net.rtdnn

Now the group "softmax" can be treated as a group with one single layer of softmax units.