SOME TOY EXAMPLES

Scripts and data for the toy examples can be found in the directory toy_examples of the NICO distribution.

The XOR problem

This classical problem serves as an introduction to the basic commands of the toolkit. Don't expect a particularly efficient solution, the NICO toolkit is not really meant to be used for this type of simple problems.

Classifying sinusoids

This example shows how dynamic ANNs can learn to classify input sequences. I created this example after a discussion on the newsgroup comp.ai.neural-nets, where the issue was whether this type of problem could be solved using artificial neural networks.

The figure 8

This small example serves as a demonstration of how dynamic ANNs can learn to output complex dynamic output patterns with only a short trigger signal on the input.

The XOR problem.

This is known to be one of the simplest problems that are not linearly separable, i.e., need at least one hidden unit. The files xor.input and xor.output printed below defines the problem.

Create a directory for this little experiment and create also the two files defining the input and output.

File: xor.input

1 0
1 1
0 1
0 0

File: xor.output

1
0
1
0

Now type in (or copy and paste it from here) the script below. It builds an ANN, trains it and evaluates it. The network has one hidden unit and the output unit is connected both to the hidden unit and the two input units. At the end of the script, the values of the output stream is printed for the four patterns. If the network solved the problem, the values should be close to 1, 0, 1, 0. If one runs this script repeated times, it is seen that the training does not always converge. I found that this happens about once every three or four times.

#!/bin/sh -v
CreateNet xor xor.rtdnn
AddStream -x input -d . -F ascii 2 r INPUT xor.rtdnn
AddGroup input xor.rtdnn
AddUnit -i -u 2 input xor.rtdnn
LinkGroup INPUT input xor.rtdnn
AddStream -x output -d . -F ascii 1 t OUTPUT xor.rtdnn
AddGroup output xor.rtdnn
AddUnit -o -u 1 output xor.rtdnn
LinkGroup OUTPUT output xor.rtdnn
AddGroup hidden xor.rtdnn
AddUnit -u 1 hidden xor.rtdnn
Connect input hidden xor.rtdnn
Connect hidden output xor.rtdnn
Connect input output xor.rtdnn
Display xor.rtdnn
BackProp -E -g0.1 -i1000 -P xor.log 100 100 xor.rtdnn xor
Excite -X OUTPUT xor.rtdnn xor
cat xor.log
cat xor.act

Plot of the four sinusoids to classify in this example.

Classifying sinusoids

Here is a simple example of how dynamic networks can learn to classify sequences. The input to the network is a sine wave, and the task is to discriminate between four distinct frequencies. Four training files with different frequencies of sampled sinusoid signals are shown below. Create a directory for this experiment and create two sub directories, data and target. Cut and paste the data files below to these directories. You also need a file listing the datafiles. Name it trainfiles, and copy it from the box below. Then create the network using the script below, and train it using back-propagation through time, with the following command:
BackProp -S -p net.log -g1e-6 -F20 30 -T2 -i5000 -d net.rtdnn trainfiles
Depending on how fast your computer is, this may take some time... You can see if the network converges by inspecting the log file, net.log.
One more thing: you can sometimes get the a warning about "Large Unit Gradient" when runnning BackProp. This means that the network is getting "overtrained".
Hopefully, the network has now converged. You can then excite the network by running the command:
Excite -X TARG net.rtdnn x3.data
and inspecting the created file x3.act. The third column of activities in this file should be close to +1.0, and all other columns should be close to 0.0. Check the other data files, x1, x2, and x4, too if you like.

#!/bin/sh -v

FEATURESIZE=25

NET=net.rtdnn

CreateNet $NET $NET

AddStream -x data -d ./data -F ascii 1 r DATA $NET

AddGroup data $NET
AddUnit -i -u 1 data $NET
LinkGroup DATA data $NET

AddStream -x targ -d ./targ -F ascii 4 t TARG $NET

AddGroup targ $NET
AddUnit -o -u 4 targ $NET
LinkGroup TARG targ $NET

AddGroup features $NET
AddUnit -u $FEATURESIZE features $NET

Connect -D -3 3 data features $NET
Connect -D 1 5 features features $NET
Connect -D -1 1 features targ $NET

File: trainfiles

x1.data
x2.data
x3.data
x4.data

File: data/x1.data

0.0000

0.8415

0.9093

0.1411

-0.7568

-0.9589

-0.2794

0.6570

0.9894

0.4121

-0.5440

-1.0000

-0.5366

0.4202

0.9906

0.6503

-0.2879

-0.9614

-0.7510

0.1499

0.9129

0.8367

-0.0089

-0.8462

-0.9056

-0.1324

0.7626

0.9564

0.2709

-0.6636

-0.9880

-0.4040

0.5514

0.9999

0.5291

-0.4282

-0.9918

-0.6435

0.2964

0.9638

0.7451

-0.1586

-0.9165

-0.8318

0.0177

0.8509

0.9018

0.1236

-0.7683

-0.9538

-0.2624

0.6702

0.9866

0.3959

-0.5588

-0.9998

-0.5216

0.4362

0.9929

0.6367

-0.3048

-0.9661

-0.7392

0.1674

0.9200

0.8268

-0.0266

-0.8555

-0.8979

-0.1148

0.7739

0.9511

0.2538

-0.6768

-0.9851

-0.3878

0.5661

0.9995

0.5140

-0.4441

-0.9939

-0.6299

0.3132

0.9684

0.7332

-0.1761

-0.9235

-0.8218

0.0354

0.8601

0.8940

0.1060

-0.7795

-0.9483

-0.2453

0.6833

0.9836

0.3796

-0.5734

-0.9992

-0.5064

File: data/x2.data

0.0000

0.6442

0.9854

0.8632

0.3350

-0.3508

-0.8716

-0.9825

-0.6313

0.0168

0.6570

0.9882

0.8546

0.3191

-0.3665

-0.8797

-0.9792

-0.6181

0.0336

0.6696

0.9906

0.8457

0.3031

-0.3821

-0.8876

-0.9756

-0.6048

0.0504

0.6820

0.9928

0.8367

0.2871

-0.3976

-0.8952

-0.9718

-0.5914

0.0672

0.6942

0.9946

0.8273

0.2709

-0.4129

-0.9026

-0.9677

-0.5777

0.0840

0.7062

0.9962

0.8178

0.2547

-0.4282

-0.9097

-0.9633

-0.5639

0.1007

0.7180

0.9976

0.8080

0.2384

-0.4433

-0.9165

-0.9587

-0.5499

0.1174

0.7296

0.9986

0.7980

0.2220

-0.4583

-0.9231

-0.9538

-0.5358

0.1341

0.7410

0.9993

0.7877

0.2056

-0.4732

-0.9295

-0.9486

-0.5216

0.1508

0.7522

0.9998

0.7772

0.1891

-0.4879

-0.9355

-0.9431

-0.5071

0.1674

0.7631

1.0000

0.7665

0.1726

-0.5026

-0.9413

-0.9374

-0.4926

0.1839

0.7739

File: data/x1.data

0.0000

0.4794

0.8415

0.9975

0.9093

0.5985

0.1411

-0.3508

-0.7568

-0.9775

-0.9589

-0.7055

-0.2794

0.2151

0.6570

0.9380

0.9894

0.7985

0.4121

-0.0752

-0.5440

-0.8797

-1.0000

-0.8755

-0.5366

-0.0663

0.4202

0.8038

0.9906

0.9349

0.6503

0.2065

-0.2879

-0.7118

-0.9614

-0.9756

-0.7510

-0.3425

0.1499

0.6055

0.9129

0.9968

0.8367

0.4716

-0.0089

-0.4872

-0.8462

-0.9981

-0.9056

-0.5914

-0.1324

0.3591

0.7626

0.9794

0.9564

0.6992

0.2709

-0.2238

-0.6636

-0.9410

-0.9880

-0.7931

-0.4040

0.0840

0.5514

0.8839

0.9999

0.8711

0.5291

0.0575

-0.4282

-0.8090

-0.9918

-0.9317

-0.6435

-0.1978

0.2964

0.7180

0.9638

0.9736

0.7451

0.3342

-0.1586

-0.6126

-0.9165

-0.9961

-0.8318

-0.4638

0.0177

0.4949

0.8509

0.9986

0.9018

0.5842

0.1236

-0.3673

-0.7683

-0.9811

-0.9538

-0.6929

-0.2624

File: data/x4.data

0.0000

0.2955

0.5646

0.7833

0.9320

0.9975

0.9738

0.8632

0.6755

0.4274

0.1411

-0.1577

-0.4425

-0.6878

-0.8716

-0.9775

-0.9962

-0.9258

-0.7728

-0.5507

-0.2794

0.0168

0.3115

0.5784

0.7937

0.9380

0.9985

0.9699

0.8546

0.6630

0.4121

0.1245

-0.1743

-0.4575

-0.6999

-0.8797

-0.9809

-0.9946

-0.9193

-0.7620

-0.5366

-0.2632

0.0336

0.3275

0.5921

0.8038

0.9437

0.9993

0.9657

0.8457

0.6503

0.3967

0.1078

-0.1909

-0.4724

-0.7118

-0.8876

-0.9841

-0.9927

-0.9126

-0.7510

-0.5223

-0.2470

0.0504

0.3433

0.6055

0.8137

0.9491

0.9998

0.9612

0.8367

0.6374

0.3813

0.0910

-0.2073

-0.4872

-0.7235

-0.8952

-0.9869

-0.9905

-0.9056

-0.7398

-0.5079

-0.2306

0.0672

0.3591

0.6188

0.8233

0.9543

1.0000

0.9564

0.8273

0.6244

0.3657

0.0743

-0.2238

-0.5018

-0.7350

-0.9026

-0.9895

-0.9880

File: targ/x1.targ

1 0 0 0

File: targ/x2.targ

0 1 0 0

File: targ/x3.targ

0 0 1 0

File: targ/x4.targ

0 0 0 1

In the figure 8 example, we see how the network can be taught to draw this figure using its two output units.

The figure 8

This example shows how the output of a dynamic ANN can have temporal structure. The input to the network is simply a trigger signal that is zero most of the time, but at one point in time it is one for one time-frame. After this trigger signal, the network is requried to draw the figure 8 using its two output units: x and y.
The following two data files define the problem:

File: fig8.trig

0.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

File: fig8.pos

0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
-0.25    0.50
-0.50    1.00
-0.75    1.00
-1.00    0.50
-1.00    0.00
-1.00   -0.50
-0.75   -1.00
-0.50   -1.00
-0.25   -0.50
0.00    0.00
0.25    0.50
0.50    0.50
0.75    0.00
0.50   -0.50
0.25   -0.50
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00
0.00    0.00

Create the two data files in a new directory, and run the following script:

#!/bin/sh -v

HIDDENSIZE=10

NET=net.rtdnn

CreateNet $NET $NET

AddStream -x trig -d. -F ascii 1 r TRIG $NET

AddGroup trig $NET
AddUnit -i -u 1 trig $NET
LinkGroup TRIG trig $NET

AddStream -x pos -d. -F ascii 2 t POS $NET

AddGroup pos $NET
AddUnit -o -u 2 pos $NET
Rename pos#1 x $NET
Rename pos#2 y $NET
LinkGroup POS pos $NET
NormStream -c 0 1 -s POS $NET

AddGroup hidden $NET
AddUnit -u $HIDDENSIZE hidden $NET

Connect trig hidden $NET
Connect -D 1 5 hidden hidden $NET
Connect hidden pos $NET

BackProp -p net.log -g1e-2 -E -i400 -d net.rtdnn fig8.trig

The script both creates a network and trains it using the BackProp command. There are recurrent connections in the hidden layer with up to five frames time-delay, but the figure-8 cycle is more than three times longer than that, so the network has to learn to use the activities of the hidden units to keep track of where in the cycle it is. Note that after 10 frames of the cycle, we're back in the origin, and at that point, the network has to remember to do the upper loop instead of simply stopping.

Check in the logfile if the training converged. If the global error is not less than 1.0e-02, try it again (I have run the script several times and it almost always converged).
Now you can check what the network actually outputs, by running:
Excite -X POS net.rtdnn fig8.trig
This generates the output file fig8.act that should be very similar to the target fig8.pos. Type:
cat fig8.act
to convince yourself.

SOME TOY EXAMPLES

Contents

The XOR problem

Classifying sinusoids

The figure 8

The XOR problem.

Classifying sinusoids

The figure 8