COMP9444代写、Python编程语言代写

日期：2023-06-29 09:32

COMP9444 Neural Networks and Deep Learning

Term 2, 2023

Project 1 - Characters and Hidden Unit

Dynamics

Due: Wednesday 5 July, 23:59 pm

Marks: 20% of final assessment

In this assignment, you will be implementing and training

various neural network models for three different tasks,

and analysing the results.

You are to submit two Python files kuzu.py and

encoder.py, as well as a written report hw1.pdf (in pdf

format).

Provided Files

Copy the archive hw1.zip into your own filespace and

unzip it. This should create a directory hw1, subdirectory

net, and ten Python files kuzu.py, encoder.py,

kuzu_main.py, encoder_main.py, encoder_model.py,

seq_train.py, seq_models.py, seq_plot.py, reber.py and

anbn.py.

Your task is to complete the skeleton files kuzu.py and

encoder.py and submit them, along with your report.

Part 1: Japanese Character Recognition

For Part 1 of the assignment you will be implementing

networks to recognize handwritten Hiragana symbols. The

dataset to be used is Kuzushiji-MNIST or KMNIST for

short. The paper describing the dataset is available here.

It is worth reading, but in short: significant changes

occurred to the language when Japan reformed their

education system in 1868, and the majority of Japanese

today cannot read texts published over 150 years ago.

This paper presents a dataset of handwritten, labeled

examples of this old-style script (Kuzushiji). Along with

this dataset, however, they also provide a much simpler

one, containing 10 Hiragana characters with 7000

samples per class. This is the dataset we will be using.

Text from 1772 (left) compared to 1900 showing the standardization of written

Japanese.

1. [1 mark] Implement a model NetLin which computes

a linear function of the pixels in the image, followed

by log softmax. Run the code by typing:

python3 kuzu_main.py --net lin

Copy the final accuracy and confusion matrix into

your report. The final accuracy should be around

70%. Note that the rows of the confusion matrix

indicate the target character, while the columns

indicate the one chosen by the network. (0="o",

1="ki", 2="su", 3="tsu", 4="na", 5="ha", 6="ma",

7="ya", 8="re", 9="wo"). More examples of each

character can be found here.

2. [1 mark] Implement a fully connected 2-layer network

NetFull (i.e. one hidden layer, plus the output layer),

using tanh at the hidden nodes and log softmax at

the output node. Run the code by typing:

python3 kuzu_main.py --net full

Try different values (multiples of 10) for the number

of hidden nodes and try to determine a value that

achieves high accuracy (at least 84%) on the test

set. Copy the final accuracy and confusion matrix

into your report, and include a calculation of the total

number of independent parameters in the network.

3. [2 marks] Implement a convolutional network called

NetConv, with two convolutional layers plus one fully

connected layer, all using relu activation function,

followed by the output layer, using log softmax. You

are free to choose for yourself the number and size of

the filters, metaparameter values (learning rate and

momentum), and whether to use max pooling or a

fully convolutional architecture. Run the code by

typing:

python3 kuzu_main.py --net conv

Your network should consistently achieve at least

93% accuracy on the test set after 10 training

epochs. Copy the final accuracy and confusion

matrix into your report, and include a calculation of

the total number of independent parameters in the

network.

4. [4 marks] Briefly discuss the following points:

a. the relative accuracy of the three models,

b. the number of independent parameters in each

of the three models,

c. the confusion matrix for each model: which

characters are most likely to be mistaken for

which other characters, and why?

Part 2: Encoder Networks

In Part 2 you will be editing the file encoder.py to create a

dataset which, when run in combination with

encoder_main.py, produces the following image (which is

intended to be a stylized map of mainland China).

You should first run the code by typing

python3 encoder_main.py --target star16

Note that target is determined by the tensor star16 in

encoder.py, which has 16 rows and 8 columns, indicating

that there are 16 inputs and 8 outputs. The inputs use a

one-hot encoding and are generated in the form of an

identity matrix using torch.eye()

1. [2 marks] Create by hand a dataset in the form of a

tensor called ch34 in the file encoder.py which, when

run with the following command, will produce an

image essentially the same as the one shown above

(but possibly rotated or reflected).

python3 encoder_main.py --target ch34

The pattern of dots and lines must be topologically

identical. But, it is fine for it to be rotated or reflected,

compared to the image above. Note in particular the

five "anchor points" in the corners and on the edge

of the figure.

Your tensor should have 34 rows and 23 columns.

Include the final image in your report, and include the

tensor ch34 in your file encoder.py

Part 3: Hidden Unit Dynamics for Recurrent

Networks

In Part 3 you will be investigating the hidden unit

dynamics of recurrent networks trained on language

prediction tasks, using the supplied code seq_train.py

and seq_plot.py.

1. [2 marks] Train a Simple Recurrent Network (SRN) on

the Reber Grammar prediction task by typing

python3 seq_train.py --lang reber

This SRN has 7 inputs, 2 hidden units and 7 outputs.

The trained networks are stored every 10000 epochs,

in the net subdirectory. After the training finishes,

plot the hidden unit activations at epoch 50000 by

typing

python3 seq_plot.py --lang reber --epoch 50

The dots should be arranged in discernable clusters

by color. If they are not, run the code again until the

training is successful. The hidden unit activations are

printed according to their "state", using the colormap

"jet":

Based on this colormap, annotate your figure (either

electronically, or with a pen on a printout) by drawing

a circle around the cluster of points corresponding to

each state in the state machine, and drawing arrows

between the states, with each arrow labeled with its

corresponding symbol. Include the annotated figure

in your report.

2. [1 mark] Train an SRN on the anbn language

prediction task by typing

python3 seq_train.py --lang anbn

The anbn language is a concatenation of a random

number of A's followed by an equal number of B's.

The SRN has 2 inputs, 2 hidden units and 2 outputs.

Look at the predicted probabilities of A and B as the

training progresses. The first B in each sequence and

all A's after the first A are not deterministic and can

only be predicted in a probabilistic sense. But, if the

training is successful, all other symbols should be

correctly predicted. In particular, the network should

predict the last B in each sequence as well as the

subsequent A. The error should be consistently

below 0.01. If the network appears to have learned

the task successfully, you can stop it at any time

using ?cntrl?-c. If it appears to be stuck in a local

minimum, you can stop it and run the code again until

it is successful.

After the training finishes, plot the hidden unit

activations by typing

python3 seq_plot.py --lang anbn --epoch 100

Include the resulting figure in your report. The states

are again printed according to the colormap "jet".

Note, however, that these "states" are not unique but

are instead used to count either the number of A's we

have seen or the number of B's we are still expecting

to see.

3. [1 mark] Briefly explain how the anbn prediction task

is achieved by the network, based on the figure you

generated in Question 2. Specifically, you should

describe how the hidden unit activations change as

the string is processed, and how it is able to correctly

predict the last B in each sequence as well as the

following A.

4. [1 mark] Train an SRN on the anbncn language

prediction task by typing

python3 seq_train.py --lang anbncn

The SRN now has 3 inputs, 3 hidden units and 3

outputs. Again, the "state" is used to count up the A's

and count down the B's and C's. Continue training

(re-starting, if necessary) for 200k epochs, or until

the network is able to reliably predict all the C's as

well as the subsequent A, and the error is

consistently in the range of 0.01 or 0.02.

After the training finishes, plot the hidden unit

activations by typing

Rotate the figure in 3 dimensions to get one or more

good view(s) of the points in hidden unit space.

5. [1 mark] Briefly explain how the anbncn prediction

task is achieved by the network, based on the figure

you generated in Question 4. Specifically, you should

describe how the hidden unit activations change as

the string is processed, and how it is able to correctly

predict the last B in each sequence as well as all of

the C's and the following A.

6. [4 marks] This question is intended to be more

challenging. Train an LSTM network to predict the

Embedded Reber Grammar, by typing

You can adjust the number of hidden nodes if you

wish. Once the training is successful, try to analyse

the behavior of the LSTM and explain how the task is

accomplished (this might involve modifying the code

so that it returns and prints out the context units as

well as the hidden units).

Submission

You should submit by typing

python3 seq_plot.py --lang anbncn --epoch 200

python3 seq_train.py --lang reber --embed True --model lstm --hid 4

give cs9444 hw1 kuzu.py encoder.py hw1.pdf

You can submit as many times as you like - later

submissions will overwrite earlier ones. You can check

that your submission has been received by using the

following command:

9444 classrun -check hw1

The submission deadline is Wednesday 5 July, 23:59pm.

In accordance with new UNSW-wide policies, 5% penalty

will be applied for every 24 hours late after the deadline,

up to a maximum of 5 days, after which submissions will

not be accepted.

Additional information may be found in the FAQ and will be

considered as part of the specification for the project. You

should check this page regularly.

Plagiarism Policy

Group submissions will not be allowed for this

assignment. Your code and report must be entirely your

own work. Plagiarism detection software will be used to

compare all submissions pairwise (including submissions

for similar assignments from previous offering, if

appropriate) and serious penalties will be applied,

particularly in the case of repeat offences.

DO NOT COPY FROM OTHERS; DO NOT ALLOW

ANYONE TO SEE YOUR CODE

Please refer to the UNSW Policy on Academic Integrity

and Plagiarism if you require further clarification on this

matter.

Good luck!