Variable-length input to an ANN || kuro5hin.org


	create account \| help/FAQ \| contact \| links \| search \| IRC \| site news

Everything

We need your support: buy an ad | premium membership

Variable-length input to an ANN

By trane in trane's Diary
Fri Jan 07, 2005 at 03:20:40 PM EST
Tags: User Diary (all tags)

I'm taking a class in Computational Intelligence this quarter. There's a final project. I think I'm going to try to duplicate some of the syntactical functions of my bot with a neural net.

ADVERTISEMENT
Sponsor: rusty

This space intentionally left blank

...because it's waiting for your ad. So why are you still reading this? Come on, get going. Read the story, and then get an ad. Alright stop it. I'm not going to say anything else. Now you're just being silly. STOP LOOKING AT ME! I'm done!

comments (24)

active | buy ad
ADVERTISEMENT

So, for example, with the code i have so far, I can take a simple S-V-O type sentence and put it into a Case Grammar representation:
Jack ate an apple ->
Verb = eat
Agent = Jack
Patient = an apple
Time = past
(I don't have time implemented yet, but you get the idea.)
Can I duplicate this behavior with an artificial neural network?
One immediate problem that comes to mind is, how do you handle variable-length input to an ANN? A network has a fixed number of input neurons, but sentences can vary in length.
One solution is to set a limit on length of input and pad with blanks. But this seems unsatisfying.
Another solution is to use a recurrent network, where you submit one word at a time, along with the saved hidden layer from the previous input. This page has a more complete description.
However, reading further in the CLASPnet paper, I see that each word he's submitting to the network has a 12-letter limit; if the word has fewer than 12 letters, he pads it with spaces (he's doing some strange duplicationg stuff as well, but still there are spaces for many shorter words). So the problem of variable input is not really solved. Unless you submit only one character at a time to the network, and let it build up its own representation of words? That seems to involve a lot of extra processing...
Anyone got any ideas on how to submit variable-length input to a neural netword?

Sponsors

Managed Hosting
VoxCAST Content Delivery
Raw Infrastructure

Variable-length input to an ANN | 2 comments (2 topical, editorial, 0 hidden)

this is a classic ANN problem (none / 0) (#1)
by Delirium on Fri Jan 07, 2005 at 03:28:19 PM EST

One approach is to "compress" syntax trees to a fixed representation, and then "uncompress" them at the end. This obviously must lose information somewhere, and a sufficiently opaque compression would result in compressed representations that the neural network couldn't operate on, but there've been a few surprisingly good results. There's a brief overview here of the Recursive Auto-Associative Memory (RAAM) approach.

Recurrence, but... (none / 0) (#2)
by Entendre Entendre on Sat Jan 08, 2005 at 04:21:26 AM EST

But what I wanna know is how you plan to represent the words themselves.

I wrote backprop net years ago that learned some music - the input and output layers had one node per note; the input layer used recurrence to 'remember' the notes that had been played previously; the highest value in the output layer determined/predicted what note to play next. (I poached the idea from an old issue of Computer Music Journal.)

The simplest application of this approach to your problem would be for the input and output layers to have one node per word. You feed each new word by setting all of the inputs to zero except the current word, then propogate the activations up to the output layer and find out what the next word is. Then do the recurrence thing on the input layer (multiply every node's activation by .5 or so). Then repeat.

That's going to be a pretty big net if you want an interesting vocabulary, but hey, RAM is cheap these days. Try it.

--
Reduce firearm violence: aim carefully.

Variable-length input to an ANN | 2 comments (2 topical, 0 editorial, 0 hidden)

All trademarks and copyrights on this page are owned by their respective companies. The Rest © 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.