So, what’s new:

• android version
• manage the android device from within the application
• updated asset editor
• a new chat-log history view (view/tool windows/chatlog history)
• some internal fixes
• improved voice input (new setting available for this in the ‘chatbot properties’ window)
• a new import of wordnet has been made (+ the importer updated)
• thesaurus search has been improved.
• thesaurus import has been improved
• thesaurus paths (in input patterns) have an extra pos  filter type: ‘any’, for allowing any part of speech.
• thesaurus paths (in input and output patterns) have an extra optional part for selecting the relationship type (ex: ^w->pertainym:any.something)
• Rule layout has been changed: the ‘Response for’ field has been moved from the ‘output’ items to the root of the rule
• the ‘questions-answers’ demo has been updated.
• InvalidRespones: the list can now be assigned sequential or random (for selecting an item)
• InvalidResponse: a single invalid response can now determine if the bot needs to continue waiting for a valid response or not.
• thesaurus variables in input patterns
• the layout of topic editors has been updated (use the first toolbar button to toggle between editing modes)
• generic data import has been added which uses topics to declare how data needs to be imported.
• changed the labeling from ‘bot says’ to ‘output’ and ‘you say’ to ‘input’ to better reflect all the usage cases and not just for defining chat statements.
• websites can now be installed to any server you like (path can be defined) (servers must support .net 4, MVC3)
• the css file is properly used in the websites (which allows customization)

Known issues on the android:

• contact info needs to be partially imported at the moment, hoping to change this in the future.
• the contact info sync screen closes extremely slow when the process is done (most likely due to  garbage collection).
• the first input is processed much slower then the rest. This is due to disk access latencies.
• Accessing the disk is generally very slow so better caching needs to be added.
• The contact info search algorithms are very limited at the moment, this will be fixed

Coming up: lots of things are currently ‘in the docks’, sort of:

• My own text-to-speech engine is half way. When done, this should allow me to provide some proper lip-sync on android and web + a path for future improvements. So far, I’ve only been working on this now and again. Soon it will get my full attention and things should start to move in this area.
• I have finally made a proper programming language to go with the neural network engine. In the past, I programmed it directly by laying out the neurons manually. This was starting to become a problem: upgrading big projects (like those that contain a wordnet db) is faster if only the code can be replaced instead of all the data. I also needed some more power in the pattern matcher statements + an easy way to extend the functionality. And finally, I would like to recover some old code I wrote for parsing images. Hence this side step. Both parser and source code generator have already been made, only the neuron renderer still needs some work.
• I’d like to add a new view to the designer for better managing conversation flow
• I’ve starting doing testing with my own STT. This is definitely a long term project.

input: define ^value:any
calculate: $result = $value[0]:wordnetmeaning
Output: ‘$value’ means $result

Pretty simple.  ‘$value[0]’ is used to select the first occurrence in the thesaurus.

And here’s the bot itself:

I thought this was an interesting setup, so I created a small test bot that tries to find a word in the word-list that is somehow related (in wordnet) to 1 or more words in the question as a fallback when it can’t find the correct response. Besides this rule, it currently has 1 other rule to deal with some simple date-time questions. I’ll most likely add some more in the future, for testing things through. There are still some issues, but results can sometimes be interesting, surprising and bizarre.

Try it out for yourself here:

In the mean time though, I think I’ll prepare a simple release, which will run pretty slow.

• The spell checker has been cleaned up some. The custom dictionary is now shared across the entire application.
• There was still a bug in the core split functionality that sometimes resulted in incorrect weights for an object when the same object was the result of multiple paths, but with different weights.
• Asset editors now have a child topic-editor by default in the project view.
• Some cleanups in the import/export functionality of the thesaurus and assets.
• a bug in the handling of variables in patterns (which resulted in invalid value refs sometimes) is also fixed.
• a new tutorial that shows how to do questions, answers and redirections.

Get the latest release from the download page.

Retrieving, or scraping info from the internet can be done remarkably easy with the chatbot designer. Here’s a screencast of a bot that retrieves weather information from the google weather api.

In the video, a .net plug-in is used to retrieve information from the internet by means of XPaths. This plugin is included by default in the application. Note though that plug-ins are only supported in the pro version.  Basic users will be able to use these projects, but they can’t create or edit any patterns that rely on plug-ins. Also, plug-ins are loaded on a project by project basis. So if you want to use the scraping features in your own project, you will first need to make certain that the correct .net functions have been loaded. Once this has been set up though, all plug-ins will automatically be loaded when the project is opened.

Loading

To load a plug-in, go to view/communication channels/OS. This will bring up a view like the one on the right. From here, you can load and unload dlls, classes and functions. First up is the dll. This can be loaded with one of the buttons on the toolbar. The first one gives access to the cache (dlls that have already been loaded). With the next button, you can select a file from disk. Note that, even though the ‘CmdShell.dll’ file (which contains the scraping functions) is part of the installation, it isn’t guaranteed that it’s already loaded in cache, so you might have to select it from the ‘program files/Chatbot designer pro/’ path. By the way, you can remove a dll by selecting it and pressing delete. Functions can be selected/deselected with the checkbox in front of the name. You can alternatively (de)select the entire class or lib at once. Notice the blue label behind each function name: this is the name that you can use in the patterns. You see, the do-patterns evaluator has no knowledge whatsoever of namespaces, classes or functions, it just knows a single name. This means that all function names should be unique across a single project. If you try to enter a duplicate name, a red box will be displayed round the newly mapped name.

There are quite a few functions available for scraping. Basically though, there are 3 groups: some functions to open/close web-pages, some functions to get data from those opened pages and finally the same functions that don’t require you to first open/close any files but which can do a scrape directly.

Short scrapes

Depending on how much data you need to retrieve, you can use one or the other technique. If there is only 1 xpath that you have to run on a page, then you can probably best use the short/direct functions that don’t require you to first open the web-page. Instead the address is supplied as an argument, together with the xpath. Here’s a list of the available quick scrapers:

And a short usage example to get the temperature info from the google API for a city that’s defined in ‘$place’:

$value = ScrapeText(“http://www.google.com/ig/api?weather=$place:interleaf(+)”, “/xml_api_reply/weather/current_conditions/temp_c/@data”)

As you can see, the first argument specified the web-page to open. The second is an xpath to the data attribute of the ‘temp_c’ element. Note that we use ‘:interleaf(+)’  cause the google API expects city-names that contain multiple words to be separated with a ‘+’ like: New+York.

More scraping

The second scraping method is primarily useful if you need to run multiple xpaths on the same content. In this case, it’s far more economical to first retrieve the page, run all the queries on the cached file and finally, when done, release it again. This can be accomplished with the remaining scrape functions.

You open a file or webpage with either ‘OpenScraper’ or ‘OpenScraperHTML’. The first works on xml content, the second on html. That is, the second will convert html to xml so that the xpath can be run on it. Both return an integer that needs to be used in subsequent calls. Basically, the integer replaces the filename as a reference. It allows you to have multiple files open and to have the system run multi-threaded and let it serve multiple people at the same time.

The scraping functions themselves are almost identical as the quick versions, except that they take an integer as first argument instead of a path. Other then that, usage is exactly the same, with the same types: one for text, integers, doubles and dates.

Once you are done with the file, you have to call ‘CloseScraper’ with, as argument, the integer that was returned by ‘OpenScraper(HTML)’, so that resources can be cleaned up. This is important, if you forget to do this, the system will eventually buckle, crack and give up.
In a normal usage situation, you would do a short salvo: open a page, do a few scrapes and close it again, all in 1 block, but this is not a requirement, you can keep the page open across multiple inputs. As long as you maintain a reference to the scraper (the integer) somewhere in memory so that you don’t loose track of it.

Html scraping

As already mentioned, html scraping is done by first converting the page into xml before the xpath is executed. This conversion can cause some ‘changes’ in the structure of the file. In other words, the path that you would calculate, based on the html file might not be correct for the xml version. This means that you best build your xpaths based on the xml version of the HTML pages.

The conversion routine that’s internally used by the chatbot designer is based on the SGMLReader library. This provides a command-line tool to manually convert html to xml files. This can be very useful for building the correct query. I’ve included a direct download for the command line html to xml conversion tool. Here’s a short description on how to use it (taken from the original documentation):

sgmlreader <options> [InputUri] [OutputFile]

Examples:

sgmlreader -html *.htm *.xml
Converts all .htm files to corresponding .xml files using the built in HTML DTD.

sgmlreader -html http://www.msn.com -proxy myproxy:80 msn.xml
Converts all the MSN home page to XML storing the result in the local file “msn.xml”.

sgmlreader -dtd ofx160.dtd test.ofx ofx.xml
Converts the given OFX file to XML using the SGML DTD “ofx160.dtd” specified in the test.ofx file.

Building an XPath

Once you have your xml file, getting the xpath to the element that you want can still be a little challenging. Html files simply aren’t designed with this type of usage in mind (and hey, if it can be easier for xml files, why not). Enter FireBug, an add-on for Firefox that allows developers to get a closer look at the html…. Or xml. After you have installed firebug and loaded up the xml file into firefox, go to tools/Web developer/Firebug/Open firebug so that you can see the debug panel. In this panel, select the element that you which to query, open the context menu and select ‘copy XPath’. And that’s it, simply paste this path in the chatbot designer and your done.

Numbers

The engine of the chatbot designer app will automatically convert every part of the input text that it can, into an integer or double. Furthermore, with the aid of the thesaurus, you are also able to pick up words that represent numbers and convert them into their corresponding number values. This allows us to easily work with and manipulate these numbers. Here are a few input pattern examples:

^n:number

^n:integer

^n:double

^n:noun.(language unit).numeral

The first 3 examples are the simplest: you can read a double (floating number), an integer or if you don’t care which, you can simply use ‘number’. The last one is a bit more interesting. Here we use the thesaurus to catch words like ‘one’, ‘two’,… Of course, that’s very limiting, cause this way, you’d have to declare a thesaurus-entry for every single number in the universe, which is, well…, impossible. So better to write the last pattern like so:

^n:noun.(language unit).numeral { [and] ^n:noun.(language unit).numeral}

The {} operator is used to find a repeating block in the input. So this catches things like: two thousand and five hundred.  As you can see, we use the same variable name for the number in the front and repeating part. This means that most likely, the ‘n’ variable will contain a list of words instead of just 1. Also, we don’t collect the ‘and’ cause it would screw up the conversion procedure. A small warning about the {} operator though: The application can’t find duplicate patterns when the {} operator is used, so watch out for this.

For completion, you could also include a pattern that handles mixed numbers and words. The conversion algorithm is perfectly capable of handling this mixed type of input.

Working with numbers

So, now that we can read in the numbers, lets look at how we can use them. let me just pick up where we ended: with word-numbers. Right now, the variable is still containing words (or a mix of words and numbers), but we actually need a single number, so this needs to be converted. Luckily, you don’t need to do this manually, but there’s already a function available for this. Here’s a usage example:

$value = $n:ToNumber

:ToNumber will convert the content of the variable into a number (only integers are currently supported, a ‘.’ is not yet recognized). Each individual word is converted using the thesaurus and finally added/multiplied together to form a single number. Now, this conversion algorithm is language dependent and at the time of writing, there is only an implementation for English. If you need it for a different language, best to contact me.

Of course, the primary purpose for having real numbers, is so that we can do calculations with them. As you’ll soon see though, this is done a little bit different compared to a traditional programming language. That’s because the chatbot designer’s pattern definition language is text oriented and not math. This means that if you were to type something like $n + $w, it would not perform the calculation, but print/store the ‘+’ sign in between the 2 numbers. So, to perform arithmetic, you need to use functions. here are some examples:

$result = $n:add

$result = $n:add($w)

$result = $n:subtract($w,$d,$f)

$result = $n:subtract($w,$d,$f:Add($h))

…

The basic principle is always the same: since a variable can contain a list of items (a set), it doesn’t really need arguments to perform the arithmetic operation, as long as it has enough items in the list. Because of this, most mathematical operations can be performed on more then 2 items. So you don’t write a+b+c, but if all 3 are in the same variable, you simply write :add.

If the variable doesn’t contain enough values or you want to include additional values, you can add those as arguments. For most functions, this list is unlimited and can always be nested, so you can do mathematical operations inside others.

For completion, here are all the currently supported mathematical operators:

Date and time

Next on the menu are dates and times. These are very closely related to numbers. Internally, the system doesn’t see a difference between a date and a time, it’s always a DateTime. Depending how you use the value, you can regard it as a date, time or DateTime.

First of, there is a special variable to get to the current system date/time: $time. This returns a DateTime-cluster which contains a list of numbers that together form the date and time. The order is predetermined by the system to make it language independent. For ease of use, there are functions to get to each part of the date, so you don’t need to memorize each position in the format (though it helps for putting dates back together). Here’s a table of all the functions to get each part of the date list (+ the index position of the item in the list):

And here are some output patterns to show how you can use them:

$time:day/$time:month/$time:year

$time:hour : $time:minutes : $time:second

The current month is ^noun.month[$time:month-1]

Note that I deliberately put spaces between the variables and the  time indicator (‘:’)  to avoid using the escape character (\). Also, in the last example, we get the name of the month from the thesaurus through the index number.  A thesaurus child at a specific index position can be accessed with the [] operator in the variable path. Because there’s no point of putting words between the brackets and this so often requires a small calculation (0 vs 1 based indexes), I have opted here for a more traditional calculus notation. Supported operators are: +-*/%.

Often it’s also useful to recompose a date, based on a series of numbers. For instance, to build the birthday of someone. As you probably already suspected, there are functions to do just that. Here are some examples:

$date = $list:ToDate

$time = $list:ToTime

$date = $year:ToDate($month,$day)

For building the date, the first value in the list has to be the year. This is the only required item, but more can follow: month, day, hour, minute, second. In that order.  To build a time value, the first item has to be the hour. Minutes and seconds are optional, but have to be provided in the previously mentioned order. If you want to omit the minutes (or any other) part , use 0 instead.

Why not simply store the date values as individual numbers in memory fields yourself, you might wonder instead of going through the hassle of converting a number into another number (which these conversion functions basically do). Well, that will become apparent in the next section.

querying on date

The date and time values are stored in such a way that a year, month, day, minute, second or week in a date is always represented by the same neuron. This means that you can easily find all the dates (and therefor the attached assets) that meet certain criteria (all in the month June for instance). This is very useful in queries like “Who else is born in the same year and month as I am?”. Getting the date objects is real easy. Here’s an example:

$people= #user.birthday:year:GetDates:ValueFor(birthday)

$people = $people:Complement(#user)

$people = #($people).name

–or, for year and month:–

$people= #user.birthday:year:GetDates(#user.birthday:Month):ValueFor(birthday)

$people = $people:Complement(#user)

$people = #($people).name

Note: For a full example, check the Date-Time demo found at {documents}\nnd\demos\dateTime.dpl.

In the first line, we get the year section of the user’s birthday. With the ‘:GetDates’ function, we retrieve all the date objects that the system has stored for the same year as the user’s birthday.  If you supply extra arguments, those values will also have to be included in the date, as is shown in the second example. With ‘:ValueFor’ we retrieve all the assets that link to the date. The ‘birthday’ argument, specifies that we only want the assets that link to the date value with the ‘birthday’ attribute. We could have omitted this, in which case we would have gotten all the assets, no matter how they are linked (it could be a ‘when’ reference for instance). Or, we could also have given more arguments, specifying a list of possible attribute values.

At this point, we have a list of assets that reference a date of the same year as the user’s birthday. There are still 2 things that need to be done: first we need to remove the ‘#user’ from the asset list, cause the ‘GetDates’ returns all objects, including the date object that represents the user’s birthday, but since the bot was asked ‘who else’, we need to remove the ‘#user’. This is done with the ‘:complement’ function. This is a mathematical function (borrowed from set-theories) and will return a list that contains all the items from the input with all the arguments removed.

We do the call to ‘:Complement’ on a different code line and not on the first one for a very specific reason: asset paths, like in the first assignment, always provide a single value as input to each function in the path, so even if ‘GetDates’ returns a list of items, ‘ValueFor’ will always work on a single value. At the end, the asset path joins all the results together. This, of course, is of no use for the Complement function, cause we need to remove the items from the list and not from a single value. Luckily, regular variable paths do provide a list of items as input values. That’s why we started a new assignment, using a variable path to calculate the complement. There are a couple of other functions (like Interleaf, Filter, Add, Multiply, Avg, Min, Max,…) which require a list of items as input. These should never be used in an asset path, but only in variable paths, for the reason explained above.

In the last line, we get the name value of all the assets that we found. This is again on a different line cause we need an asset path to get the ‘name’ of an asset. Regular variable paths don’t know how to do this. Hence the extra 3th line.  And thus, we get the names of the known objects that have a birthday in the same year as the user.

Time ranges

Besides the Date-Time type, there is also a Time-Span type. This specifies a length of time instead of a specific moment. You get a Time-Span by building it with a function, or as the result of some Date-Time calculations (see later). To build a time range (or time-span), you can do something like:

$range = $list:ToTimeSpan

$range = $days:ToTimeSpan($Hours)

Unlike a DateTime, a timespan contains the number of: days, hours, minutes and seconds (no year or month sections). so there are only 4 elements in the list. The first item in the list should always be the nr of days, possibly followed by the hours,… Missing data is set to 0. Like with :ToDate and :ToTime, you can also specify missing data as arguments. The same order applies.

For retrieving values in a TimeSpan, you can use the same functions as for the DateTime. They operate in exactly the same way, but the index position of the actual items in the list is different compared to a DateTime. here’s an overview:

At the time of writing, TimeSpans don’t yet support the same level of querying as the DateTime object does. If anyone has an urgent need for this, let me know, and I’ll see what I can do.

Calculating with dates and times

Like with regular numbers, you can also perform some calculations on dates, like subtracting one date from another. The syntax for arithmetic with dates/times is exactly the same as with numbers. Here’s an overview of what’s supported:

In general, the principle is always the same: the function input determines the type of the result, except for ‘Subtract’ which always returns a range and only works on Time values.

Boolean operations

As a final topic, perhaps a word about logical or Boolean operations on numbers, dates and ranges. Well, for the most part, these work as you would expect both for regular numbers and times/ranges. There’s just 1 small twist. The mathematical people out there will probably have noticed that set-theory is very present in this system: all variables can contain a list of items. This means that Boolean operators need to be prepared for this. Here’s an overview on how each operator handles lists:

• Equality (==): If the left part contains a list, the right side must have the exact same list: with the same nr of items, in the same order.
• Difference (!=) If the left part contains a list, the right side is different if it contains a different amount of items, the sequence is different or one or more of the items differ.
• Contains (contains): The right side of the operator should be part of the list found on the left side of the operation.
• doesn’t contain (!Contains): The right side of the operator should not be part of the list found on the left side of the operation.
• >, >=, <, <= (Bigger, bigger or equal, smaller, smaller or equal): If the left part contains a list, each item in the list will be evaluated to each item found on the right side. The operation succeeds if all items are compared and none failed.

There is plenty more to say about this topic. But my time-range has run out, a time to move on has been reached. All that’s left for me to say is: Stay tuned and CY later.

Because

Let me first give an example of a ‘because’ statement that a user might say:

I have a dog because I like animals.

This statement is a bit of an oversimplification, perhaps just right for an example.   It doesn’t only specify why I have a dog, but also that I have a dog and that I like animals. All this data should be stored and retrievable so that the system can answer questions like:

Why do I have a dog?

Do I have a dog?

Do I like animals?

So, before we can store the data, we need to find an input pattern that’s able to handle ‘I have a dog because I like animals’. To start with, we could do something like:

I have a $value because $reason [.]

This can work for many possible input values, except that the ‘I’ vs ‘you’ change is difficult to do since there is no knowledge of the context (‘you’ can change into ‘I’, ‘me’,… depending on the context), so this is the least usable pattern, but there are other possibilities.  For example, lets take:

I have a $value because I like $object [.]

or better:

I have a ^value:noun because I like ^object:noun [.]

With this input pattern, we are able to extract all the valid data so that it can be stored in memory and we also have static text for ‘I’, which can easily be transferred manually. A small improvement might be to replace ‘$value’ with ‘^value:noun’ so that only a proper noun can be captured instead of things like ‘I have a pretty big keyboard on my lap sitting there because I like typing’. The same goes for ‘$object’. As in the previous example, ‘typing’ could probably better be transformed to ‘type’ before you store it, so in the pattern you could replace ‘$object’ with ‘^object:verb’ or ‘^object:noun’,… Anyway, lets take a look at how the do-patterns would look like for storing this info:

#user += $value

#user.like.($object) = yes

#user.($value):why = “because you like $object”

In the first line, we create a ‘has’ data item for ‘$value’. The second line adds the ‘object’ to the ‘like’ list, with the value ‘yes’ so that it’s clear the user likes it. And the third line stores the reason why the user has a ‘value’. ‘:why’ is a function that provides access to the ‘reason’ data-tree. What you store in ‘:why’ is completely up to you. In this example, we store a text, between brackets, cause I wanted to preserve the spacing, but you could also treat it as a sub asset (or anything else, like a link to another rule), allowing you to write paths like #user.($value):why.object.

Why

Once you have stored  the information, you can retrieve it again if the user asks a why question. Here’s a possible input pattern that you could use for capturing this question:

Why do I have a $value [?]

or better:

Why do I have a ^value:noun [?]

And the output pattern could look something like:

You have a $value #user.($value):why

or better:

Bot says when:  #user.($value):why

You have a $value #user.($value):why

else:

I don’t know why you have a $value

That was easy enough. The first line is the most basic approach, the only problem with it: if the bot doesn’t know, it returns ‘You have a’. Which is no good. Better to put it in a conditional and check if there is a ‘why’ value, if there is non, let the bot say it doesn’t know. To check if there is a ‘why’ value, simply put the path in the ‘when’ close. This will check if the result of the path has a value. To check if it doesn’t have a value, use the not (!) operator in front of the pattern, like so:

Bot says when: !#user.($value):why

I don’t know why you have a $value

else:

You have a $value #user.($value):why

Improvements

Right now, we have a working system to handle because/why type of things, but it’s not very flexible, I mean, there are tons of reasons why you can have something, and not just because you like something else. Also, there are more people then just ‘I’, so lets improve the input patterns a bit and make things more flexible.

Using the Thesaurus

Lets start with ‘I’. As you had probably already guessed, the thesaurus variables are ideally suited for this. So, here’s an improved input pattern:

^s:pronoun.subject (have|has) a ^value:noun because ^w:pronoun.subject like ^object:noun [.]

Ok, that’s already a bit more complicated. The ^s variable can capture any child of the pronoun ‘subject’.  If you check in the thesaurus, under the pronoun POS, there are a bunch of greyed-out items, one of which is ‘subject’. These are ‘placeholders’, that is: these thesaurus items don’t actually contain a word, only a label. So our ^s variable can’t actually catch the word ‘subject’, but only it’s children: I, you, he, she, they, we, it.

The do-patterns also become a bit more complicated, since we will now have to calculate the inverse of the pronoun and we also have to extract the asset out of it. Luckily, there are some helper functions for this:

#($s:ResolvePerson) += $value

#($w:ResolvePerson).like.($object) = yes

#($s:ResolvePerson).($value):why = “because &w:InvertPerson ^verb.like:conjugate($w:ResolvePerson) $object”

:ResolvePerson is able to extract the asset (or concrete representation) out of a word.  At the time of writing: I and you  are supported. Names, ‘He’, ‘she’, ‘it’,… need some further testing.
:InvertPerson uses thesaurus links to jump from one person to it’s inverse, if there is one. So ‘you’ becomes ‘I if ‘you’ was used as a subject.
:Conjugate will try to find the correct conjugation of a verb, based on the argument that is supplied. This has to be an asset, usually the asset representation of the sentence-subject. So if you want to conjugate for ‘I’, you pass in ‘#bot’. ‘You’ becomes ‘#user’, and so on.

Split the patterns

The second improvement that we can make to the pattern is a bit more radical and relies on a special feature of the pattern matcher. You see, the pattern matching process is not restricted to finding just 1 pattern in the input. It will try to find the longest possible sequence of patterns that it can. That is, if the same words can be caught with a single pattern, this pattern will get precedence over a sequence of patterns. But if the pattern matcher can do a longer match by using 2 patterns instead of 1, that will become the result. This allows us to split up the first part of the sentence: ‘I have a $value’ from ‘I like $object’. opening up a whole new range of possibilities, and more importantly: saving use lots of duplicate work. The do-patterns become a little different though. First the input-patterns:

Topic: HAVE

^s:pronoun.subject (have|has) a ^value:noun

because ^s:pronoun.subject (have|has) a ^value:noun

 

Topic: LIKE

^w:pronoun.subject (like|likes) ^object:noun [.]

because ^w:pronoun.subject (like|likes) ^object:noun [.]

As you can see, not much has changed except that they are now 2 sets of patterns and each set has a version with and without ‘because’ in the front. We could also have written them as:

Topic: HAVE

[because] ^s:pronoun.subject (have|has) a ^value:noun

Topic: LIKE

[because] ^w:pronoun.subject (like|likes) ^object:noun [.]

But we aren’t. Instead, we keep each input-pattern in it’s own rule so that each can have it’s own set of do-patterns. This way, the ‘because’ version can be treated differently. Basically, what it comes down to is this: when there is no ‘because’, we simply store an extra memory field called ‘subj’ which allows us to recall the left part of the asset operation (excluding the attribute). When we have a because, we check if the ‘subj’ field is set and if so, we store the ‘:why’ in this field. Here are the do patterns:

Topic: HAVE

#bot.who = $s:ResolvePerson

#bot.Inverted = “#bot.Inverted because $s:InvertPerson ^verb.have:conjugate(#bot.who) a $value”

#bot.attribute = $value

#bot.who += $value

#bot.value = #bot.who.($value)

#bot.Subj = #bot.Who

#bot.Subj.(#bot.attribute):why = “because $s:InvertPerson ^verb.have:conjugate(#bot.who) a $value”

Topic: LIKE

#bot.who = $s:ResolvePerson

#bot.Inverted = “#bot.Inverted because $s:InvertPerson ^verb.like:conjugate(#bot.who) $value”

#bot.value = yes

#bot.attribute = $value

#bot.who.like.($value) = yes

#bot.Subj = #bot.who.like

#bot.Subj.(#bot.attribute):why = “because $s:InvertPerson ^verb.like:conjugate(#bot.who) $value”

The full code example can be found in {documents}\NND\Demos\Why_Because2.dpl. As you can see, it’s a bit more code then where we originally started. Though if you look a little closer, a lot of it is boiler-plate stuff: store who, attribute, value, subj, inverted.
Key advantage here: with only a few patterns we can have any type of combination: have because like, like because have,  have because have, like because like, because have, because like, have, like, have because like because have,….

The basic set-up is always the same, we dissect the sentence into it’s parts so that they can be reused in other parts. here’s what we need:

• calculate ‘who’ (the asset form of the subject part in the sentence = I, you, he,…) so we can reuse it and don’t have to recalculate it each time.
• store the the inverted sentence, for output generation (can always be useful)
• the attribute and value (‘color’ is the attribute of ‘yellow’). In this example, strictly speaking not really required, but other parts of the concept rely on this data, so best to get used to it.
• subj: sometimes, the ‘who’ isn’t enough to find out where some data needs to be stored. ‘like’ (and most other verbs) is a good example of this. So if we want to get to the correct data path later on, we need to store this new path, hence the existence of ‘subj’.
• If ‘when’, ‘where’, ‘how’,… is also present in the sentence, these are all data parts that can be stored like attribute, value or more complex sub structures.

There are a few extra do-patterns required to get this working correctly. The ‘who, inverted, value, attribute, subj’,… fields of the #bot are all temporarily, that is: they are supposed to be used as mid-term memory, for as long as the current input is being processed, so that the information can be passed along from one pattern to another. This means that we need to remove the data after the input has been processed so that it wont interfere with any of the next input. This can be done from the Chatbot’s properties view (select the menu item view/chatbot properties, next go to the ‘Do after output’ tab.). here’s how it would look like for this example (can be simplified, which we will do in the next example):

#bot –= Inverted

#bot –= Who

#bot –= Subj

#bot –= attribute

#bot -= value

Also, in this example, I only used thesaurus variables. You can achieve similar results with regular variables, but there has to be a small change in the pattern definition for it to work properly. Everything has to do with the fact that a regular variable can only determine it’s end by what is defined after the variable, and if the pattern definition ends with a regular variable, it will collect the remainder of the input and never jump to another pattern. So we need to put something behind the variable if the ‘because xxx’ needs to be handled correctly. This can be done by moving the ‘because’ from front to back like so:

Topic: HAVE

I have a $value [.]

I have a $value because

Topic: LIKE

I like $value [.]

I like $value because

The biggest disadvantage: you need extra patterns to handle a ‘because xxx’ (not shown or included in the demo), but on the plus side, the do-patterns become a little simpler using this type of pattern definition style. For a complete example, see: {Documents}\NND\Why_Because.dpl.

Sub topics

There’s one more trick we can use to make the patterns more flexible and which will also resolve a final problem caused by using multiple rules as we did in the previous step. You see, finding a list of unrelated rules is primarily done to recognize multiple sentences in the input and not for splitting up a single sentence. That’s because all the outputs from those rules are always automatically combined. So you can’t change the order or content. This can make output management a little tricky.

The solution comes in the form of sub-topics or sub-rules. With this technique, it’s possible to reference a topic or rule by it’s name from within a pattern in the same or other topic. This allows us to split the ‘because’ out into a third topic and then group them all back together into a single rule, which will be the final, single result. So, on the input side, it becomes more flexible, and thanks to a special variable ‘$output’ and a switch in the chatbot properties, we can also take control of the output side. more on that later, lets first start with the input patterns, how do you declare sub-topics?

Topic: HAVE

^s:pronoun.subject (have|has) a ^value:noun

Topic: LIKE

^s:pronoun.subject (like|likes) ^value:noun [.]

Topic: BECAUSE

because

Topic: BECAUSEHANDLER

~have

~like

~becauseHandler ~because ~becauseHandler

The first 2 input patterns should be familiar by now, nothing has changed since before, only the ‘because’ version has been eliminated. The first new topic is also nothing special, just a single word ‘because’.  The magic happens in the last topic, which groups all the other topics together.

~ is used to indicate a Topic reference. It’s always followed by the name of a topic and possibly a dot followed by the name of a rule. It’s meaning is simple: include all the patterns in the rule(s) of the specified topic at the location of the reference. Here’s an example for sub rules:

~have.statement because ~like.statement //presumes that topics ‘have’ and ‘like’ have a rule labeled ‘statement’

Now, if you look at the last topic: ‘BECAUSEHANDLER’ you’ll notice that it only contains references to other topics, it’s a root topic. In this example, it’s purpose is to provide a place to declare the output for every topic, so we can combine things correctly + it also stores the ‘why’ relationship. Also, if you look at the last pattern, it references it’s own topic, 2 times. This is recursion and allows us to recognize a sequence of ‘becauses’ like: I have x because I like y because I have z,….

You might be wondering why it’s the BECAUSEHANDLER that stores the ‘:why’ link and not the ‘BECAUSE’ itself. That is because at the time of the ‘Because’ pattern, the reason is not yet known (this is defined in the next pattern, which hasn’t been processed yet), so it can’t link anything up yet. To overcome this, we make certain that there is an extra rule that gets executed after every other part of the sentence: the BECAUSEHANDLER.  In other words, the ‘becauseHandler’ is a way to perform some code after all the patterns have been processed.

By the way: topic names can be edited in the ‘project view’ (select the topic, press F2 or right mouse click/rename). A topic name should be unique within the project if you want to use it as a sub-topic, otherwise it’s not that important, but the UI will always warn about duplicate names and those topics will have a red icon instead of blue. The name of a rule is always visible in the ‘description view’s title when the rule is selected.  It can be changed in the topic editor: select the entire rule (don’t click on a pattern, but on the background of the rule). With F2, you get a dialog to change the name.

Let’s continue with the do patterns.  For ‘have’, ‘why’ and ‘like’, they remain very similar: the patterns are used to store the inverted sentence, who (asset), the attribute, the value and the ‘subject’. In this example though, instead of directly storing it under the ‘bot’, it is stored underneath ‘mem’ so that we can move the entire result set with 1 statement later on. So here’s a small example of the ‘have’ do-patterns:

#bot.mem.attribute = $value

#bot.mem.who = $s:ResolvePerson

…

And so on. The really interesting stuff happens in the ‘BECAUSE’ and ‘BECAUSEHANDLER’ topics. Note that this time, we have some do-stuff in the ‘calculate’ area and others in the ‘do’ section. The major difference here: ‘Calculate’ is done, just before any of the conditions are evaluated, so this allows us to do pre-calculations that can be used in those conditions.

Topic BECAUSE:

#bot.because = #bot.mem

#bot -= mem

Topic BECAUSEHANDLER:

Rule ~becauseHandler ~because ~becauseHandler

Calculate:

$result = “because #bot.mem.Inverted”

$path = #bot.because.Subj.(#bot.because.attribute)

Output when: #($path):why == $result

Yes, I now #bot.because.inverted because #bot.mem.inverted\.

else

I see, #bot.because.inverted because #bot.mem.inverted\.

       Do

#($path):why = $result

Rule ~have

$output

First off, BECAUSE: this moves the ‘mem’ field to ‘because’ and ‘mem’ is cleaned/deleted. Basically, we store the data of the previous sentence and prepare to collect the data for the next sentence. The ‘because’ field will later on be used to link to the newly collected ‘mem’ field. Note that this rule doesn’t generate any output.

Secondly comes the ‘BECAUSEHANDLER’. This builds up the result that needs to be stored in ‘:why’ and calculates the location where this info needs to be stored (in $path). Before actually committing the data to memory though, a check is done to see if it was already known. If so, a different response is given compared to when it is not yet known. In the latter case, the info is also stored.

For the other rules (~have and ~like), we simply declare the $output variable in the output section, indicating that we want to reproduce the output of previous topic.  Note that the use of this $output variable can be controlled in the chatbot’s properties view. When turned off, it will function as a regular variable, and the output of all the patterns will simply be combined. This allows you to select between a simple styled bot or something more advanced.

Finally, as already mentioned, we still need to do some clean-up after the input. Since we have grouped all the mid-term memory in 2 fields: ‘mem’ and ‘because’, cleanup becomes a little simpler (Note: sometimes the ‘because’ part doesn’t exist, but that’s ok, nothing will be removed in this case):

#bot –= Mem

#bot -= because

I guess that’s about it for now.

Well, we went from simple, straight forward, fire-cracker-leveled patterns to something that’s more akin to ‘rocket science’. The combination of memory, thesaurus and sub-topics might just turn out to be a very explosive mix. I for one, am very interested to see where all this will eventually lead too… Stay tuned.

support for multiple speech engines

The first, and most fundamental change is the fact that the system now supports multiple speech engines.  At the moment, there is an implementation for the managed and unmanaged Microsoft SAPI versions. Others might follow. The major reason for this feature is that the SAPI libraries behave differently on different systems and OS’s. This was most apparent in the fact that some voices aren’t properly supported in the managed version. That’s why the unmanaged version has become the default, since it is the most powerful and most stable (also in terms of timing), but XP for instance doesn’t support SAPI 5.3, so this requires the managed version. To select the engine you want to use, go to the menu item: tools/options, select the ‘Speech’ tab and use the combo box to select one of the engines.  Note: before switching engine, it’s best to turn off audio input/output on all chatbot channels.

Alternate input

Also new in this version is the ability to select between alternative input text for voice recognition. The system wont yet automatically be trained with this information though. You can turn this on/off with a new button on the chatbot-local toolbar, to the right of the Audio-in button. When turned on, and the speech engine can provide alternate text values, a dropdown-box will be displayed just below the text (one for each speech salvo). You can select an alternate text from this box. Note that if there aren’t any alternatives, no dropdown is shown. Also, the managed engine can give different results compared to the non-managed version.

New demos

Finally, there are a few new demos included in the installer. 3 new demos that show different ways to implement why-because (and also show how to do have and like). The previous release also had a demo for why-because, but it still had some issues.

That’s it for now, more things to follow…

For the curious, there are actually a number of fixes/updates all over the place:

• Importing/exporting part of a thesaurus has been fixed.
• There was a problem in the ‘rebuild project’ procedure for the thesaurus which has been fixed.
• A do-statement of the form #x.z = #y  now gets a correct y value.
• There’s a new CmdShell extension for scraping xml and html files (from the internet or disk)

2 new demo projects:

• weather.dpl: uses the scraper extension to get the weather info from the Google weather api.
• Why_because.dpl: shows a way how you can correctly handle why/because statements. In the process, it also shows how to do ‘be’, ‘have’ and ‘like’. 2 more versions of this demo will follow shortly.

Short term memory

Or also called ‘volatile’ as it’s content is lost in time, can be accessed through the use of variables in the patterns.  Input patterns support 2 types of variables: regular variables, which can collect any type of content and thesaurus variables, which can only collect words that are equal to or are children of the thesaurus item referenced by the variable (in other words, they are filters). Asset variables, a third type of variable, is theoretically also possible, but not yet implemented. These are also filters, like thesaurus variables, except that they filter on concrete (asset) data instead of abstract (thesaurus) data.

The basic usage of this short term memory is simple: to provide a mechanism for collecting values of variable parts in the input patterns so that these values can later be used in the output and do-patterns for providing a response and feeding the long term memory.

Regular variables

As already mentioned, regular variables can’t filter on the values that they collect, but they are optionally able to limit the number of words that they collect, either as a specific number or a range. Here are some short input-pattern examples:

I’m called $var[.]

I’m called $var:1[.]

I’m called $var:1-3[.]

I’m called $var:4:CollectSpaces[.]

Copy $from:collectSpaces to $to:collectspaces

The $var constitutes the variable (‘$’ is the variable operator, followed by the name). Every word (except spaces, if ‘CollectSpaces’ is not specified) is collected by this variable until the pattern matcher finds a word in the input that follows the variable in the pattern or until the range is fully used. This variable can then be used in the output and do-patterns of the same rule (and also from other rules, if you are certain that the input-pattern is part of the result set, which can be checked upon, more on that  later).

Thesaurus variables

Thesaurus variables are a special type of input variable: they provide a mechanism for filtering possible input to a sub-branch of the thesaurus. If the input can’t be found in that branch, the pattern wont be activated. So, this is a filtering mechanism. The actual value that was found, can be accessed like any regular variable, through it’s name. There is no mechanism for providing length or range values though since they have no meaning here: it’s either an exact match with a thesaurus node or it isn’t. Here are some examples:

I’m called ^var:noun [.]

I’m called ^var:noun.name [.]

I’m called ^var:noun.(first name) [.]

I’m ^var:number years old[.]

A thesaurus variable always starts with  a ‘^’ followed by it’s name. The ‘:’ indicates the start of the thesaurus path and should always be followed with a POS (part of speech). These are the supported POS values:

noun, verb, adjective (or adj), adverb (or adv), article (or art), pronoun (or pron), conjunction (or conj), interjection (or inter), preposition (or prep), number, integer (or int), double.

You can stop there, which would indicate that you want any word of the specified part of speech. You can also continue the path with a ‘.’ followed by a text value (put into brackets if it’s multiple words).  This allows you to further refine the thesaurus path. Note that you don’t need to start at the root of the thesaurus that you are using, just as long as you are comfortable that it will point to a unique word within the tree (otherwise you can have multiple matches,… which might also be desirable). Note that the last 3 POS values (number, int and double) can’t have any further path specifiers, they have to stop at the POS value.

Collecting multiple values

It’s possible to use the same variable name multiple times in the same input pattern. This allows you to collect a list of values for the same variable. Thesaurus and regular variables can be intermixed. Here are some examples:

{$name ,} and ^name:noun.name are here  //catches something like: Tom, Flint and Warner are here

Note that there is a difference when a regular variable collects multiple words at a single location compared to when it collects single words at multiple locations in the pattern. When a single location collects multiple words, this group of words is combined into a compound word (as in ‘baby gear’), but when words are collected at multiple locations, a list is created. This list can later-on (in the long term memory) be labeled as AND, OR or LIST (unspecified).

Using short term memory

Up until now, we’ve only been talking about how to collect the values for the short term memory.  Of course, there’s no point in doing that unless you can actually do something with these values. That’s done in the output and do-patterns. As already mentioned, you access the content through the variable names. Here are some output-pattern examples:

Ok, I see, your name is $var\.

So you are $var, nice to meet you!

So, you can $verb:Infinitive, can you?

I see, $name:interleaf(“\, “, ” and “)

At it’s most basic form you specify the ‘$’ operator followed by the name of the variable that you want to render. Note that you should always use the ‘$’ operator while rendering, even if the value was collected using a thesaurus variable ( ^ ). This is because the ^ operator is used to access the long-term, abstract memory (the thesaurus data itself).

Rendering the value as it was collected, is useful but often we want to do a little more, sometimes we need to do some kind of change or transformation to the values, like conjugating a verb, get the plural of a noun or find the attribute for the value (see later),…  This is done through functions that you define in the path. A function starts with a ‘:’ followed by the name of the function (a list of all the available functions will come shortly) and optionally a list of arguments for the function, specified between brackets and separated by a ‘,’. Note: if you use the ‘,’ sign as an argument value, it must always be escaped with a \ Also, if you need to preserve spaces, the argument should be placed between brackets (as in the last example).

Long term memory

The second major type of memory that’s available to the bot is used to store and retrieve values so that they can cross the boundary of the single-shot input/response system, in other words: long term memory. Currently, there are 2 types: a thesaurus structure for storing abstract information and assets which maintain concrete knowledge.  Typically, you use this data to compare against short-term variables, render previously stored data or store newly acquired knowledge.

The thesaurus

As already mentioned, thesaurus variables are used in the input-patterns so that the valid content for a variable can be filtered.  When the ‘^’ operator is used in output, conditional or do patterns however, it behaves a little bit different: it becomes a value generator instead of collector.  Consider the following output patterns:

We are in ^noun.month[$time:month-1]

I like ^noun.food.(Italian food):random

I ^verb.be:conjugate(#bot) trying something complicated   //render: I am trying something complicate

As you can see, a thesaurus output-path contains a mix of statics and functions which eventually result into 0, 1 or 2 values. Because they render values and don’t collect it, no name is required. You can use the [] operator to select a child at a specific index position, like in the first example, which is used the generate the name of the month instead of a number. Note that the index is 0 based. In case that a static path item contains multiple words (like ‘Italian food’), use () brackets to group them. Also, if there are no values found for the path, any spaces that follow it in the output are stripped.

You can also store new data in the thesaurus. This is done in the calculation or do-sections. There are basically 2 operations that you can do at 2 different levels: you can add or remove values either as thesaurus children or as conjugations/references. To explain the difference between children and conjugations or references, take the following examples and how they are stored:

In the first example, we are declaring a child relationship: house is a building. If you have done any coding before, the syntax might be vaguely familiar: the left part of the statement contains the thesaurus path, the ‘+=’ operator to indicate that we want to create an ‘is child’ relationship, and on the right-side comes the value that needs to be stored. This could be a variable reference, an asset, another thesaurus path,….

The second and third examples look identical and for all intent and purpose, they are. The only difference is on the inside: in the first example ‘plural’ is a known conjugation form, ‘opposite’ is not. The statement used for storing this information, is a little bit different. First of, the thesaurus path ends with a ‘->’ followed by the name of the relationship that you would like to edit. Next, we use the ‘=’ assign operator instead of ‘+=’ to indicate that we want to change the relationship value.

The 2 last examples demonstrate what happens when you use the –> operator together with the += assignment or when you use multiple –> operators. When combining += with –>, you will first calculate the full result of the left side.  So in our example, we first take the singular value of ‘birds’, then we add a child to this result, which is ‘bird’. A similar thing happens when you use multiple –> operators: the value is calculated.

Except for the POS value at the start of the path, every other item in a thesaurus path can be a static, a variable reference, an asset path or another thesaurus path. This allows for tremendous flexibility in the way that you store data. We could generalize some of the previous statements like this:

^noun.building += $value

^noun.($singular)->plural = $value

^adj.good->($relationship) = $value

Removing values from the thesaurus is done using the ‘-=’ operator or by assigning to the ‘null’ value. Like with storing, all parts can be static or variable. This is probably best explained with some examples:

Assets

As already mentioned, assets could theoretically also be used in the input, but that’s not yet supported. If someone has a need for this, let me know, it’s not that tremendously difficult to add, it just creates a little more overhead.

Anyway, like thesaurus paths, asset paths can be used in output, do and conditional patterns. They are declared in much the same way as thesaurus paths by using the ‘.’ (dot) or ‘:’ (function) operators, except that they start with a # and ‘–>’ (links) are not supported.  For the thesaurus path, the ‘.’ (dot) operator selected a child node, for assets, this selects an attribute value. Here are a few output examples:

My name is #bot.name

your children are called #user.child.name:interleaf(“\, “, “ and “)

a book is made of $(#(^noun.book).component.name):interleaf(“\, “, “ and “)

Bot and User are hardcoded assets and refer to me and you respectively, from the bot’s point of view. In the third example, the first value in the asset path, is actually a thesaurus path. This results in concrete information about abstract data (a book is made of paper, ink, glue,…).

Also in the last example, the entire asset is the first value in a normal variable path, because an asset path will always calculate it’s result based on 1 value, if the previous path item resulted in multiple values (like ‘component), the next part of the path is calculated as if there was only 1 result (internally, a split is done), and only at the end of the path, all results are joined. This doesn’t work for ‘:interleaf’, it expects a list of values to combine. A variable path can do this, hence this construct.

To store asset data, the = (assign), += (assign add), != (assign not) and !+= (assign add not) operators are used. Removing data is done with the –= (assign remove) operator.   Take the following examples (input statement to the left, how to store/remove it to the right):

When you use the ‘=’ (assign) operator, you declare an ‘is’ relationship: ‘color’ becomes the attribute, ‘blue’ the value. Since ‘blue’ is not an asset, but just a word, we have a terminator: blue can’t have any more children.  But, there is a way to cross this border, by using the ‘:extra’ function, as in the 5th example.
If instead, you want to declare that something is not y, you can use the not-assign operator (!=). This allows you to still store the information that something is not. Be careful though, there is a thin line between being and not being, if you don’t check on this in the conditions, you might say that something is, while it isn’t (sounds familiar?).

The ‘+=’ or assign-add operator is used to create ‘has’ relationships, like in the second example. The major difference with the first one is that ‘dog’ becomes the attribute and the value becomes a new asset that will represent the dog. There is also the not version:  !+= which is used to indicate that the asset doesn’t have something.

If you want to create a list of values, you can use either the ‘;=’, ‘|=’ or ‘&=’ operators. The first one creates (or adds to) a generic list, the second is for OR lists and the last for AND lists. The generic list operator can add to any type of list without modifying it’s type. The |= and &= operators will change a generic list to OR and AND respectively. When you try to add an item to an OR list with an AND operator, you create a new list object that contains the original OR list and the newly added item. the same goes for an OR operator with an AND list.

Finally, you can also actually remove an attribute value. This is done with the ‘-=’ operator. The  right-side should be the name of the attribute that you want to remove. The value that is removed get’s cleaned up automatically, so if the value was another asset which isn’t referenced anymore after the remove, the entire asset will be destroyed. (Removing items from a list has to be done with the ‘:Remove’ function.)

As already mentioned, every asset value can always use the ‘:extra’ function to get to a sub-asset. There are a few other functions worth mentioning which allow you to expand the dataset that the asset can store. These are used to declare things like when, where, why, how, amount,… Functions are:

Mid term memory

Some functions, like the ‘:attribute’ function (which is able to extract, for instance,  ‘color’ from ‘blue’, or ‘name’ from ‘Jan), make use of context, if it is declared. This context is usually a list of asset paths that point to some memory region of the bot. The idea is that, together with a response, you also generate the meaning of what was said and store this information in the asset that you declared as context. If you refresh this context on each run, you effectively have simulated mid term memory.

The basic setup for using mid term memory consist out of:

• a global context declaration so that the system knows where to go look for contextual info.
• some global do-after-each-statement patterns. These are responsible for erasing the previously collected data and possibly creating an echo.
• some global do-on-startup patterns which will remove any data from the previous run.
• do-patterns on each rule to actually collect the knowledge about what is being said.

Context:

#bot.memory.subject

#bot.memory.attribute

#bot.memory.object

#bot.PrevMem.subject

#bot.PrevMem.attribute

#bot.PrevMem.object

 

Do after output:

#bot.prevmem = #bot.memory

#bot –= memory

 

do on startup:

#bot –= memory

#bot –= prevmem

 

on pattern (example pattern = I like $value [.])

#bot.memory.subject = #user

#bot.memory.attribute = like

#bot.memory.object = $value

 

Mid term memory also becomes very useful once you start working with recursive sub rules/topics. This technique allows you to rebuild the extracted data.

Patterns

All the different types of patterns (input, output, conditional) could also be considered as a form of long term memory. Internally, they are stored in exactly the same manner as all the other data. As such, they can also be manipulated in a similar manner as the other long term data. Although, at the time of writing, there is still limited support for this.  More on that to come.

Images

The animations and visemes are stored as a series of images which are displayed in rapid succession, much like how old school film works. Before you can display these images though, they need to be created. Now, in the olden days, they used to have big chunky camera’s for that. Things have changed a bit since. We can use tools like DAZ3d, Poser, Blender,…

Content

What you put in the images is entirely up to you. There aren’t many limits with respect to content in the sense that there aren’t any ‘expected’ parts. That said, since we are dealing with chatbots, I’d try to put in something resembling a mouth somewhere, just to get some lip-syncing working. Most 3D tools these days, come with some way to manipulate mouth positions. The designer uses 21 different images for lip-sync (+ 1 for silence, which is actually the background image). Most 3D tools only provide morphs for 16 mouth positions. The remaining 5 visemes can be created by reusing other images or by creating your own, if the 3d app allows it. Here’s a good visual overview of all the visemes.

Animations like blinking eyes or moving hair also need to be exported from your 3D package as a series of images. Luckily most 3d software provides a feature to just that. Also, some animations can use the same image multiple times, it’s always best to reuse the same image in those cases, to save memory. For instance, an eye blink can be done with 2 or 3 images: eye fully open is provided by the background, half closed and fully closed.

File formats

Currently, the following file formats are supported:

It should also be possible to use xaml files (to create flash typed characters) , but this is not yet tested and will most likely not yet work. Give me a shout if you would like to do this.

Processing

Once you have created all the images, it’s best to process them a little further. At this stage, every image always uses up the full viewable area of the character. That’s ok, if all you want to do is lip-sync and possibly a little animation during idle times. But in this setup, your character can’t blink while speaking. To accomplish this, we need to cut out only the valid parts of each image and make the rest transparent. This way, the system can overlay multiple images and create the illusion of different parts moving at the same time.
Another added advantage of this process is that it usually (depending on the file type) shrinks the file. PNG files, when edited with the correct tools, will use less space if there is a bigger transparent area.

There are multiple ways you can crop the images. You can use the eraser tool found in most bitmap editing tools like paint.net.  Though, I’ve noticed that at least some versions of Photoshop don’t shrink png files when using the eraser, so that’s maybe not the best tool for the job. Personally, I used a little home-made app to do the job. You can download it from here.

It’s very simple to use. As you can see on the screenshot, there are 2 big buttons to the left which contain the source images. Press on each button to load the images. The top image should contain the root, in the bottom, you put the images that need to be cropped. To the right, on top, you see the result for the currently selected target image (seen in the bottom button). With the slider, you can scroll through the loaded images.
On the next line, there is a checkbox and a slider. These control the calculation process: do you want the parts of the image that are different or the same, and with the slider you select the tolerance level used to calculate the difference. When you put the slider fully to the left, there is 0 tolerance, meaning that there can be no difference between the pixel in the source image and that of the image that needs cropping. Put the slider fully to the right and the difference has to be very big.
Finally, with the button labeled ‘save’, you can save the processed images to a directory that you select (the original files are kept).

Depending on the way that the images are rendered in the 3D package, the file format and the quality of the rendering engine, there can be a bigger or smaller difference between parts of the images that should be the same. That’s why you normally have to play a little with the tolerance level. The lower, the better. Sometimes it’s better to keep a lower tolerance level and manually remove the remaining dots with an eraser in a bitmap editing package.

CCS file

When you’re ready with your images, it’s time to put them all together, a bit like a collage. Unfortunately, there currently isn’t yet a ready-made tool for this, so you are gonna have to do a little xml writing. Fortunately, there’s a lot of copy paste involved. The outer xml tag, the start of the file, is a <Character> tag, like so:

<?xml version="1.0" encoding="utf-8"?>
<Character xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
</Character>

Now, Let’s go over each section in order of the file content:

Character Info

The first section, and the easiest, is the character information. This is how it looks like:

  <CharacterInfo>
    <Name>Mika</Name>
    <Author>Ady Di Pierro</Author>
    <Copyright>Copyright 2011 Ady Di Pierro</Copyright>
    <License />
    <AuthorWebsite>http://www.laticisimagery.com.au/</AuthorWebsite>
    <CreationDate>2011-08-07T14:43:26.602+02:00</CreationDate>
    <LastUpdateDate>2011-08-07T14:43:26.602+02:00</LastUpdateDate>
    <Rating>
      <Rating>Unknown</Rating>
      <Sexual>false</Sexual>
      <Violence>false</Violence>
      <Other>false</Other>
      <Description />
    </Rating>
    <OnlineOptions>
      <OnlineCharacterBaseUrl />
      <PreferredWidth>150</PreferredWidth>
      <PreferredHeight>150</PreferredHeight>
    </OnlineOptions>
  </CharacterInfo>

Just copy the above and paste it into your xml file, just under <Character>. This part only contains reference information about the character: what’s it’s name, who designed it, what’s the licence, possibly a website,… All of the info in this section is optional, you can leave the tags empty, but it’s best to include them. This information, by the way, is displayed in the popup window on the ‘chatbot window’ (the button in the lower-left corner on the images window).

Online options are currently still skipped but will probably be used in future, online versions.

Background

<Background>
   <ImageResource>images\Kima00.png</ImageResource>
   <ImageResource>images\KimaEars.png</ImageResource>

</Background>

The background section defines the images that should be used as background (obviously). 2 things worth mentioning: a background is defined as an ImageResource. This is used often throughout the file. Whenever you want to reference an image file, you use this element. The text part of the xml element defines the relative path to the image (that is relative to the CCS file).

Also, and that’s perhaps the weirdest part, you can declare multiple backgrounds. Every image in this list will be displayed as a background, unless an animation turned one of the images off. And that’s the main usage of having multiple backgrounds: so that animations can turn part of the background off while playing. A good example are Mika’s ears.  They are drawn in a separate background image, so that the ‘flip-ears’ animation, can hide the background-ears while it’s playing. We do this cause some images in the animation sequence are smaller then the ears in rest, which would otherwise give ugly results with half an ear overlapped and the rest still visible. I’m certain there are plenty of other cool tricks to done with this feature. Hiding a background image is explained in the ‘Animations’ section.

Animations

The ‘Animations’ section is one of the bigger parts of the file. This is where you declare all the animation sequences available to the character for emotional expressions and idle times. The background animations (which run all the time, not jus at idle times), are declared somewhere else. Anyway, here’s how an animation definition looks like:

  <Animations> 
    <Animation>
      <AnimationFrames>
        <AnimationFrame>
          <Duration>5</Duration>
          <ImageResource>images\other\Kima Ears (01).png</ImageResource>
          <VisemeGroupName/>
        </AnimationFrame>
          <Duration>5</Duration>
          <ImageResource>images\other\Kima Ears (01).png</ImageResource>
          <VisemeGroupName />
        </AnimationFrame>
      </AnimationFrames>
      <EnableFrameSpeaking>true</EnableFrameSpeaking>
      <HoldLastFrameForSpeak>false</HoldLastFrameForSpeak>
      <FirstFrameUnderlay>false</FirstFrameUnderlay>
      <BackgroundSuppress>KimaEars.png</BackgroundSuppress>
      <Name>earsmove1</Name>
      <ZIndex>0</ZIndex>
    </Animation>
  </Animations>

The section starts with an <Animations> element, which can contain 0, 1 or more <Animation> elements. Each animation defines a series of ‘frames’, where a frame represents a single image in the animation. A frame contains a Duration section, an ImageResource and a VisemeGroupName. The last one, you can forget about, it’s there because of backward compatibility reasons with the original CCS file format and should always be empty (for now). We’ve already been over the ImageResource element and the duration simply declares how long that the image should be displayed. This is expressed in milliseconds.

Underneath the frames, you need to declare some more info about the animation. Besides the name of the animation, which is used to start the animation in a rule’s output patterns, you have:

It might seem a tremendous task at first to declare every frame in an animation like this. But it turns out that most animations can be build using between 3 and 7 images, sometimes reusing images to build sequences of about 10-20 frames. So all in all, this is still manageable.

VisemeGroups

The next section declares the images used for lip-syncing. The basic structure looks like the following xml snippet (note that it doesn’t contain an entry for all 22 viseme images, just the first 2).

  <VisemeGroups>
    <VisemeGroup>
      <Name>Default</Name>
      <ZIndex>1</ZIndex>
      <VisemeImages>
        <VisemeImage>
          <VisemeIndex>0</VisemeIndex>
          <ImageResource>images\Kima00.png</ImageResource>
        </VisemeImage>
        <VisemeImage>
          <VisemeIndex>1</VisemeIndex>
          <ImageResource>images\Visemes\Kima03 EH.png</ImageResource>
        </VisemeImage>
      </VisemeImages>
    </VisemeGroup>
  </VisemeGroups>

The file format already allows for multiple viseme groups to be declared, although at the time of writing, only the first one is used (and supported). Multiple viseme groups could be useful for moving heads: a viseme group for each head position.  As such, the ‘name’ element for each group would be used to reference a group. But, as already mentioned, this is something for the future. 

The ZIndex element defines the Z-order that should be applied to the viseme images. This allows you to manipulate the order of the images. For instance, you could use this to move a part of the background image on top of the viseme images.

Next comes the ‘VisemeImages’ group, with a ‘VisemeImage’ for each mouth position + optionally 1 extra image for the silence position.  Each VisemeImage contains an ImageResource (as described above) and an Index (VisemeIndex), which determines the letter that the image represents (so the order in which the VisemeImages are declared, is irrelevant).

The silence is primarily for backward compatibility with verbot characters (which don’t have a separate background section). If you have a ‘Background’ section, the viseme image at index 0, will be skipped, otherwise it’s used as the background. That’s why the ‘background’ section has to be declared before the VisemeGroups.

IdleLevels

Idle time is the time when a bot doesn’t have anything to say or any emotion to show. In other words, nothing’s happening. In order to create the illusion of being alive during this period, you can use idle levels to start animations (that were declared in the ‘Animations’ section) when the bot is idle. Here’s the definition:

  <IdleLevels>
    <IdleLevel>
      <MinStartDelay>5</MinStartDelay>
      <MaxStartDelay>15</MaxStartDelay>
      <MinDuration>5</MinDuration>
      <MaxDuration>15</MaxDuration>
      <MinInterval>2</MinInterval>
      <MaxInterval>8</MaxInterval>
      <AnimationNames>
        <AnimationName>earsmove1</AnimationName>
        <AnimationName>eyes squint</AnimationName> 
      </AnimationNames>
    </IdleLevel>
  </IdleLevels>

Like most other sections, you can again declare multiple IdleLevels. In this example, we only have 1 though. The idea behind multiple IdleLevels is this: when the idle time starts, the first idle level is activated, but when the duration of the level has ended, the next one is activated until the last one is reached, which remains running until some activity happens. This way, you can have a bot act differently after x amount of idle time, progressively.

Each idle level contains 4 bits of information: 3 time ranges and a list of animation-names that the idle level can use. Each range has a Min and Max component, indicating the lower and upper part of the range. The different times are used for:

BackgroundAnimations

Idle levels are very useful to create a sense of liveliness, but they can be expanded upon. Sometimes, it’s also useful to have animations run all the time in the background, even while speaking. A good example of this could be breathing or eye blinking. For this purpose, there is the final section, called ‘BackgroundAnimations’, which contains a series of special animation definitions, all of which will run all the time, in a loop. Here’s the definition:

  <BackgroundAnimations>
    <Animation>
      <AnimationFrames>
        <AnimationFrame>
          <Duration>10</Duration>
          <ImageResource>images\other\Kima Nose Flare (01).png</ImageResource>
          <VisemeGroupName/>
        </AnimationFrame>
        <AnimationFrame>
          <Duration>10</Duration>
          <ImageResource>images\other\Kima Nose Flare (02).png</ImageResource>
          <VisemeGroupName />
        </AnimationFrame>
      </AnimationFrames>
      <Name>nosebreath</Name>
      <LoopStyle>VarTimer</LoopStyle>
      <MinStartDelay>1</MinStartDelay>
      <MaxStartDelay>3</MaxStartDelay>
      <ZIndex>2</ZIndex>
    </Animation>   
  </BackgroundAnimations>

Unlike idle levels, the animation is declared inline this time.  That’s because there is a small difference in declaration between this type and idle/emotion animations. Another reason is to make certain that background animations can’t be used as emotions (they run all the time anyway, no need to start them separately).

The way that the individual frames are declared is the same as with regular animations: a list of ‘AnimationFrame’ objects that define the image, the duration and an optional viseme group. The difference is in the extra options: there is an extra ‘LoopStyle’ and ‘StartDelay’ range. ZIndex is also the same as in animations: It determines the Z-Order at which the animation is displayed, the higher the number, the more to the top it will be.

The ‘StartDelay’ range is used to build in a small idle time between each animation loop. Each time that the animation needs to be started, a value is selected at random from within the range. That’s the delay which will be used. Not every loop-style makes use  of this delay, some do, others skip it.

Finally, the LoopStyle element can have the following values:

And that’s it. Once you’ve got all these sections laid down, you’ve got yourself a character. All that remains, is to copy the files to the {my documents}\NND\Characters dir and restart the chatbot designer. Enjoy.

Projects

A project contains the entire neural network (in binary form) together with a bunch of configuration files in a directory. This directory is accompanied by the main project file. Both have the same name.  When moving around a project, make certain you have both file and directory. When opening and saving projects, you only need to worry about the file, the directory will be created/managed automatically (done in such a way that extra directories like for version control systems, will remain functioning).
The project files themselves (extension ‘.dpl’), are simple xml files which contain various configuration data for the designer. In general though, you don’t need to edit this file manually, all settings can be managed from within the designer.
The most important files in the project directory are binary and should never be manually edited. Some files are xml based, but apart from the test-cases, should be left ‘as-is’ since they contain vital configurations for the internal network.

Topic files

Topics form a major component of a chatbot designer project. They define what the bot can do. Internally, topics are stored as neurons, but in order to share a topic across multiple projects and users, Topic files were introduced. This is an xml file (extension ‘topic.xml’) that contains all the rules defined in the project.

The designer provides functionality to import and export topics one by one or in bulk from the File menu or the context menu on the Project tool-window. So ideally, you should never be confronted with the internals of these files. For the die-hard text-editor fans, here’s a little more details on the structure:

A minimal topic file should contain a topic, name, rules and questions sections. Like:

<Topic>
  <Name>Who has</Name>
  <Rules />
  <Questions />
</Topic>

And a bit more challenging, containing a single rule with 1 input and output pattern pattern:

<Topic>
  <Name>Time</Name>
  <Rules>
    <Rule>
      <Name>new rule 1</Name>
      <Patterns>
        <Pattern>
          <Expression>What time is it [?]</Expression>
        </Pattern>
      </Patterns>
      <Conditionals />
      <Outputs>
        <Output>
          <Expression>It is $time:hour\:$time:minute\.</Expression>
          <QuestionCanFollow>True</QuestionCanFollow>
        </Output>
      </Outputs>
      <IsOutputSequenced />
      <DoPatterns />
      <Calculate />
  </Rules>
  <Questions />
</Topic>

For the curious, the expression: $time:hour\:$time:minute\.  is used to render something like 17:30. The expression has to be split up in 4 parts:

time:hour	returns the hour
\:	we need to escape the : cause it is just after a variable path. If we don’t do this, the parser thinks that we will specify another function call.
time:minute	returns the minute part of the time value.
\.	The . also needs to be escaped, otherwise the parser thinks we are trying to further specify the path.

Global patterns

In the ‘chatbot properties’ view (From the menu: View/chatbot properties), you can specify a bunch of global patterns like the opening statements, fallback values, ‘do-patterns’ that need to be executed at startup, just after each statement,…  This also includes all the OS function-mappings that were declared in the ‘OS channel’.  All these can be exported/imported using 1 file, with the extension ‘global.xml’. Import/export can be done from the menu: ‘file/import/global patterns’ or ‘file/export/global patterns’. 

As you probably already guessed, this is also in xml format, but like with the topics, the content can be fully managed from within the designer, so there is no need to edit it manually.

Thesaurus files

Thesaurus files are used to share, exchange or replace part of, or the entire thesaurus that is used in a single project (thesauri are project specific and not shared across different projects within the designer). It’s the same routine again as for the previous file types: the extension is ‘thesaurus.xml’, yes, it’s an xml file format and can be fully managed from within the designer. In fact, for the thesaurus, it is more than advised to use the designer and not edit it manually since a thesaurus node can be hard to define as there are multiple levels at which words are stored. The only scenario where you might have to do some manual editing, is when you create a new thesaurus, completely from scratch when all the functionality needs to be preserved and you don’t have access to a network-designer version of the chatbot designer. In that case, the ‘bindings’ section on certain words (numbers, pronouns like I, me, myself,.. needs to be done manually. Here’s a small example of a binding for the object ‘myself’:

  <Object Name="myself">
    <Text Value="myself" Index="0" POSGRP="false" />
    <Bindings>
      <ref>I</ref>
      <ref>Singular</ref>
    </Bindings>
  </Object>

Importing/exporting is done from the context menu on the thesaurus. If no node is selected, the entire thesaurus will be exported, or the content of the imported file will be added to the root. If a node is selected during an import operation, all items will be added as a child of the selected node. For export, only the selected item and all it’s children will be exported recursively.

It’s also possible to import or export just a single level of children of a selected node, in the form of comma-separated-value (csv) files.  For imports, you don’t even need to get the data from a file, but you can just as easily use the clipboard: just copy a csv list to the clipboard, go to the context menu of the parent node and select ‘Import\wordlist from clipboard’. This feature allows you to quickly build up trees of related words.

Asset files

Assets store concrete information like ‘you are reading this now’ and ‘I have written this text’. By now, the theme should have become clear to you: asset files are stored in the xml format, they have the extension ‘asset.xml’. But here’s where we stray a little from the path: currently asset import/export is only supported in the pro version (that is, it’s still a little bit under construction, so it will soon be available). The same for editing: the pro will have an asset editor, but not the basic version. The latter only has the chatbot window for adding input statements.

Test cases

Test case files aren’t independent files like ‘topic thesaurus, asset or global patterns’ files, but are rather part of the project directory. As such, you can’t currently export or import them automatically from within the designer (though you can fully manage them as in copy, delete, edit,.. from within the designer). That said, they do deserve a special mention since it could sometimes be useful to extract some test-cases and copy them to other projects. And, because a test-case file is actually a stand-alone xml file, this is certainly possible.

Test case files are located at {project directory}\DesignerData\  and have the extension ‘TESTCASE’. If you copy a test-case from one project to another, or you want to delete some manually, make certain that the project is closed.

CCS files (characters)

As a final file format, I’d like to talk a little about characters and their files. A character is stored as an xml file (extension ccs) together with a bunch of images. These images can be located in a subdirectory of the ccs file, as long as the path can be described, relative to the ccs file.

For the chatbot designer to be able to use a character, the files (and possibly directory structure) of the char need to be copied to the directory: {My documents}\NND\Characters\CharName,  where ‘Charname’ is the name of the character. Best to do this when the chatbot designer isn’t running.

Unfortunately, ccs files can’t yet be edited in the designer (or any other UI tool). They need to be created manually. Luckily, the structure is relatively simple and allows for lots of copy/paste. More info on the file structure will come shortly.

So, with no further ado, here’s the basic version and also the pro (the latter will remain active until the end of the year). I will most likely also release a demo of the designer version – which allows full debugging and extending/replacing of the network – in a couple of days, after I’ve cleaned up some more demo projects.
I might also release a 32 and 64 bit specific version in order to support voices that were compiled for a specific platform, other than the one you are running on. The currently released versions will run at 32 or 64 bit, depending on what system you have.

Please, if you experience any ‘hanging’ situations (no reply is coming and in the lower-right corner, the second nr – with tooltip ‘The total nr of still active processors’ -  never goes back to 0), let me know. Different processors can give different results and I don’t have the resources to spend on different hardware setups, so I’m expecting some ‘issues’ in this area. Your help is much appreciated.

a word about the included demos

• Name & age: contains some patterns that demonstrate how to access the ‘name’ and ‘age’ settings which can be supplied in the ‘chatbot properties’ window.
• SysMan: demonstrates how to access external .net functions. It provides access to most of the File, Path and Directory functions.
• Thesaurus operations: shows how to manipulate (add, remove,…) thesaurus data  using do-patterns.
• Asset operations: shows how you can manipulate the memory.
• Complete the sequence: from the previous demo, shows how to perform the ‘complete the list’ trick.
• All: this is the start of a common, reusable library of patterns, most of which don’t even have output, but only manipulate the memory. I’m hoping that this can become the basis of a new approach to pattern matching.

Also, none of the demos includes thesaurus data. this can be imported from this thesaurus file. I’ve done this cause this data is still ‘under heavy construction’. The thesaurus currently contains a little more than 2000 words (not much), but can easily be extended using different import methods.

Thesaurus variables, sub-topics and InvertedWho

Next week, I’ll probably be spending some time putting together the documentation. In the mean time, there are a few tricks which were used in the common library that I’d like to mention.

Firstly, the whole thing is full of statements like:
^subj:noun.name
^subj:adj.possesive
These are ‘thesaurus’ variables. The first word (in this case ‘subj’) is the name of the variable (so you can access the values in the output or do patterns). The other words describe a path into the thesaurus. Any child of this path will give a match.
Thesaurus variables are very powerful, but also more taxing to the system compared to statics (though, usually less than regular variables). When you have lots of patterns that use thesaurus variables, like the common lib, it’s best not to let the system auto resolve synonyms. More on that later.

A second feature:  sub-topics. This packs a serious punch, in all ways you look at it. In short, it’s possible to reference a single rule or an entire group of rules (a topic) from within another input pattern, like this:
~subject (am|'m|is|'s|are|be) ~object
This single pattern can capture anything from: I’m Jan or my name is Jan over My aunt’s name is Rita to The big tree is a little bigger and anything in between. Even more interesting, this technique allows you to do something that I call ‘topic-inheritance’, which basically means you can extend or overwrite the behavior of patterns. I plan to use this technique to build a ‘Watson’ like chatbot on top of this common lib.

Finally, ‘InvertedWho’ simply refers to how the memory is used in the common lib. The basic topics like ‘subject’, ‘object’, ‘location’, ‘time’ and ‘numbers’ don’t generate any output, but store the data in 2 memory streams: the first forms the ‘inverted statement. So ‘I’ becomes ‘you’ and ‘mine’ ‘yours’ (guess for what that’s used). The second stream tries to store the actual meaning (which is a collection of references to other memory addresses or words grouped in an organized and structured way). These things are done in the ‘do patterns’. These are hidden by default in the editor, but can be made visible on each pattern (or shft + ctrl + d to expand/collapse them all at the same time).

There are many, many more details and cool features to talk about. Stay tuned.

Note: the video is best viewed in max resolution and full screen to see all the details.

I’ve been having a huge smile on my face all day

For the interested, here’s a screenshot of the rule that enables this trick (click to enlarge):

The important bit is the :complete after the variable $ToComp which performs the calculation.

Here’s another screencast that shows what’s happening behind the scenes (basically, it’s a walkthrough of the neural code in the designer):

Pretty cool He!. I think so as well. The character is another of Laticis Imagery’s creations. Ady provided all the images and I assembled them into a single character. The video demonstrates all the available expressions, which can be activated in the output using ‘mark’ ssml tags.

Perhaps some more information on the project: The first release, the basic version should be ready in a short while now, when I have created some content (which should also be a perfect opportunity to work out some of the final details). The basic edition will be a free (as in beer) version. After that, the pro will be prepared for release which will contain some more functionality like user interface automation, and/or home automation (not certain yet what to do first).

Some of the features that will be available in the first release:

• Select if the bot starts the conversation or waits for some input on startup. Opening statements can be declared in the bot’s properties page.
• You can declare custom memory operations that need to be performed each time the bot starts.
• ‘Do patterns’ are also executed each time output was generated.
• Input repetition is recognized (stored in memory as a counter) and can be handled with custom, conditional output patterns.
• When no patterns matched, the system will use one of the custom fallback outputs.
• Input patterns are grouped together into a single rule. These patterns share the same set of possible output patterns.
• Multiple output patterns can be declared for a single rule. You can select if a random item needs to be selected from the list or if each item needs to be used in sequence (useful for story telling bots).
• Each rule can have it’s own do patterns, which are used to manipulate the memory.
• Rules are grouped together in topics (the 2 files that are imported in the video, each represent a topic), which are responsible for providing context. This allows you to declare the same pattern in multiple topics (useful for short statements like ‘why, when, yes, no,…’
• Additional context can be added through do patterns and can be queried in conditions.
• It’s possible to declare conditional questions at the level of a topic, meaning that multiple output patterns can share the same questions. The first one who’s condition matches will be used for outputs that don’t declare their own question.
• A single output pattern can link to other output patterns, indicating that It should be used if the rule it belongs too, is the answer to a question declared in one of the linked outputs. This is useful to properly handle responses or when the user doesn’t respond as expected.
• Time and date are supported in the output and conditionals through a variable. When used in combination with the thesaurus, some pretty powerful things can be done.
• Test-cases for running automated tests on your bot.
• Synonyms are automatically resolved in the input. This is a very powerful feature that’s able to recognize and replace compound words in the input. For instance,  if an input pattern contains ‘what is’ and the system knows the synonyms for ‘what is’ are ‘whats, what’s, wats, wat is, wat’s’, then you only need to declare 1 input pattern to recognize all of the possible synonyms.
• Synonyms can be managed from the thesaurus editor.
• The following operators can be used in the input patterns:
  • () group input together
  • [] option: words between the brackets are optional, not required to be present in the input
  • {} loop: words between the brackets can be found 0, 1 or more times (useful for lists)
  • | choice: the input needs to contain either the left part or the right part of the choice. This can be combined with an option, group or choice, like: [I | you | he | she | we | they]
  • $name: variable declaration: collects words that can be used in the output or conditions.
  • ^path: thesaurus variable declaration: the input needs to contain a word (or compound) that is a child of the specified thesaurus path (very powerful). The actual collected word can be used in the output/conditions like a regular variable.
  • && the and operator allows you to declare groups of words that need to be present in the input, but which can have ‘holes’ in between them, ex: (hello) && (what’s your name)
• conditions and outputs can also use:
  • #path: declares a data-path into the memory.
  • ~name: to reference topics.
• There is a built in topic-editor or you can edit them directly in xml format.
• The built-in topic editor has a spell checker.
• Patterns with errors have a red line, making them easy to find. The error text can be seen as a tooltip or in the log.
• …

Looks pretty cool he? So, what happened? Well in short, I took the neural network designer, removed everything complicated from view that remotely had anything to do with ‘neurons’ but kept all the memory functionality, added a character engine and a simple but powerful pattern matcher (implemented in neural code, pretty cool I think,… and simple stuff).

The character in this demo (actually called Tara – and still in development) is designed and named by Ady Di Pierro from Laticis Imagery.  The drawings were made with DAZ3 and assembled in something very similar to verbot’s CCS file.  In fact, all verbot characters should work in this character engine as well, cause I only added features to the file format but didn’t change any existing.    The images were manually assembled for this character cause there is no char-editor yet.  That is scheduled.

Also check out what Roger Davie (aka Freddy, Admin of the AI  dreams forum) did with the forum’s bot:

The images are also rendered with DAZ. He definitely has the visemes already worked out better than in the first demo.

So, how did all of this come to be, you might wonder. Well, I think you can thank Wendell Cowart from the chatterbox challenge for this. He originally contacted me with a request for a new ‘pattern based’ chatbot project. I took this as a nice challenge to demonstrate exactly just how flexible resonating neural networks are. Soon, Patti Roberts of Bildgesmythe also joined in. Together, they basically told me how they would like to have things, which features they were looking for and such. Thus, this little project was able to come into existence at record speed.
Patti and Wendell also helped out a lot with the initial ‘mid development’ testing, which I’m sure you understand is a pretty frustrating job to do, as things are usually not yet behaving the way they are expected to. So many thanks for cracking out those basic ‘issues’.

Anyway, for those who would like to play with it for themselves, a first public beta release will most likely be coming shortly. Just keep in mind that any first release will be a ‘technology preview’ sort of say.

Note: Deprecated!

The next screen cast is ready. It demonstrates some of the features found in the ‘asset’ editor.

Note: Deprecated!

I’ve uploaded the first 2 in a set of screencasts on how to use NND. They can be reached from the Quick start page or you can view them directly from here:

Hope you liked them.

Note: Deprecated!

The time for a new release has finally arrived. Lots of new things:

The designer and core:

• lots of speed improvements and deadlock fixes.
• Added a new asset editor. With this editor, you can declare things like ‘An eye has a color and the color of an eye is usually blue green or brown’. This is done with a graphical editor, instead of entering text, so you can directly manipulate the network.  Of course, this can also be used to view recorded ‘asset data’.
• Added a new dialog box which allows you to merge 2 neurons into 1. This can be very useful in some situations.
• Extended the ‘frames’ editor, so that it can also edit semantic frames (which are new in this version of Aici).
• Added a ‘bool filter’ to the frames editor
• As I’ve previously mentioned, there’s also a new ‘test case’ debugging tools, which is great for automated testing and validation (using regular expressions) of large input volumes. Running big tests can be useful to pick up unexpected behavior that got introduced because of newly added frames.
• I’ve also added support for ‘code search’. You can now search for all occurrences of any neuron within a code cluster in the entire network. To search for a neuron, select it and press ‘Ctrl+F3′.
• There’s a new version available of the ‘echo words’ demo, which uses less code (3 statements).
• New instructions: GetCommonParents, GetCommonParentsFiltered, GetCommonParentsWithMeaning
• Lots of bug fixes + introduced some new ones

Aici:

• A brand new semantics stage (again), which supports complex sentences and provides more possibilities.
• Added a bunch of semantic frames for things like ‘place’, ‘time’, some verbs, nouns and actions.
• Added a bunch of new grammar frames, mostly for complex sentences, but also for things like ‘where’, ‘who’,…
• Added new actions, mostly for complex sentences, but also for better support of verbs.
• cleaned up/improved the ‘be’ and ‘have’ action handlers a bit.
• lots of new words
• … (I sort of forgot to note everything I did near the end) For a good overview of what’s currently possible, check the conversations category.

And naturally, he latest version can be retrieved from the downloads page.

Well, it’s an answer

Note: Deprecated!

Debugging time!!! (when isn’t it.) I sort of forgot about this one, until a couple of days ago, when I ran into a similar problem as for which I originally created this (ahem) simple little trick.  To explain you what attached neurons are for, perhaps first a little explanation on how I have been designing my neural algorithms. (Caution: this is advanced stuff, not for the faint-hearted).

As with all algorithms, I take some input and process it to generate some output. The input with neural algorithms, is always expressed using neurons. The process to execute is defined through the meaning of a link (which also maps to a neuron). The link starts from the input neuron and usually points to some value that needs to be compared against or searched for,…, but this can also be something else.

The problem is the ‘split’ instruction. This is a very powerful thing indeed, which allows me to write simple, massively parallel code. It does, exactly as the word says, it ‘splits’ the current process into x nr of similar processes. They are similar, but not exactly the same: ‘the input neurons’ have been duplicated so that they contain the same values, but using new neurons, so that the process will eventually lead to a different result neuron. Also, the ‘variable’ that was supplied with the split instruction, has a unique value in each process (the value items are declared in the split instruction, they  determine the nr of new processes in the split). And finally, globals can be declared  as having to ‘duplicate’ their content when a split occurs. All this means that each process is similar (most variables point to the same neurons), but not exactly the same. (Mount Everest stuff, I know.)
The key thing to keep in mind here are the duplicated input neurons: when a split occurs, the idea is that each process will work on it’s own input and result data. But, sometimes, for various reasons, one or more of these neurons can spill over to other processes. And that’s when ‘contamination’ can occur, or when one process modifies the input/result of an other process, which is very bad indeed and excruciatingly difficult to trace using traditional debugging methods like breakpoints.

Thus come attached variables to the rescue. The idea is very simple: Whenever a ‘variable’ that has been signaled in the designer as ‘attach required’, gets it’s first value assigned, it attaches this ‘value’ to the  process in which this assignment occurred. If, from that point on, another  process tries to change  this value-neuron, all alarms go off as if it were WOIII. And that’s basically it. From there on, it’s up to you to use the presented information (where and who), to figure out the why.

For practical use, I have found that the ‘CurrentFrom’ system variable, really is the most useful variable for this trick, though it is possible to put it on any other. You can even attach neurons manually to a process, if you’ve paused and selected one. This is done on a code editor, using the context menu item ‘attach to  processor’. To activate the monitoring of a variable, make certain that it is added to the left most pane in the debugger view and select checkbox  in the ‘attach’ column before you provide the network with some input.

Like I’ve already mentioned, it doesn’t take much to use and activate it. This little trick’s usage field isn’t very broad, it only traces down 1 specific type of bug, but one that is very hard to find otherwise. Also, if not used correctly, you can get a lot of false errors, since it’s very common for multiple processes to modify the same neurons. So use with caution!

• I had never made something like that, while I had plenty of experience designing languages,
• I didn’t really know what the end product would look like or how it would behave,
• and apart from some ‘UML designers’ that render classes, I had never used such a thing for editing code myself.

So then, what made me decide to use this approach instead of the more conventional and traditional way of using text?  First of, a major part of the answer lies in the previous statement . Other reasons were:

• Guided editors (can) lower the entrance level, when done properly (something I needed,… seriously).
• I new I would be needing some good debuggers (lessons learned!!). I could try and incorporate these into some existing editors, or turn out one of my own. At the time, it seemed a better idea to have full control from top to bottom, if I wanted to do this properly.
• I had a bit of experience working in WPF, which gave me some confidence that it was possible.
• Also, I came across this, rather serious attempt at structured editors (originally for C#), which raised my hopes as well.
• Did I already mention you that I was hoping this would make things a bit simpler for other people to understand?

In the end, it turns out I am glad I decided to use this approach, though not exactly for the reasons as previously stated. Except perhaps for the debugging, which really was fairly easy to implement this way. No, the thing that convinced me was the way that it changes how you program. This didn’t happen over night, I only began noticing this the last couple of months, after some 1,5 years of (trying to) program this way.
So what changed: well, over time, when I needed a ‘variable’, instead of thinking ‘variable’ I began to think ‘purple’. This is a shift from thinking in words to thinking in color. The best evidence for this, came in the form of me, continually trying to put a conditional in an assignment as if it were a global (cause I thought I had picked up a global). If you look at the picture to the right, you can probably understand why I erred: the blue colors are just to close to each other.
What’s more, over time I also began to think more and more in visual structures instead of ‘if-else’ or ‘while-do’. When I think about an ‘if’, I think:

In short, when programming, I am thinking more and more in terms of: blue square, red hexagon, white circle with line below (the conditional part),… Or in other words, it feels as if I am programming like building with Lego blocks!!!
I can’t explain exactly how or why this is, but it appears to free the mind for thinking about other stuff. At times, I find myself, just mindlessly putting blocks together to build up the picture that I had previously constructed in my head.  Very pleasing.

So, as a first try, I am very happy with the result, well enough for me to continue this path, and with some minor corrections here and there, keep the design. That is not to say there isn’t room for improvement, I have bucket-loads already of those, just not the time, so most of these will not be for the near future.  Others might be implemented sooner, depending on needs.

Basically, most of the concerns are centered round work speed (the number of movements and clicks required for an operation), maintainability and overview:

Speed

• To start with, the toolbox needs some redesign. I currently have 2 pages: one for general items, and a separate tab for all the instructions. I find myself continually switching between the 2, which is time consuming and unnecessary. The solution is simple: split them up in separate toolboxes.
• The toolbox currently only supports drag-drop operations. So to get 5 objects of the same type, you need to perform 5 drags. Better would be to have a ‘paint’ tool, like graphical apps, and many (UML) designers have.
• The code designer really could use some more keyboard support, so it can be used like a regular text editor (sort of). This seems a logical next step to do, just haven’t gotten there yet.
• I have been thinking a lot about using WPF’s new multi-touch libraries to add support for using 2 mice while coding. I don’t really make use of the left hand during the GUI coding, apart from the occasional copy/paste or editing a label. At times, I could definitely work with 2 hands at the same time. It’s a bit like using both hands for searching and placing blocks.
  I had also thought about using this new kinect toy, which appears to have been hacked already for PC usage. This would even resemble more like building with Lego (or like Tom cruise is doing in Minority report) . But then I figured: ok how would you feel after 2 weeks of keeping  your arms up in the air? Ok, I don’t know how I’d feel after 2 weeks like that, but my shoulders and neck definitely do!!!
  So that plan was canceled.

Overview

• As I’ve previously mentioned, I often simply recreate the picture that’s already in my head. The annoying thing is that the designer only shows a small part of the object (algorithm) that I’m working on, while at times I have more a need for a birds view approach. That’s why I added the zoom support, but more can be done such as:
  • make the code templates (the drawings) a bit smaller. I’m no good at designing, which shows.
  • provide a bread-crumb like ability (for drilling down and going back up again), much like the windows file explorer provides using it’s breadcrumb approach for directories.  I could use this for sub blocks.
  • See what I can do about providing support for multiple screens.
  • Buy a new big sized monitor, but given the massive resources at my disposal, that’s not an option
• Also previously stated: some of the colors can use a little tuning, the same perhaps for some shapes, to make them more distinguishable from each other.
• Currently, the drop area for the arguments is displayed as a simple, single square. The thing is: the argument requirements are known for each instruction, so it’s perfectly possible to display a custom set of boxes, with correct labels, for each instruction. This, by the way, would also free some space, since no drop area needs to be shown when all arguments are provided.
• I could also try to implement some sort of 3D-view on the network model, but I think I’ll need to extend my skill-set for that one.

Maintainability

The single major issue I have with code maintainability is concerning documentation. I added support for this by allowing to add some ‘rich-text’ string to each neuron. The idea being that you’d put all the comments concerning what’s going on, in there. Unfortunately, it doesn’t really work very well for code documentation because:

• neurons can be reused. This is different compared to regular text code: when this is compiled, each line of code will get it’s own output (or better, each compiled instruction corresponds with exactly 1 location within the text). With this neural network however, things are a bit different: it’s perfectly possible to use the same assignment in 2 (or more) different places. This is something I do extensively, by the way, to speed up coding and to keep the nr of links down.  What I noticed is this: though the code is shareable, the reason usually isn’t. I am currently solving this by stacking the different comments in the same text-blob, when needed. But better solutions are possible.
• I also find it annoying having to select an object in order to see it’s comments within the code view. It appears to be easier/better when you can quickly scan a document for all comments and read the one you need, without thinking about selection.

To solve this, I have been thinking about adding support for comments within the ‘project’ documents. These would maintain and display (as call-out balloons) all the comments for the neurons that they contain. The location of the comments would be determined by the comments themselves, so they can be floating, closest to the neuron that they are attached to. This would also allow me to display the comments, even if the actual neurons aren’t visible, but collapsed.

Final note

As a final note, for those who would like to build their own structured editor in WPF, perhaps a word of advice: stay away from ItemsControl or any of it’s descendants. The paradigm sounds grate: manage a list of items and display them any way you want through styling and templates. What’s not mentioned is that this only works for 10 objects or so. Any more, and your app slows down to a crawl or worse: crashes just as bad as the good old win32 apps would, perhaps even worse at times. Instead, go lower level: inherit from panel and roll out your own Measure/Arrange or go even further down and overwrite the rendering routines.

Don’t ask me how I figured it out, but yesterday, I decided  nothing could be wrong with those lines of code and instead, something in the neural code has to be wrong, more specifically, with the variables. So I built myself a better view on the memory content of the processors, and lo and behold, there they are: 3 innocent globals that get carried over from one algorithm to the next, causing all sorts of mayhem. In total, the fix took about 20 seconds. Which leaves me with mixed feelings: I’m soooo glad I finally found it. If only I had looked in the right place 3 months ago.

Note: Deprecated!

An introduction

In short, a flow is a template definition of recognizable data. It is used to transform a stream of neurons into another stream of neurons (doesn’t sound very useful, but trust me, it is).  To get a visual image on how a flow works, you can perhaps think of those toddler toys where the child needs to fit different types of blocks into different types of wholes (see image).

Well, a Flow is the yellow filter at the top of the box. As the blocks pass along, a flow tries to find a set that fits it’s filter. When it finds one, it collects this set, tags it, and stores it for further use. This process is done by the flow recognition algorithm.

A flow is a template definition, used to recognize sequences of data in a stream.

flow items

So, what is this filter that’s supposed to recognize the data, actually constructed of? Well, flow definitions are very (mmm, perhaps not so very) similar to Coco/R definitions. In short, you use constants, other (nested) flows, loops and options to build up a definition. Here’s a short example:
This flow can recognize the sequence: Agent verb possibly followed by a not. It’s actually the definition for the simple past tense (there’s an extra hidden filter on ‘verb’ which I’m not showing yet for this example). ‘Agent’ is underscored, indicating that it is another flow, verb is not, so it’s a static, like not. Which is surrounded by strait brackets, indicating that it’s optional. So basically, this flow can recognize statements like: I swam, you dug, the brown men found not,… Without the filter (like here), sentences like: I fish, they eat not,… would also be recognized.

Statics and nested flows

Most of the items in a flow definition tend to be statics and other, nested flows. This forms the meat, the data that can be found in the input stream itself. When 2 statics are declared in sequence, the same sequence needs to be found in the input, without anything in between.

Nested flows can easily be recognized. They are underlined. If you double-click on them, you will jump automatically to the flow (you can navigate back and forward between the selected flows using the navigation commands). Behavior-wise, nested flows are the same as statics. If you declare 2 nested flows in sequence, the input stream should contain the data for the 2 flows in the same sequence.

Floating flows

A floating flow is a special type of flow that can be recognized anywhere in the stream without having to be specifically declared in any other flows. They are mostly used for things like spaces and newlines. You usually don’t integrate them into other flows, as nested.

Floating flows can be destructive or not. When a destructive flow is found, it will break up the recognition of other data in the stream. So this type of flow can not be found between 2 statics, but can be between 2 nested flows or between options, loops or conditional parts (see further). If you need to have a floating flow between 2 statics, it needs to be non destructive. The type of flow can be assigned from a context menu on the flows: you can select for normal flows (not floating), floating (visualized by a green bar in front of the flow) and non destructive floating (displayed using a blue bar).

You can also specify if the data that a floating flow finds is discarded or not. Discarding the data can be useful in cases where you are really not interested in it, like for empty space. This is also selected from the context menu on the flows and is visualized by using an extra bar at the end of the flow.

Options

With an option you can define a number of different paths, from which 1 can or has to be selected. The following example of options defines the ‘can’ verb:A blue block (also called ‘conditional part’) represents a single path. If the frontal bracket of the options contains a vertical line, 1 path has to be selected, if there is no line, no path has to be selected but 1 can be. The content of a conditional has to be recognized in the same sequence as the way that it is defined and can consist of statics, flows, loops and other options, but no conditional parts. The option itself can only contain conditional parts.

Just like statics and nested flows, you can also reuse the same option in different places. In fact, the more you can do this, the more processing power can be saved. The same goes for the conditional parts (the blue blocks). These can also be shared by different options (and loops). This can be done by drag and drop from within the editor.

Loops

Loops are very similar to options, they also consist of conditional parts and they can also be defined with a vertical line in front, indicating that 1 path has to be selected or not. The main difference with options is that a loop can recognize 0, 1 or more of it’s conditional parts in sequence. They are declared using curly brackets, like in the following example: This loop simply declares a sequence of digits, with at least 1 required digit. In other words, it’s the definition for an integer.

Loops can have some more lines in the front to declare extra info. If there is a green line present, to the right of the black one, it means there can’t be a floating flow in between 2 conditional parts, otherwise this is allowed (see floating flows). A red line to the left of the black one, indicates exactly the opposite: 2 conditionals need to have a floating flow in between them for a valid parse (not often used).

As a final example, here’s the entire scanner definition which is used to recognize words, numbers (integer and doubles), signs and spaces:

Note: Deprecated!

It’s time again to do a new release, which was long overdue to tell you the truth. Lots of things have been changed, though mostly in the background, both for the designer and Aici. Here’s a non exhaustive list of the changes:

The designer + core:

Basically, I began the process of improving the feel of the application, to get the prototype taste out of it. It’s not done yet, but things are improving considerably. I am now able to work with the designer for a full day, while keeping memory reasonably under control and with the same speed at the end as at the start. here’s what else got changed:

• Lots of speed improvements (more to go)
• fixed some memory leaks, there are still some more left to plug.
• New instructions: IsClusteredByAny, ChangeChild, ChangeInfo, ChangeParent, Avg (average), Max, Min, StDev
• Added system events OnStarted, OnShutDown, OnSinActivity.
• Added a new system variable: Time (to get the current system time).
• Added back/forward commands to the flow editor.
• Added the ability to manage 1 to 1 thesaurus relationships (from within the thesaurus tool window).
• Added ‘go to unfreeze location’ functionality in the memory profiler.
• Created a custom treeview, currently only used by the thesaurus. There are still a few issues left with the new view, but it fixes some unresolvable issues found in the default (wpf) treeview, so all trees will be replace with this new one in the next releases.
• lots of bug fixes

Aici

Aici has undergone a major upgrade, though mostly in the background, that is to say, to improve already existing functionality or extend it. Here’s what’s new/updated:

• There’s a new stage, just after scanning and before parsing, used to retrieve the different sentences and groupings from the input. This allows the system to process multiple sentences in the input (at the same time). 1 small note: the final stage hasn’t been tested yet, so even though the sentences are recognized, only input with 1 sentence is processed correctly at the moment.
• I’ve changed the semantics processing from a simple transition/translation to a full stage, so the semantic information can be used to do more filtering.
• Yet another new stage has been added to do grammatical analysis (things like figuring out whether an ‘a’ is most likely used as a noun or article.
• The output stage has been adjusted so that the output isn’t directly sent out, but instead stored as neurons that can be used for further analysis while being sent out at a later stage. The analysis itself however isn’t done yet.
• Some new frames have been added, mostly to handle invalid grammar and present/past articles.
• The flow definition has been simplified a bit.
• Lots of bug fixes

Aici-Web

And finally, the newcomer: Aici-web which is a web-based interface of Aici (go figure!). It’s currently implemented in Silverlight (other web technologies should follow). The basic purpose of this website version is 2 folded: as a technology showcase and as a way for me to easily get conversation logs that I can use to improve the network. So go ahead and test the limits, see how fast you can break the thing (shouldn’t be to hard ). You can find it here. (Note: I plan to reset the backing database on a regular basis, to keep the thing fresh and with the latest Aici version.)

• Scroll & load speed: Even with full virtualization turned on, scrolling crawls almost to a halt when you have 60 000+ root items and a few of them are expanded some levels down the tree. The same for loading large lists.
• Memory usage: probably related with the previous item, in short: kaboem!!!
• BringIntoView: Here’s the killer, when virtualization is turned on, and the UI TreeViewIem hasn’t been rendered yet, you can’t bring it into view. Translated: you can’t do a search and show the found items in large trees, only in small trees (go figure).

Note about my test data:  The thesaurus demo contains about 100 000 words (syn-sets actually), almost all of which have multiple relationships, so  in total, there are +500 000 nodes in the tree.

Ok,… and how do you fix this? Well here’s how I did it:

Container rendering

Virtualization means that you only render the UI elements that are currently visible, and not the whole tree (or a large part like microsoft does). This means that, for each UI object that got rendered, you somehow need to keep track of the data position in the tree. I do this by using 2 cursors: one for the first and another for the last visible element. This cursor is nothing more than a list of indexes into the tree.

All items in between those 2 cursors are flattened out (by rendering UI elements for them). The nr of indexes (or nr of parents) for each item, is used as the level for the UI element. This determines the amount of indentation to the right that is applied to the UI element.

When the user scrolls down, the bottom cursor is updated (advanced by the amount that the user scrolled). This is a basic tree walking algorithm. From this bottom index, we render objects while going up the tree until the panel is full (again, the same tree walking algorithm). The index path of the new top item becomes the new top cursor. The same technique is used to scroll up, but instead, you start from the top cursor and fill downwards.

Remark: Watch out while calculating the scroll difference: The scrollbar value is usually a double while we only consume integer values, so there might be some left over in the scroll difference which has to be consumed the next time you calculate the dif.

Of course, this technique only works if the TreeView has some knowledge of the data structure, how else can it find the children of the root items and see if they are expanded or not? This requires an interface. Here’s how mine looks like:

   public interface ITreeViewItem
   {
      bool IsExpanded { get; set; }
      bool IsSelected { get; set; }
      bool HasChildren { get; }
      IList TreeItems { get; }
      ITreeViewPanelItem ParentTreeItem { get; }
      bool NeedsBringIntoView { get; set; }
   }

Each tree node should implement this interface so that the TreeView can properly traverse the tree. A few properties are mostly for the visual parts, like ‘IsSelected’ or ’HasChildren’ (to display the toggle button). The last property is perhaps a bit of an odd one: ‘NeedsBringIntoView’. It’s a quick method to let the Tree know that an element needs to be brought into the viewable area. You could also use a function on the TreeView (like microsoft does), but then you are forced to go find the UI element from your search code to bring the item into view, which I wanted to avoid.

Using this technique, you do create and destroy a lot of containers since each time the user scrolls, the entire visible area is re-rendered. To minimize the impact of this on the garbage collector, it’s probably best to use some recycling scheme. This doesn’t have to be fancy: a simple queue to put the items in when clearing the list which can be consumed again when rendering the containers. I chose a queue since it gives best chances to reassign the same container to the same data object when possible, which minimizes the impact of re-rendering on the binding system.

Events

The second important thing for virtualized TreeViews, is the event system. You can not simply go and hook up event handlers to every data element. This would blow up the memory. Also, you’d have to write a lot of code to keep track of those events as data objects are being created and destroyed (trees of such a size can only work if you combine UI virtualization with data virtualization, but that’s an entire subject all together). On the other hand, events are VERY important to get UI virtualization working for keeping track of the maximum scroll count, tree nodes that get expanded/collapsed, objects that need to be brought into view, and UI elements that need to be updated.

Are you seeing the bubbles already? Yep: use bubbling events, like WPF does. The only problem: that doesn’t exist for regular objects, it’s only available in the WPF world. You could inherit all your objects from DependencyObject, but with 200 0000 objects in memory, that somehow becomes bloated. Better to roll out your own sleek, mean and fast event mechanism, which isn’t hard at all to do. Here’s a little sample code to bubble events up a tree:

public static void OnPropertyChanged(IOnBubblingChanged sender, BubblingPropertyChangedEventArgs e)
{
   while (sender != null)                                  //use a while loop to avoid recursive calls.
   {
      sender.OnBubblingPropertyChanged(e);
      IOwnedObject iOwned = sender as IOwnedObject;
      if (iOwned != null)
         sender = iOwned.Owner as IOnBubblingChanged;
      else
         sender = null;                                    //need to make the loop stop
   }
}

This function is called whenever a property on a tree item is changed. As you can see, a lot of interfaces are at work here. IOnBubblingChanged defines a function that can be called which will raise the event. IOwnedObject is one of my core interfaces for all objects that can be ‘owned’ by another object. This maps to the ‘ParentTreeItem’ property of the first ITreeViewItem interface.

A similar thing needs to be done for the CollectionChanged events. A remark on the ‘reset collection’ event: make certain that you can pass along the list of items that got cleared, otherwise the TreeView will have a hard time figuring out exactly how much the count needs to be decreased (were any of the removed items expanded?). The default WPF NotifyCollectionChanged event doesn’t do that, so you can’t simply use it’s event arguments.

With this type of bubbling event system, it’s probably best to work with some kind of ‘root’ object that defines the events for the property and collection changes in the entire tree. Again a nice job for an interface, I suppose. This root interface can be used as the ItemsSource for the treeview instead of a list, like WPF does. This way, you can have lots of root objects without overtaxing the .net event management system.

Using this technique, you can make Treeviews that aren’t only fast to load and scroll, but also use very little memory. I am silently hoping that microsoft will eventually pick up on this and begin implementing proper UI virtualization in the coming versions of WPF (thus relieving me from the burden). In the mean time, I guess we are left to roll out our own stuff.

Note: Deprecated!

The designer is a strange beast: the project that is opened for editing, is also running in the background as a neural network. This means that you can use the input/output channels just like in a regular (neural network enabled) application, like the AICI client app. Grate for debugging and trying out stuff, but you don’t want to get your test data mixed into the actual network, enter sandboxes:

A sandbox is a complete copy of your project, running in a new, separate designer.

So, after you have opened the Aici project in the designer (installed at {Documents}\NND\Demos\AICI_1, note that you need to select the directory, not the file), you can run your project in a sandbox. Simply click on the button in the toolbar, or select ‘debug/sandbox’ from the menu. This will start a new designer with you sandbox project loaded. Once the sandbox is open, you can use the already declared text-channel to communicate with the network. if not yet opened, switch it on by selecting ‘view/communication channels/Text channel’ from the menu. The Echo words demo has more info on this type of channels (though the view has been updated a bit). here’s the short version:

The toolbar:
1: Send the input text to the network.
2: a combo for selecting the method to send text to the network.
3: copy the conversation log to the clipboard.
4: save the conversation log to file.
5: clean the conversation log.
6: toggle audio on/off.
The edit area:
1: All the neurons that were sent to the network as input.
2: All the neurons that were sent back from the network as output.
3: The conversation log.
4: The input box

In short, simply type some text in the textbox at the bottom, press enter and watch what happens.

When you are done testing, simply close the sandbox app and you are done (it wont ask to save any changes). If you would want to do some more testing later on with the same data, you can first save it to a different location (otherwise the sandbox data will be deleted when you start a new sandbox).

Note: Deprecated!

It’s been a while, a bit longer then planned actually. As usual, there was a little unexpected detour required. This time, it was called the ‘neuron profiler’. Don’t ask, in short: I couldn’t find a bug, which would be traceable with a profiler thingy. The good news: found the bug, the bad: found some new. Anyway, here’s what’s added, updated or fixed (sort of):

The Designer:

• A new neuron profiler, very similar to a memory profiler: it tracks down leaked neurons (which were created but not destroyed). This is a powerful little debug tool indeed.
• the new flow editor is working. This is a lot faster now. Some bugs should also be fixed (maybe some new were added as well).
• Search on all editors and tool windows has been made into an asynchronous process.
• I’ve added a new clean up dialog box (can be found under ‘tools’), called ‘clean orphan flow data’. Which can be used to trace down and remove flow data that should have been deleted, but wasn’t and is now simply causing the parse to fail.
• cleaned up the module import and export a bit, but not yet completely ready.
• The Window manager has been updated.
• The UI has been cleaned up a bit. I’ve tried to make it esthetically a little more appealing, hope you like it.
• Some speed improvements in the core, more to come though.
• I’ve reloaded the thesaurus demo, for 2 reasons: 1, the first demo wasn’t loaded correctly (had forgotten to turn on to include items with multiple words) and secondly, I mixed up the meaning of hyponym and hypernym in the thesaurus view (not in the Wordnet-channel view). A typical mix-up for me, a consequence of flying solo, I guess. Anyway, this project should be very useful for stress testing.

AICI:

• You can now call .net functions from within the network. As a test-case, I’ve implemented the ‘file copy’ function (System.IO.File.Copy). At the moment, Aici requires some code for each known  .net function to extract the arguments from the network’s input data. I plan to change this into something more generic, so that Aici knows that a function requires arguments ‘x’ and ‘y’ without having to lay out custom code for each function.
• There’s a lot less neuron leakage going on. The flow recognition algorithm is almost completely clean (except for 1 remaining issue). The latter part of the process still needs some work.
• I’ve cleaned up a lot of the code, removed bugs all over the place so that it becomes more responsive.
• The flow recognition algorithm has also been updated to remove some more bugs that were causing the parse to fail.
• The flow definition itself has also been updated considerably. In part so that it would be able to parse complex sentences, but mostly to get the speed up (dramatic results can be achieved with a correct filter, early on in the parse).

All in all, I think I’m starting to see some light in the distance, finally. Perhaps it’s almost time to move to beta stage. )

For instance, the abstract can be ‘a house’ while the concrete is ‘The house that I live in,… my house. Aici needs a way to distinguish the 2 and be able to find relationships between both. This is done through the use of different data structures.

Objects

Lets start with the objects, these are the simplest. They represent abstract knowledge. Every interpretation of every word is represented by an object. For instance, ‘house’ can mean the house that you live in, but could also refer to a musical style. So there is an object for each interpretations of the word. Furthermore some words can have synonyms, words that have the same meaning. For instance, ‘house’ and ‘home’ could be considered as synonyms, or house and ‘house music’. With ‘house music’ being a compound word: 2 words joined together to form a new meaning. Other examples of compound words are: car factory, fire hose, film studio,…

Another bit of information that can be useful to know about a word, is how it should be interpreted in the context of a sentence: can it be a noun, verb, adjective,…. Sometimes, a word can have multiple interpretations, while being used as the same sentence type (like house), sometimes multiple sentence types are allowed for a single meaning (colors for instance can be used as adjectives or nouns: ‘my eyes are blue’, ‘that blue is pretty’ ). As a speed optimization, we group all the objects together that have the same part of speech, but with different meanings in ‘Pos-groups’, though this is not required.

As a visual example, let’s take the objects ‘hello’ and ‘goodbye’ (as in a greeting). Both are nouns (it’s an hello and a goodbye). Both have multiple synonyms: hi, howdy, bye,..  So here’s how this would look like:

Relationships between objects: the Thesaurus

Objects are stored in a thesaurus like structure to indicate relationships between them. For instance, both ‘aici’ and ‘jan’ are names, so  this could be expressed in a thesaurus relationship like in the 2 images below:

Notice (to the left) that the ‘noun’ filter was selected in the thesaurus together with the ‘is a’ relationship. To the right, you can see how this is represented internally: Name points to an ‘is a’ cluster, which contains all the related objects. So the ‘Noun filter works by only displaying objects that point to ‘noun’ or are clustered by a ‘noun’ pos-group while the ‘is a’ relationship will filter out all but the clusters attached to the roots using this ‘is a’ neuron.

Sometimes a root object doesn’t have any children (as in cluster-children, not in the thesaurus). This makes the object a dummy, a placeholder that shouldn’t be used for input or output directly, but only for filtering. Some examples of these dummies can be seen when the ‘pronouns’ are selected.

The thesaurus relationships can be used in frames to filter the input in a general way and to perform lookups by the actions. For instance, the ‘be name’ frame (in the ‘names’ editor) has an ‘object flow’ element that filters on ‘name’ through the ‘is a’ relationship, as seen in the image below.

This means that the frame is only selected when the input contains the result of an object flow that is the child of an ‘is a’ cluster, attached to ‘name’ or one of it’s thesaurus children (so multiple levels are allowed).

To create this type of filter by the way, you simply drag ‘name’ from the thesaurus to the filter area.

Actions also make use of these thesaurus relationships in a very similar way. For instance, an action could check if an object  is allowed to/can be the name of something (‘my name is Jan’). Or, in sentences like: ‘this is blue’, the thesaurus is used to find out what ‘blue’ actually is: a color (I’m jumping here, keep with me, It will become clear very soon).

An object can also be the child of multiple items in a thesaurus. For instance, ‘blue’ is both the child of ‘colorful’ and ‘sad’ (child of ‘emotional’). Note that these are all adjectives, hence we say ‘colorful’ instead of ‘color’. To find out which relationship to take, we can make use of a log of previously made statements or a list of focused objects. The most important usage of either color(ful) or emotion(al) in this log, triggers it for the new input. Most important can be interpreted very broadly: it could be ‘newest log item’, or ‘most talked about’,… If it is impossible to solve this, a question needs to be asked.

Anyway, once you have figured out which path to take in the thesaurus: colorful, we still have an adjective. If the same logical concept/idea can be expressed in multiple parts of speech, using different words (like color-colorful) and you want to be able to jump between the 2, you need to have a relationship between them. As I already stated, we interpret ‘blue’ as a ‘color’, not ‘colorful’. Why, will become more clear when we talk about the assets, for now lets just take it that we need to change from adjective or verb to noun. As you can see in the image, the thesaurus has a special section to define this type of relationship: the left part of the table contains the link meaning, the right section is the ‘from’ part of the link.

The same can be done for verb conjugations. In English, a verb usually has 3 different conjugational forms, other languages have others, so the thesaurus allows you to define this type of relationship free form, much like the part of speech relationships. When you create a new verb though, the designer will try to fill in all the conjugations (if it finds neurons titled ‘present particle’, ‘past particle’ and ‘third person present’. It’s always best to check them before storing the default though, since it doesn’t know about irregular verbs and sometimes it/I may simply goof up.

The verb conjugations, compared to the part-of-speech relationships, are used a bit differently. These serve their purpose while parsing the input and generating output. Some flow elements try to search for conjugation relationships while filtering (in the ‘FilterCode’ tab) in order to get the correct parse. For instance, in the ‘model verbs flow’, this is used to make certain that a verb is the present particle. The same goes for generating output: when a verb is used in certain situations (ex: ‘what’ contains a verb instead of a noun), we need to generate it’s present particle form (or another, depending on the exact situation).

Assets

Up until now, all relationships were between objects, so between abstract knowledge only. To record a network’s experiences (the data that it records), we use assets and asset relationships.

At it’s core, an asset is a cluster with meaning ‘asset’ (we’re very original here), which contains a number of child neurons that represent property-value pairs for that asset. This is done through links to objects and/or other assets, using the meanings: attribute, value, amount,… From this basic structure, we can start to play and create meaningful relationships.

As usual, a picture says a thousand words, so to the left, I’ve displayed a debug view of the ‘text channel’ neuron (the text input channel of AICI). Why this neuron? If you look closely, it has a link that points to an ‘asset’ cluster, using the meaning ‘Pr:You’. So with this image, I’ve got 2 birds with 1 stone: the asset and how to find the asset that AICI is using for the person on the other side of the channel .

A text channel stores a reference to the asset that represents the entity it is communicating with using a link with meaning ‘Pr:you.

As a side note, linking ‘you’ to the text channel, allows the system to talk to multiple users at the same time: each on their own channel.

So what’s stored in this asset anyway? As you can see, there are 2 child neurons in the asset, so 2 data items: the first has an attribute (property) of ‘entity’ and a (property) value of ‘animate entity’. The second asset-child  points to ‘eye’ for the attribute, the value is another asset and it has a general amount of ‘plural’.

Note that the attribute should always be a noun. Having a fixed transformation point facilitates the search.  Why a noun: well, all adjectives and action-verbs can be converted to a noun, but not the other way round. This conversion can be done using the part of speech relationships that were previously described.

The ‘entity’ property is how AICI tries to classify assets: is it an object, animate, animal, human,… If you look at the thesaurus (in the designer), ‘entity’ is a noun-root and the asset-child’s value is one of it’s children. This is a rule: when the value is an object, it must have a thesaurus relationship with the attribute. This type of child asset represents the ‘x is an y’ type of relationship. Here, it is ‘I am an animate entity’, an other example could be:  ‘I am a human’, which would/should map to the same asset-child.

- An asset-child with an object as value represents an ‘x is an y’ sentence structure.
- In this case, the value must have a thesaurus relationship with the attribute. This relationship can span multiple levels.

AICI at the moment,  presumes that an ‘entity’ that is able to communicate through a text channel, must be animate (mammal, AI,…). So, even if this information is not yet provided, it is presumed to be the case. This is done cause the property and value play a role in other parts of the network.

The second asset-child has another asset as value. This type of relationships represents the ‘x has an y’ sentence structure. More specifically, in this example we said: ‘I have blue eyes’. This also explains the undefined ‘plural’ amount: eyes is plural and was converted into a single value, but since we don’t know exactly how many, we keep a ref to the multiple. The sub asset also has it’s own child-asset, which defines the ‘color-blue’ part of the example sentence. Through this nesting, it is possible to describe complex entities that consist out of multiple parts of knowledge.

An asset-child that references another asset as value, represents an ‘x has an y’ sentence structure.

Objects referencing assets

Assets aren’t exactly islands, independent of everything else. No, they too are referenced in many different ways. We already saw 2: assets attached to child-assets, and to the text-channels.  There’s one more reference worth mentioning at this moment: objects that reference assets. Yes, assets don’t just reference objects, it can also go the other way round. This is used to represent general knowledge about general things, like: ‘a human has hands’, as depicted in the image to the right. The same thing is also valid for statements like ‘an x is a y’: they are represented as assets, linked from objects.

Note: I did a bit of recycling with regards to the meaning of the link from the object: it also uses ‘asset’, which is the same as the meaning for the cluster.

An object that references an asset cluster, using ‘asset’ as meaning for the link, is used to represent statements of the form: ‘an x is/has a y’

The important difference between stand alone assets and assets that are linked to objects, are the information that they represent about the agent of a sentence: stand alone assets represent concrete agents: I, you, the book,… while object referenced assets represent abstract agents: a book, a human,…

These 2 data structures: objects and assets, are simple, but, I believe, flexible enough to represent a whole world of information. It is not as much the the querying and reshaping of the data as it will, in the end, be the automatic creation of these structures which I believe will ultimately be the truly interesting thing. With a query, we can retrieve data, reshaping it allows us to find truths, fallacies and falsities in the data, but a correct automatic creation of a new, complex asset, is in fact, a new entity.

In the end, all is just structure.

Note: Deprecated!

The thunk to .Net is finally working:

So it’s now possible to call static functions that take value types as argument and/or as return type.  In this example, I am calling the standard System.IO.File.Copy function to copy a file on my disk. Exceptions are also supported, as you can see from the first line, when I am trying to copy a non existing file. The output comes from the exception generated by the copy. Note that you can specify the arguments in different ways. In this example, I am using the ‘to’ word. ‘From’ would also be recognized. Both determine the extraction order of the arguments.

At the moment, I have hardcoded this extraction of the function arguments for the ‘copy-verb’ manually. The idea is to change it to something more general that can work for any .net function using some meta data of the function (like which neuron to send to which output, how to recognize the arguments, how many arguments,…). I’m not there yet though. First some finishing touches to the designer, so I can trace some neuron/memory leaks. After that I’ll probably do a new release, so you can play a bit.

Oh, and it’s still way to hot outside

Note: Deprecated!

Perhaps a mid-dev cycle release would be appropriate this time round, since some changes were done to the core and some people might be interested in taking a peek under the hood. Here’s roughly what’s changed:

• There’s a new expression: the LockExpression, which does what you would expect: lock 1 or more neurons and/or 1 or more links for a short period of time so that other threads can’t get to them during the lock. Watch out with this one, try avoiding to use this with split or duplicate instructions since that combination can easily create deadlock situations.
• A new instruction: SplitWeighted, which is the same as the normal split, but gives each processor a weight, based on the position of the split variable that the processor received and the accumulator value. Example, if you were to split over 3 vars, which an accumulator of 1, the first processor would get a weight of 0, the next 1, an the last would get 2. This is an easy shortcut for search routines.
• I have began work on the new flow editor, but this is still a work in progress.
• Lots of bugfixes.
• Cleaned up the interface a bit.

As usual, you can get the download from here.

Put into sequence:

1. Thread 1 locks x and continues execution (no waiting)
2. Thread 2 locks y and continues execution (no waiting)
3. Thread 1 requests y and waits
4. Thread 2 requests x and waits,… oeps

The most often way to locate this situation is by examining the execution stack of the CPU. But what if you do out of sync locking: the request in function 1 and the release in function 2, without any relationship between the 2 functions. As an example, take the network-core’s lock expression, which is new. This locks the items when the statement is called, but releases them only after all the child statements were called. The interpreter obviously can’t do this in the same function call.

So how do you trace this type of bug? Well, slowly, painfully and with lots of debug code to generate application dumps. here’s the smallest dump that I generated for the core (usually it was several 100 lines longer):

Links in
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Links out
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Values
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
Processors
Parents
ID: 7461 ReadCount: 0  WriteCount: 0, Waiting: 2
Children
ID: 7432 ReadCount: 0  WriteCount: 1, Waiting: 0
ID: 7461 ReadCount: 2  WriteCount: 0, Waiting: 0
ID: 7288 ReadCount: 0  WriteCount: 1, Waiting: 0
LockExpressionCounter: 2
Single lock count: 0

Basically, it allows me to check all the locks that are still active (in order of age) when the deadlock occurred. From then it’s simply a matter of figuring out who has the oldest lock and why it wasn’t released. To do this, it’s best to keep track somehow of the threads that do the locking and make certain that they are named, so you can find them again in the execution stack of the debugger.

Note: Deprecated!

No, this post doesn’t explain what AICI is, check here to read more about that topic. Today, I’m attempting to explain how to add the functionality to AICI so that it can respond to the statement ‘What’.

As previously mentioned, there are a number of stages that we need to get through to go from input to output. The first is the scan phase, which translates the letters into words, numbers and signs. When this is done, we try to find sentence constructs, like the subject, verb,…. This is matched to a frame, which determines the action that needs to be triggered. Finally, the action might also activate an attribute, but this is out of this post’s scope. So let’s make our network respond to ‘what’.

Recap:
First check if the flows can parse the statement: both scanner and grammar. Next comes the frame and finally the action and attribute.

Scanner

This stage will transform the letters into a single word ‘what’, when it reaches the ‘Sentence’ flow. The result of this stage is a cluster, with the meaning ‘Sentence’ (the same as the name of the flow that produced the result). The child will be a textneuron, representing ‘what’

‘What’ is a simple word, which is already handled by the scanner, so no worries there. If you are wondering how words are parsed, take a look at the preceding image, which is the flow for a word. Underlined words are references to other flows,  words that aren’t underlined represent static values. Curley brackets mean a repetition, strait ones define an option. If a conditional has a vertical line after it’s first bracket, a selection is required, this is valid for both repetitions and loops. This basic syntax was inspired by parser generator languages like Coco/R, Lex and Yacc.

So in the above example, a word can start with number (a sub flow), letters or capital letters. A word must always have at least 1 letter or capital, may have middle numbers and can end with a number or an alpha numeric.

Note: at the time of writing, there is still a little optimization required to get words with many mixed digits and letters working properly.

Grammar

Moving on to the grammar section. This is also defined in a flow editor, called ‘English grammar’. So open this (in the ‘Flows’ folder) and immediately select the ‘Interrogative pronouns’ (if you are, like me, not that very fluent in English grammar, and could use a mental aid now and then, there are plenty of resources on the net, check out Google dictionary, the English club, Wikipedia,…). As you can see, our ‘what’ is already in the ‘Interrogative pronouns’ flow.  So perhaps no extra work for the grammar stage as well? As it turns out, this will be the case. For those who want some more details though, here goes:

If we  look more closely to the flow name itself (on the first picture), we can see there is an overlay ‘FL’, which is short for flow-code. If you hover over the ‘FL’, you get a tooltip with more info. This says ‘References code that is executed when the item is recognized in a data flow’. This means that the flow does something when it is recognized in an input stream. Usually, it formats the recognized data in such a way that the next stage knows what type of sentence part it was. If there is no ‘FL’ overlay, it means that the data will be recognized in the input stream, but no extra actions are taken, when found. So the input is simply copied over to the result which leaves it unrecognizable. The ‘Reflective pronouns’ are a good example of a flow without overlays.

But what to do in such a case? Before you start to code, first see who uses the flow anyway, maybe they do something with it.

Tip:
Use F8 to browse all relationships of a neuron.

If you select the ‘Reflective pronouns’ Flow (click 2 times, the first time we automatically focus on the editor) and press F8, you can see which clusters it belongs to (and some other stuff too, not important for now). In this case it’s only 1 cluster: a conditional part (we need flows), so if we drill down a bit further, until we eventually get to the ‘Object flow’. This is used to find the object of a sentence. These flows do have an overlay (in the editor), and tag the recognized data. Usually, this is enough. Only when you need to tag this sub result as well, do you need to add some code, which is generally not the case.

Anyway, to get back to our ‘Interrogative pronouns’, lets see what the attached code does, and open the code editor (right-mouse button on the flow, and select ‘View code’). Most often, as with this flow, it simply duplicates the content of the ‘Result’ variable (which always references a cluster at this stage), and removes all outgoing links on this duplicate (sometimes, result clusters can link to some extra data which we don’t want,  but is used by the algorithm) and stores this duplicate back into the result cluster.

To better explain what this actually means, lets take the following input stream: ‘you are here’. After the scan stage, we have a result that looks like image 5: a cluster with meaning ‘sentence’ and 3 text neurons as children. Once the grammar stage is done transforming this dataset, it looks more like image 6: a cluster with meaning ‘statement’  and 3 children again, but with some more info. The first 2 represent the agent (sentence subject) and the verb, both flows use the previously described transformation code. The last child is the object representation of the textneuron ‘here’. It is the result of an adverb that was recognized somewhere in a flow, which didn’t transform the data, so content simply got copied to the final result.

AICI:
When a flow has been recognized, the ‘flow-code’ callback of the flow is executed. The ‘Result’ variable will contain a cluster with the flow as it’s meaning and the recognized data as it’s children.

AICI:
Use the ‘TransformFlowResultCallback’ cluster as a default callback for flows so the result can be recognized by the frames.

Frames

Up until now, not much had to be done, with the frames though, this will change, here we actually need to do something to get it working. Frames can be seen as the conduit between the front (scanner/grammar) and back (actions/attributes). A frame defines which sentence parts it can contain and their order. Multiple orders can be defined, so that you can have the same sentence parts to form a statement or a question. You can also have optional parts, to make frames even more flexible. Each order is defined in a frame sequence. This is also able to render itself back as output, completing the circle.

Just like for the scanner and grammar, the network is already capable to extract the correct frame sequence, based on the set of already defined frames and their sequences (so if it’s not yet defined, the network can’t recognize it). Once, the sequence is found, the framework performs some calls to attached code that you have to provide so that the network also knows what to do with the input statement. The following code blocks can/should be attached to a frame sequence:

• ExtractAction: this code block is called to determine which action(s) the frame should trigger. Multiple actions are allowed, in which case each will be triggered in the order as they were returned. This is done through the variable ‘ExtractedActions’.  The attachment is required.
• ConvToKnowledge: this code block should transform the result data of the grammar parse, into the input data for the actions and return this cluster through the variable ‘KnowledgeConv’. (edit 11/July/2010) The attachment is required. This attachment is optional. If you define a value for the new column ‘ResultType’ in the frame editor and the ‘role’ neuron has a ‘ConvToKnowledge’ code cluster, the network can automatically generate the result cluster for an entire frame sequence, and so this code is no longer required. You can override the auto rendering by defining this code cluster.
• RenderCode: This code block is called by an action to render output according to the structure of the frame. The output to render is found in the ‘CurrentFrom’ system variable. The format of the data is the same as that of ‘ConvToKnowledge’. The attachment should only be defined if there is an action able to call it is output. This is not always the case, for instance, frames that deal with ‘unknown’ word types, don’t need to be generated.

Recap:
A frame determines the action to execute, provides the data for the action and is able to render data provided by an action.

As previously mentioned, a frame needs to prepare the data for the action. This should be a cluster without meaning (it will automatically be assigned, once the correct frame sequence is found, and will eventually indicate that the cluster is an input statement). The content of this cluster needs to be well defined, so that the action can handle it correctly. In theory, you can use any type of structure that you want, as long as the action uses the same. In practice, it’s best to use a predetermined structure, shared by all the actions and frames, for easy sharing of default actions. So here’s a small overview of the possible content:

• The first child neuron should indicate the time value: current, passed or future. Though most actions don’t specifically expect it on the first pos, all current frames use this.
• The child list can also contain a ‘not’ neuron, to indicate that the statement should be interpreted as a negation.
• The sentence agent should be stored in a cluster with meaning ‘who’
• You can store the sentence object in a cluster with meaning ‘what’ or ‘value’, depending on the situation. Sometimes, ‘what’ can be extracted, from ‘who’, a previous statement or the ‘value’.
• a cluster with meaning ‘verb’ can be used for sentences that a verb. Originally, this wasn’t done for the basic verbs like ‘to be, ‘to have’, since it can be inferred from the action that is triggered. However, it can always be useful to take the actual verb that was used to the action, in case it needs to do some analysis on the ‘subject’ indicator of the word. For instance, ‘am’ is usually for ‘I’.

Frames can play a little bit with these structures: fill in missing data, switch who, what and value,… Though most of the code in the current frame sequences is shared: if there is an agent in the sentence, 9 out of 10 times it is transformed in the same way, so lots of copy paste (or drag drop) possibilities here. Also perhaps for future automatic learning.
(Update 11/July/2010) In the new version, it is no longer required to provide this code for each frame. It is moved to the level of the ‘role’ (as defined in the frame element), usually these are the flows like ‘agent’ or ‘be verb’,…

The translation results for the input ‘i am jan’. In the final stage, ‘what’ is filled in by the frame.

For our example, we will have to create a new frame to handle the input ‘what’. Since this is a sentence that only contains an interrogative pronoun, we should probably best put it in the ‘Pronouns’ frame editor together with all the other pronouns (for easy lookup later on). So open up this editor (double click on the editor’s name in the project view), or create a new one if it doesn’t yet exist (the button that looks like a rectangle, you know ‘frame’).

To create a new frame, click on the button with the square, on the editor (tooltip: Create a new frame) or press Ctrl+Shift+F and provide a name for the new frame. Lets call it ‘what frame’. To quickly build the frame, you can drag neurons from the toolbox, thesaurus, flow,… and drop them on the frame editor. To start with, find an ‘Interrogative pronoun (to get it in the explorer, press ‘F4’),  and drop it somewhere in the top part to create a new frame element. This indicates that the sentence should contain an ‘interrogative pronoun’ sentence part.

Tip:
To quickly sync the explorer with the currently selected item, press F4 from anywhere in the application.

Make this element an evoker. Only evoker elements are used to find all possible frame candidates. This is a small optimization for the search algorithm. Each frame should have 1 evoker element. Usually, this is the verb, but in this case there isn’t one so we use what’s available.

At this point, we have a frame that will be activated for all sentences that contain an interrogative pronoun, like: what, who, where, when,… We need to further specify that we only want to respond to ‘what’. This is done by adding a filter to the frame element. So make certain that the element is selected, and from the thesaurus, drag ‘what’ to the bottom part of the frame editor and drop it there. this will automatically create a thesaurus filter for the currently selected relationship. So if ‘what’ had one or more children in the thesaurus, these would also be valid for the filter. In this case we don’t have that, so the relationship field could have been left empty.

Also notice the ‘RestrictionMofidierInclude’ in front of the filter. This is to indicate that input values that contain the filter value are allowed. If this was set to ‘RestrictionModifierExclude’, only sentences that wouldn’t contain ‘what’ would be allowed (so, where, who, when,..).

Finally, there is also the small button. When this is checked, the content of the sentence part should only contain what is defined in the filter, and no other words. If not checked, other words may still be present. This is an important little property which I almost got wrong starting out this post. Do we want to handle sentences like ‘what color’, ‘what time’, what the hell’,  ‘what in the name of sweet Jesus’, or only ‘what’. If you go back to the very first image, you can see that it’s allowed to put some words after the pronoun. This is to allow correct recognition of sentences like ‘what size are you’ where ‘you’  is the subject. This means that the grammar will also recognize ‘what color’ as an interrogative pronoun. It turns out this is actually ok for us (see frame sequence code). So we leave this not checked to allow other words in the flow result.

The Frame sequence

Now that the frame has been defined, lets move on to the sequence. To create this, press the ‘Add frame sequence button’, or ‘Shift+Ctrl+S’ and provide a name for the sequence (make certain that the frame editor is active when pressing the button, otherwise it won’t be active). Lets call it ‘What statement’. You create the sequence by moving the frame elements from the left part to the right part of the editor. You can use the arrow buttons for this. Multiple elements can be moved at the same time. The up and down can be used to change the order.

All pretty strait forward. The cool part comes in the code editor. To open this, select the frame sequence in the editor (or somewhere else, like in the explorer), open the context menu and select ‘view code’. First, lets make certain that this new code editor is also available from the project. All you have to do is open the context menu on the bottom tab items of the code editor, and press the ‘Show in project’ menu item. This will add the neuron to the currently selected project folder (or root if there is non selected). So, when you do this, it’s best that the correct folder is selected first. If you forgot to select the one you wanted, you can always still move it later on.

Tip:
To add a code editor to the project view, right click on the bottom tab items and select ‘Show in project’. The item will be added to the currently selected project folder.

By default, there are only a limited number of pages available in the code editor. The rules and actions pages are always available (though the code clusters are only created when you actually drop something on them). But not the ones that we need. Those will have to be added manually. So open the context menu, again on the bottom tab items, and select the required pages listed under ‘add new code page’. We need ‘ExractAction’, ‘ConvToKnowledge’ and ‘RenderCode’.

Tip:
You can add any type of code page to a code editor, make certain that the required neuron is tagged as ‘a default meaning neuron’ in the explorer. Open the context menu on the bottom tab items of the code editor and select the new page from the ‘Add new page’ menu item.

Frame sequence code

RenderCode: The render code for the ‘what’ statement as displayed in the above picture is very simple, but not completely correct (see below), we basically write “what?”. I added the question mark, just to show a little trick of the text sins: When you send them multiple neurons, by default, each one will be rendered on it’s own line. To group neurons together, use the ‘BeginTextBlock’ and ‘EndTextBlock’ neurons.  These are recognized by the text sin as special formatting tokens.

We use the ‘output’ statement to send data to a sensory interface. The fastest way to declare this expression, is by dragging it from the toolbox. This has 2 pages: one for common expressions like clusters, assignments and commonly used neurons, and a second page, exclusively for all the instructions.

Tip:
You can quickly expand or collapse all toolbox items from their context menu. All instructions display a tooltip with a short description, the expected parameters and output.

The first argument to all the output statements is always the same: ‘OutputSin’. This is a global that is filled in by the framework and points to the sensory interface to which the output should be sent. You can also hardcode which sensory interface it needs to be sent to, but that is less flexible. This way, you can actually have multiple text-sins in 1 network and have multiple users chat at the same time. As a small side note: because the output statement can take any number of arguments, we could have written it shorter: in 1 statement. This wasn’t done because I forgot, before taking the screenshot

Also, if you look closely, some of the statements appear to have a double border, while others don’t. This is to indicate that it is used multiple times. This is important to keep in mind, if you modify these statements, you are also changing them in some other part of the code. When dragging code around, it is not like text that you duplicate. When you drag and drop, there is no deep copy, only a new reference made. This can keep the code size pretty compact, allows for some deep future analysis and most importantly should allow for easy auto generation later on. But it is something to keep in mind while changing code.

Caution:
Statements with a double border are used multiple times, be careful while modifying these.

The complete render code looks something more like the following image. This is able to handle not only ‘what’, but also ‘what color’, ‘what on earth’,… Looks scary? Don’t worry, it’s actually very simple, and most of the code is copied over from other parts anyway.

The second statement, the big yellow monster, is actually a query, who’s result is stored in the ‘what’ variable. ‘Get children filtered’ returns all the child neurons of a cluster that meet a specified filter. In this case, we only want neuron clusters that have a cluster-meaning equal to ‘sg:What’, in other words, we want to retrieve the ‘what’ part from the input. (no surprise there).
The next statement is an ‘if’ construct: if we found something (count(what) > 0), render the contents of the ‘what’ variable, using the ‘Render object’ code block. If there is no cluster found, simply render ‘what?’. The ‘what’ variable isn’t passed over to the ‘Render object’. This is not possible and not required: a code block doesn’t have it’s own variable space, so all the variables maintain their value, hence, the ‘what’ variable is passed along implicitly.

And thus, you have mastered your third little peace of code already

ConvToKnowledge: (update 11/July/2010: this code cluster is no longer required, but is still allowed) The transformation for ‘what’ is also fairly strait forward. This  type of attached code tends to have a fairly similar structure across frame sequences. In this one, we check if the interrogative pronoun contained anything other than ‘what’. We do this by checking if it has more than 1 child. When this is the case, we create a new cluster, assign it the meaning ‘sg:what’ (this is a tag operation) and we copy all the children, except the first, from the cluster that we found to the newly created one. Finally, we add this new cluster to our result. We don’t copy the first child, since that is the the ‘what’. We presume this, if you know any situation in which it is not the case, the code needs to be updated to something more secure (remove the actual ‘what’ neuron).

ExtractAction: Lastly, we also need to provide some code that returns the action we want to execute. Before we can do this, we actually need to know which are the possible actions that can be performed.

So lets try and come up with a few possible dialogs, so we can analyze the different possibilities. Here’s what I came up with on short notice:

PC: I have hair
You: what
PC: I have hair –> this is a simple repetition of what was last said.

PC: I am in the black room
You: what color
PC: black-> return the first value that was found through an ‘is a’ relationship (black is a color), from any of the words of the prev statement.
You what room
PC: the room of your house –> return more relationship info on the actual result of ‘the black room’, if there is any. In this case, we know the owner of the house.

You: what the fuck
PC: No swearing please –> catch a special case.

So Basically, I have 3 different groups of the conversations (if you can think of more, le me know). In the first case, when the user simply asks ‘what’, we return the last output statement. The 3th conversation, when the user wants to express surprise/disagreement through an insult, we return a standard message to keep it clean. Finally, in the middle example, some analysis is done on the previous statements, to find a proper match. We can check the actual words, see if there is an ‘is a’ relationship to be found. In this  case, the user might have forgotten the actual value, but still remembered the attribute. The user could also be requesting more info about something, in which case we need to return some more links if there are any.

Exactly how you pour this information into actions, is a bit up to you. Me, I would go for 3 different actions. In fact, I would handle the last conversation type (aka: ‘what the xxx’) as a special case in a different frame, so we can filter on these specific word groups. I guess that would be a good exercise for you. The first and second should be handled by this frame, so these are 2 new actions that should be returned by our frame sequence. To determine the exact action that needs to be returned, we can use a similar ‘if’ construct as in the ‘ConvToKnowledge’ code cluster: when there is more than a ‘what’ in the input data, use ‘Find Prev out statement’, which does the heavy searching. If there is only ‘what’ in the input data, use the light weight ‘Return last out statement’. To return the action we want to execute, simply assign it to the ‘ExtractActions’ variable, like in the statements to the right.

Recap:
Return your actions in the ‘ExtractedActions’ variable.

Actions

Rejoice, the end of this post is almost in sight: Actions, the final step. As already mentioned, we are going to need 2 actions, though to keep the post a bit within the limits of sanity, I am going to restrict the explanations to the simplest action. The more complex search algorithm needs another one of these posts to explain it all. In fact, I still have to finish it myself. Perhaps, I’ll leave it like it is, as an exercise for you

Before we start to create our action though, perhaps a small word about actions in general. These are actually nothing more then regular neurons with 2 different types of code clusters attached. The first type contains code that is called when the action is executed. You can specify a different code cluster for all the different states that the action needs specific responses for. The default state is ‘Normal state’, which is currently the only one defined, but you can create and set as many of your own states as you want. The current state of the network is defined as a link between the text sin using the meaning ‘state’ to the current value (see image). This means that you can have as many concurrent network states as communication channels. Also, when an action doesn’t have an implementation for the current state, or when the text-sin doesn’t define a state, the fallback ‘Normal state’ is used.

The second type of code clusters that are attached to actions, are used during the conversation, to check for and handle possible responses to questions. That is, some actions require some sort of response. These types of actions need to check if a response is valid, determine if an action is a valid response action and can also specify extra ‘response’ code. This though is again far outside the scope of this post, so lets leave  it at that.

Back to our example, we still need to create the action. Since this is just a neuron, there are many different ways of doing this. If you are already in the correct project folder, a quick way is to use the toolbar button above the project overview, called ‘Create a new neuron with attached code’. This will create a neuron, open it’s code editor and add it to the project. If you press F2, you can change the name (make certain the project node has keyboard focus). To add the ‘Normal state’ code page, just do the same as with the frame sequences: open the context menu on the bottom tab items, select ‘add new code page’ and click on ‘Normal state’.

And that’s how the action should eventually look like. It’s a nice bit of code to end with, with just a little bit more meat to look at. Though in the end, it’s still very strait forward. Before we delve into the specifics of the code, perhaps first a small word about the conversation log, since that’s our primary data source in this action.

The conversation log is a cluster, attached to the text sin for which it is keeping track of the previous statements made by the network and the user of the text sin. This means that the network has a log cluster for each active user. It keeps track of a 1 to 1 conversation. A Part of the content of the log consists out of the clusters that we created in the frame sequence, in the ‘ConvToKnowledge’ code. These have a cluster-meaning of ‘in’. The other part consists out of the clusters that were generated in the actions, as output.  These are labeled ‘out’. Both ‘in’ and ‘out’ children still have a link to the action(s) that was/were executed so we know what they triggered.

When we translate this information to our action, we get something like the previous picture. Basically, we try to retrieve the log from the ‘OutputSin’ global. We have already encountered this global.  It is the one that is always pointing to the Text-sin for whom we are currently processing, remember. If this doesn’t have a log, something is wrong, so we show a message to the user, log it as an error (this is useful for the debugger) and exit the link. We exit the link and not the neuron or the entire processor, since there might still be other actions that need to be executed.
If there is a log file, we walk through each log entry, from back to front, using a counter and the ‘GetChildAt’ instructions. You can do this in different ways (using the reverse instruction combined with 1 or more selects, for instance). I choose this way, since I think it’s the fastest solution for this situation.
If we find an item, with a cluster meaning of ‘out’, we simply push it back on the stack and exit the loop. By putting the cluster back on the stack, we actually let the network execute the action again, since it is still linked to the data. Note that we decrease the counter each time that we don’t have an item with meaning ‘out’. The ‘-‘ instruction might be written a bit strange at first. You can probably best think of it as a polish notation. This is actually a result of the fact that ‘-‘ is just an instruction and at present, all instructions have the same UI input format. And that’s it.

Ok, slowly breed in, and out. You’ve just reached the top of the Himalayas (or something comparable). It’s done, you’ve absorbed all the info, and your still alive. Time now to let it all breed a little, let it digest, and sleep on it a bit. Don’t worry about it, it will come to you too.

Note: Deprecated!

The tail side is still red hot from the afterburners. At full throttle, 2 editors were completely rewritten, the drag drop system cleaned up and a rare deadlock removed. Both the mindmap and code editors now support all the mouse functions: horizontal/vertical scrolling and zooming. Oh, and I’ve also cleaned up the AICI code a little, though that needs more work.

Get the latest installation from the download section.

Note: Deprecated!

The new code editor is beginning to work:

Note: Deprecated!

I have created a new small network host demo application which you can use instead of the full designer to run AICI. You can get it from here, or from the new download section.

It’s basically a general purpose network host that only supports the text sensory interfaces for input/output. You specify the network that needs to be run at the command prompt. The installation program will do a default setup to work with the AICI demo, but you can change the shortcut to use your own if you want.

Note: Deprecated!

Finally, it’s time for a new release. Most of the changes are in the backend this time, though there are a few visual changes. So here goes:

What’s new

Designer and core

• The storage system has been updated to a binary, flat file structure, which improves the operation speed considerably in all areas. This essentially makes the core a true database system, which should be fairly easily scaled (see thesaurus demo project), even over multiple hardware systems (eventually).
• I  have finally implemented the namespaces. These are currently maintained in the background, so you don’t need to worry about them. The only change you’ll notice, is an improved browser popup in the frame and flow editors (see image). Basically, the thesaurus, timers and all the project editors (with their entry points) are included and automatically updated.
• The frame editor has been updated a bit cause the previous datagrid didn’t handle the custom namespace selection object. So back to listview for the frame elements (will be updated when the datagrid is fixed).
• I have also added a new restriction type for frame elements: a custom filter, which allows you to do determine the filter result through code.
• Support for modules has also been included, though not yet properly tested, so the imports will definitely not yet work.
• The speed increase brought some more threading issues to light, I hope they are now solved, but one or two things can still pop up here or there.
• Somehow, I managed to show the thesaurus in reversed order (so the view for the hypernyms was showing that of the hyponyms). While fixing that, I also tried to get it to work with an arbitrary number of elements (so you can also browse a huge thesaurus, the search hasn’t yet been updated though).
• The thesaurus can now also be built automatically, based on the content of the database.
• I have added a dialog to remove clusters that have no children, based on the ‘remove orphans’ dialog. This will probably grow out to a general purpose query window, but that’s for the next version.
• New instructions (some were long overdue):
  • Substract: remove 1 or more neurons from a set, contained by a variable or global
  • Select: perform a filter on the content of a variable or global.
  • Remove Links In: removes all the incoming links of a neuron that have a specified meaning.
  • Remove links out: same as ‘Remove Links In’, but for the outgoing links. These 2 instructions allow you to do without an extra loop, which makes it all more thread save and faster.
  • GetAt: retrieves the neuron at a specified index in the content of a variable/global.
• I have changed the parameters of the split instruction a little bit: I have removed the ‘result cluster’ which was originally used to retrieve the results of a split. This worked ok, but could cause problems when multiple users would be performing the exact same algorithm. The solution was to simply use the already existing ‘GetSplitResults’ function, which now also works after the split (and not just when the split is still running). This means that you manually need to clear the split result cache once you are done with it, otherwise the next split will give some bogus results.
• I have also removed the possibility to assign a weight value to a single neuron. Instead you now assign it to the entire processor. This will assign it’s weight to all of it’s result values when it is done. This way, you can change the weight before you know the actual result.
• Attached neurons are working.
• some cosmetical touch-ups here and there.
• Lots of other bug fixes.

AICI

The aici network demo is slowly starting to function. That is, it is beginning to record data into understandable structures. Though it’s still very early, so lots of stuff still goes wrong or hasn’t yet been implemented. Some fun responses guaranteed. Here are the major changes:

• I’ve managed to simplify the flow recognition algorithm (while fixing some tough bugs), so no more synchronization points for the loops. The designer still allows to change this prop at the moment, but this will be removed soon.  Don’t worry about existing values, they are simply skipped, no longer used.
• Some bugs in the frame recognition section have also been solved.
• I have implemented a first version of the noun path resolver algorithm. This is used to understand statements like ‘my brother’s sister’, ‘I’, ‘my favorite car’, ‘sister in law’. This algorithm is also responsible for generating new ‘compound words’, like ‘brother in law’.
• Compound words are recognized again, at different stages. Some compounds can’t be recognized yet (if they have verbs).
• New frames:
  • Be-unknown noun: sentences with an agent, be verb and a possible noun. Example sentences: I am Jan. It is a cat. My name is Jan. I was Jan. It will be a cat. (preconditions: Jan and cat are not yet  in the network). The frame is triggered when it receives a sentence that might contain a noun, and there is an open question that can consume this value.
  • some more, to much to write all down.
• If AICI knows several different synonyms for the same word, it will now mimic your preferred one. So if you just said ‘bye’, it will also say ‘bye’ next time. If you say ‘goodbye’ next, it will switch to this.
• I have introduced the concept of ‘attributes’. These provide an extra step in the semantics stage. For instance, ‘color’ is an attribute. It’s possible values are: blue, red, black, white, green,… Color is a special attribute, since it changes meaning when applied to different types of subjects. For instance, the sentence: ‘I am blue’ usually means: ‘I am a bit down’. So that’s the semantic part: an attribute can inspect to who it is applied and, when needed, can change the attribute, the owner or the value.

The latest release can be downloaded from here, or from the main menu.

Coming up

Next in line will be the editor views (finally). These are currently built using standard WPF controls (listboxes, treeviews,…), which are, well, not performing as desired: they generally can’t handle more than 2-300 items (if you can’t use virtualization) and, more importantly, have a hardcoded recursion limit that causes serious problems, primarily in the code editor. So I need some custom controls to get the speed and reliability up.

Before this though, I will be spending a little more time with AICI. The algorithms are beginning to work the way I want them to, so it’s time to complete the flows, in order to get a larger part of the English language parsed. And off course, I will be adding a bunch of new frames, actions and attributes.

Note: Deprecated!

An annoying side effect of this type of resonating network, is, when you have a bug in the code, you generally don’t get 0 answers, but a whole lot. This can sometimes be very confusing to debug. Why does a certain path give a positive result, when it shouldn’t? And more importantly (from NND’s point of view at least), how can you follow the construction of that specific result, without the clutter of all the other processors? Enter ‘split paths’.

A split path represent the route that a result took, from start to now, expressed in the neurons that the path chose during all the splits.

As example, if only 1 split was executed to get to a result, and there were 2 neurons in that split, the path of the result only has 1 node and can have 2 possible values, those that were used in the split. In the real world though, a path usually has many, many nodes.

To record a split path, you first need to put a break point at a position where you know you will get many invalid results. That’s because a split path can only be recorded when the processors are still running, not when the result has already been calculated.

In the aici demo, it’s best to put a breakpoint in the ‘Stage 1.1’ code, in the if, second part (GetClusterMeaning(CurrentTo) == flow), if you drill down a bit further, you get the ‘Add split result’ instruction, which is basically the end point of the flow recognition  algorithm. (see picture below).

Put breakpoints on ‘Add split result’ instructions to find invalid split paths.

Once you have a processor that is producing an invalid result at your breakpoint, you can store the split path. Select the processor, open the context menu (right mouse button on a processor in the debugger overview) and select ‘Store split path’.  This will produce a new entry in the tree on the ‘Debugger’ page (see 3th image). The root item will be ‘path for x’ where x is the name of the processor. The children are all the neurons that were selected for the path during a split instruction.

I usually kill everything after I have the path, the other processors will produce invalid results anyway, and you need to start a new run before it’s  possible to actually use the path anyway.

So, finally we can start using the data to debug our network. The basic idea is to  use the path for finding the processors that are following the specified path, so we can take a closer look at the code. In short, a split path provide a shortcut: instead of manually stepping through all the code and individually deciding which processors to follow, we can jump to the ones we are interested in. These will be highlighted in green and will pause when they have reached a split that is selected in the path. Off course, we first need to select the path in the debugger, and at least 1 child node, otherwise there wont be anything to see.

Processors that follow a selected split path will show up in green and pause whenever they reach a split that produces a neuron which is selected in the path.

 One last note perhaps, once you have the processor you want, you can use the ‘Kill all but this’ command to stop all the other processors. This way there is less distraction in the debugger.

In case you are looking for a mental image to visualize this whole concept, try this: Resonance. When a link gets activated (for instance, the one to the very first neuron), it creates a resonance that triggers one or more other neurons. This excitation in turn can cause another link between 2 neurons to be activated, causing more resonance and so and and on until the whole thing settles down.

I have absolutely no prove or idea that this is how it works in the real world, that’s just how my model can be interpreted. Truth be told, this is not how I conceived the thing (aka lets try to create a model that uses resonance), it was rather more like: I have 2 neurons, they are linked, how can I make something happen? Well, I can attach some other neurons (that behave like code) to the link and execute that. Cool. But, wait a moment, that looks like resonance…

and we just passed ‘culminate’, pfff

Note: Deprecated!

The new release is finally ready.  The Aici demo took a bit longer than planned. Also, lots of things have been fixed and updated. Here’s a non exhaustive list:

• There’s a complete new lockmanager running in the background. This is much more secure (thread-wise, that is) and a lot faster. It’s still a bit of a diesel though, it takes some time for it to get going, but once running, it should be pretty fast. The slow start is due to the storage mechanism (all xml files currently). This is the major drag on the entire system at the moment, and will be fixed next.
• Wordnet import has been seriously updated, a lot more info is retrieved, and it’s now also possible to import the entire db in one go (although not yet advisable, due to a memory bug in the designer, it still takes a major byte out of the hard disk and it simply takes ridiculously long).
• The thesaurus has been given a make over to allow for editing and filtering. He can now also display / edit non recursive relationships. Drag drop is also supported.
• The frame editor has been updated considerably: drag drop support has been added and frame element filters/restrictions have also been introduced (will probably be extended in the future).
• I changed the function of the ‘contains’ operator a bit. It now only checks the contents of a variable. For clusters/children, there are the new instructions (IsClustesteredBy, LinkExists, ContainsChildren, GetInFiltered,…).
• I have added the ‘not contains’ operator.
• New instructions:
• Arithmetic group (+-/*%): I finally caved on those. My original plan was to see how far I got without using any arithmetic in the neural code. I guess, this is as far as I got with that.
• Get-at group: get child at, get cluster at, get out at, get in at, get info at.
• Distinct
• get Incoming, get outgoing, get info, Get in filtered, get out filtered, Get info filtered
• Is clustered by, Link exists, Contains children
• Remove-at group: Remove child at, remove info at, remove link in at, remove link out at
• many, many bug fixes, updates and little improvements.

Aici 1

This demo is a small chat interface. Though, in it’s current state, it’s not much more than a framework. It can initiate a conversation, close it, ask for the name of the user and store the data. It’s not yet able to fully recognize a recurring user since I haven’t defined the neural code for this yet.

I will be explaining how it works and how you can expand on this functionality, shortly. For those who can’t wait and want to get a peek under the hood, here are some pointers to get started:

• There are 3 main stages:
• Flow recognition, which is basically the syntactical stage: check the word types and order. Project pages are:
• the flows (aici/flows),
• the code that is attached to these flows (aici/code/flow code)
• the code that recognizes the flows in the input (aici/code/flow recognition)
• Frame recognition, or the semantics stage. This is where we try to find meaning in the words. Project pages are:
• the frames (aici/frames)
• the code that is attached to the frame sequences (aici/code/frame seq code). The frames don’t have code (yet).
• the code that recognizes the frames in the flow results (aici/code/frame recognition).
• action execution, or the response of the network to the input. Project pages are:
• Actions: all the different actions that the system knows (not yet a lot, should be extended).
• Output: some common code blocks for rendering output. This is also used by the frame sequences, since they are also used to render data in a predefined format.
• Action helpers: code that the action neurons can use to perform common tasks, like stopping a conversation or controlling the timers. Timer callbacks are also stored here.
• The transition between the different stages can be located in aici/code/transition.
• More specifically, the ‘Transition’ neuron is used to go from the flows to the frames and finally to the actions. 
• An action is started using the ‘Execute action’ neuron as meaning for a link from a data cluster to the action that needs to be started.
• The project also contains some mindmaps that describe the inner data structures and functionality.

New release coming shortly.

Note: Deprecated!

Intro

Time for the second demo overview: the Scanner.  It builds on most of the ideas found in the first demo but it goes way further, and actually does something very useful (although you wouldn’t say it at first).  It’s probably going to be a lengthy piece so I’m thinking of cutting it in 2 or maybe even 3 parts. Anyway, lets first start it up, either through the start menu shortcut (in the Demo’s sub folder, conveniently called Scanner demo), or by opening it in NND (File/Open, select the ‘My documents/NND/Demos/Scanner’ folder). Once the project is loaded, you should see a single text communication channel open (called Text sin), if this is not the case, go to View/Communication channels/Text sin and make certain that it is is selected.

Overview

Lets get a taste of what it does, so enter some text (or leave the one that’s already there) and press the ‘send’ button (or enter).

You’ll notice that it basically does the same thing as the echo demo: the input text is echoed back, except that it’s a bit slower. If you had the debugger tab open, you probably also noticed a lot more activity, so something more must be going on.

And indeed, if you take a closer look to the text sin channel, in the upper section, you can see the neurons that were send to the network (on the left) and those that were returned (on the right), which are different. This was not the case with the echo demo, it simply sent all the incoming neurons back out, as they were. In this demo though, we get back something completely different: TextNeurons, that represent the same thing as the int neurons that were sent as input (if you regard them as ASCII characters). Hence the name of the demo, it’s a scanner.

Converting a stream of ASCII chars into words, integers, doubles and signs is a pretty useful feature and that’s all this demo is capable of doing, but that’s only because I stopped there. You see, in the background, is a general purpose algorithm that converts an input stream of neurons into a single result cluster, using any and all of the flows that are defined in the network. This means that you can use the same algorithm with many different flow definitions. I have simply defined some to recognize words, integers and doubles. You could go further and add flows to find verbs, sentence subjects,.. (in fact, that’s what the AICI 1 demo does). You could even go further still and create flows for visual objects or audio fragments, the same algorithm can be used. Unfortunately though, the editor doesn’t yet support such types of displays for flows (will probably be added somewhere in the future though).

Details

So how is the translation actually performed? To explain this, let me first recap some of the basic concepts of neurons and flows:

• The different types of  data available to a neuron are: incoming and outgoing links, possibly one or more parent clusters, for clusters possibly 1 or more children and a meaning. And finally value neurons also have their value of course. This is important, cause when the translation process starts, this is all the available information.
• Flows are nothing more than clusters that contain flow items, which can be statics or conditionals (loops and options). These in turn can only contain conditional parts. They represent a single branch of the decision tree. Parts can again have the same data as flows: statics or conditionals. So if you are a neuron (a static, part, conditional or flow), you can always look up into your list of parents to see in which flows and parts it is used.

Now, if you recall from the first demo, an input starts by creating an IntNeuron for each ASCII value, which is linked to another neuron using the ‘Letter’ neuron (ID 109). These are all put on the execution stack and the processor starts (the Rules code on the Letter neuron is executed).

So both neurons are new and only have each other as links. This means that we can’t use links or parent/child relationships to resolve the first step, but instead must use something different. The only other thing that remains is the value of the int neurons, so this is compared against some constants to see if they are digits (0..9) alpha numeric (a..z+A..Z), spaces | returns, or something else.  This comparison results in the creation of 1 new neuron per input neuron: the result cluster, in which we store the integer. One of these clusters (or it’s duplicate, due to a split) will eventually store the end result.  This cluster is linked to one of 3 static neurons: Digit, Alpha or Space (signs like . or , are handled a bit differently, this will be explained later). Naturally, if the integers would represent color values, we would use other starting points than digit or alpha. In other words, this first part is variable according to the type of input and the required accuracy of the algorithm. As meaning, we use  the start of the ‘flow recognition’ algorithm, called ‘Stage 1.1’ and put the result cluster back on the stack.  It’s important to put this one back on the stack, and not the item we are looking for. That’s because the algorithm can perform numerous splits and we want the result to be duplicated not the searchable, cause the contents of the list are continuously modified and we don’t want the result of one processor to be modified by  another one (I had to learn this the hard way).

After this initial step, the actual recognition algorithm kicks in. This consists out of 4 stages, grouped by 2. Meaning that  stage 1.2 is executed immediately after stage 1.1 for each neuron (this is the same for stage 2,1 and 2.2), but at the end of stage 1.2 and 2.2 all the result neurons are collected into a single global. Only after the last link of the last item on the stack has  been processed, are all the result clusters put back on the stack, with links for the next stage.  This is done for allowing to group items together.

The different stages are:

• Stage 1.1 (search parts/flows): Find conditional parts or flows in the list of parents of the searchable. If there are multiple results, perform a split for each, after the result list has been filtered (these are the shortcuts). If there are no results and this is the only and last item still on the stack, the end result has been found.
• Stage 1.2 (sequence-combine and filter): Check if items are sequential (2 flow items declared after each other in the same parent list, which is a part or flow) and handle floating flows, which are allowed to appear anywhere in the input stream, but which break up the sequence of other items.  Different actions can be performed if the order of the items is not ok: try to solve further or exit without result. Results of sequential items are grouped together.
• Stage 2.1 (search conditionals): Find conditionals in the list of parents of the searchable if this is a conditional part, otherwise the stage is simply skipped. If there are multiple results, perform a split for each after the result list has been filtered (not yet completely implemented at this stage). If there are no results, there is an error in the flow definition.
• Stage 2.2 (process loops and sync-points): If there was a conditional found in the previous stage, check if this is a loop. If so, and the previous item is of the same loop, combine the results. Also start a sync-point (will be explained later) if this was defined on the conditional.

These 4 stages are repeated until there is only 1 result cluster on the stack that represents the end result of a flow which is no longer used in any other flows. This cluster is made the result of the  split for the processor it ran on.  Off course, because there were possibly many splits, there could be many results. These will all be presented in the split-callback cluster (you need to provide a code cluster to the Split instruction, which will be called when all sub processors are done). In this demo, the result is sent back to the sin that caused the input, in a normal situation thought, this will simply start another process, as is done in the AICI 1 demo.

During this whole process, the algorithm is capable of executing callback code (attached to the statics, conditionals, parts and flows) at certain specific moments in the code. This is where the magic happens. The following types of callbacks are possible (together with  their execution time):

• Flow code: this code cluster is executed when a flow has been recognized in the stream. The ‘Result’ variable (ID 1822) contains all the neurons that match the flow. It’s here for instance that the int neurons are converted to a single word using the CiToS (Cluster with ints to string) instruction.
• Filter flow code: this code cluster is called from stage 1.1  (or 2.1, but this is not yet completely implemented) just after all the next items were retrieved from the list of parents of the searchable (stored in the CurrentTo variable). It allows the flow item to determine if it is a valid result, given the current state of the network. This is done by checking the contents of a number of globals, like ‘Prev stage item’ (ID 1956), which contains the previously processed neuron.

In the next post, I’ll go deeper into the specifics of the algorithm itself, for as you’ve guessed by know, it’s a bit funky, and I don’t want to forget all the subtleties, since it’s definitely still a work in progress (there are many improvements still possible).

The engine appears to be stabling out, although it is still acting fishy on single core machines, where there are errors I don’t have on my multi core dev system, so I need to move to a different machine to test this out. The designer is also still very much lingering behind the engine when this is processing at full speed, but the UI should remain responsive now.

I have also included an extra demo project called ‘Scanner’. It is able to transform an input stream containing integers, representing characters, into words and numbers (ints and doubles).  This doesn’t seem much, and it isn’t, except that it is doing this using a couple of flows and a general purpose algorithm (processing is still slow, mostly because the UI is trying to catch up).  This was the guts of the older ‘English language definition’ demo, which I have split into 2: the scanner and the language definition, which is no longer able to do any processing (all code removed). It’s an example of a more complex flow.

The scanner demo also has the number scanning problem fixed (numbers longer than 2 came out with multiple results). This appeared to be caused by deleting a couple of neurons to many (in the scanning algorithm).  I had already experienced the dangers of deleting neurons I thought were no longer used, but which were because of the splits. I will probably have to implement some sort of a garbage collection system to clean up unused neurons (but that’s for later).

Anyway, here’s the latest download (best to remove previous installation before installing this one).

Note: Deprecated!

This is the very first debugger I have written, and I am pretty proud about it! It’s not a masterpiece, but functional.  The code definitely could use some tidying up and some speed tuning wouldn’t hurt at all, but you can trace bugs, inspect values and follow the program flow, and that’s already something I guess. To explain the debugger to you, I thought it perhaps best to do it using some of the demo’s.  Simply open the ‘Echo words’ project to get started.

Set up

Before we send some input to the network, we need to set up the designer so that it is ready to debug:

1. We need to  put a breakpoint  on a statement. You can do this in a code editor. The very first code block that gets called in this network is the rules code on the ‘Contains Word’ neuron, so simply double click on the ‘Code: Contains Word’ node in the Projects tab.
   To set the breakpoint on the first statement, expand the ‘CheckStartTextBlock – space insertion’ code block and click in the little circle on the ‘if’ statement. This should make it turn red.
2. The designer also needs to be set in design mode.  You can do this by the drop down box on the main toolbar or on the debugger tab’s toolbar. There are 3 possible modes:
   • Off: no debugging is possible.  This mode runs faster.
   • Normal: The debugger stops whenever a breakpoint is encountered. When a breakpoint is encountered or a processor was paused, it is possible to inspect values.  This is how most other debuggers provide debug capabilities.
   • Slow motion: In this mode, the debugger automatically pauses and continues on every statement, creating a movie of the execution process. You can use the throttle bar on the left of the combo box to select the speed, which is updated real time. It’s possible to pause and continue this movie using the pause and play buttons on the debugger tab’s menu.  I have included this type of debugging mode because I believe it is sometimes useful to see the path that is taken to a specific point of interest. With this feature, you don’t have to keep pressing F6 to advance.
3. It’s also possible, but not required, to let a processor stop when an error was encountered. This is done by activating the ‘Break on error’ button. This only works when the error happened on a processor in debug mode.

Note that it is not possible to switch between different modes while a processor is running. You can only specify the debug mode for newly created processors.

An overview of all the breakpoints in the project can be found on the right side of the debugger tab. Currently it simply lists all the breakpoints, but new features should be added to this list shortly.

Start a processor

To get started, we need to send  some data to the network through it’s text-sin (sensory interface), so make certain that the ‘Echo channel’ is opened (a communication  channel is the visual interface for a sin). Go to View/Communication Channels/Echo channel, and make certain it is checked (and that the tab is selected).

Once the channel is open, type some text in the input section and send it to the network by pressing enter (or the send button). You should see a neuron appear in the left upper screen, which represents the input event (the neuron that will be solved by the processor). Your text will also appear as a string in the centre dialog screen. There will also appear an object in the centre screen of the debugger tab. This represents the processor that was started and which is handling the input event.

The processor overview contains 2 numbers. The first number represents the name of the processor. This value can be changed and is used to identify it between multiple processors. The second number represents the number of neurons that are left on the stack + the current neuron that  is being solved.

The buttons represent, from left to right:

• a toggle button that can be used to open and close the detailed view for the processor. 
• An indicator that is selected when the processor is paused. This is used to inform you on the run state, it’s not an interactive button.
• An indicator that lets you know that the processor is still running or not. When this is no longer selected, an irrecoverable error occurred in the processor and it is actually dead (in other words, you’ve got a problem when this is no longer selected).

Once a processor is started, the middle square in the right side group on the status bar will be blue.  This will remain so for as long as the network has still a processor running.

Detailed view

If you press the first toggle button, a detailed view of the processor should open in a new tab (using the same name as that of the processor).

This tab is divided into 3 sections (from left to right):

• The content of the execution stack.  This stack contains the neurons that have to be solved by the processor.  You can add and remove neurons from/to this stack using the Pop and Push instructions (The first neuron is automatically added). In our demo, there is only 1 item currently on the stack, that’s the one being solved. This item displays  a square in the stack UI element.
• All the neurons that were assigned as meaning to the links starting from the solvable neuron. The top one will be executed first.  A special case is the ‘Actions’ neuron, which will only be solved when all the others have been, no matter where it is found in the list.
• A stack containing all the processor frames.  A frame represents a single cluster’s code that is being executed by the processor. Since some statements call other clusters, like conditional statements, code blocks,… a processor will usually have a number of frames active. This provides an exact view of the processor’s current execution location.
  The following info is provided for each frame, from top to bottom
  • The code, in textual form. Each code unit can easily be selected separately for debugging purposes (check out the context menu on each item to see what you can do with it).  It’s not possible to edit in this window though (you have to open the editor for this).
  • The name (or id when no name is defined) of the neuron that defines all the code.
  • The relationship between the neuron that defines the code and the actual cluster containing the code.  This can be:
    • Children: in this case, the neuron that defines the code, is the cluster that contains the code.
    • Rules: the neuron that defines the code has a link with meaning ‘Actions’ to a code cluster.
    • Actions: the neuron that defines the code has a link with meaning ‘Actions’ to a code cluster.

The detailed view is great to have an overview of where we are in the processing stage, but it isn’t very useful to  debug a program flow. It’s better to use the code editor for this because it provides a better overview of the connection between the statements.  To do this, open the editor containing the code and make certain the the processor in the debugger tab’s center screen is selected.  This last action is very important for the following reason: a detailed view is opened for a specific processor, so a single tab always links to a single processor.  Code however is not specific to a processor, it can be called by many processors. So it needs to know the processor context for displaying debug information.  It uses the processor that is selected in the  debugger overview for this purpose.  This has a nice  side effect: when you have multiple processors running, you can quickly switch between active processor and view where each one is in the code. Note that the code editor will put a red square around the statement that will be executed next, to indicate execution location.

Do the debug

Code flow

We are ready now to start debugging the network. The first thing you can do is walk through the code using the following commands:

• Run (F5): continue execution until the next breakpoint or until the processor is finished (only active when paused).
• Next step (F6): execute the next step only and  pause again (only active when paused). Note, when  you do a step next on a split instruction that creates multiple sub processors, all sub processors will also wait on the next statement, they wont run wild, but wait until you tell them what to do, which is very useful for inspecting stuff.
• Pause: stop execution and wait until the Run or Next step command have been given (only active when running).
• Stop: Ok, this sounds silly, I know, but I haven’t yet implemented the stop.  I’ve always been able to stop in other ways (I’m running the debugger inside another debugger which is great for creating a schizoid coding mind). If you feel an urgent need for this command, let me know, otherwise, it’ll get in there,  I just have absolutely no time frame for this.

Inspecting values

When a processor is paused, it is possible to inspect the value of any item that returns a result, this includes: variables, globals, result statements, bool expressions and ByRefs. You can do this by selecting the item you want to inspect with the mouse, and pressing F7 or through it’s context menu (Inspect value). This opens a dialog with the debug info of the result.  This can be empty, 1 or multiple neurons.

Debug info for a neuron contains the following info:

• All the incoming links, where they point to and the meaning of the link (note that the meaning UI element still has to be changed into a debug neuron).
• All the outgoing links, where they point to and the meaning of  the link.
• All the clusters that contain the parent node.
• When the parent node is a cluster, all it’s children.
• When the parent is a cluster, the meaning of the cluster is depicted, in brackets, after it’s name.  This is also a debug UI element.

The debug info is depicted recursively for all neurons. This type of debug visualization is used in multiple places throughout the designer.  You might have noticed that the echo channel uses this UI element to depict the incoming and outgoing neurons. The detailed view of a processor also uses it to depict most of the neurons.

Watches

A final feature of the debugger is watches.  These allow you to observe the content of variables and globals in a list for a single processor or across all processors for those that are paused. 

This feature is best experienced using the ‘English language definition’ demo since this performs some splits while processing text input. I have put a breakpoint in the ‘Code: Stage 1.1’ page on the second if statement  (first child if in the left-side path of the only root if) and ran it until ‘Found == Sentence(flow)’ to get the screenshot on the left.

To add watches, drag a variable or global and drop it in the left part of the debugger tab.  This will add it at the bottom of the list. If you are dragging it from a code editor, it is best to hold the ctrl key pressed, so that the item stays at it’s original position.  Note that it’s currently not yet possible to drag from the toolbox.  This is a small bug that still needs fixing. Also note that it’s not yet  possible to remove variables (except by editing the designer file).  This command will be added very soon (just goes to show how fresh the debugger still is).

By default, a list of watches with their values for the selected processor is depicted.  The left side is the name of the variable (or it’s id if no name has yet been assigned). To the right of the name, the content of the variable is displayed. This can be empty, 1 or more neurons, all of which are depicted using the debug UI element for neurons.

If you switch to variable view, the left section of the debugger tab will only display a radio button for each watch. The middle section, which contains processor info, will now display the contents of the selected watch for each processor. 

Also note that, when a processor gets split up into multiple sub procs, the view will switch to a tree.  The root node shows the number of processors contained by the node. If a new input event is sent to the network before the previous has been processed, it is added as a list item, at end of the list. If sub processors split up, more sub nodes are created as children of the nodes that triggered the split (tree structure). This allows you to see how processors are related to each other.  At the moment you can only see the current state, through the tree/list structure.  In the future, an extra view might be added that shows a line view over time to show when and how may processors did a split and when they died out, although that’s just an idea at the moment, so don’t put your hopes up to see it any time soon, there are still far more important things to do.

Errors and warnings

If there are any errors generated by the code, either through the Error or Warning instruction or because of an error in the code, you can view exactly where it occurred. All messages are stored in the log tab. When they are blue, you can double click on them. This will open a code editor with the statement selected  that caused the log item (note: if it is somewhere in a sub section, this is not expanded automatically) Because you can use the same statement in multiple locations, only the first few will be selected, this is to take care of some problems with WPF’s standard controls (will be fixed in the future).

As you can see, it’s all still fresh,  but functional. The debugger has already become a central point of usage for me and I suspect this will only increase, so this will get some extensive testing early on. There are plenty of things that still need adding like conditional breakpoints, counts on breakpoints, enabling-disabling of breakpoints, more commands on the breakpoints-list (like clear, enable/disable all,…),… These things will probably be added as needed.

I am certain there is plenty more to say about NND’s debugger, I just can’t think of anything  anymore, so I guess I’ll leave it by this for today. It’s already turned out a long enough post.

Note: Deprecated!

Note: this is the second part of in a group of 3.

N2D is still in proto type stage, so there are plenty of idiosyncrasies to work around. Here are some tips and tricks that might make things a bit easier.

• When there is lots of code visible, things can slow down fast. (fixed)  To avoid this, use the drill down/up arrows on some statements like conditional statements to close as much as possible.  If you are drilling down and need to see more code, you can always open sub items into a new code editor.  This will speed up things dramatically. You can do this on conditionals, their parts and code blocks
• Be careful with copying items around (or using shift drag). The editor displays items that are used in multiple places, but you can still be surprised when you change something to an argument only to see it changed it multiple places. The same goes for breakpoints: if an item is used in multiple places and it has a breakpoint defined, it will break everywhere the statement is used.  Keep this in mind while using copy or shift-drag.
• I use the sync with explorer command (F4) a lot. This was mainly because the copy paste system wasn’t working yet.  But it’s still useful I think: it’s a quick route into the explorer.
• The Toolbox is really useful to build code.  If you select the ‘Instructions’ section, you get a good overview of all the available instructions, grouped by functionality.  To quickly collapse and expand the groups, you can use the context menu of the toolbar.
• It’s possible to add items to the toolbox, although this currently has to be done by hacking the “XXXdesigner.xml” file.

Final note

There are still a number of shortcomings in the editor that need to be worked out, more specifically:

• CPU and memory usage are a problem resulting in a very slow view when there is lot’s of visible data. This needs be solved by a custom control. (Done)
  
• Keyboard entry functionality, similar to the flow editor must still be implemented, this should speed up code editing significantly, compared to the all drag-drop or copy-paste solution.
• Clean up the views some so that they are clearer, compacter,…

Next up will be the debugger, I guess.

Note: Deprecated!

Note: this is the second part of in a group of 3.

All the different statement types, the general neuron types, some common static neurons, all the operators and the instructions can be accessed from the toolbox. You can easily drag and drop them on the editor.

Assignment

An assignment is used to assign one or more neurons to a variable. After the operation, the variable will reference the result of the right part of the assignment. The left part must always be a variable: regular or global, but no system variables are allowed.  What exactly is assigned to the left part, depends on the type of the right part.

• If it is another variable, the contents of the right part are assigned to the left part.
• It can also be a result statement, in which case the statement is executed and it’s result is assigned to the variable.
• ByRef statements are also resolved: the content of the ByRef statement will be assigned to the variable. This is useful if you want to assign a variable or a result statement to a variable and not it’s contents or results.
• In all other cases, the right part is assigned to the left part, so if you put a code block in the right section, it will not executed, instead, the variable will point to the code block.

(Result) Statement

A statement is used to call an instruction and provide all the parameter values for this instruction. You can call any type of instruction with a statement, even those that return values, although that’s not very useful cause any result values are lost when used in a statement.

You select the instruction using the drop down box.  The instruction itself is also just another neuron, that the statement links to.  This has some consequences.  For instance, if you want to find all the statements that use a specific instruction, just check the incoming links on the instruction.

Most instructions require 1 or more arguments. The values for these arguments are displayed in a list that comes after the instruction. This is a regular drop list, like a code list. Each  instruction should have a little description (you can see/edit this in the description tool-frame if you select the instruction in the toolbox).  This description should say all the required arguments and any possible result values.

A result statement works similar to a normal statement, except that it return a value and tend to be used as part of another statement, like an assignment, a Boolean expression or as an argument value.  Result statements are depicted using the same template, but with a slightly different color to make a distinction.

Code block

A code block is used to group other statements together. This allows you to reuse common functionality.  It is a bit similar to a function in traditional programming languages (like C, Pascal,..), except that there are no arguments allowed and it doesn’t allow for local variable values (scope locality), see Variables for more info on this one. Internally, a code block stores all the statements in a cluster that is linked to the statement.

It is possible to see the statements of a code block inline, by using the expand button in front of the name.  Note that this can slow the editor down dramatically.  Better is to open the code block in a new editor frame (make certain that ‘Statements‘ tab is selected).  You can do this through the context menu’s ‘View code’ command (can also be triggered from the main toolbar). If you want to reuse a code block, you can simply drag it from the explorer frame to the editor.

Even though you define parameters on code blocks, it’s still possible to pass along argument values.  Because the content of a variable isn’t reset when a code block is called, you can create some Variables that function as parameters, fill them before you call the block and access them inside the subroutine.  The same can be done for result values, you can return as many as you want, simply  define some variables and assign a value to them in your code block.

Conditional statement and it’s parts

A conditional statement is used by the processor to perform execution jumps (if, case, while, foreach,…).  Each conditional should contain at least 1 conditional part, but can have more. This depends on how it is set up. You add parts by dropping them in the ‘Children’ section of the UI element (which moves to the right).  You can also move them around, just make certain that the entire part is selected, and not just the ‘conditional’ section. When correctly selected, the part will have a selection border.

Visually, a conditional statement is always represented as a horizontal split in the vertical code branch.  When there is a repetition in the statement (for-each, looped, until, case-looped), 2 vertical lines are also visible, making the conditional statement boxed in. This is an easy visual queue to identify repetitions (so you can have a bit of a birds eye on the code to see the structure, without the details).

There are currently 5 different types of conditional statements (which can be selected from the drop down box): these are:

If statement

An ‘If’ statement can be compared with a structural language’s if statement. It can contain multiple parts, with only the last one allowed to have no condition. The first part who’s condition evaluates to true will have it’s code executed, all other parts are skipped from then on (so there is no fall through possible like in C’s case statement). After all the code in the part has been executed, the next statement after the if is performed, so there is no repetition in this type of conditional.

Adding statements to a part is done by dropping them in the ‘children’ section of the UI element. Because a condition must evaluate to either true or false, a Bool expression tends to be used, but a result statement can also be used. Simply drop an item in the ‘condition’ section.

Case statement

The case statement is, like the ‘if’ statement, very similar to it’s structural equivalent: the content of a value is compared against a number of different possibilities.  Each possibility is provided through the ‘condition’ of the part, which can be a static or another statement that returns a result. Only the last part can be empty, which will be executed if none of the previous values matched the variable.  An empty part is not required though. There is no fall through possible (unlike the switch in C).

When you select this type of conditional statement, the UI element will display an extra drop target for you to specify the variable  who’s content needs to be checked.  This has to be a variable (or global).

Looped statement

A looped conditional is comparable with  the ‘while’ statement, but with a twist: it is possible to define multiple parts.  It’s actually a bit like C’s while and if combined. The loop will run for as long as the condition in one of the parts evaluates to true.  Like the if statement, this part’s sub-statements will be executed.  The last part doesn’t require a condition, in that case, the loop will run for ever and will only stop when a break or one of the exit instructions is called. These instructions will always terminate a loop. If there  is no ‘empty part, the loop will stop when none of the conditions evaluates to true.

Until statement

The until conditional is similar to the looped variant in that it is repetitive, but it only allows 1 part that must always have a condition which is evaluated after all the sub statements have been executed.  When the condition evaluates to true, the repetition is stopped. This type of conditional is very similar with C’s do until loops.

Case looped statement

This conditional is a looped statement, but instead of using an ‘if’ kind of testing style, a case is used. This means that you need to provide an extra case item variable, like the case statement. The statement will loop for as long as a condition in one of the parts equals to the content of the case variable. The conditions in the parts should be static values, result statements, ByRefs, bool expression, variables or globals (so anything that can return a result). You can also specify an empty condition in the last part which will be activated if all of the other parts were different than the case variable.  This results in a never ending loop, which will only stop when the break or one of the exit instructions is called.

The case looped statement is a bit of a strange beast.  It’s origins can be traced back to Gene, where it made a lot of sense for parsing or generating stream.  Id don’t know if it’s useful in this context, so I simply left it in, you never know where this might end up: could be the dumpster, could be some cool trick.

For each statement

This type of conditional statement can probably best be compared to C#’s ForEach loop: it executes a peace of code for each neuron  in a list. Only 1 part is allowed. It’s condition should be a statement that returns some neurons.  This can be a variable or global that contains multiple items, or a result-statement that returns a number of items, like the one that returns all the children of a cluster.

When this type of conditional is selected, an extra drop target is visualized, which should reference a variable (or global) that will receive the current item from the list, as it’s content. This allows you to reference the loop item from within  the subroutine.  After the loop is done, this will still reference the last item in the list.

Bool expression

A Boolean expression is evaluated to the ‘True’ or ‘False’ neurons. It can be used as a conditional part’s condition. The left part is compared with the right part, using the specified operator. This statement type is probably recognizable from more classical programming languages.  There are a couple of important twists to this type of statement though:

First of all, because the operator is also a neuron, it can just as well be another result statement, or variable.   As long as you have something that can be evaluated to 1 or more operators, it’s ok. And this brings us to the second wrinkle:

Because variables, globals and result statements can return a list of neurons, so that the left, operator and/or right sides can have multiple items, the expression evaluates the lists using the following algorithm:

• If the left part has no result (no neurons) and all the operators are all ‘!=’ and the right side is not empty, true is returned, otherwise false.
• Otherwise: for all items in the left part, compare each item in the right using every operator.

Note, when there are multiple operators, it is like having an ‘and’ operator for conditions that work with the same left and right side, but different operators.  This will probably not be used a lot (I haven’t yet).  It’s just a result of the structure, so I let it be available.

All the operators are represented by statically declared neurons (so you can’t add your own). Most are recognizable from classical programming languages like C(++). You have:

The ‘Contains’ operator is a bit different. First of, there is no equivalent in traditional programming languages and it  requires a little more information to work properly: it needs to know which list to search: in the child list (in which case the left part should be a cluster),  or in the clustered-by list, which contains all the clusters that contain the neuron (this list is available in every neuron type, so the left part can be any type).

There is no operator precedence defined as in most traditional languages, since it is the structure of the neurons that defines the order.  This is visually verifiable by the statement’s container: first the left part is evaluated, next the operator and finally the right part.  If left or right has a sub bool expression, this is recursively repeated.

Search expression

This type of statement has already become deprecated. It’s been surpassed by the instructions, which provide more functionality and flexibility.  Nothing more to say about this, I guess.

Variable

A variable is a neuron that is able to store a reference to other neurons in the context of a processor. That is to  say, the content of a variable is only valid for as long as the processor running the code is still alive.  The contents of a variable can also be different for each processor. It will also be reset after the processor has completed evaluating all the links on a neuron and when a split callback function is called (scope locality).

A variable doesn’t have an initial value by default. It is possible to provide one though.  You need to unfold the drop location to make it accessible. This value can be another result statement, variable, global byref, or bool expression, in which case it is first resolved. In all other cases, the value is treated as a static.

Global

A global is similar to a variable, except that it has a different scope.  Globals are able to retain the same value for as long as the processor is alive while normal variables are reset at different times during the execution process of a processor. This is useful to pass along data generated during the execution of one link to another one.

Globals are also able to define what should happen during a split operation.  Possible values are:

• Empty: clear all values from the global in the new processors
• Duplicate: create duplicates from the content of the globals. A duplicate is a neuron with the same incoming and outgoing links, same value (in case of Int-, double- and Text-Neurons) and same children, but a different ID and not clustered by any item (so the clustered-by list is not duplicated, cause the clusters don’t get new neurons added because of a split).
• Copy: simply copy the content of the global to the new processors.

ByRef

The ByRef statement can best be compared with C’s ‘&’ operator, but instead of getting the address of the memory location, in a neural network, it prevents a statement that return a result (like a variable, result statement, bool expression) from calculating that result, but instead returns the statement itself.

Although this statement probably wont be used that much, it is very important for the split instruction to work properly, since this requires a variable neuron as one of it’s arguments.  Often, this statement is the only way to provide this value.  There are other situations where this statement can be used, but less often.

Lock

(Added 11/July/2010) The lock is used to combine a number of statements into a single  unit of execution, relative to 1 or more neurons and/or links. This statement can best be compared with C#’s lock keyword. When you lock a link, you need to specify from and to, but not the meaning, so each link lock actually requires 2 neurons to lock.

When you lock a neuron, the entire object is locked: incoming and outgoing links, value, parent cluster list and children. When you lock a link, only the incoming or outgoing section of the neuron is locked. This is important to keep in mind, if the locked items are operated on from other processors.

Don’t use the ‘duplicate’ instruction or perform any splits (with the split instruction) inside the lock. Although technically possible and allowed, it is not advised since this can and probably will cause deadlocks. Any object that needs to be duplicated during a split and which references the locked object, will cause the network the hang in a deadlock (although you can still terminate all the processors from the designer).

This concludes the overview of all the different statements.  Up next, some tips and tricks.

Note: Deprecated!

Today, I’d like to talk a bit about the code editor which is used to create and view executable data. Notice that I used the verb ‘designing‘ instead of ‘writing‘ code in the title. There’s a very simple reason for this: N2D doesn’t yet define a syntax for textual input of code, instead, there is the code designer which provides a visual view on the raw assembly code (so there is no complex conversion required). The idea is that it will eventually (I hope sooner rather than later) be able to understand natural language, making a custom syntax not necessary.  Because designing a good language, with accompanying parser can be tricky and time consuming, I opted to skip this step and instead rely on WPF for me to build a powerful designer with.  This has sort of worked.

Before you get started with this post, might I suggest you to check the Echo words demo explanation for a more general introduction on how to use the code editor. This provides a good introduction for most of the general concepts. This post will deal with some more technical details.  Because of the length, I decided to split it into 3 posts:

• editing techniques: that’s this post
• all the different code statements
• and: tips and tricks.

Editing techniques

Code editing is currently based on a drag drop paradigm in which you drag statements to the editor and drop them at the appropriate location.

• Drop locations are indicated using a gray, rounded border, containing the name of the drop target, like ‘Args’, ‘Children’,…
• To add statements at the bottom, simply drop them somewhere in the white space below or at the side.
• To insert items, drop them on the little black line above the statement.  The previous statement will be moved down.
• When you drag an item and drop it somewhere else, it will be moved, except if the ‘shift’ key is pressed, in which case the item will remain at it’s original position and will be copied to the drop target (no duplicate, but the same neuron is referenced. This is important!)
• You can also copy and paste items. During the copy process, the id’s of all the selected items are placed on the clipboard.  A paste will get the id’s from the clipboard, transform them into neurons which are put in the selected place.  You paste in drop targets, simply click in the drop target (there is no visual queue yet to indicate that the drop target is selected, this still needs some work) and paste. Note that copy-paste doesn’t yet work across multiple applications.
• If you press delete while an item is selected, it’s reference will be removed from the editor.  If the underlying neuron is no longer referenced anywhere else, it is removed.  All neurons that it references which are also no longer referenced are also removed (cascading delete).  This is the most used deletion method in code editing.  This means that when you delete a statement, all the parameter values that are no longer used, are also deleted. Or, if you delete a code block, all the statements that are no longer used anywhere else, are also deleted from the network.
• If you want more deletions options, use the ‘Delete special’ (ctrl+del)  command.  This shows a dialog which allows you to specify the deletion method you want:
  • simply remove the reference, but always leave the neuron in the network
  • Remove the reference and when  no longer used, delete it (as with the normal delete)
  • Always delete the neuron.Referenced neurons can be
  • left alone: they are not deleted
  • deleted when no longer referenced
  • always deleted
• All neurons can have a ‘display title‘. When the designer finds one for a neuron, this is displayed when possible.  You can easily change this name by pressing ‘F2′ or through the context menu of the selected item.  This is very useful for variables and globals.
• Most statements have a little circle in the front of the image.  This is a toggle button to enable/disable a breakpoint on the item. This is part of the debugger integration into the editor (more on that later).
• It’s also possible to change a statement from one type to another one through the context menu.  A warning though about this command, you might loose data with this command, that can’t yet be undone (no proper undo support yet for this command).
• If you simply need to move a statement up or down, you can also use the context menu, which has a submenu for moving items around.

Anyway, here are some of the most significant changes:

• Cut/copy paste system: this was still missing.  The backbone is done, the code and flow editors already have a working implementation, the rest should follow.
• Deletion system: I updated the deletion system so as to better reflect common editing practices.  This is again done for the code and flow editors, the rest will be done later.
• undo system: There were still some issues with undoing network operations.  This should be fixed.  The mind-map editor still requires some redoing in this area (and many more by the way).
• Debugger: This is starting to become really useful.  I would not have been able to get the scanner demo working without this. I’ll explain more about this one in a later post cause it definitely needs some explanation.
• Demos updated:
  • I have split the echo demo in 2: a word echo demo and letter echo demo. Both give a very basic implementation of 2 different coding techniques to use in neural networks: a singular operation, and a collector.
  • The English language  def demo is currently able to scan incoming letters and turn them into words and numbers (although there’s still a problem with the numbers).  This scanner algorithm (which really is a general purpose flow interpreter) will be explained later, cause it’s a bit funky (to say the least). You’ll also probably notice that the UI needs some serious speed tuning in order to keep up with the engine.
• Execution engine: this is where the major hacking had to be done, the axe is still glowing red hot, but it’s running, oh yea, it’s running.
• Some new instructions: More will probably follow, but I’m beginning to get a better picture of instruction requirements.  This is starting to stable out.
• lots and lots of small fixes.

The new release can be downloaded from here, or through the usual links in the menu. I’ll probably be spending some more time on posting new entries to explain the debugger and some algorithms. I might also need to start explaining how you actually do you’re own coding in NND. I’d say: stay tuned.

Note: Deprecated!

Just a quick ‘in between’ note. I had to make a small change to the flow editor because a flow can now have an attribute ‘FlowIsFloating’ assigned (through a link pointing to ‘True’ or ‘False’). It is used by the scanner and parser routines to find flows that can appear anywhere in a stream.  This provides a generalized manner of defining this property without having to create custom routines for each set of flows. Spaces between words are good examples where this is used. The context menu for the flow name in the overview list on the editor contains a toggable menu item to activate/deactivate this attribute.  A green line in front of the flow indicates it is floating.  Here’s a screenshot:

Note: Deprecated!

After using the flow editor a bit, some of the more annoying errors became pretty obvious, so it’s time for a new update (well, full install again) to the flow editor. Besides the more obvious bug fixes, I’ve also introduced some new functionality, notably:

• shortcuts ‘N’ & ‘ctrl+N’: add/insert a new neuron.
• shortcut ‘R’: toggle ‘selection Required’ for current option/loop.  This is an important feature that I missed (in gene this was no problem since you could use an empty condition which is not possible here).  The problem is this: sometimes,
  an option or loop has to require at least 1 selection between a part (so you can’t skip the condition or flow), sometimes this is not required.  To allow a distinction between the 2, an extra ‘attribute’ is attached to the conditional (the loop or option), much the same as how a selection between loop and option is done.
• I changed the layout from wrap-panel to horizontal stack-panel, which is more logical to work with.

Note: I have updated the table on the first post about the flow editor so it contains the new shortcuts.

I will probably also have to implement a new statement type, to allow for grouping very soon.  It’s not really needed for defining the flows, but I think it can be useful later on, during the parsing. For instance, the expression: (verb "ing") means that you are expecting a verb, with ing behind (standing, listening,…).  You don’t need the brackets to define it, but it makes more sense while parsing, if the grouping statement has an attribute attached to it that indicates how it needs to be interpreted.

I have also noticed a ‘logical’ error in the way that I implemented keyboard navigation: some keys have been reversed. So when you are on an item inside a part and need to get to that part, use the left arrow instead of the right.  Being able to select
a part is important to continue adding items at the end again when you have added a conditional like in the next image. It’s annoying, but fuck it, this is one for later on.

I’m also not yet very happy with the deletion functionality.  As it is at the moment, you can remove or delete.  Removing simply takes away the reference to a neuron in a list, deleting, will remove the neuron from the brain.  This is annoying, cause you usually want to remove statics and delete conditionals and parts.  If you want to remove an entire condition, you need to first delete all the parts separately.  A better scheme would be as that of the code editor, which will check for this type of situation.  Again, this is a minor thing, so I will add it to the feature list. At the moment, there are more important things to do though.

A bit more interesting, I have also began work on an English grammar definition.  Here’s a screenshot of a part:

It’s off course not a definition that covers the complete language, but I think it should be enough to build a natural language interface (update: Oh boy, was I wrong with this one ), which should be interesting. I’m certain I missed a bunch, I still need to define adverb handling (which will be treated in a pre parser, a bit like how comments are handled in more traditional parsers) and the scanner also needs commencing.  And expect some shuffling around and updates as I implement the parser for this definition.  At this stage, extra attributes will probably be added, to handle more semantically oriented parsing. I have no idea how long this is going to take me, but I expect some issues with the debugger, so I might be out for a week or 2 for the next update.

Note: Deprecated!

I’ve added support for flow editing, which should be the final ‘big’ editor required to create useful networks. From now on, only finishing functionality and bugfixes. It took a little longer than I had originally planned, mostly because of a struggle
with WPF’s keyboard navigational system, which is crap. I finally got it working though (it also supports drag and drop, copy-paste still needs implementing), hope you like it.  You can download the program from here. Unfortunately it’s still a full install, so it’s the whole 40 and some megs download.  I’ll try to find some time to create an update installer so that the download size can be minimized.

The editor is still a bit rough, but it should be useful. I’ve tried to make it more text oriented, so you can easily navigate/add/remove items from the keyboard.  In the background though it’s still listboxes so coding the view and data was easy and fast, just that creepy WPF navigation system, off course more importantly, it makes certain no illegal input can be provided.

Anyway, here’s a screenshot:

Basically, this Flow editor describes how a noun can be found in a stream: a noun should start either with an article (which is either ‘a’ or ‘the’) or a number, followed by 0 or more adjectives finalized with a single noun.  Ok, there are gaps here (it’s still a sketch to show the editor): what’s a number, adjective or noun and how to find them.  These things will be explained in the demo, but basically, you use converters (functions that can transform a neuron into another one, like the object ‘house’ into the neuron ‘noun’) or some other info that can be used in a search function which can be attached to the flow items.

To create one yourself, go to ‘Insert/Flow editor‘, use the toolbar button ‘Create new flow editor‘, use the toolbar button on the ‘Project‘ overview tool frame or use it’s context menu. Press ‘F’ to create a new flow followed by a ‘.’ to select a static or ‘['/'{' for an option/loop (press '|' to add new parts in a loop or option). Here's a complete list of available short cuts:

A final note perhaps on how to use these flows in a neural network.  The thing is, this is really up to you, the application doesn’t make any hard-coded use of them. Though, there will probably be a couple of default algorithms that can be reused. 
The basic idea is relatively simple: when the first neuron comes in, search all the clusters to which it belongs with the meaning ‘Flow’, ‘FlowItemConditional’ or ‘FlowItemConditionalPart’ and store the result list in a cluster.  When the second comes in, try to find all the clusters of the previous result set that allow the new neuron to follow the previous one, all clusters that don’t allow this are removed from the result set.  Various clean ups / lookups can be performed during 2 incoming points. When a ‘flow’ cluster is found with code attached to it, execute this. This off course, can be made as simple or as complex as you want. More on this in a later post. Fuel is depleted for today.

The solution was very simple and pretty elegant: use a different list to store the result of a split. So now a processor has 2 internal lists: the execution stack used to build the code that should be executed next and a list with the split results (unique neurons) and their weights. A couple of new instructions had to be created to control this list.  No biggie: an Add, Remove, Get and Clear.

Anyway, I’m updating all the documentation now.  If I mist something, don’t forget: 2 separate lists. The next post will probably be a Demos explained on the split.

The problem, you see, is the fact that N2D is a Designer, running on top of a neural network, accessing it’s services.  So when you are designing the network, it is also running.  This means that all the stuff you type into the text-channels, will also be stored in the project. Something you don’t want while performing tests.

To overcome this issue, N2D can use a special directory as a sandbox in which it can copy projects that need testing. The sandbox project will be started in it’s own designer so you can play with it, with all of it’s features (including neuron streaming) without messing up the actual data.

To run the current project in a sandbox, use the ‘Sandbox‘ toolbar button or menu item Debug/Sandbox. This will save the project, copy it over to a temp directory (selectable in the ‘Tools/Options‘ window) and start a new session.

When N2D is started in Sandbox mode, it will show this in the lower right corner, like so:

Note: Deprecated!

Most development tutorials start out with an ‘Hello world‘ type of application.  This one doesn’t.  The issue I had was the fact that a neural network is inherently a responsive system, that is, everything it does is a reaction to input.  So, I figured, to illustrate this better, I could show a simple echo application.  It’s just as simple to make as a ‘Hello world’ thingy, but actually does something:
echo back everything sent to it.

To start the demo, go to {Windows start}/N²D/Demos/Echo words if N²D isn’t already running, otherwise, do File/Open and browse to the folder {My documents}/NND/Demos/Echo. Note that a project is opened by selecting a folder, not through a file (this might still change in the future, if required).  If the project has already been opened, you can reopen it using File/Recent projects. Once opened, you should see something like this:

Note that the echo channel (a text channel) is shown by default. This is because I saved the project with the channel opened: visibility of a channel is stored, not for other items at the moment.  You can close this window by right clicking on the tab and selecting the close command. You can also control which channels are visible through the menu: View/Communication Channels. The designer will
create a communication channel for each sensory interface in the network.

The Text channel contains the following items:

• In the bottom: a Text box to enter text you want to send to the network
• in the middle is a list box containing the current conversation: all the text you sent interleaved with the answers of the network.
• the top row contains 2 listboxes which contain visualizations for the neurons that were processed as input by the sin (left one) and received for output (right part).  Note: selecting a line in any of these 2 lists will also select the corresponding line in the conversation section and visa versa.
• with a toolbar on top of it containing
  • a button to send the text to the sin.
  • a combo box to select the way the sin splits up the input strings into neurons
  • Some editing commands to cut, save or clean the conversation log
  • a toggle button for turning the speakers on/off

If you make certain that ‘Speaker‘ is pressed and your loud speakers are turned on, you should hear ‘Type here’ once you press the ‘send‘ button (and haven’t changed the input text). Some items should also appear on the screen, something like:

What just happened? Well, a whole lot really:

• The text channel displays the string ‘Type here’ in the central dialog overview.  This is done by the designer.
• next, it sends the string to the Echo channel’s backing sensory interface which splits it up into 2 words: ‘type’ and ‘here’.  This is because it is in ‘Use Words’ mode.  There is another echo demo, called ‘echo letters’ which is programmed to ‘Use letters’ by the way.  If you want, you can open this and try it to see the difference. In the ‘echo words’ demo, the 2 words are converted into text neurons (by searching a dictionary of already created text neurons or by creating a new one, which is stored in the dictionary. These 2 text neurons are linked to a newly created neuron which represents the input event.  This conversion is done by the sin itself.
• The sin also raises an event to let the channel know about the neuron it just created as input value which is displayed in the top left list.  A processor is created and the neuron representing the input event is pushed on it’s stack, which starts of the network. Up until now, we haven’t really used the neural network yet, everything was done using traditional code. This changes once the processor is started.
• The processor ‘executes‘ the code attached to the links between the ‘input event’ neuron and the text neurons (how this code gets there and how it looks like will be dealt with later).  This is the neural network doing it’s thing which will eventually execute several ‘Output‘ instructions, sending neurons back to the sin.
• When the sensory interface receives these neurons, it converts them back into a string and raises an event. And so we get back into the world of regular code.
• The communication channel, which receives this event, displays the text in the conversation dialog and the neuron(s) that represent the string in the top right list box.

Ok, but what exactly happened in the neural network itself? Truth is, not much as you will see. It’s time to show some code. If you select the ‘Project‘ tool-frame, you will notice 3 items: a mind map (more on that later) and 2 code items, each one is simply a reference to a neuron in the network. To open them, double click on the item.  Let’s start with ‘Code: ContainsWord(in)‘.

Mmm, interesting. All GUI-like is probably the first thing you’ll notice. Indeed, there is currently no textual method for entering code, as more traditional development environments tend to use.  I choose this way for 2 reasons: first, it’s faster to develop (at least, in WPF that is) and it lowers the entry barrier since you can visually put items together instead of having to conceptually create a peace of text in your mind, requiring a grater level of knowledge. In the long run though, this technique tends to be slower than writing code and it also somewhat limits the amount of code displayed, but be patient, there is a solution for this.

Lets first go over the 2 lower tabs.  These represent the 2 different code lists that can be assigned to the neuron: the ‘Rules‘ and ‘Actions‘ list. Note that these lists are only created if there is code, for as long as you don’t drop any objects on the design surface, there isn’t actually a code list assigned to the neuron. The bottom tab-strip also has a context-menu which allows you to toggle wether this neuron is shown in the project tree (under the currently active folder, or the root if there is no folder). You can also create new code clusters from the sub toolbar, but remember these are code clusters (which can also point to other code clusters).

Both lists are called at different times, and serve a different purpose. Code in the Rules section, is called when this neuron is used as the meaning for a link that is currently being processed by the processor. The Actions code list is called by the processor when this neuron was processed by the processor and all other links have already been handled. Usually, you don’t program on the last tab (actions), it is provided to ‘view’ the code that was attached to it by the sin that generated it (see further). Note that there are other tab possibilities, but these have no importance for now and will be explained later on.

So what does this actually mean for our example: well, the ‘ContainsWord(in)‘ neuron is used by the sin as the meaning for the link between the generated neuron and each word in the input string (this is pre defined), so for each word in the input text, it will call this little peace of code.

As you can see there are only 2 statements in this code list. The second statement calls an ‘Output’ instruction which will send the content of the ‘CurrentTo‘ variable to the content of the ‘CurrentSin‘ variable, in other words, this is the actual echo. Both variables are system defined and are filled in by the processor, they can not be modified, unlike normal variables. You’ll notice that they appear to be multiple variables stacked on top of each other.  This is to indicate that the same neuron is used in multiple places.  It’s a visual cue to indicate if a code item is used somewhere else or not (ahh yes, unlike regular code, these are neurons, all linked to each other, there is nothing preventing you from literally using the same statement in different places). That’s it for doing the echo really.  The rest of the code is simply to make certain that the sin will put all the words in 1 line with spaces in between them. This is spread out over 2 locations: the first statement and the ‘Input actions‘ code list of the sin.

The first one, is a ‘Block‘ statement, which can best be compared to a ‘Function call’ except that there are no parameters or return value allowed. To view it’s content, use the drop down arrow for an inline visualization, or ‘View code‘  in the context menu to display the code in a new window. You should see something like this:

Note that the code is located in the ‘Statements‘ tab this time if you opened a new view. That’s because a code block has an extra known code cluster attached to it: the code for the block.  It can also have a rules and actions list, but these are not used in this example (and probably will rarely be used this way on code itself).

Ok, and what exactly does this code do? To explain that, perhaps a note first on how a text sin converts output data. Whenever it receives a neuron, it is converted to a string which is than sent as an event to the outside world.  Of course, in our example, we don’t show each word on a separate row, instead, we show the entire input statement on a single line with spaces between the words.  This can only be done if the sin knows where a sentence starts and where it ends, otherwise it simply sees a stream of output data. So to accomplish this, there are 2 special neurons that it recognizes: ‘BeginTextBlock‘ and ‘EndTextBlock‘. When these are sent to a text-sin, they are not converted, but instead, BeginTextBlock will make it accumulate all the strings until the EndTextBlock neuron is received. So that’s what this little bit of code does: it checks if this is the first word being processed, if so, the ‘BeginTextBlock’ neuron is sent and we store a switch so we know of this event.  If it wasn’t the first one, we sent a space to the sin, so we don’t concatenate the words. The ‘EndTextBlock’ is handled in a different way, we’ll get back to that later on.

First a bit about the different statements that were used in this little code block.  The top most one is a conditional statement, set to ‘if’.  N²D takes conditional statements even further than Gene and completely unites all different conditionals into 1 and the same statement type, with an option to switch between them.  So a Do-while, While-Until, For-Each, For, if, case, and case-loop (not possible in other languages) are all combined into 1 statement, with a single neuron that switches between the 2. This approach is taken because of it’s ease in coding: it becomes a lot easier to change an if statement to a loop with multiple if’s: simply make it a loop (yes, you can have multiple conditions in a while loop). To design this bit of code, you can simply drag a ‘Conditional statement’ to the designer and drop ‘Conditional parts’ in the ‘Children’ area to the right. A conditional part forms a single trunk of the evaluation tree. It contains a list of code items (which can also be dropped on it’s respective ‘Children‘ drop area) and possibly a condition that determines if the part’s code is executed or not. This condition should evaluate to the ‘True’ or ‘False’ neurons.  Usually this is done by a Boolean expression (as has been done in this example), but this is not required, it could just as well have been a result statement that returns ‘True’ or ‘False’.

A word perhaps on the little arrows found on the variables to the right of the name.  These allow you to assign a default value to the variables.  In our example, we can’t use this cause all of them are system variables who’s values are controlled by the processor.  Regular variables however don’t have a value the first time they are accessed in the context of a processor.  You can use this little drop target to assign a default value.  This can be a constant, or another statement that can be resolved to one or more neurons.

Another important fact about these variables is that they can point to 1 or more neurons.  They actually function as buckets.  This is very useful in that there is only 1 known type: a list of neurons.  This list can contain 1 item or more.  Often times, you will use a split to resolve multiple values to 1.

The last part of this demo can be found in the ‘Code: Echo channel‘ Project item, which is actually a reference to the text sin.  If you open the ‘Input actions’ tab, you’ll see the final 2 statements for this demo. This tab represents a code cluster that the text-sin attaches to each input neuron it generates so the code can be called after all other links were executed.  In our example, we do 2 things: let the sin know the sentence is closed so the string can be displayed and delete the input neuron, so we don’t let the network grow indefinitely. By default, all data is always stored, if you don’t want this, like in this example, you will need to manually delete the neuron.  All of it’s links and stuff will be cleaned
up automatically.

The notion that the code is spread out over different locations and called at different times is an important point: a neuron network can not be programmed in a monolithic fashion as a single code block. Code is always divided into small chunks, which run at different times, in an undetermined but predictable order.  This is where a lot of it’s power comes from.  Once you grasp this, you are a long way into designing something truly beautiful.

Gosh, dear all mighty, this was a climb indeed. I’m glad you were able to get till here, all the way to the end. Even though this is a small demo, it packs a punch when it comes to new ideas. There are plenty more areas to cover, but I am certain that once you become more familiar with the basic concepts explained in this demo, the rest of the ride will be downhill. More demo explanations to come soon…

Editorial Note: This was one of the first neural algorithms I ever wrote. In fact, it was even used to get the initial engine running. As the core and my skills progressed, it became clear that this little example can actually be written in many different ways, some of which better and faster then this example. As of version 0.6 of the designer, I’ve included an improved version of this algorithm, which uses the minimum nr of statements (I can think of) to achieve a simple echo: 4

• Output(CurrentSin, BeginTextBlock)
• Output(CurrentSin, CurrentTo, ‘ ‘)
• Output(CurrentSin, EndTextBlock)
• Delete(CurrentFrom)

Note: Deprecated!

Despite it’s youth, N²D is already quite a large and rather complex application. Things like this tend to have a steep learning curve.  Of course, climbing a mountain is a lot easier if you have a map and proper control over your equipment.  I am working on the map, tools are up to you though.  There are many ways you can play with the application without actually having to design a proper neural network. Here’s a short guide.

Viewing and importing thesaurus data

N²D has access to the English version of the WordNet database through a sensory interface (input/output port). Apparently there are WordNet databases for other languages, but these are not yet available. To open the view for the WordNet data activate the menu item: View/Communication Channels/WordNet (note: opening for the first time can take a little time as the database needs to be opened, should get fixed somehow in the future:done).  This should show something similar to (although with less text):

You can search for words in the database.  All known meanings will be displayed. You can also select to show a specific relationship type with the drop down list.  If ‘As Part’ is pressed, all entries that contain the specified text will be shown.  You can either import a single meaning separately or the whole lot, including all the relationships (advised) and see the network grow.  The system should be able to do multiple imports at the same time (you’ll notice that a single import can take a long time, depending on how many relationships there are for the word). While the network is learning, the green box in the right bottom corner is filled.

You can browse through all the imported items from the thesaurus view.  You can select the type of relationship you want to see.  All root tree items represent root neurons for the specified relationship. With the text box, you can search for words in the tree. All neurons can be dragged to other locations (like mind maps for instance, see further).  It is also possible to add items to the tree by dropping them on the level they should reside.

Update: Aici doesn’t use words that are imported from this data source. I have found the thesaurus tree a bit awkward to use, and more importantly, lots of relations, useful to Aici are missing, while hordes of others remain unused.

Creating Mind maps

Mind maps allow you to draw neurons in a free form manner, similar like vector graphics. There are multiple ways to create new mind maps, you can:

• use the menu: Insert/Mind map
• use the toolbar button (4th from left: only valid in image, App has been updated)
• or when on the ‘Project’ tool window, use the local toolbar button.  From here, you can also create sub directories, change the name of the project items (F2),…
  Note: Mind maps (and all other project items) are added to the currently selected folder or the folder that contains the currently selected item.

Most of the mind map commands are currently only accessible through it’s context menu. For creating new items, you can also use the toolbox. You can add different types of neurons or change them to another one.  Watch out with changing the type though, cause this might loose data sometimes (for instance, when you change a cluster into a regular neuron).

If you select a neuron, you can use different ways for showing ‘related‘ items, like children, incoming or outgoing links and clusters to which the selected item belongs. You can also manage the clusters to which it belongs.  If you have multiple items selected, you can quickly create a cluster for them through the ‘Make cluster‘ command.

The easiest way to create a link between 2 items is by selecting them both (the first will be ‘From‘, the second ‘To‘) and activating the ‘Link‘ command. This will show a dialog with From and To already filled in, so you only have to select the meaning of the link.

As a side note, ‘Sync with explorer‘: a very useful command, accessible from the main toolbar, allows you to quickly find the selected item in the explorer.  This can be used from most places in the designer (all items that have a neuron as backing).

Drawing

Note: Drawing features have been moved to the next version.

Yes, N²D is also a drawing application, although some further evolution is clearly needed in this area.  Drawing capability is provided so you can work with the visual sensory interfaces. To create a new image channel, activate the menu command Insert/New Image channel.

Perhaps a word about the difference between a sensory interface and a communication channel: the underlying network has sins, which are also just neurons and have no ‘functionality‘. N²D, the designer, creates a ‘communication channel‘ for sins so you can work with them. If you use the network in a different application, the communication channel might look completely different, but the sin is still the same.

An image channel currently has 2 sections: the left part to manage/import multiple images (which is not yet fully functional) and a right part that will eventually contain 2 sections: an input and output section.  The output is currently not yet visible, but is supposed to function as the output part of the sin, so you can see images/videos that the network generates.

The input section works as an ink canvas, so you can use a pen table, and apply pen pressure (this can be toggled). The shape, size and color of the pen point can be selected. It is also possible to switch to a gum or selector.

Commenting

All neurons and some other types (like mind maps and frame editors) can have a description.  This is a FlowDocument (rich text format), so you format the text. Spell check is also provided. The description of the currently focused item is shown, you will have to make certain that the ‘Description’ Tool window is visible.

Standard text editing functionality is present:

• Select the font type
• and font size
• Bold/Italic/underline and Outline
• Bullets and numbering
• Spell check

As a side note: The selected item is a NeuronCluster, so it can have child neurons.  The explorer in this screenshot, is set to display it’s side panel which contains all the children of the selected cluster. Just like from the explorer, you are able to drag items from the left panel.  It’s also possible to drop neurons on the panel, which will add them as children to the selected cluster.

Creating frames

Frames are a form of compound objects (a group of neurons that form a single unit) which can be used to translate input from one form to another and to connect code to these compound objects. They are created in a similar way like mind maps, and can also be managed from within the project tool window.

To create a new frame, you basically push the toolbar buttons in sequence:

• The first one will create a new frame (a frame editor can contain multiple frames,usually they are related to each other somehow).
• Next, you create a frame element for the selected frame (make certain there is one selected, there are still some bugs in this area). You can also drop items to be used as frame elements.
• The third button creates a new frame evoker (all neurons in this list are put in a cluster ‘Evokers’, so you can easily do searches on them, more on that later). Like frame elements, you can also drop already existing items in there. Evoker specifications are now done on the same page as the frame elements.
• On the last tab, you can edit all the sequences for the frame.  Use the 4th toolbar button to add a new sequence, make certain it is selected and use the buttons to add/remove or move up and down the frame elements from the first tab to build a sequence.
• The last toolbar button is used to display the ‘Import from FrameNet‘ dialog, with this frame editor already selected. FrameNet has proven to be somewhat incompatible. You can still view this data, but no longer perform an import.

Note: although frames have expanded considerably since this was written, most is still valid, except where indicated (5th august 2010)

Importing frames

As already mentioned, FrameNet’s data structure hasn’t been found very useful in the creation of Aici. Frame format definitions have drifted since, so import is no longer available.

If you don’t feel like creating your own frames, you can always see if they have already been included in FrameNet, which you can import, either from the menu Tools/Import from FrameNet or from the
toolbar of a frame editor.

You can search FrameNet using the textbox in the upper right corner. The first time you select a row in the upper list might take a little time, this is because he is loading the WordNet database (which takes a little time). This is required to provide a mapping between FrameNet and WordNet data (not provided by default). Some mappings have already been created but most still need to be defined.  Each time you display this dialog and change some mappings, these will be saved.

There are many more things to discover in the designer, but I’m all out of words for today, so I’ll continue this on another occasion.  In the mean time, lot’s of fun playing with your new toy.

The whole thing is written in C#, the designer in WPF.  So for the time being it is strictly windows stuff (XP or vista). The core neural network and sensory interface libraries should be portable to Mono but
the designer will be a tat difficult. I don’t know yet how to tackle this one.

With regards to processors I can be simple: the more the better.  This is a massively multi threaded application, so the more cores the faster it should run.  I only have a dual core here, so I haven’t had the change yet to see it running on lots of threads simultaneously, but I would love to find out.  As I am writing this, I am sure there will still be a lot of bugs to filter out.

So to recap:

OS: Win XPSP2/Vista
Platform: .net/WPF
Hard Disk: + 450 MB
CPU Cores: as many as you can find