Having Trouble with Tantalizers?

During the 1970s I did a lot of work testing a heuristic problem solver called Tantalize - which was written in CODIL. The following news item was published in the New Scientist of 21 August 1975 and as I will be referencing this in the next blog post [insert link] I have decided to reproduce the original item.

For those who have trouble solving the Tantalizers that run each week in New Scientist, Dr Chris Reynolds of BruneI University has developed a computer programme.

A Chatbot spouts rubbish to the media about the Turing Test

Alan Turing has set a hard test
To find out which computer chats best
About speedboats or fiddles
Or solving hard riddles
And fooling a real human guest

At the end of last week the media were full of stories of how Eugene Goostman, a guinea pig-owning, 13-year-old boy living in Odessa, Ukraine, (actually a computer program) had passed the Turing Test. I suspected it was another case of academic hype relating to those magic words "Artificial Intelligence" fooling the press and waited a few days for the reaction. As I expected the criticisms soon appeared - such as Celeste Biever writing in the New Scientist, the report That Computer Actually Got an F on the Turing Test on Wired, and Mike Masnick, who tears the whole publicity stunt to ribbons. Among many well-directed comments he says.

The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction.

The Turing Test - Picture from 1clicknews

It helps to understand that Eugene Goostman is one of hundreds of bastard progrm descendants of Eliza, and early chat-bot program which attempted to carry out a simple conversation with a human being via a simple teletype. The "rules" in writing a chatbot seems to be that you keep on adding ad hoc rules to try and conceal the fact that your program does not really understand the human, and that the program writer does not understand how the human brain actually works. If all else fails the program takes approach that politicians use when asked an embarrassing question - you repeatedly try to change the subject of the conversation. The difference is that a politician knows what the answer is but does not want to admit to the truth, the chatbot doesn't want to revel that the human input means nothing to it - and is trying to evade revealing that it is merely a stupid computer.

It is worth looking at a hypothetical visual version of an interrogation test, as it might have been carried out a few decades ago. Via a normal TV screen a human is shown views of different places around the world. Some are real photographs and others are computer generated images, and the aim is to see if the computer can generate images from a general verbal description and fool the human into thinking the computer images were taken by a camera. The person programming the computer knows that the task will be difficult and narrows the scope of the test - for instance by artificially confining the views of the Rocky Mountains. He does this because he knows all about the mathematics of fractals - which can be useful in generating variations in landforms - and knows that it is easier to computer produce realistic images of forests of conifers - rather than mixed deciduous woodland. Because at least some of the landscapes should include signs of human activity special code is added to generate log canons, roads wandering along valley bottoms, and even tiny images of cars on those roads.

While such computer graphic techniques, when fully developed, have proved very valuable in constructing realistic sets in the film industry the guessing game tells us nothing about how camera lenses and photographic films combine to produce the "normal" images. Similar limitations apply to Eliza-style computer programs such as Eugene, which actually say very little about haw the brain works. Perhaps 40 or 50 years ago some people honestly thought that by writing such programs would tell use something about how humans think. Since then many millions of man hours by academics and students in university departments have written chatbot descendants of the Eliza program usually driven by a competitive urge to write something better than their rivals. The comparative lack of progress (especially if the effects of the increases in raw computer power are discounted) has clearly demonstrated that the Eliza approach is a blind alley as far as understanding human intelligence is concerned.

Alan Turing Statue at Bletchley [From Geograph]

 Of course it was interesting that the Royal Society ran a competition to help remember the 60th anniversary of Alan Turing's death, but the competition has really demonstrated is that human brains (and not just media reporters) are not very good at critical thinking and will happily accept the hype of a well presented public relations story without any understanding of the real science that lies behind it. 

THis problem kis nothing new. One can criticise many of the early Artificial Intelligence researchers as being more interested following an academic career based on playing games than in understanding the real world problems that a human brain has to deal with. In an ideal world scientists should be free to criticise weaknesses in other people's research, and to accept such criticism of one's own research in good faith. However this is not an ideal world and it may be that my failure to get research grants and papers published in the 1970's was because my views on the establishment research meant that doors were being slammed in my face.

At the time much of the research was into game playing (especially chess), solving formal logical puzzles, and writing Eliza style packages which generated text which superficially resembled natural language. My research had a very different background. The trigger was a study of a massive commercial sales accounting system where the trading rules were always changing due to a range of market forces, and where there needed to be really good two-way communication between the computer and the human user. I realised that the approach could be generalized to handle a wide range of real world tasks where it would be useful to have a system that could work symbiotically with the user. In the early 1970's my ideas were still embryonic and, fooled by the hype, I initially ignored Artificial Intelligence research on the grounds that it was tackling a different kind of difficult problem. When I mentioned this to a colleague he said that once you got under the glossy cover most A. I. research was trivial compared with what I was trying to do, and he loaned my a copy of a newly published Ph.D. thesis on the subject.

Perhaps I should not have been surprised with what I found. My research involved tasks involving many and dynamically changing rules, with data which could be incomplete or poorly defined, and when there might be no immediate answer, or many. The Ph.D. thesis looked at formal logic problems which could be characterised by a fixed number of well defined rules, well ordered data, and a guaranteed single solution - in effect a trivial subset of what I was trying to do. Having spent the weekend reading the thesis I made a couple of minor tweaks to my research software and got it to solve most of the problems in the thesis. While my approach was basically a pattern recognition one, I found it was possible to morph it into a powerful language for processing patterns - and the problem solving package TANTALIZE was the result. This was used to solve 15 consecutive Tantalizer problems as they were published weekly in the New Scientist, and also many of the similar problems in the  A.I. literature. 

There was only one problem - peer review! Papers which included descriptions of my system processing logical problems (including copy listings and timings) were rejected with a bald "too theoretical - will never work". On one or two occasions I was told that I couldn't expect a paper to be published if I used used CODIL [which is a pattern-recognition language which doesn't distinguish between program and data] because all papers on A.I. had to be written in pop-2 [a conventional rule-based programming language popular in the leading A.I. departments in UK at the time]. Finally a paper aimed at a top American journal came back with a rejection slip and four reviews. One was favourable, one reviewer admitted he didn't understand what I was doing, and two were about as insulting as an anonymous critical review can be.  By this stage I was so depressed I decided to abandon all A.I. research, and switch to other application areas ofn CODIL. It was only some years later that I read the end of the coveringrejection letter. The editor (who would have known who had written the reviews) ended by urging me to continue the work because he thought there was something in it to annoy the reviewers so much!

A Simple Guide to the Relationship between Neurons, Natural Language and CODIL

I have posted the detailed discussion paper Fromthe Neuron to Human Intelligence: Part 1: The “Ideal Brain” Model and my idea is to supplement it with brief notes examining various topics, including any raised by comments. This is the first of those notes

A noun such as Macbeth, or Dagger, or Author is represented in the brain as a somewhat amorphous network of neurons which I have called a memode.

Memodes contain other lower level memodes. Thus Murderer will contain Macbeth and Crippen, while Author will contain Shelly and Shakespeare. People will contain sets such as Murderer and Author and individuals such as Churchill.

A memode may also represent a context where several nouns are associated. An example of a context would be Macbeth; Duncan; Dagger. Another might be Macbeth; Shakespeare.

The ideal brain model connect up the links – so the above two examples can be merged as Macbeth; Duncan; Dagger; Shakespeare.

As Macbeth is a Murderer we can expand the above to the context Murderer  Macbeth; Victim  Duncan; Weapon  Dagger; Author  Shakespeare. While we are only using nouns it is easy to relate this to a natural language statement such as “According to Shakespeare Macbeth used a Dagger to kill Duncan.”

CODIL was a blue sky project to try and provide a fundamentally human friendly information processor for handling a range of non-mathematical tasks. In MicroCODIL (a demonstration version that runs on the BBC Microcomputer and uses colour) the above example would be represented as

1 MURDER = Macbeth,

2   VICTIM = Duncan,

3     WEAPON = Dagger,

4       AUTHOR = Shakespeare.

While the ideal brain model works by making links within a network of neurons, and CODIL works by moving symbols around a digital store, the two processes are equivalent.

The CODIL idea was triggered by research on a very large commercial data processing system, and has been trialed in medium sized poorly structured data bases (medical and historical data), providing online tutorial material for classes in excess of 100, as a schools package for demonstrating a wide range of information processing ideas, and in the area of artificial intelligence. A package called TANTALIZE used CODIL to solve 15 consecutive Tantalizers (now called Enigma) published weekly in the New Scientist.

The parallel between the ideal brain model and CODIL suggests that the ideal brain model could probably support a reasonable level of natural language skills – but more research is required. The bottleneck as far as the basic ideal brain model is concerned relates to the speed of learning – and this issue will be addressed in Part2: Evolution and Language.

TANTALIZE - the School Colours problem - and peer reviews

TANTALIZER No 226     NEW SCIENTIST 

 SCHOOL COLOURS 

    "Tell Me, Professor Pinhole, which school does your daughter Alice go to?"
    "Let me think. Is it the one with the orange hat and the turquoise scarf? or with the khaki blazer and orange emblem? or with the pink blazer and orange scarf? or with the khaki scarf and pink emblem? or with the khaki hat and turquoise emblem? I fear I cannot recollect."
   "Good Heavens, Professor! However many schools are there?"
   "Just four and I have one daughter at each. Bess goes to St Gertrude's, Clare wears a turquoise hat and Debbie wears a khaki emblem. St Etheldreda's flaunts a pink scarf, St Faith's an orange blazer and St Ida's a pink hat."
   "And whose are those clothes flung down on the floor over there?"    "The turquoise hat and the khaki blazer belong to different girls. As for the turquoise blazer, well, I think you might work out whose that is for yourself." 

Martin Hollis

(For solution see the paper on TANTALIZE)

In fact some of the work I did with the TANTALIZE package in the 1970s is relevant to the brain modelling work I am doing now - and a little of the history is relevant. 

In 1972 I started the work of implementing the second version of the CODIL interpreter on the 1903A computer at Brunel University with a view to concentrating of open-ended commercial and data base tasks once I had got the system up and running.  One day I had a discussion with a colleague, Roland Sleep, and he pointed out that while there was a lot of hype about Artificial Intelligence what was actually being done was comparatively simple - and he lent me a copy of a Ph.D. thesis on one of the leading problem solver packages. Within three days I had CODIL up and running the key examples in the thesis. I followed this up and used CODIL to implement a problem solving package which I called TANTALIZE - which, among other things solved the Tantalizer "brain teaser" puzzles for 15 consecutive weeks as they were published in the New Scientist. The first paper I wrote was TANTALIZE with included a number of examples of CODIL on its own and using the problem solver.

The reason for mention TANTALIZE now is that CODIL was not designed to be a programming language, but as it is designed to reflect the user's view of his information processing task it has to accommodate users who want to use it to "write programs". TANTALIZE is by far the biggest CODIL "programming" task written and can be considered as a sophisticated production rule system, written in, processing, and obeying production rules. The first phase is to ask the user a series of questions about the task, and also any general information on the type of task and the resources needed. The second phase turns the user input into a set of production rules and in some cases the package uses dynamic learning to sort the rules into a "most likely to succeed" order - which can lead to orders of magnitude reductions in the time needed in the third stage. The third stage take the optimised production rules and uses them to search the problem space and present the answer.

I continued solving problems with TANTALIZE but immediately ran into difficulty with the peer review system in getting A.I. papers accepted - so I simply switched to other application areas and dropped the work on heuristic problem solving. After all CODIL was not designed to handle small well-defined closed problems - but naturally t can do them because they are a subset of the bigger less well-define open-ended real world problems with which it is really concerned.

In retrospect it is interesting to look at why, for example, a paper was rejected as "Too theoretical - will never work" when I had reported in detail the way the package actually solved a wide range of problems. Or why I was told about another  that if I wanted to get papers accepted I should use the POP-2 programming language. A paper sent to a leading journal in the USA came back with two vitriolic reviews, one reviewer admitted to not understanding it, and there was one favourable review. I was so cheesed off by multiple rejections at this stage I just junked it and only some years later rediscovered the covering letter from the editor (who would have know who the reviews were) which ended with the advice that I should continue as he felt there must be something in it to have annoyed two of the reviewers so much.

Of course the real problem is that we are all trapped in the mental boxes we have constructed for ourselves during our lifetime and my mental box did not overlap with the mental boxes of the majority of the  A.I. establishment. For instance I approached the problem from the angle that there are many very complex open-ended problems - with no simple solutions - and to me the logic puzzles were a trivial artificial subset of the real world - where there were precise pre-defined rules and unique answers. The A.I. establishment at the time concentrated on applying formal mathematical models to closed tasks - such as game playing - in the belief that this was the way forward to modelling intelligence. My papers did not fit in as they were not expecting a solution coming from the area of open-ended and poorly defined tasks. Looking back it is clear that I was not really aware of how counter-intuitive some of my ideas were. I suspect that most genuine "outside the box" research has similar problems with peer review systems for both academic publication and research grants.

--------

If you read the TANTALIZE paper earlier you will find the missing sections have now been added.

An account of the TANTALIZE package published in the New Scientist is below the break.