Erik Pukinskis

Foundations of human-computer semantic transfer

Theorem 1: Thought is simulation from episodic memory

Some parts of memory are classified as "episodic". These are related directly to particular experiences of the rememberer. I posit that semantic memory is also episodic in nature; that use of semantic memory is simulation based on schematic episodic memories. Thus episodic memory is the predominant content of thought and simulation the predominant process.

If the objects of thought are percepts, episodic memories and semantic memories, then no object of thought is not directly related to particular experiences of the rememberer, though possibly only schematized experiences.

Note: can we change "directly related" to something stronger? Perhaps "similar to" or "a model of".

Theorem 2: Language models experiences (action and perception)

There are two arguments to suggest that language models experiences. An ecological argument is that language serves to inform others, and information embodies afforances in an environment. Language must then model afforances, which are the stuff of perception and the cause of action.

The second argument comes from the perspective that language serves to elicit similar thoughts in others. Hypothesis 1 suggests that experiences are the stuff of thought, thus language must model experiences.

Theorem 3: Transfer of ideas through language requires similarity of experiences

It has been long assumed that transfer of ideas requires only similarity of thoughts and proper language processing. But though the thought process is built on experiences and simulated experiences, it is not true that thoughts model perception and action accurately. Instead, perception and action and simulated perception and action are an integral part of the thought process over the lifetime of an individual. These are not embodied completely in the memories of an individual, but in the way those memories are manipulated by an environment or internally simulated environment.

Thus similarity of experiences (perception of an environment and action within an environment) is necessary for transfer of ideas through language.

Lemma 1

Transfer of ideas through language between human and computer requires similarity of experiences (perception and action) between human and computer.

Discussion

The failure to recognize Lemma 1 is the reason current approaches to natural language processing have failed at accommodating semantics. Attempts to add a semantic layer to our robust syntactic processors have failed largely because semantics are based on perception and action, and attempts to align semantics without aligning percepts and facilities will be fatally brittle.

Futher evidence that such alignment of percepts and facilities is necessary for transfer of ideas between humans and computers is that this task is currently performed by humans when using computers, at the expense of great cognitive load. Every piece of software in use today demands that the user understand the system's faculties and perceptural abilities. This is embodied perhaps most clearly in programming, but it shows in all types of interfaces. This is what separates contemporary computer use from linguistic communication.

Aligning the percepts and facilities of humans and computers is not an easy task, but the explosion of valid, expository psychological theory that has come out of the cognitive and ecological revolutions over the last fifty years provide us for the first time in the history of computing the resources to solve this problem.

There are two possible approaches to aligning the perceptural and manual faculties of humans and computers. The first is embodiment. The domain of robotics, this would physically endow computers with the sensory and motor charictaristics of humans. Unfortunately, such an approach is science fiction. It is debatable whether it is even possible to duplicate much of the functionality of the human body in aluminum and silicon, and even if we get there, it will take hundreds if not thousands of years to create a robot that was similar enough developmentally and behaviorally to allow for transfer of ideas.

The second approach is modelling. This involves giving the computer a model of human perception and action with ties to the computer's model of data. The model could be limited in scope by considering only the percepts provided by the computer and the actions afforded.

I believe both approaches should be used together in limited capacities. Giving the computer a sense of embodiment by giving it an episodic memory and simulation capabilties will align its experience to some degree with that of its user. And simple models of the percepts and afforances provided by the computer will allow the computer to align the user's language more closely with its own data.

These are very small but necessary steps toward the goal of strong, reliable alignment between the experience of computers and humans which will pave the way for right linguistic communication.

Problems

One problem with this approach is that episodic storage of semantic information makes use of this information more or less computationally intractable. Storing up all of the episodes of one's life and searching through them every time one must make a judgement is ludicrous from both storage and processing standpoints. This is, perhaps, an instance of the Frame Problem.

The obvious baby-with-the-bathwater solution is to bring back semantic memory, and suggest that each new episode trains the semantic network. This of course is somewhere between undesirable and wrong given the arguments above.

The second posible solution is the one outlined in my discussion of the Frame Problem itself--that we do analysis on episodic data constantly, as the data comes in. So instead of waiting to do processing when we need an answer, we do processing as we receive bits of the solution.

But I'm not entirely convinced that this is substantially different from the semantic memory approach though. We have to store what we process somehow, in some form other than raw episodic memories. Is that necessarily semantic memories? It could also be what I call "schematized experiences" above. But still two questions remain: are schematized experiences useful enough to replace semantic memory, and are they fundamentally different?


 
This page was last updated August 25, 2004 at 11:19pm.