Snowed In

I had a meeting with my advisor Jim today, and he asked what the punchline for my second year project is going to be.  Here’s my best guess:

There’s a rich tradition developing of studying the role our bodies play in cognition (Nunez, Goodwin, others).  I have presented evidence that we can take these analysis methods and theories into the digital realm to study the digital extensions to our bodies.  Put another way, I have tried to show that our “digital body” is subject to the same sorts of analysis as our body body.  This is in line with Andy Clark’s notion of “radical embodiment”, where the body does not end at the skin, but extends opportunistically to include any structures that we can command.

This evidence points at a rich space for further inquiry.  The programmers I’ve been studying use a pretty impoverished “digital body”, composed of a pointer, a carat, a selection and a few windows.  Given that embodied activities in the digital realm are an important part of our problem solving, how can we create richer digital body parts that will enable more advanced cognition?

One approach to answering this question is to use the bridge I’ve created in my work between the digital realm, where we know very little about embodiment, and the physical realm, where we know much more.  We can analyze the work of Chuck Goodwin and others to look for common embodied activities that could be pushed across that bridge to help do digital work.

Another approach is to look closely at digital tasks and use embodiment theory as a foundation for the design process for better digital tools.

Anyway, not super punchy, but that’s the bush I’m beating around.

(title courtesy of Robert Burns, and thanks Niamh for setting me straight on the proper wording of the phrase way back when…)

Here’s a first crack at a new research plan for my 2nd year project proposal. It’s entirely unedited and needs work. But I think it’s a step in a better direction, more along the lines of what Ed is looking for.

Procedure:

Participants are asked to recreate a structure out of blocks. The purpose of this exercise is to give participants “ownership” of the blocks, where they feel like they can use them freely. After the structure is built, they are asked to knock it down so it is not in the way. Participant A is then asked to watch a video of a complex event. Possible examples include multiple car traffic accidents, or football plays. Participant A would be able to watch the video twice, both times having access to the blocks. Participant A would then be asked to explain the event to Participant B. Participant B would then take a quiz about the event.

Analysis:

I expect that participants will use the blocks to help them enact the events. From video footage, I will look for examples of participants using their biological bodies to enact events, and I will attempt to discern the purpose of such enactments, and the affordances provided by their body parts. I will then observe the ways participants use the blocks and the table to enact events, and will similarly discern their purpose and the affordances they provide.

Hypothesis 1:

It is my hypothesis that there will be overlap. I expect that in some cases, if not most, the purpose of the block enactment will fall into a well established category of hand enactment. If this is true, it lends support to the idea that physical objects can play a role similar to biological body parts in cognitive activities, and that the same cognitive structures are likely to be used for physical objects as for body parts.

Hypothesis 2:

By varying the complexity of the events being enacted, I will be able to study the way in which the affordances of the blocks diverge from the affordances of participants’ hands. In particular, I will be able to observe when participants choose to use their hands to enact an event, and when they choose to use blocks. My hypothesis is that participants will opportunistically switch to using blocks instead of hands in cases where they need to represent more than two or three positions simultaneously.

Hypothesis 3:

By offering different shapes of blocks, I will be able to shed some light on the phenomenon of tool choice. When choosing blocks to use to enact an event, participants might choose based on similarity between the block and the entity being represented, they might choose based on ease of grasping, or they might choose based on similarity to other body parts simultaneously being used in the enactment. If participants are re-using cognitive structures meant for their biological parts, then we might expect some preference for blocks with similar structure.

Example stimulus videos:

Car accident: http://youtube.com/watch?v=-r5bClOqhjQ

Football play: http://youtube.com/watch?v=9oWtAsRxR34

Similar video stimuli could easily be created in a video game environment, with varying degrees of complexity. Both driving and football video games exist which could easily be used to create high quality videos of these phenomena.

I just finished a very short, very drafty draft of my second year project proposal. I wanted to put it together before asking Rafael, Ed, and Jim to be my committee members. It’s rough, but I’m pretty excited about where it’s going. Here’s the introduction:

A small but convincing body of mounting evidence suggests that tool use is, in some cases at least, a misnomer. Tools are not merely “used” by a distinct and well-defined “user”, but instead are actually assimilated into the body schema, occupying the same role as a hand, finger, or eyeball. There is both imaging (Maravita and Iriki, 2004) and behavioral (Yamamoto and Kitazawa, 2001) evidence suggesting that the same cognitive machinery used for biological body parts is used for tools. These results point towards a new, radical theory of embodiment and tool use, where body parts and tools are not distinct categories, but a unified pool of physical structures which a “profoundly embodied” agent (in the sense of Clark, 2007) opportunistically engages and reconfigures in order to adapt to its environment. This view is just beginning to achieve mainstream attention in cognitive science, notably in the aforementioned paper by Andy Clark.

These new insights should be particularly exciting to researchers studying human-computer interaction. HCI has long suffered a lackluster theoretical foundation both in academia and in industry, instead relying mostly on moderately effective but poorly understood methodologies and heuristics for designing digital systems. Theories of Psychology, and more recently Cognitive Science, have had limited success in explaining the “H” in HCI, and theories of computation and of external cognition have provided insight into the “C”, but rich theories of the actually interaction between the human and the machine which are predictive, prescriptive and generative have not yet materialized. [cite Rogers?]

The theory of “profound” embodiment is exactly that. The idea that tool use and embodiment are one in the same, supported by our burgeoning understanding of embodied cognition and the way it develops in humans and primates throughout their lifetimes, provides a solid platform for explaining the actual interaction between human agents and computing machines.

The broad research question behind this project is: by what mechanism does the human body change to include non-biological structures, and how can we use our understanding of that mechanism to inform the design of human-computer systems?

Read the rest

I’ve been slowly working through my readings for my 2nd year project literature review, which is growing bit by bit. I’ve found a few more interesting experiments, like Yamamoto and Kitazawa, 2001, which gives some evidence that we can actually perceive sensations at the tip of a tool, rather than at the location of our grip where our actual nerve endings are.

I’ve been updating my spreadsheet as I go too. Particularly exciting is a brand spanking new article I just found this afternoon by Andy Clark:

Clark, A. Re-inventing ourselves: The plasticity of embodiment, sensing, and mind. Journal of Philosophy and Medicine (in press). [PDF]

Looks like it might be interesting.

I’m heading up to L.A. right now. I’m going to take the Surfliner up and visit Tracy. Pretty excited about getting 3 hours of solid work done on the train. So fancy.

Just a quick update that I’ve re-read the big three papers I cited in my literature review and updated the review. I’ve also started building a database of papers for my 2nd year project in a spreadsheet.

I’m working on doing a clean literature review that will serve as a base for my second year project.  I wrote a paper called “Where The Body Ends” last fall trying to pull together an argument that in some cognitive activities–like tool use–we should consider physical objects legitimate body parts, just like a finger or a callus.

I thought I did a good job in the paper of looking for support in a variety of fields, from anthropology to biology, but in the end my argument from a cognitive science perspective was quite weak.  Thus I’m going back to the basics here, trimming away the fat and starting anew, trying to build a base of literature with only hard cognitive science evidence.

A writeup of what I have so far is below the fold. All of the works cited need to be revisited so I can expand my discussion. This is just a state-of-the-research writeup.

(more…)