Interface Problems and the Role of Experience

Click here and press the right key for the next slide.

(This may not work on mobile or ipad. You can try using chrome or firefox, but even that may fail. Sorry.)

also ...

Press the left key to go backwards (or swipe right)

Press n to toggle whether notes are shown (or add '?notes' to the url before the #)

Press m or double tap to slide thumbnails (menu)

Press ? at any time to show the keyboard shortcuts

Interface Problems and the Role of Experience

Two Interface Problems

selection

primary motivational states vs preferences

execution

motor representation vs intention

Experience is key ...

rather, animals have to learn about the value of a commodity in a particular motivational state through direct experience with it in that state’

(Dickinson & Balleine, 1994, p. 7)

‘primary motivational states have no direct impact on the current value that an agent assigns to a past outcome of an instrumental action; rather, it appears that agents have to learn about the value of an outcome through direct experience with it, a process that we refer to as _incentive learning_’

(Dickinson & Balleine, 1994, p. 8)

A role for experience in solving the interface problem.

Why are rats (and you) aware of bodily states such as hunger and revulsion?

Because this awareness enables your preferences to be coupled,
but only losely,
to your primary motivational states.

Isn’t it redunant to have dissociable kinds of motivational state?

‘the motivational control over goal-directed actions is, at least in part, indirect and mediated by learning about one's own reactions to primary incentives.

By this process [...], goal-directed actions are liberated from the tyranny of primary motivation’

(Dickinson & Balleine, 1994, p. 16)

Dickinson & Balleine , 1994 p. 16

Summary so far

Two (or more) kinds of motivatational state dissociate,

leading to an interface problem

that is solved by experience of our own bodily reactions.

But this was only one of the interface problems we encountered

can experience also play a role in solving other interface problems?

Two Interface Problems

selection

primary motivational states vs preferences

execution

motor representation vs intention

I was just saying that Experience is what enables there to be nonaccidental eventual alignment of largely independent cognitive systems. This is what experience is for.

Can we think along these lines in the case of action?

Two complications make it appear difficult ..

Let me first mention two complications facing any attempt to link motor representations and intentions ...

Third consideration which complicates the interface problem: dynamics.

It’s ‘not just how motor representations are triggered by intentions, but how motor representations’ sometimes nonaccidentally continue to match intentions as circumstances change in unforeseen ways ‘throughout skill execution’

(Fridland, 2016, p. 19).

Fridland, 2016 p. 19

Here we need to distinguish different kinds of change. Some changes can be flexibly accommodated motorically without any need for intention to be involved, or even for the agents to be aware of the change. This includes peturbations in the apparent direction of motion while drawing (Fourneret & Jeannerod, 1998). But other changes may require a change in intention: circumstances may change in such a way that you wish either to abandon the action altogether, or else switch target midway through.

**KEY**: in executing an intention you may learn something which causes you to change your intention; for example, you may learn that the action is just too awkward, or that the ball is out of reach. So motor processes can result in discoveries that nonaccidentally cause changes in intention.

This also shows that we can’t think of the interface problem merely as a way of intentions ‘handing off’ to motor representations: in some cases, the matching of motor representations and intentions will nonaccidentally persist.

phase shift

period shift

Second consideration which complicates the interface problem: scale. This shows that we can’t think of the interface problem merely as a way of intentions setting problems to be solved by motor representations: there may be multiple intentions at different scales, and in some cases an intention may operate at a smaller scale than a motor representation.

Suppose you have an intention to tap in time with a metronome. Maintaining synchrony will involve two kinds of correction: phase and period shifts. These appear appear to be made by mechanisms acting independently, so that correcting errors involves a distinctive pattern of overadjustment. Adjustments involving phase shifts are largely automatic, adjustments involving changes in period are to some extent controlled.

How are period shifts controlled? Importantly this is not currently known. One possibility is that period adjustments can be made intentionally (Fairhurst, Janata, & Keller, 2013, p. as][p. 2599 hint); another is that there are a small number of ‘coordinative strategies’ (Repp & Keller, 2008) between which agents with sufficient skill can intentionally switch in something like the way in which they can intentionally switch from walking to running. But either way, there can be two intentions: a larger-scale one to tap in time with a metronome and a smaller-scale one to adjust the tapping which results in a period shift.

EP: Skilled piano playing means being able to have intentions with respect to larger units than a novice could manage. But in playing a 3-voice fugue you may need to pay attention to a particular nger in order to keep the voices separate. So you need to be able to attend to both ‘large chunks’ (e.g. chords) of action and ‘small chunks’ (e.g. keypresses) simultaneously.

BACKGROUND: Because no one can perform two actions without introducing some tiny variation between them, entrainment of any kind depends on continuous monitoring and ongoing adjustments (Repp, 2005, p. 976). % \textcite[p. 976]{repp:2005_sensorimotor}: ‘A fundamental point about SMS is that it cannot be sustained without error correction, even if tapping starts without any asynchrony and continues at exactly the right mean tempo. Without error correction, the variability inherent in any periodic motor activity would accumulate from tap to tap, and the probability of large asynchronies would increase steadily (Hary & Moore, 1987a; Voillaume, 1971; Vorberg & Wing, 1996). The inability of even musically trained participants to stay in phase with a virtual metronome (i.e., with silent beats extrapolated from a metronome) can be demonstrated easily in the synchronization–continuation paradigm by computing virtual asynchronies for the continuation taps. These asynchronies usually get quite large within a few taps, although occasionally, virtual synchrony may be maintained for a while by chance.’ % Repp & Su (2013, p. 407): ‘Error correction is essential to SMS, even in tapping with an isochronous, unperturbed metronome.’ One kind of adjustment is a phase shift, which occurs when one action in a sequence is delayed or brought forwards in time. Another kind of adjustment is a period shift; that is, an increase or reduction in the speed with which all future actions are performed, or in the delay between all future adjacent pairs of actions. These two kinds of adjustment, phase shifts and period shifts, appear to be made by mechanisms acting independently, so that correcting errors involves a distinctive pattern of overadjustment.% footnote{ See Schulze, Cordes, & Vorberg (2005, pp. 474–476). Keller, Novembre, & Hove (2014) suggest, further, that the two kinds of adjustment involve different brain networks. Note that this view is currently controversial: Loehr & Palmer (2011) could be interpreted as providing evidence for a different account of how entrainment is maintained. } Repp (2005, p. 987) argues, further, that while adjustments involving phase shifts are largely automatic, adjustments involving changes in period are to some extent controlled. % (‘two error correction processes, one being largely automatic and operating via phase resetting, and the other being mostly under cognitive control and, presumably, operating via a modulation of the period of an internal timekeeper’ (Fairhurst et al., 2013, p. 987{repp:2005_sensorimotor}) One possibility is that period adjustments can be made intentionally \citep[as][p. 2599 hint); another is that there are a small number of ‘coordinative strategies’ (Repp & Keller, 2008) between which agents with sufficient skill can intentionally switch in something like the way in which they can intentionally switch from walking to running.

So it is not that intentions are restricted to specifying outcomes which form the head of the means-end hierarchy of outcomes represented motorically. They can also influence aspects of outcomes at smaller scales.

object indexes

what is an object index? Formally, an object index is ‘a mental token that functions as a pointer to an object’ (Leslie et al., 1998, p. \ 11). If you imagine using your fingers to track moving objects, an object index is the mental counterpart of a finger (Pylyshyn, 1989, p. 68).

The interesting thing about object indexes is that a system of object indexes (at least one, maybe more) appears to underpin cognitive processes which are not strictly perceptual but also do not involve beliefs or knowledge states.

Object indexes are belief-independent. In this scenario, a patterned square disappears behind the barrier; later a plain black ring emerges. If you consider speed and direction only, these movements are consistent with there being just one object. But given the distinct shapes and textures of these things, it seems all but certain that there must be two objects. Yet in many cases these two objects will be assigned the same object index (Flombaum & Scholl, 2006; Mitroff & Alvarez, 2007).

What do object indexes contribute to experience? Structure!

nonmodal. Not arrangement of surfaces in space and their boundaries (vision, touch). Instead we should think of them as giving **structure** to experience.

When a patterned square moves behind the occluder and a solid ring emerges on a spatio-temporal trajectory compatible with a single object, you can report the experience of a single object moving. But what is this experience? It’s not an experience of the surfaces, but nor is it an experience of something independent from the surfaces or of an entirely different element of experience. What you’re commenting on is something about the structure of your experience.

object indexes

belief-independent

structure experiences

subject to limited, indirect cognitive control through attention

There’s one major disanalogy with object indexes. Object indexes are about things which are actually present, whereas the motor representations we are interested in specify possible future outcomes.

This appears to be an objection because on the face of it, it seems that there could not be expeirence of future actions any more than we can experience future events.

So, you might object, the idea that motor representations structure experience is ok if you are merely observing someone act, but it will not help with performing actions.

Costantini et al, 2010 figure 1b

Part of the answer to the objection is that motor representations of outcomes may structure our experiences of objects. The existence of affordances suggests that this is at least possible.

So here is the idea: What is experienced is an object, not an action or an outcome. So the fact that the action lies in the future and the outcome has not yet occurred is no objection. But the motor representation of the outcome structures the experience of the object in some way. So the overall character of the experience of the mug differs when an action is represented motorically compared to when it does not.

And this difference is structural in the same sense that the difference object indexes make is structural. It’s about how elements of experience are organised rather than about any particular sensory modality.

‘Action index’ conjecture

Motor representations of outcomes structure
experiences, imaginings and (prospective) memories

in ways which provide opportunities for attention to actions directed to those outcomes.

Forming intentions concerning an outcome can influence attention to the action,

which can influence the persistence of a motor representation of the outcome.

A second objection: intentions and motor representations need to match in situations where you merely imagine acting or merely imaginary objects. In such situations there are no objects to experience.

Reply: motor representations structure experiences associated with imaginig things as they do experiences associated with actually perceiving things. To imagine acting on a mug (say), you need to imagine the mug.

A third objection: close your eyes, put yourself in a sensory deprivation chamber. Let your hand rest palm down on a table. Now intend to turn your hand palm up. Often enough, the intention will succeed. But by hypothesis there is nothing you experience (you are in a sensory deprivation chamber).

Two points in reply to this objection: first, we haven’t removed proprioception and other somasomatic senses. It may be that motor representations structure experiences of the body just as much as they structure experiences of mere objects.

But what if you remove somasomatic senses too? This is likely to impair action, but unlikely to make it impossible. Perhaps the ability to act in such situations depends on memory and imagination.

Predictions?

So far I’ve suggested that this conjecture (a) might contribute to solving the interface problem and (b) isn’t obviously wrong. But how could we tell whether it is right? What predictions does it generate?

1. Not all action-related changes in experience are merely changes in bodily configurations, movements and their sensory effects.

Prediction 1: it is possible to vary which action someone experiences while holding fixed her perceptual experiences of bodily configurations and joint displacements and their sensory effects.

2. Memory for objects should be influenced by their affordances.

Two Interface Problems

selection

primary motivational states vs preferences

execution

motor representation vs intention

conclusion

In conclusion, ...

One action can involve multiple, dissociable
motivational and
effective
states.

We do not understand how these ever nonaccidentally match,
although experience
of our own bodies and
of action possibilities
may play a role.

(And this may be one reason why we have such experiences in the first place.)

Keyboard Shortcuts?f

Interface Problems and the Role of Experience

Keyboard Shortcuts`?f`