NetNews Usenet Archive 1992 #31/NN_1992_31.iso/spool/bit/listserv/csgl/1941

Text File | 1992-12-22 | 6.6 KB | 130 lines

Comments: Gated by [e-mail]
Path: sparky!uunet!zaphod.mps.ohio-state.edu!howland.reston.ans.net!paladin.american.edu!auvm!BEN.DCIEM.DND.CA!MMT
Message-ID: <[e-mail]>
Newsgroups: bit.listserv.csg-l
Date:         Mon, 21 Dec 1992 20:32:52 EST
Sender:       "Control Systems Group Network (CSGnet)" <[e-mail]>
From:         [e-mail]
Subject: Re: Info theory
Lines: 119

[Martin Taylor 921221 20:00]
(Bill Powers 921221.1500)

Bill, your posting just arrived, as I was on the way out to go home, so this
must be short (our phone lines have got very bad recently, and I can no
longer try to do it from home).

As usual, you are an acute observe, especially in relation to the question
of model types.  But I really do think that I can use information theory
to identify that the PCT structure was correct, at least feasible.  When
you put in the appropriate perceptual input functions, gains, and delays,
you get the same model that you and/or Tom would produce without information
theory, so it should make the same predictions in any specific case.  So
why should I try to do better, when I anticipate the result being identity?

What I do want to do is to get some deductions about the structure and its
behaviour that are not obvious, even though they may (should) agree with
what you have found to work in practice.  I find that it makes much more
sense to me to have a good theoretical underpinning that allows me to
generalize from a practical result than just to see the practical result
and wonder what might happen if some little thing were changed.

>Martin, the difference that Tom is talking about, I believe, is
>between a descriptive model and a generative model. A descriptive
>model provides a general picture of which a specific behavior is
>only one example. A generative model actually generates
>(simulated) behavior for direct point-by-point comparison with
>real behavior.

Yes, I understand.  I have a bit of a problem with limiting myself to
either kind of model exclusively, though, and it is a problem that has
been with me since undergraduate days.  If a generative model does
predict reality well, without excessive use of parameters, then it
produces strong evidence of the plausibility of the theory that
underlies it.  But if the generative model fails, it does not give
evidence against the underlying theory, because the failure could
have been only in the choice of parameters.  So the generative model
is a one-sided kind of support.

On the other hand, the theory by itself is only plausible unless it
can be shown to predict reality, and that can be done only through
generative models or mathematical analysis.  In the case of your and
Tom's models, the prediction is very good.  So I see little point in
trying to create generative models from what I see as a theoretical
support for the same structure on which your models are based.  It
is conceivable that in some situations the information-theoretic
approach might produce numerical statements of more precision or using
fewer parameters, but those situations probably will not be easy to
find.  They will be at higher levels in the hierarchy, most probably.
I'm not even going to look for them at present, at least not until
I can see some problems with your practical approach that are resolved
in the information-theoretic approach.

>In the end, the generative model will explain behavior, while
>descriptive models show that the behavior thus explained and the
>structure of the successful model are consistent with general
>laws.

Wise.

I obviously must either drop the information-theory thread or take it
much further back to first principles.  In response to what Tom seemed
to want in his revised version of his challenge, I anticipated doing
just that.  But perhaps it would help if I contradicted one of your
assumptions:

>You seem to be taking the position of an external observer who
>has one probe on the reference signal and another on the
>perceptual signal.

No.  Throughout, I am trying to take the position that the only
probabilities that can be observed are based within the observing entity.
Sometimes I slip, I acknowledge.  But that's a simple mistake, not
a failure of principle.  In this case, the reference signal and
the perceptual signal are both known within the ECS.  If you remember
a long way back, this came up.  There is no need for an external
evaluation of the probability distribution, any more than there is a
need for an evaluation of a neural current that is based on a rate
of neural impulses.  I suppose it might be possible for an external
observer with a probe to make the analyses, and sometimes it is
didactically easier to posit such an observer.  But in practice there
isn't one, and it is not necessary to think of one.

One didactic problem is that WE are external to the ECS in question, so
that WE externally observe (imagine) what is going on.  But we have to
try to imagine ourselves being in the ECS.  It's not easy.

As you can see, we do not
>begin with the
>phenomenon of messages passing between behaving systems.

I've been playing with information at an intuitive level for as long as
you've been playing with control systems.  It's hard for me to get back
to basics (or even to exact formulae, since I don't use them much), but
it will be a good exercise for me to try.

>If the reference signal and the perceptual signal are both
>varying in a pattern that requires a bandwidth of, say, 2 Hz,
>doesn't this mean that both signals are carrying information at a
>rate corresponding to that bandwidth?

They could be, but they need not be.  The problem is in thinking of
information as being carried, as if an external observer could see it.
That's the same problem as the "codingism" problem.  The information
arriving at a receiving point depends on the probabilities of the
received pattern as believed by the receiver, which are unknown to
any other party.  The information the originator thinks is being sent
is based on the originator's beliefs as to the probabilities held by
the receiver.  In the 2Hz case, the presumption is that all these
probabilities are flat (actually Gaussian) distributions, maximizing
the information that could be transmitted.

Anyway, I'll try to put together a discussion of this, but it may take
some time.  I'm sorry I annoyed Rick with my comment.  Information
theory clearly isn't as easy to understand as I thought, but in any
case I had no right to say soemthing that could be perceived as insulting.
I'm afraid I may have done it again, in a somewhat hasty reaction to
Rick's riposte.  Rick--if you are reading this--let's try to keep things
more technical, if we can.