Holistic Model

of Sound Perception
THOST
May 2015
Presented by: Vedanta Sutra dasa
Ideas by: HG Lila Purushottama dasa
HG Rasaraja dasa

Semantic Processing
Artificial general intelligence (AGI):
a machine can successfully perform any intellectual task that
a human being can.
from understanding language to writing award-winning books
from comprehending music to composing bandishes in different
ragas

Source: Wikipedia

Semantic Processing
What are our computers doing ??
At present, digital computers only perform token manipulation
(0/1 bits).
It does not lead to understanding or cognition.
The computers manipulate the symbols using a form of syntax
rules, without any knowledge of the symbol's semantics (that is,
their meaning).

Source: Gomatam R., 2009

State of the Art - what is lacking?

Computers can perform complex computations at lightening
speeds. E.g.,
finding logarithms
computing square roots
optimization using gradient algorithms
They do not feel exhausted or bored by this labour

State of the Art - what is lacking?

Computers can perform complex computations at lightening
speeds. E.g.,
finding logarithms
computing square roots
optimization using gradient algorithms
They do not feel exhausted or bored by this labour

But they are only performing bit operations, they way they
have been programmed

State of the Art - what is lacking?

What about AI ?
Automatic temperature control
Automatic OCR (optical character recognition)
Adaptive signal processing
Automatic music transcription
GPS navigation

State of the Art - what is lacking?

What about AI ?
Automatic temperature control
Automatic OCR (optical character recognition)
Adaptive signal processing
Automatic music transcription
GPS navigation

Only a bit more sophisticated programming

They are only doing as they have been programmed

State of the Art - what is lacking?

Automatic OCR (optical character recognition)
pixels of 0/1
learn features which are again 0/1

Source: CS, Boston University

State of the Art - what is lacking?

Hierarchy of simple computational models give rise to complex
understanding

State of the Art - what is lacking?

Hierarchy of simple computational models give rise to complex
understanding

This is REDUCTIONISM, an old argument

It has brought some empirically successful systems upto
certain accuracy, but no understanding of understanding
itself

State of the Art - what is lacking?

What about a computer program defeating a human
grandmaster in playing chess?

Source: Deutsch D., 2012

State of the Art - what is lacking?

What about a computer program defeating a human
grandmaster in playing chess?

The algorithms used by human and computer are not even

remotely similar.
The grandmaster can give exciting reasons for his steps and
can write a bestseller book on the game. But a computer does
not even know what it is doing apart from following a set of
instructions.

Source: Deutsch D., 2012

Semantic Processing
But no brain on Earth is yet close to knowing what brains do in
order to achieve any of that functionality. The enterprise of
achieving it artificially the field of artificial general intelligence
or AGI has made no progress whatever during the entire six
decades of its existence.

Source: Deutsch D., 2012

Is it possible?: two views

Machine can only act like it thinks and has a mind
AGI is not possible
Machine can think and have a mind
AGI requires a major breakthrough

I call the core functionality in question creativity: the

ability to produce new explanations

Source: Deutsch D., 2012

Goals of Semantic Processing

Creativity: the ability to produce new explanations
Causal Reasoning: for taking decisions

Current Sound Processing

We sing
Air column vibrates
Diaphragm of microphone
vibrates
Vibration converted to
electric signal

Electric signal moves the coil

Diaphragm of speaker
vibrates
Air column vibrates
We hear

Current Sound Processing

We are processing in the visual domain, not the sound domain
[Sir]
Even touch is playing an intermediary role
Corollary: frequency, phase are an artifact

Can we process sound in the sound domain?

Law of the Instrument:
if all you have is a hammer, everything looks like a nail
- Abraham Maslow

Quantifying Experiences
Experience is irreducible/holistic
Modern science tries to reduce it to elementary entities. Why?

Quantifying Experiences
Experience is irreducible/holistic
Modern science tries to reduce it to elementary entities. Why?
May be the main motivation for this is to be able to QUANTIFY
the experiences.
Quantification gives better
understanding (notes of music/raga)
portability (reproducing the experience)
control (manipulability)

E.g., the experience of seeing an elephant can be represented as numerical values

corresponding to color, shape, size, etc. Once this is done, it opens the world of unlimited ways
this information can be utilized, be it for image processing, object detection, tracking,
animation and so on.

A Gap in the Understanding

The machine can do speech recognition, then how are we
different?
Machine can do symbol manipulation, not semantic one.
It can recognize symbols/shapes, but it does not know what to
recognize; we have to tell it what to look for.
But humans can automatically detect what to recognize or
learn, because they know the meaning associated
E.g. One can speak a sentence in many ways - low pitch, high
pitch, whisper, alaryngeal etc. But no one has to train me how
to make sense out of it. But a system has to be trained
separately that this signal (symbol) corresponds to this
phoneme (another symbol)
Humans can reach meaning even with fragmentary input

A GAP in the Understanding

The modern sound processing methods can manipulate lower
level handles like frequency, phase, etc.
Music theory can manipulate/explain higher level concepts like
instrumentation, pitch, etc.
There is a big gap in between
No single theory can explain all the levels coherently

Goals of Semantic Processing

Creativity: the ability to produce new explanations
Causal Reasoning: for taking decisions
Single theory for explaining different levels of perception
E.g. a note is important because it gives a different form to the
composition, and not because of statistics of notes

How is it possible?
Form vs Substance
Every instrument has a form which identifies it. This form
manifests through the audio that we hear.
A message has a form which manifests through the alphabets
written on paper
Form is simultaneously same and different from the substance
Modern science is good in manipulating substance, but no idea
of form

Source: Gomatam R., 2014

How is it possible?
Hierarchy of form
Alphabet, word, sentence, article, idea
Note, composition, raga, idea/mood
Similar to the levels of sabda in Vedic understanding
The higher level guides the lower level, not the other way
round

Modeling the form

Let the form be modeled by a mathematical entity called infon
Infon carries the information from the source to the perceiver

Modeling the form

Let the form be modeled by a mathematical entity called infon
Infon carries the information from the source to the perceiver

SOURCE
INFON
PROJECTED FROM
PERCEPTUAL
SPACE

OBSERVER

GROSS MEDIA
CARRYING INFON

INFON
PROJECTED TO
PERCEPTUAL
SPACE

Modeling the form

An Infon is a relationship - between different objects, sense
experiences
Model (MQM?)
infon as a wave function
information transfer as projection operator
INFORMATION is a scalar measure associated with an INFON

Modeling the Experience

Earth, water, fire, etc. are gross media to carry infons
The medium has some infon (form)
The source puts its infon onto the medium, resulting in a
different infon
The observer has own perceptual space
The infon in medium gets projected onto the perceptual space
of the observer

How to define Information?

According to the present understanding, information is defined
w.r.t. the quantity the observer is interested in
E.g. in Shannons information, the probabilities are the ones
associated with the patterns the receiver is expecting
E.g. given a green tennis ball
Red vs Green
Tennis vs Ping-pong
Size

Further directions
Dimension of infon?
Can we model it completely?

Conclusion

Present computation (AI) is syntactic, not semantic

AGI requires creativity and causal reasoning
Distinction between substance from form
Model form as infon, which carries information
Experience as projection on perceptual space
Information depends on what the perceiver is interested in
Higher level in the hierarchy guiding the lower level

Bhagavat Sankhya approach of

H.G. Rasaraja Prabhu
Syntactic TOKENS to Semantic SYMBOLS
A different world view: we are interpreting things as TOKENS, but
they can also be conceived of as SYMBOLS
physically characterize macroscopic objects in terms of their
relational properties, as SYMBOLS with semantic content

Source: Gomatam R., 2009