Information theory-how does entropy make sense for my example?

Thread Starter

Vikram50517

Joined Jan 4, 2020
81
Hello y'all, i have a doubt on what is the physical significance that entropy has in my example. According to a famous prof from EPFL, entropy is basically the average number of binary questions that you should ask to get an answer. OK, say there is an employee who asks his manager whether tomorrow is a holiday or working day , say the probabilities are Ph=0.99 and Pw=0.01 (Ph is probability of a holiday and Pw is the working one). Now if you calculate the entropy the answer turns out to be 0.08 bits. WHAT? that means 0.08 questions? How?

Let me post a link to the professor's claim , see 2:39

 
Last edited by a moderator:

Ya’akov

Joined Jan 27, 2019
6,561
Fractional bits occur because probabilities are not quantized. If you need ≤1 bit, you need a whole bit to encode, but you can still compare the entropy of two systems in fractional bits.

So, entropy might be expressed in fractional bits but encoding is always in integer bits since bits in encoding are atomic.
 

Thread Starter

Vikram50517

Joined Jan 4, 2020
81
oh, but it does make sense when the same entropy scales for 100 days i.e 0.08*100 =8 bits . But for one day it does not make sense as it is a fractional bit, so he has to min send one bit. Hmm I kinda get it i guess
 

Ya’akov

Joined Jan 27, 2019
6,561
I think the confusion is concerning how the unit “bits” operates in the context of entropy versus its operation in encoding. Two systems could have fractionally different entropy but describing them would require the same number of bits.
 

Ya’akov

Joined Jan 27, 2019
6,561
It might be helpful to consider this from a different angle. These two videos talk about entropy and practical data compression which is about probabilities but requires actual bit-wise representation.

 

BobTPH

Joined Jun 5, 2013
5,759
Entropy and information theory depend on the statistics of large numbers. They do not make always make sense for small numbers.

When a baseball players’s batting average is 0.333 does that mean he gets 1/3 of a hit each time he bats?

Bob
 
Top