Information Theory part 12: Information Entropy (Claude Shannon's formula)

Share this & earn $10
Art of the Problem
Published at : 19 Dec 2020
123564 views
1880
30

Entropy is a measure of the uncertainty in a random variable (message source). Claude Shannon defines the "bit" as the unit of entropy (which is the uncertainty of a fair coin flip). In this video information entropy is introduced intuitively using bounce machines & yes/no questions.

Note: This analogy applies to higher order approximations, we simply create a machine for each state and average over all machines!
entropyinformationinformation entropy

How to Stop Caring About What Others Think of You

How to Stop Caring About What Others Think of You

Christina Aguilera - Dirrty (VIDEO) ft. Redman

Christina Aguilera - Dirrty (VIDEO) ft. Redman

How To Make A Traditional Mixed Mother’s Day Bouquet

How To Make A Traditional Mixed Mother’s Day Bouquet

DIY Notice Board for Home | Pin Up board using Cardboard | Best Out of Waste | Kids room decoration

DIY Notice Board for Home | Pin Up board using Cardboard | Best Out of Waste | Kids room decoration

Speaker Baleka Mbete Is Scared Of White People - Julius Malema.

Speaker Baleka Mbete Is Scared Of White People - Julius Malema.

Solly Mahlangu - Obrigado 1

Solly Mahlangu - Obrigado 1

Mafia Boss Tells All - Jimmy Hoffa, JFK Assassination and Much More

Mafia Boss Tells All - Jimmy Hoffa, JFK Assassination and Much More

Swallowed the guy - Playing darts - Flying inside the bubble / Zoom-Zoom Shrunk Challenge

Swallowed the guy - Playing darts - Flying inside the bubble / Zoom-Zoom Shrunk Challenge

A Review of Durability of Joints

A Review of Durability of Joints

АЛОХА И РЕЗОЛЬ ПЫТАЮТСЯ ВЫИГРАТЬ НА 4к // ALOHADANCE + RESOLUTION // DOTA 2

АЛОХА И РЕЗОЛЬ ПЫТАЮТСЯ ВЫИГРАТЬ НА 4к // ALOHADANCE + RESOLUTION // DOTA 2

30 THINGS THAT MUST STOP IN 2021

30 THINGS THAT MUST STOP IN 2021

Improve English Pronunciation & Speaking Technique | Schwa | Go Natural English Lesson

Improve English Pronunciation & Speaking Technique | Schwa | Go Natural English Lesson

1 Hour of Beautiful Relaxing Snow Falling in a Mountain Village

1 Hour of Beautiful Relaxing Snow Falling in a Mountain Village

$1,000,000 Question - Who Wants to Be a Millionaire?

$1,000,000 Question - Who Wants to Be a Millionaire?

This Secret Food Can Very Quickly Build Collagen In Your Skin- It's works

This Secret Food Can Very Quickly Build Collagen In Your Skin- It's works

Nico Collins - On Accident

Nico Collins - On Accident

DIY Transparent Screen

DIY Transparent Screen

[ 식스틴 ] SIXTEEN Final Mission minor1

[ 식스틴 ] SIXTEEN Final Mission minor1

5 Signs The Universe May Be Blocking You From A Relationship

5 Signs The Universe May Be Blocking You From A Relationship

Jordan Peterson - Diversity, Inclusivity & Equity

Jordan Peterson - Diversity, Inclusivity & Equity

Doodle Video - Everything at Once by Lenka

Doodle Video - Everything at Once by Lenka

СДЕЛАЛ 50 СПИНОВ ПО 1000 РУБ В СЛОТЕ HELLOWEEN // ЗЕРНОВКА ЗАНОСЫ НЕДЕЛИ

СДЕЛАЛ 50 СПИНОВ ПО 1000 РУБ В СЛОТЕ HELLOWEEN // ЗЕРНОВКА ЗАНОСЫ НЕДЕЛИ

New Secret Christmas Robot Hatched! - Bubble Gum Simulator Roblox

New Secret Christmas Robot Hatched! - Bubble Gum Simulator Roblox

CUTE (NABILA RAZALI REMIX) - HARITH ZAZMAN, MFMF., LOCA B, NABILA RAZALI

CUTE (NABILA RAZALI REMIX) - HARITH ZAZMAN, MFMF., LOCA B, NABILA RAZALI

Alida made grilled shrimp with scallion sauce with unique mayonnaise, the taste is so great!

Alida made grilled shrimp with scallion sauce with unique mayonnaise, the taste is so great!

What is Electrical Engineering ? (What do electrical engineers do) | Explore Engineering

What is Electrical Engineering ? (What do electrical engineers do) | Explore Engineering

[PAKTORY X Handspeak] Primary - Bless You (feat. Sam Kim, WOODZ, pH-1) Sign Language Ver.

[PAKTORY X Handspeak] Primary - Bless You (feat. Sam Kim, WOODZ, pH-1) Sign Language Ver.

My dog is protecting my newborn baby||guard dog

My dog is protecting my newborn baby||guard dog

Hottest Female Referee's in Football

Hottest Female Referee's in Football

The Cash Budget

The Cash Budget

No Account Casino - Slots 6s

No Account Casino - Slots 6s

College/University is NOT WORTH the Time, Money, Effort | WHY? | Advice from me

College/University is NOT WORTH the Time, Money, Effort | WHY? | Advice from me

What Do You Consider Cheating?

What Do You Consider Cheating?

COLUMN WITH BOTH AND FIXED: EXPRESSION FOR BUCKLING LOAD

COLUMN WITH BOTH AND FIXED: EXPRESSION FOR BUCKLING LOAD

What Does a College Lecture Hall Look Like?

What Does a College Lecture Hall Look Like?

CHOOSE YOUR OWN APOLOGY VIDEO

CHOOSE YOUR OWN APOLOGY VIDEO

Best singers on tiktok

Best singers on tiktok

Aşk-ı Memnu 75. Bölüm

Aşk-ı Memnu 75. Bölüm

🚑 Nurse and Patient in Plaster Leg Casts!

🚑 Nurse and Patient in Plaster Leg Casts!

QUIZ: What Kind Of SPIRIT GUIDE Do You Possess?

QUIZ: What Kind Of SPIRIT GUIDE Do You Possess?

Floh - A backpack that's also a scooter

Floh - A backpack that's also a scooter

Using the Glorious Model O Wireless for PvP | WATCH BEFORE YOU BUY!!!

Using the Glorious Model O Wireless for PvP | WATCH BEFORE YOU BUY!!!

Calvin Harris - Drinking From the Bottle (Lyric Video) ft. Tinie Tempah

Calvin Harris - Drinking From the Bottle (Lyric Video) ft. Tinie Tempah

Bring Back a Legend | Music Video | The Lion Guard | Disney Junior

Bring Back a Legend | Music Video | The Lion Guard | Disney Junior

SIMI - UNDESERVING (Visualizer)

SIMI - UNDESERVING (Visualizer)

5 Easy Motorcycle Tricks to Learn Pro Skills

5 Easy Motorcycle Tricks to Learn Pro Skills

PLAYING RECORDER WITH NOSTRIL

PLAYING RECORDER WITH NOSTRIL

Driving on the Surface of Mars

Driving on the Surface of Mars

We Attempt EXPERT Paper Mario Origami

We Attempt EXPERT Paper Mario Origami