Multimedia in HTML5

Prehistoric HTML

Before HTML5, browsers could only natively display one type of multimedia.

<img src="omgcat.gif">

The plugin era

Plugins proliferated:

  • 1991: Apple Quicktime (.mov)
  • 1994: MIDI (background music)
  • 1995: RealAudio (.ra, .ram)
  • 1997: RealVideo (H.263)
  • 1998: Windows Media Player
  • 1999: Quicktime for Windows (.mov, .mp3, .mp4, .swf)
  • 2002: Macromedia Shockwave Flash (.swf)
  • 2007: Microsoft Silverlight (flash, vid, etc.)

HTML support for plugins

To embed a file for a plugin to open, you'd use:


Introduced in Netscape 2.0, a simple tag.


Introduced in HTML 4.01 spec, this tag supports more complex configuration.

The embed element

The embed tag should specify src and type attributes.

<embed type="application/x-shockwave-flash"

The object element

The object tag should have one or more param child tags that give info to the browser on what to embed.

<object width="300" height="150">
  <param name="movie" value="zombie_walk.swf"/>
  <param name="bgcolor" value="white"/>

object and embed

The object tag can include content that is executed if the browser doesn't understand the object tag. This content could be text, another object tag, or an embed tag.

<object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B"
 height="400" width="500">
 <param name="src" value="" >
 <param name="autoplay" value="true" >

  height="400" width="400"
  src="" autoplay="true" 

The object element (Example)

The code from the previous slide produces:

Embed Codes

The (old) Youtube embed code includes an object and an embed tag.

<object width="480" height="385">
  <param name="movie"
  <param name="allowFullScreen"
  <param name="allowscriptaccess"

  <embed src=""
         allowfullscreen="true" width="480" height="385"/>


Embeds: Youtube (Example)

The code from the previous slide produces:

The problem with plugins

  • Bad user experience (pop-ups)
  • Didn't play nice with eachother (crash!)
  • Opened up security holes (malware)
  • Didn't work everywhere (Flash on iPads)

The dawn of native multimedia

In HTML5, you can embed audio or video using native HTML tags audio and video, and if the browser supports the tags, it will give users controls to play the file.

Why is native better?

  • No plugins needed
  • Better performance
  • Native, accessible controls

The audio element

The audio element is used for embedding an audio player inside a page for a particular audio file.

You can use various attributes to customize the player: preload, autoplay, loop, and controls.

<audio src="argentina.ogg"
  controls="true" preload="true"></audio>

The video element

The video element embeds a video player for a particular video file.

You can use these attributes to customize the player: poster, preload, autoplay, loop, and controls.

<video src="chrome_japan.ogv" controls width="390"></video>

Codecs & Containers

Audio codecs

An audio track is compressed, stored, and decoded according to a codec. Most audio codecs support two channels of sound, some support more.

The most relevant audio codecs are:

  • MP3: Patent-encumbered.
  • AAC (Advanced Audio Coding): Patent-encumbered. Used in Apple products.
  • Ogg Vorbis: Free, open-source, patent-free

Browser support

There is no official codec for browsers to support, so they vary:

HTML5 Audio Support

Video container formats

A video file, like an ".avi" file, is really a container for multiple related files that describe a video, like:

  • a video track
  • one or more audio tracks with synchronization markers
  • metadata (title, album art, etc).

The most popular video containers are:

  • MPEG4: .mp4, .m4v
  • Ogg: .ogv
  • WebM: .webm

Video codecs

A video track is compressed and stored in a particular way, and a "codec" describes how to turn the stored data into the series of images that you see.

The most relevant video codecs are:

  • H.264: Patent-encumbered.
  • Theora: Royalty-free.
  • VP8: Patented but licensed royalty-free.

Browser support

Each browser supports a different combo of formats and codecs, due to patents issues. This is the current situation:

HTML5 Video Support

Multimedia Encoders

There are now many online services, desktop apps, and command-line libraries for encoding, like:

Multiple media sources

As we saw, different browsers support different formats. Thankfully, HTML5 lets us specify multiple sources for the video and audio elements, so browsers can use whichever works for them.

<video height="200" controls="">
  <source src="chrome-japan.webm" type="video/webm"/>
  <source src="chrome-japan.mp4" type="video/mp4"/>
  <source src="chrome_japan.ogv" type="video/ogg"/>

Fallback options

There are some browsers that don't support these new elements. In that case, you can provide a fallback option via a browser plug-in, like Java or Flash, and put the fallback inside the element. For example, here's a Flash fallback:

<video src="video.ogv" controls>  
    <object data="flvplayer.swf" type="application/x-shockwave-flash">  
      <param value="flvplayer.swf" name="movie"/>  



The track tag can be used to specify subtitles, captions, chapters, descriptions, and metadata:

<video src="video-file.mp4"  controls>
  <track src="en.vtt" kind="subtitles" srclang="en" label="English subtitles" default>
  <track src="de.vtt" kind="subtitles" srclang="de" label="German subtitles">
  <track src="ch-en.vtt" kind="chapters" srclang="en" label="English chapter listing" default>

Video Track · HTML5 Rocks Demo · Elephants Dream

Track files

The WebVTT or SRT file format describes timed information.

Tags and annotations can be used to specify class names, timestamps, positioning:

00:00:10.500 --> 00:00:13.000 A:start
Elephants Dream

00:00:15.000 --> 00:00:18.000 A:end L:10%
And thus it <b>begins</b>... 

JavaScript APIs

API Attributes

Many of these attributes can be set via JS, and trigger events on change:

  • duration
  • autoplay
  • loop
  • controls
  • muted
  • src
  • currentSrc
  • networkState
  • preload
  • buffered
  • readyState
  • volume
  • playbackRate
  • paused
  • ended
  • width
  • height
  • poster
var video = document.getElementsByTagName("video")[0];
video.playbackRate = video.playbackRate * 2;

MDN Video Attributes reference

API Events

  • loadstart
  • progress
  • suspend
  • abort
  • error
  • emptied
  • stalled
  • loadedmetadata
  • loadeddata
  • canplay
  • canplaythrough
  • playing
  • waiting
  • seeking
  • seeked
  • ended
  • durationchange
  • timeupdate
  • play
  • pause
  • ratechange
  • volumechange
<div id="time">0</div>
var video = document.getElementsByTagName("video")[0];
video.addEventListener('timechange', function() {
  document.getElementById("time").innerHTML = video.currentTime;
}, false);
W3C Video Events Spec

API Functions

  • load()
  • play()
  • pause()
  • canPlayType()
  • addTextTrack()
<button id="play" title="play" onclick="playVideo()">play</button>
function playVideo() {
  var video = document.getElementsByTagName("video")[0];;

The Future!