————————————————————————————————————————————————————————————————————
Java Media Development with QuickTime for Java
by Chris Adamson12/23/2002
Now that Sun's Java Media Framework can't even play MP3s anymore — support was removed in August due to what Sun calls a "licensing issue" — its collection of supported media formats and compression schemes (codecs) has dwindled to near-uselessness. The JMF's powerful plug-in architecture allows developers to expand JMF's capabilities, however, and that's exactly what this article will do, by using the rival media API, Apple's QuickTime for Java.
In this first part, we'll consider how JMF's design leaves the door open to improve its capabilities. In the second part, we'll get into the details of QuickTime for Java to deliver more media support.
Opening up Java Media Framework
To handle a given piece of digital media, an application has to know how to handle the following:
- Format — how the contents are arranged in a file or network stream.
- Multiplexing — how multiple media streams are put mixed together into a single byte-stream. While it might be convenient to write all the video data to a file and then all the audio, the resulting file could be difficult or impossible to play at a consistent speed, so you "multiplex," or interleave, the streams together, putting the pieces of each stream that represent the same time close to one another.
- Encoding — how the media is encoded and (usually) compressed.
- Rendering — how to present the decoded/decompressed data to the screen, speakers, etc.
| Source Code Download the source code for this article. |
Sun's JMF implementation comes with classes that can only handle a handful of the possible formats and codecs you're likely to encounter on the Web. According to the supported media types page, the most popular video formats JMF supports in its all-Java version are the deprecated AVI format and QuickTime's .mov file format ... not Windows Media WMA and WMV, or RealMedia .rm formats. And just because a format is supported doesn't mean a given clip in that format will play. For example, JMF can't handle DivX AVIs or QuickTime files that use the popular Sorenson video codecs.
Extending JMF
The good news is that you're not stuck with this modest media support, thanks to the plug-in architecture. The design of JMF allows it to decide at runtime what code to use to handle a given format, multiplex scheme, and encoding. It does this by carefully parceling out responsibility for format handling, demultiplexing, decoding, and rendering into different classes, then using reflection to discover what kinds of handlers are available.
Extending JMF often means writing new DataSources and/or Players or Processors. A DataSource represents both the media's organization and access to it. In other words, you'd look to a DataSource to determine the media's content type and to open a stream to the media. A Player represents the ability to "play" the media, that is to say, to start reading and decoding the time-based data, and presumably to render it to the screen and/or speakers. Player has a subclass called Processor that allows lower-level access to the decoded media, which you might use to add effects or "transcode" to another format. To keep things simple, in this article, I provide code for only a small subset of Player-defined functionality.
You don't always instantiate DataSources and Players directly (although we did force the issue of our own special .jar-file-handling DataSource in a previous ONJava.com article). Instead, you ask a class called Manager to try to find an appropriate DataSource for a MediaLocator (which is basically a wrapper for a URL), and then an appropriate Player for a DataSource.
How Manager Works
In each case, Manager takes a list of known package prefixes, combines that with a standardized subpackaging and class-naming scheme, and looks for the resulting class in the CLASSPATH. For DataSources, the subpackage path is media.protocol.protocol-type, where protocol-type is the protocol part of the URL — for example, http or file. So, for the default package prefixes javax, com.sun, and com.ibm, the Manager would try to handle an http:-style URL by searching for the classes, respectively:
javax.media.protocol.http.DataSourcecom.sun.media.protocol.http.DataSourcecom.ibm.media.protocol.http.DataSource
The scheme is similar for Players, except that the subpackage path needs to incorporate the content-type, such as video.mpeg or audio.midi. The basic subpackage path is media.content.content-type.Handler. Yes, Handler, not Player ... it's kind of weird that way. There's also a special rule for Players: if no class is found for a given content-type, the CLASSPATH is searched again with unknown for the content-type. Generally, this "unknown" class is expected to be a "catch-all" player, quite possibly an OS-specific native player. So for video.quicktime content-type and the default package-prefixes as above, Manager would search for six classes:
javax.media.content.video.quicktime.Handlercom.sun.media.content.video.quicktime.Handlercom.ibm.media.content.video.quicktime.Handlerjavax.media.content.unknown.Handlercom.sun.media.content.unknown.Handlercom.ibm.media.content.unknown.Handler
Each time it finds one of these classes, Manager attempts to call the Player's setSource() method. The search is over when it finds a Player that doesn't throw an exception when setSource is called.
By the way, in our implementation there's no difference between the DataSource for the http, file, and rtsp protocols, so there's a single com.mac.invalidname.punt.media.protocol.DataSource class, and the protocol-specific subpackages have trivial subclasses of this class.
Using the JMF Registry
To control the behavior of Manager, JMF provides the JMFRegistry application, which allows an end user to inform JMF of new plug-ins and to control some of Manager's behavior.
To use the JMF Registry, use the executable that comes with the OS-specific JMF release, or in the all-Java JMF, enter at the command line:
java -classpath $JMFHOME/lib/jmf.jar JMFRegistryOf interest to this article is the "Packages" tab. This defines the list of package prefixes used above, in the order in which they will be tried.
When you're ready to try the JMF-to-QuickTime bridge, run the JMF Registry with the punt.jar file in your CLASSPATH. Use the Add buttons to add com.mac.invalidname.punt to both the Protocol Prefix List and the Content Prefix List, then select the package and use the "Move Up" buttons to make it the first choice in the lists. The result should look like this:
JMF Registry: Packages tab
Click "Commit" to save your changes to the registry. The Manager will now make com.mac.invalidname.punt the first package it tries when searching for DataSources and Players.
Trying it Out
To get a taste of the JMF-QTJ bridge, make sure you've downloaded and installed the SDKs for JMF (at least version 2.1, latest is 2.1.1c) and QTJ. Since we want to play MPEG-4 files, and since the code uses the new JQTCanvas class, be sure you have QuickTime 6. Windows users should do a custom-install to make sure QuickTime for Java gets installed or updated (it's not installed by default on Windows). After installing, try out the simple demos like JMF's JMStudio and QTJ's PlayMovie, just to make sure you've got any CLASSPATH issues resolved. Note that QuickTime is only available for Windows and Mac — sorry, Linux folks.
On Windows, the Makefile I wrote assumes you're running Cygwin; be sure to export OSTYPE=cygwin to get file and path separators handled correctly. On Mac OS X, there's a curiosity about the QTJava.zip file: it's in your CLASSPATH when running a Java application, but not when compiling with javac or jikes. The Makefile deals with this by always putting QTJava.zip in the CLASSPATH for you.
The code includes a "sanity check" application, launched from the Makefile with make sanitycheck. It shows off the supported methods by setting up an AWT Frame with the movie and its control bar in a non-standard location, playing the media for a few seconds, muting the sound, blasting the sound, stopping the movie, jumping halfway into the media, grabbing the current frame, and then playing backwards. Here's what it looks like:
Screenshot of make sanitycheck with MPEG-4 iMac ad
It takes a URL on the command line — the default in the Makefile is an MPEG-4 clip of one of Apple's iMac advertisements, but the SANITYURL argument is easily changed in the Makefile, overridden by its companion private.mk file, or you can just call com.mac.invalidname.punt.SanityCheck directly.
The code can also be used in JMF's demo media-player, JMStudio. Use make runjmstudio to try it out.
This should handle media formats unsupported by JMF, including MPEG-4, MP3, QuickTime movies with Sorenson Video, and even user-added QuickTime components like On2's open-source VP3 or Apple's optional MPEG-2 component. Curiously, though, while QTJ supports Flash 5, JMF still seems to grab .swf files before I can, and tries to play them with its obsolete Flash 2 player.
Part 2: Closing the Deal with QuickTime for Java
QuickTime Basics
The first thing a QuickTime for Java application has to do is to initialize QuickTime with the QTSession.open() call. Subsequent QTJ method calls will fail if this hasn't been done. It's also important to shut down QuickTime when your app is done. Mac OS X handles this for you when you use the default Quit menu item, and on Windows you typically add a WindowListener on your main window to close down QuickTime before terminating the application. In the case of our bridge, we don't know when the application is actually being shut down, so our static initializer that opens QuickTime also registers a ShutdownHandler to shut down QuickTime when the calling application is going away.
Unsurprisingly, while both JMF and QTJ do some similar things and even have some common method names, their structural approach is rather different. In JMF, a DataSource represents a reference to media (its content type, the ability to start reading data from it, etc.), while a Player represents the ability to actually decode and present that media. In QuickTime, the Movie object represents some of both of these roles. The docs opaquely state, "The movie is not the medium; it is the organizing principle." True enough, but don't get the idea that a Movie is some inert object to be passed around ... in fact, it's the object that has the start() method, and is probably the class you'll get most familiar with.
Because Movie contains functionality represented in JMF by both DataSource (getDuration(), the Positionable interface's setPosition(), etc.) and Player (start(), stop(), setRate(), etc.), it makes sense for our bridge to share a Movie between a DataSource and a Player that we define.
One note before continuing: the included code is an absolute bare-bones implementation of a JMF-to-QTJ bridge, and completely no-ops many methods that exist solely in the JMF world, such as the managing of multiple Controllers and the careful detailing of JMF states. On the latter issue, our Player is always in the Realized state, meaning it's ready to go. Extending the idea to a full-blown JMF implementation, possibly implementing Processor and providing capture ability with the QTJ SequenceGrabber API is, as always, left to the reader as an exercise.
|
Loading a Movie
To create a Movie in QuickTime, you can call the new Movie() constructor to create a new Movie in memory, one that you'd then add tracks to, but for our purposes, we want to quickly get a Movie loaded with the media from some file or URL. The most general way to do this is to use the static Movie.fromDataRef() method, which takes a DataRef argument. The DataRef class is a generic reference to a media source — a file, a URL, even a location in memory in the form of a QTHandleRef (and you didn't think Java had pointers!).
The second argument to the Movie.fromDataRef() is an int called flags. The use of behavior-modifying flags, typically combined with logical ORs when you're using more than one, is seen throughout QuickTime, and it can be very aggravating to the Java programmer, as appropriate values are almost never detailed in the javadocs. In the case of Movie.fromDataRef(), the method's javadoc documentation does recommend the newMovieAsyncOK flag (defined in StdQTConstants4, one of several massive classes that define pseudo-consts), but doesn't mention the useful newMovieActive flag that is commonly set for this method. For the real story about appropriate flags, you have to see the method's native API documentation, and I suppose we're lucky to have the convenient hyperlink from the javadocs, as the full QuickTime API reference currently weighs in at a table-cracking 3,304 pages.
But there's one more surprise when loading a movie this way. Using the newMovieAsyncOK method means that QuickTime won't block on loading media, and while that's good for user-experience reasons (we don't want to block interminably, especially if we're in Java's AWT event-dispatch thread), it has a nasty side-effect for "quick start" movies, which are non-streaming movies that QuickTime can nevertheless start playing before it has all the bytes, so long as it thinks the download will stay ahead of the playback. The problem is that when our call returns immediately, our movie has no width and no height, and won't until some amount of media is downloaded. That messes up the creation of our GUI widget (below) by defaulting it to a zero-by-zero size.
The native API has a method called GetMovieLoadState that we could use to tell if we have at least a little bit of the movie downloaded — a return value of kMovieLoadStatePlayable means we have enough to start playing — but the method seems not to have a QuickTime for Java equivalent (although its return values are defined in StdQTConstants5). So instead, the included code handles the problem by blocking until the movie has more than zero seconds of media loaded. This is determined by a call to maxLoadedTimeInMovie(), and while it's zero, we call task() to give QuickTime more time to load.
The Movie-loading code is below (with comments and printlns removed), and merits two more small comments. The first is that Java on Mac OS X seems to like URLs of the form file:/, while QuickTime needs file:///, so we need a workaround for that. Secondly, we call a pair of methods, prePreroll and preroll, to coax QuickTime into getting ready to play the movie sooner.
public void setLocator (MediaLocator ml) { super.setLocator (ml); String urlString = ml.toString(); try { java.net.URL url = ml.getURL(); if (url.getProtocol().equals("file")) urlString = fixFileURL(url); else urlString = url.toExternalForm(); DataRef urlRef = new DataRef (urlString); qtMovie = Movie.fromDataRef (urlRef, StdQTConstants4.newMovieAsyncOK | StdQTConstants.newMovieActive); qtMovie.prePreroll (0, 1.0f); qtMovie.preroll (0, 1.0f); while (qtMovie.maxLoadedTimeInMovie() == 0) { qtMovie.task (100); } } catch (QTException qte) { System.out.println ("Couldn't get a QT Movie from " + urlString); qte.printStackTrace(); } catch (java.net.MalformedURLException murle) { System.out.println ("Bad URL: " + urlString); murle.printStackTrace(); } }Getting a GUI
In JMF, you get the media into the GUI by calling getVisualComponent on the Player. If the media has images or video, this returns an AWT Component. To get a Swing-friendly component, you first call Manager.setHint(Manager.Manager.LIGHTWEIGHT_RENDERER). Getting a control widget, such as the typical time-slider with popup volume control, is done with getControlPanelComponent().
In QuickTime, the visual representation does not come from the Movie. Rather, you create a GUI component, create a Drawable from the Movie, and add the Drawable to the component with setClient(). QuickTime offers a heavyweight QTCanvas for AWT use and a lightweight JQTCanvas for Swing use, though performance in Swing is so poor that you might be better off using QTCanvas and working through any AWT-Swing collisions (one good hint from the quicktime-java mailing list: if your only issue is with JMenus disappearing under the QTCanvas, call setLightweightPopupEnabled(false) on the menus).
The Drawable that's passed to setClient comes in several different flavors for different purposes. Here are a few popular ones:
QTPlayer— shows media with the standard QuickTime controller.MoviePlayer— same as above without the controller.MoviePresenter— draws movie to an offscreen buffer (so you can apply effects) and then draws buffer to the screen.SGDrawer— shows the stream from a capture device such as a webcam.
In our case, we use the MoviePlayer, using two separate QTCanvases or JQTCanvases, one for showing the movie and another for the controller, which allows us to return the movie and its controller as separate components with the JMF Player's getVisualComponent() and getControlPanelComponent().
Here's our implementation from Handler (which is what JMF requires us to call our Player implementation), which instantiates an instance variable called movieCanvas:
public java.awt.Component getVisualComponent() { System.out.println ("getVisualComponent()"); Movie movie = getMovieFromDataSource(); if (movie == null) return null; if (movieCanvas != null) return movieCanvas; try { MoviePlayer moviePlayer = new MoviePlayer (movie); Object swingHint = Manager.getHint(Manager.LIGHTWEIGHT_RENDERER); if ((swingHint != null) && (swingHint instanceof Boolean) && ((Boolean)swingHint).booleanValue()) { JQTCanvas jqtc = new JQTCanvas(); jqtc.setClient (moviePlayer, true); movieCanvas = jqtc; } else { QTCanvas qtc = new QTCanvas(); qtc.setClient (moviePlayer, true); movieCanvas = qtc; } } catch (QTException qte) { qte.printStackTrace(); } return movieCanvas; }Small JMF-QTJ differences
A few notes on areas where JMF and QTJ have similar ideas but different details:
setRate() — both APIs have a similar concept of representing the playback rate as a float value, where 1.0 is playing forward at regular speed, 2.0 is forward double-speed, -1.0 is backwards at regular speed, etc. In JMF, the rate can only be set when a Player is stopped, in effect saying "this is the speed to play at when started." In QTJ, starting and stopping is implicit in setting the rate, meaning that setRate(1.0) is equivalent to "start playing at rate 1.0, regardless of whether or not the movie is currently playing." Similarly, setRate(0.0) in QTJ is equivalent to stop(), though actually calling stop() is recommended. Another point to note is that JMF's setRate() is not guaranteed to work for any arbitrary value other than 1.0, and in practice, negative rates never seem to work with the default JMF components.
Volume / Gain — a JMF GainControl has a concept of a "mute" that is independent of gain. Like the "mute" button on a television, calling setMute(false) brings up the volume to the level it was at before you hit "mute." QuickTime's volume has no corresponding concept, so we remember the "old" volume on a setMute(true) and restore it on setMute(false).
QT time scale — JMF Time objects work with seconds or nanoseconds. QuickTime movies report duration and current position in integers, in a scale that is unique to the Movie. For example, if getTime() returns 1800, and getTimeScale() returns 600, then you're 3.0 seconds into the movie. In other words, timeInSeconds = qtTimeValue / qtTimeScale.
Grabbing an Image from a Movie
| Related Reading  Java 2D Graphics |
At the 2002 O'Reilly Mac OS X Conference, an attendee asked me how you grab an image from a QuickTime movie. I said I didn't remember the details right then, but that it involved drawing the bytes to an offscreen buffer and then getting the image from there. Well, half-right, and half-wrong.
Half-wrong because if all you're interested in is getting the image written to a file, it's easier than it looks. Movie has a getPict method that gets the image at any time in the movie and returns it as a Pict, a wrapper around the classic Macintosh PICT image format. Pict, in turn, has a writeToFile().
But to support JMF's FrameGrabbingControl, we have to return the grabbed frame as an AWT Image, wrapped inside a JMF Buffer, and that's where things get tricky. We need a QuickTime RawEncodedImage for transferring the bytes, so we draw the Pict to an offscreen QDGraphics, which in many ways resembles an AWT Graphics object (although the name is a trap, since Mac and QuickTime developers commonly refer to this object as a GWorld).
To get the bytes, we're fortunate that QuickTime uses a 32-bit ARGB scheme, meaning 8 bits each of alpha-channel (i.e., translucency), red, green, and blue, all packed into a 32-bit integer. We can use the DirectColorModel to represent this arrangement, and from there create an Image from the bytes in memory. (Jonathan Knudsen's Java 2D Graphics offers a full explanation of ColorModels and other low-level Image details.)
Once we have the AWT Image, we just have to use JMF's ImageToBuffer class, represented here as an instance variable called itb, to get the Image wrapped by the specified Buffer return type. Here's the complete method for grabbing the current frame:
public Buffer grabFrame() { Buffer convertedBuffer = null; try { // Grab a Pict from the Movie and draw it to a QDGraphics // note: could get a RawEncodedImage from the Pict, but // apparently no way to get a PixMap from the REI. QDRect box = movie.getBox(); Pict pict = movie.getPict (movie.getTime()); QDGraphics g = new QDGraphics(box); pict.draw(g, box); // get data from the QDGraphics PixMap pixMap = g.getPixMap(); RawEncodedImage rei = pixMap.getPixelData(); // copy bytes to an array int intsPerRow = pixMap.getRowBytes()/4; int[] pixels = new int [intsPerRow * box.getHeight()]; rei.copyToArray (0, pixels, 0, pixels.length); // now coax into image, ignoring alpha for speed DirectColorModel mod = new DirectColorModel (32, // bits/sample 0x00ff0000, // R 0x0000ff00, // G 0x000000ff, // B 0x00000000); // ignore alpha Image image = Toolkit.getDefaultToolkit().createImage ( new MemoryImageSource (box.getWidth(), // width box.getHeight(), // height mod, // color model pixels, // data 0, // offset intsPerRow)); // convert Image to a Buffer (frame rate == 0 is meaningless // because we are not generating live video) convertedBuffer = itb.createBuffer (image, 0); } catch (QTException qte) { qte.printStackTrace(); } return convertedBuffer; }Thanks to Andrew Millin and Vickie Jaffee from the quicktime-java list for the format-conversion recipe used in this method.
Conclusions
So there you have it ... a small amount of code provides a dramatic improvement in JMF's supported media formats and performance on the platforms QuickTime supports. With a similar plug-in approach, we could support other media formats; for example, by writing a JNI bridge to the recently open-sourced Helix Community code to add support for Real-format media.
On the other hand, this approach obscures the details and thus the power of QuickTime for Java, and if we're really just interested in working with QT-supported media, we should dig deeper into that API. In coming articles, that's what we hope to do.
Editor's Note -- due to reader feedback about installation issues, Chris has graciously updated the installation paragraph. We apologize for any inconvenience caused by previous versions of this article.Chris Adamson is an author, editor, and developer specializing in iPhone and Mac.