For the Wiidiaplayer, the technology choice was not that hard. I looked for a tiny-tiny-tiny while at building my own video player in javascript, but discarded this option almost immediately, flash was the way to go. Obviously there was still a choice on how much of the application should be flash (and the rest HTML/javascript I guess), but flash would always be a component.

My experinces with flash had never been extremely positive. In the Old Days of flash 6 I just didn’t get it. I could do the flash tutorial in the IDE 10 times, and still I didn’t get it. True, I could create a simple flash animation, but no, I wasn’t sure what I was doing. It was the same feeling I had using Word for the first time (having only used the alt-F3 code screen of WordPerfect before); it was the feeling I has when using Frontpage in design mode: The computer did a lot of stuff for me, and most did what I wanted, but the part that didn’t work as I needed it too, well, I didn’t have a clue why… I want to see lines of code, and I want to be sure that nothing exists except those lines of code!

Flash 7 and 8 actually made stuff a little easier for the programmer. Less designing was needed, and more programming could be used. Movieclips (but not library objects) could be created from code. This code, however, could live on a couple of hundred levels (each movieclip could have its own code, and handlers, and each object had handlers of its own, so sometimes you would spend hours figuring out why something would work in some way, only to find some old code still deep in your object structure.

One can imagine my relieve on encountering (the beta of) Flex 2 some years ago. For the first time (as far as I was concerned), code based flash programming was possible, and it seemed that finally I would be able to write a decent, extendable, reusable, flash movie, and it used eclipse, so I could use all my favorite key bindings in flash-editing. And, although it seemed like Adobe wanted everyone to buy the whole Flex-server package, the Flex IDE could be bought separately, for a price that was still “acceptable” to the individual. Still, there was one big flaw, it would be Flash 9 (Flash 8.5 then) only, and acceptance of a new flash version into the 90-95% range still takes years. Unfortunately, for this reason, I’ve not done any serious projects in Flex Builder 2 yet.

The Wiidiaplayer should run on the Wii, so should be flash 7 (or less). Flex would not be an option here. I had more or less decided to build the better part of the Wiidiaplayer in flash, and I dreaded having to do this in the flash IDE. I did build my proof-of-concept in the IDE, but decided I would start looking for alternatives.

The two serious alternatives that I could find are MTASC and HaXe; MTASC compiles actionscript files ito an existing swf file. This file should contain the library for the movie. Since the library typically only changes once or twice during the project, one could create the library in the flash IDE, and then develop in eclipse, and compile with MTASC. Yet a step further is the possibility to first compile the swf file and library out of thin air (ok, a very small XML file), using swfmill. MTASC only compiles up to ActionScript 2 files; anyone having the need to handle AS3 is directed to the successor of MTASC: HaXe.

HaXe seems to be a very ambitious project. HaXe is a programming language; programs in the language can be compiled to run serverside (though the neko virtual machine), run in the browser (it is rewritten to JavaScript), or compiled to an swf-flash file. In includes easy methods so that client-server communication is streamlined. The language is ECMA-script, so it’s really close to ActionScript, but not quite equal.

The idea to have one language that runs both on the server and on the client is compelling. In my work we recently find ourselves writing some algorithm both in PHP and in javascript. In addition, I really like the ECMA-script language, and would like to do my serverside programming in it as well. Finally, I was really surprised by the stability the language seemed to have: I only did a few tests, but it seemed to be stable, and deliver what it promised! However, the language is still very new. Resources on it on the internet are scarce, and it might well still contain bugs. There was especially little documentation on how to use it to produce flash 7 files; However, the most important reason for me not to use it, is it’s incompatibility. The MTASC homepage used to contain a section on why to use MTASC. One of the reasons was that, since it compiled normal ActionScript, switching back and forth between the flash IDE and MTASC was easy. Interestingly does the same site now promote HaXe as the way to the future. The first thing I needed (as with any large flash project) was a good logging system; Luckily the internet contains thousands, all free, and all written in ActionScript, none compatible with HaXe…..
So, for the past couple of weeks I’ve been developing using MTASC, the MTASC eclipse plugin, and swfmill, and I think I finally get Flash 🙂 .

Advertisements

Update 2009-07-28: In the final project another solution was pointed out to me, which ended up being better in most cases, see the commands here: http://code.google.com/p/wiidiaplayer/source/browse/trunk/wiidiaserver/rtmp/flvstreamprovider/convertvideoffmpeg.sh

In my previous blog I explained the format I want the flash video in (and the reasons why), however, getting my video in that format did pose some problems, most of which I hope I have tackled by now.

The actual command for converting the video itself is not so complicated:

 /usr/bin/mencoder "$1"
        -of lavf -lavfopts format=flv
        -af resample=44100:0:1 -af channels=2 -oac mp3lame -lameopts cbr:br=128 -mc 0
        -ovc lavc -lavcopts vcodec=flv:vbitrate=2500:autoaspect:vratetol=1000:keyint=1 -ofps $FPS
        -o "$2"

On the first line, it shows the command and inputfile, the second line defines handles the output container format; The third line handles all audio (44.1kHz (the next two numbers define the conversion type), use two channels, convert with lame to a constant bitrate of 128kbps. The final option disallows any shifting between video and audio frames.

The forth line defines the video options (codec flv, bitrate (max) 2500kbps (max 1000kbps tolerance), per 1 frame 1 keyframe (ie: use only keyframes), and the required framerate), the final line sets the output filename.

Now, before we continue, it’s necessary to note that I have installed 3 different versions of mplayer in the last week, and each version needed a slightly different command. Currently I’m using MPlayer SVN-r24130, as provided by Gentoo ebuild media-video/mplayer-1.0.20070824, if you have a different version, you might need to adjust some parameters. I’m using some unversioned version of ffmpeg, which is provided by media-video/ffmpeg-0.4.9_p20070330; however my experiences with ffmpeg is that it is much more stable and compatible between versions, so any recent version should probably do.

The problem with the command above is that it only works on certain video files, and obviously we would like to convert any video file that mplayer can play. It seems that there are several bugs in mplayer/mencoder that disallow this (try the command above on your average avi file and mencoder will crash with a segfault), so we have to take a work-around. We will convert the video in two steps: first convert the source to an intermediary format, next convert this format with the command above to the final result. To avoid quality loss and extra CPU load, the intermediary format is the raw, uncompressed video format. This format takes about 4.5MB per second, so we don’t want to store it on disk, but pipe it directly from one mencoder into the next. To accomplish this we need to redirect any other output from mencoder to somewhere else (so that only the audio/video data reaches the next mencoder, not the status information). Furthermore, we need to keep the first mencoder from wanting to seek in the file it writes; When writing avi-files, mencoder updates the avi header every couple of hundred megabytes, by seeking to the beginning of the file and writing the header – this obviously cannot be done with a pipe. The -noodml tag fixes this problem, but another problem persists: the second mencoder (behind the pipe) expects the maximum avi size to be 4GB (which is reached in 15 minutes), so after 15 minutes encoding stops. The only solution I’ve found to this is not using the avi container, but the asf container (the mpeg container and other containers have other problems, only asf seems to work).

So the encoding part of the file looks like this:

/usr/bin/mencoder "$1"
                -of lavf -lavfopts format=asf
                -oac pcm -af resample=44100:0:1
                -ovc raw -vf scale=400:224 -ofps $FPS
                -o /dev/fd/3 3>&1 >/var/log/mencoder/1 2>&1 |
        /usr/bin/mencoder /dev/stdin
                -of lavf -lavfopts format=flv
                -af resample=44100:0:1 -af channels=2 -oac mp3lame -lameopts cbr:br=128 -mc 0
                -ovc lavc -lavcopts vcodec=flv:vbitrate=2500:autoaspect:vratetol=1000:keyint=1 -ofps $FPS
                -o "$2" > /var/log/mencoder/2 2>&1

The first mencoder sets the correct resoltion, framerate and audio options, and outputs it to a filedescriptor called /dev/fd/3. The little filedescriptor magic afterwards redirects this filedescriptor 3 to stdout, while directing stdout en stderr to a logfile. Both the input and the output are files. For the wiidiaplayer all sources are (for now) files, and I render to a temporary directory as well, to improve caching and enable seeking in the video.

To this we add some extra padding that lowers the priority of the encoders (to avoid the computer locking up when coding), and some code that makes sure all child processes end when the parent process is killed, and that’s the full conversion script:

#!/bin/bash

function getmencoderchildids {
    ps --ppid $$ | awk '$4=="mencoder" {print $1}'
}

function stop_encoding {
 echo "now stopping"
 PIDS="$(getmencoderchildids)"
 for mypid in $PIDS; do
 	kill $mypid > /dev/null 2&>1;
 done
 sleep 2;
 PIDS="$(getmencoderchildids)"
 for mypid in $PIDS; do
 	kill -9 $mypid > /dev/null 2&>1;
 done
}

trap stop_encoding TERM;
trap stop_encoding EXIT;

FPS=18

nice -n 2 /usr/bin/mencoder "$1"
	        -of lavf -lavfopts format=asf
                -oac pcm -af resample=44100:0:1
                -ovc raw -vf scale=400:224 -ofps $FPS
                -o /dev/fd/3 3>&1 >/var/log/mencoder/1 2>&1 |
        nice -n 1 /usr/bin/mencoder /dev/stdin
        		-of lavf -lavfopts format=flv
        		-af resample=44100:0:1 -af channels=2 -oac mp3lame -lameopts cbr:br=128 -mc 0
        		-ovc lavc -lavcopts vcodec=flv:vbitrate=2500:autoaspect:vratetol=1000:keyint=1 -ofps $FPS
        		-o "$2" > /var/log/mencoder/2 2>&1 &
while [ -n "$(getmencoderchildids)" ]; do
 echo "$(getmencoderchildids)"
 sleep 1;
done

Nothing is as irritating as stuttering image or sound in the video you just put up on your big-screen TV. In the Wiidiaplayer, this stottering can come from 2 sources. Either the Wii doesn’t get the video and audio-data fast enough; this might be either the fault of the network, of the video-source, or the encoding program. Of these problems, only the encoding speed is of interest to me, the network is fast enough (11MBit), and the video source is a local video file. The problems in file-encoding wil be discussed in a separate blog entry.

The second reason for stuttering video or audio is the decoding and rendering speed on the Wii. Flash is extremely slow for showing video; on my Linux box, a flv video playing full screen in a browser might take 80% CPU (flash 9), while playing the same file in mplayer full screen results in less than 1 percent CPU load (and that is without optimized videocard drivers which will take even more load off the CPU). There might be a thousand good (or not) reasons for this, however this doesn’t change the fact that playing video in flash is slow. (There are two extra reasons to assume flash-video is slow. Firstly, the Wii plays mjpeg video’s without a problem, secondly, Adobe recently announced Moviestar, which (among other things) brings hardware acceleration to flash video (implying that it has little/none of the now).

Since the Wii has only limited processing power, it is essential to tweak our videos to make the best use of this power. It was clear from the start that 1080p videos on the Wii would not work. But as I stated before, although better image quality does look nicer, the movie usually does not improve. For me personally, what the Wii can bring is good enough.

There are several parameters that will infuence flv playback speed. I discovered the optimal values for most of these parameters through extensive testing on the Wii. Probably most tunings will work likewise on other platforms, however I never tested that. Note again, that I only tested for decoding-and-viewing performance on the Wii; The suggestions below are not valid when network bandwidth is an issue. I also do not claim to be an expert on audio or video-image quality, so while my settings are acceptable for me, others might be able to improve on them. Also, since your content is different from mine, your results might vary as well. If you get different results, please post them in the comments.

You will notice that on a lot of issues I substitute CPU on the Wii for CPU on the server (of which I claimed I had enough). These options probably need different tweaking when the server CPU is less powerfull.

Right now, I’m getting 400 x 224 video @ ~18 fps.

Testing method 

Since I do not have a reliable method to determine load on the Wii, I varied different parameters and played the file on the Wii with a too high framerate. As soon as the Wii reaches full capacity, frames will be dropped and the reported framerate will stabilize on a lower value than the actual video framerate. The flash program will regularly report the actual framerate (from the NetStream.currentFPS) back to the server; a higher framerate means that the load of each frame is lower.

Resolution

Obviously the resolution of the video file matters. A little experimenting led to a best resolution of 400 x 224. This is not exedentally exactly half of the 800 x 450 resolution that will fill the screen in Opera (adjusted for FLV resolutions which always need to be a multitude of 2).

Keyframes 

Logical reasoning would suggest that making each frame a keyframe, would be the most effective, and indeed, I seem to get a 1 or 2%  increase in framerate whe using only keyframes.

Video Framerate

Ideally, the framerate of the source video is equal to the framerate that is being displayed. When this is not the case, extra overhead is introduced by discarding frames. Also, the resulting viewing experience is less. If our video is 20 fps, and we can only show 19 fps, the result will be 18 images that are 50 milliseconds apart, followed by one that has a 100 millisecond gap. Having a continuous stream of images with 53 ms gaps looks far better.

What I found was that when video was encoded at 25 fps, the Wii would show only about 16.5 fps. Recoding the video to 18 fps allowed the Wii to play most of the video at 18 fps. However some parts of the video (especially those with heavy zooming and panning) still play at only 14-15fps. Encoding the whole video on 16 fps, would allow everything to play at 16 fps.

For now, I opt  for the higher setting of 18 fps.

Bitrate

My initial guess was that a higher bitrate would result in a lower CPU usage, since a higher bitrate would mean less compression (so less work to decompress). However, the opposite turned out to be true. Forcing the video bitrate to remain under the 2500kbps, avoids framedrops and stuttering on high-motion parts.

Sound

FLV supports mp3 and proprietary Nellymoser codec for sound. I didn’t look into the Nellymoser codec too much (don’t even know if any open source products can encode into it), some things can be tweaked on the mp3 side though. Leaving out the sound at all increases the framerate with about 2 fps compared to using 320kbps stereo mp3. Here as well it seems that a lower bitrate improved performance, so I settled for 128 kbps stereo.

Extra encoding options

There are numerous extra encoding options in mplayer, most of these seem to be concerned with exchanging bitrate for CPU on the encoding side. None of the extra options I tried increased the framerate measurably, nor could I see any improvement in image quality (and they do produce higher load on the server), so I use none of them

Overlays 

Perhaps more than obvious, but it tricked me for a while. Using extra stuff in your flash file on top of your video decreases performance, especially if these are nice semi-transparent layers. I had such a layer printing the current FPS for a while, not realizing that just that layer was keeping the fps down. A single transparent layer, occupying about 20% of the screen cost me about 25% or 4 fps performance. Having a lot of flash objects doesn’t seem to be a problem, as long as they are invisible (and, honestly, as soon as I pop up my menu, I don’t care that much that my framerates plumet).

SWF framerate

It is possible to set a framerate for SWF files. This decides how often a frame is drawn to provide nice swf animations, however seems to be totally independent of video framerate. An swf framerate of 2 fps, resulted in a really choppy interface, but the video still played at the same  framerate. I could not find any video-performance differences between 2 and 25 swf-fps (obviously, when doing a lot of swf-animations on an overlay, or having a (lot of) heavy  eventhandlers on “onframeenter”, keeping down the swf fps will improve video performance).

SWF scaling

I have experimented breefly with SWF scaling; making an swf of dimension p*q, and then telling the HTML page to resize the swf to x*y, but as far as I could see, it only decreased performance slightly.

SWF window mode

In the HTML page, it’s possible to set a SWF window mode, though the wmode attribute. The value of the wmode attribute seems to have no effect on the performance

SWF video smoothing 

There is a attribute for the video tag in flash called “Smoothing. Apparently, it exchanges CPU for quality in flash video playback. Since it only works for flash 8+, I expected the setting to have no effect, and I haven’t been able to measure any either.

Wii/Opera scaling 

From the WiiCR project, I got the suggestion that it could increase performance to make the swf-movie smaller (in dimensions/resolution) and use the Opera zooming/scaling functions to return it to full-screen size. Indeed, having the swf at only 400 x 224, and then zooming to full-screen, the performance seemed to improve about 1%. The problem however is that now I need to do manual zooming and positioning every time I want to watch a video, so I’m not sure whether this is worth it.

Lockups 

When doing frequent reloads of the wiidiplayer page on the Wii, sometimes the console locks up, any I need to press the power-button at the front of the console for 5 seconds to reset; during normal operation of the wiidiaplayer, however, I have not been able to crash it.

I’ll describe my mencoder settings in the next blog.