Media Player 128

[airship]

I've been following this with great interest, as I do all of your projects! As marvelous as it is to see anyone's projects for the C64, VIC-20, and PET, C128 projects are my favorite. Of course.

This one is awesome, BTW!

[Mathias Roslund]

It's little more blocky, but I bet that wouldn't matter much in motion.

OT: As for blocky. I made a little FMV experiment for the C64 a few years ago. My goal was to have something streamed from floppy running at 25fps, so I picked a simple animation and didn't worry too much about the visual quality. In case you decide to check it out, you need to press fire button on port 1, not port 2. Silly mistake 

[saehn]

Just checked it out... pretty cool! I'm sure Hydro's will look similar, but will hopefully the image quality will benefit from a larger screen size.

[RobertB]

As for blocky. I made a little FMV experiment for the C64 a few years ago.

Your Raycaster for the C64 with SuperCPU is interesting.

          Truly,
          Robert Bernardo
          Fresno Commodore User Group
          http://videocam.net.au/fcug

[wte]

What... no comments???

All people are still deeply impressed!

[Hydrophilic]

Thanks all for the interest and encouragment!  I was hoping to give you some color screen shots this weekend, but so far the color results are quite sad.

The only good thing in regards to color is the encoder does a good job of converting a color-source video into VIC-II monochrome.  (The images posted previously were generated from monochrome, so there were no conversion issues to begin with).

Like a schizoid, I can't do 1 thing at a time.  Besides working on color, I also worked on the RLE/Copy compression (now up from 67% to 70%) and did some regression testing...

I had not tested the new "in-visible-screen-2MHz" code on a real C128 until yesterday.  I've been doing most of my developement in VICE.  The VICE team did a good job of improving the 2MHz code I submitted years ago (in particular I/O clock stretching), but the emulator still doesn't do DRAM-refresh stretching and I think MRW instructions (INC $D030 for example) are also faulty.

I was worried the player would crash on a real C128 or run a little slower because of DRAM refresh.

To my delight, the player doesn't crash AND it runs faster!  I'm guessing it doesn't crash due to good design.     I'm guessing it runs faster because I'm using a uIEC with the real C128.  The C1581 used in VICE developement has a delay in moving from track to track... as would be expected in reality.

Well, I'm going to check out Mathias Roslund's FMV experiment and get back to Color coding.  Hopefully I'll have more screen shots next weekend.

I've been thinking (danger!) that to get a good color image, you want to give priority to objects in the foreground.  But there is no Z component in 2D source video.  So the encoder will have to guess which pixels are foreground and which are background.  That would take us to the 5^th dimension...  It's easy to get lost in all these dimensions 

[airship]

That would take us to the 5^th dimension...  It's easy to get lost in all these dimensions 

http://www.youtube.com/watch?v=fQP563gKwIU

[Hydrophilic]

Thanks for the link Airship.  I'd never heard of them before.  I see they have several songs/videos.  Is there anything special about the one you linked to?

Thanks for the link Mathias Roslund!  Your video has a much better frame rate than any of mine.  I knew I couldn't be the first to do something like this, but I couldn't find any links before I started this project.  I read your info web page and found it interesting.  My encoder is already breaking the image down into 4x4 blocks (basically... it's actually variable, see previous posts).  But what suprised me was your use of delta encoding...  I've seen this applied to audio, but never to video!

So I'm guessing it just counts bits and spits out a sequence of bit pairs?  That's a good idea in general, but with the audio IRQ constantly stealing cycles, I don't think the CPU would have enough time for anything that complex in my case.  I guess plan 83 could be a video with no audio... should be able to handle complex compression and give a better frame rate.

Or, an idea somebody mentioned was to use hardware for audio.  CIA#1 serial port is for the fast serial bus.  But you can use CIA#2 serial port output and feed it to the SID audio input.  Of course this would limit audio to 1-bit quality and require a minor hardware "device" (if you can call a cable a device), but it would reduce IRQ overhead by 75% with the same bit rate.  I have no intention of implementing that, but just thought I'd toss the idea out there.

I think I prefer the multicolor over hi-res.

Thanks for the feedback!  I tend to agree.  I mainly did that just to make sure it works in the general case... hi-res or multi-color.  All the screen shots above, and some I'm about to post all have a global palette.  This means it can be implemented in real bitmap mode (and it actually has been so far), or in text-emulated bitmap mode... but this would require a little extra effort to reduce the number of distinct cells to 256 or less.  Of course this would generally reduce the quality, but with a smaller memory footprint, it should give a higher frame rate all by itself, but moreover allow for more buffered frames increasing memory-copy compression and increasing frame rate yet again. 

Also in keeping with the general approach, there is also possibility for Extended Background Character mode.  It would have to be a really small/simple video to require only 64 cells   Besides hi-res quality, you also get 5 global colors, or 4 global + 1 custom if you include Color RAM.  Although the groundwork is there, I haven't even tried it because it would require good cell and color optimization.  I'm not that far yet...

I'm still working on the color problem.  I've tried several things but have yet to develop an algorithm that consistantly delivers good results.  I've learned quite a bit and have some results that might prove interesting, or perhaps hilarious, depending on your sense of humor...

[Hydrophilic]

One of the first things I tried for color was to have the encoder try every possibility and use the one with the least error.  Well even considering the small size of a VIC-II bitmap and the power of a 1+GHz 32-bit CPU with FPU, it would still take over 1 second to try all 16*15*14*13 = 43,680 combinations.  Which might be okay if you only had a few images, but could take an hour or more if you have 1000+ frames of video... really sad for a 5~10 minute video!

Leaving the speed issue aside, the results were highly unstable and often unpleasant.  It would often do things like mix purple and light green to make white... not pretty trust me... wish I had a good screen shot of that...

Anyway, I've attached some screen shots which show the bizarre unpredictability.  Some of the frames look like what you may have seen in other digitized VIC-II video.  But many look terrible.  The worse part, when watching a video, is the palette keeps dramatically shifting.  The last two images I've attached show 2 nearly identical frames, yet the encoder choose entirely different palettes.  I'm guessing this is due to subtle differences in the over-all hue / luminance of each frame.

But the thing I found interesting was the bizarre color artifacts constructed by the human mind (or at least mine).  In particular human-percieved 'color shift'.  Digital cameras have to take this into account in terms of 'white balance'.  Maybe it's not the exact same thing but anyway...

One example (sorry no screen shot) was if the picture used Brown as the overall (background) color instead of Black.  In this case, shades of Gray would appear somewhat Blue.

Another example (first screen shot) seems to have 3 shades of purple.  How many shades of purple does the VIC-II have??  Only one!  Now you might think this is due to color dithering, but if you look close, there really isn't any.  Well there is dithering with Black in the background, but I'm talking about the bright area around the headlight.

Now you might also think the headlight center is White or Light Gray, and the surrounding area is Cyan or Light Blue perhaps.  But no.  I took some time to analyze this one.  It turns out the center of the headlight is Light Green, and the surrounding is Light Gray.  Yet somehow the mind (at least mine) turns this into 3 shades of purple.

[Hydrophilic]

Due to the slow analysis time and unpredictible output of the "least error method" I've spent quite a while working on the color problem.  I still haven't got a good general purpose routine, but have improved the reliability and to some extent quality.

I've posted a couple of these for you.  Warning: Do not zoom in!  It only makes it look worse.  Trust me.

Although it seems to do okay for most scenes, in doesn't work well in very bright or very dark scenes.  It also often uses a LOT of dithering which mackes the image unpleasant to me, and more importantly it severly hurts compression and thus frame rate.

I'm still trying to find a nice balance between this algorithm and the "simple" (less dithered) method of the screen shots of the previous post.

[airship]

Man, you really know how to open up a can of worms, dude! LOL! I can't wait to see what you come up with next! I'm really enjoying the ride!

BTW, the 5th Dimension song I linked to uses the Declaration of Independence as its lyrics. Very inspirational to a Yank like me; I memorized most of it from the song. Their most famous song is Up! Up! And Away! (In My Beautiful Balloon). I actually saw the group live in 1969.

[Hydrophilic]

I can't believe they've been around that long and I haven't heard of them.  1969 makes me think of The Beatles, Elvis, and Pink Floyd.  That goes to show I don't have a very wide variety when it comes to music appreciation... I thought that I did... but you proved otherwise.

[Hydrophilic]

I'm making progress in the color department but it is slow going.  I had planned on simply adding 1 extra dimension to the problem -- hue, on the basis that VIC-II colors all have the same saturation.  That is according to an excellent article on VIC-II color by Philip "Pepto" Timmermann.  Of course if you read that, you'll see that Bob Yannes countered with the fact that the colors were choosen individually and are not as precise as described in the article.  Either way, my experience with several TVs and Commodore monitors (and a few non-Commodore monitors) makes me agree that all colors have the same saturation.

Unfortunately, you can't determine between gray and color without saturation.  So by mistake I took the project from the 4^th dimension to the 6^th dimension.  Oooops.

Previously I said my encoder was doing 16*15*14*13 calculations to optimize the color.  Wrong.  It was only doing 16+15+14+13 calculations.  That's several orders of magnitude difference!  I'm guessing it ran slow because I am testing with a debug build.  I've written more advanced graphic programs that run faster once compiled normally.

Anyway, because of the prior slow performance, and the highly unstable palette choices of the "least error method", I've developed a new color coding algorithm.  It starts by building a histogram of all used colors.  This takes only slightly longer than calculating the error for 1 color (which previously would be done 16+15+14+13 times).

Then it "eliminates" the less used colors and builds a palette from those that "survive".  To my amazement, many scenes end up requiring only 2 hues.  Not most, but many.

I have a confession.  The last algorithim I tried (the last set of screen shots) was not really an algorithim.  To avoid the time penalty and palette instability of the first "least error" method, the last simply choose 1 of a few hard-coded palettes.  Although some of the scenes look nice, a lot of them still looked like crap.  Naturally I posted the good-looking ones.

In particular, dark scenes looked terrible.  And bright scenes didn't look very pleasant due to massive dithering (see the last two screen shots from the previous post as an example).

With the new "histogram and eliminate" method, dark scenes look fantastic.  Bright scenes generally look better due to less dithering, but they are generally not as colorful as before.  Compromises, compromises... other important benefits are better compression and usually a consistant color palette.  I still need to tweak it, because occassionaly, about once every 30 frames, it will decide upon some totally crazy palette.  (Much better than every 2 frames with the "least error" method).

I can post some screen shots if you like.  Any suggestions would be welcome as well.

[saehn]

I have no suggestions :-) but we would always love to see more screenshots!

[Hydrophilic]

Okay here's some screen shots with some comments.  That way I can tell you what I think the problem is and how I might be able to fix it, or maybe someone will have a better idea they would like to contribute.

So I'll start with the re-rendering of the last 4 frames I posted previously.  In the 1st, you've got high contrast, with the black background and the white headlight.  These are are the hardest scenes to digitize because you can't effectively emulate black or white.  So with 2 colors used, there are only two colors left.  Note you can effectively emulate light gray.  The encoder will try to fudge the contrast a little to eliminate white so that it can use light gray.  But it doesn't try too hard because that increases dithering, which I prefer to avoid for both asthetic reasons and the reduced compressibility.  As we shall see later, sometimes it doesn't try hard enough.  Overall I am pleased with the result, as the characters are more detailed than the orginal and it has higher contrast.  The only thing I can think to make it look better would be to use yellow for the headlight, which would dither better with the characters... if you zoom in you can see how annoying it is where it dithers white with the flesh tones.

In the 2nd one, the scene is not as colorful in this rendering as in the previous post.  Although it has lower color fidelity, it has better dynamic range.  Again a 'hard scene' ranging from black to white.  So you still end up with a lot of dithering, but it is a lot less than the original.  So in that regard it looks better; except dithering orange with white is a bit annoying.  Overall I think CBM/CSG did a good with the VIC-II palette, but orange is just too dark!   

The third frame looks better overall in my opinion due to use of real white, instead of trying to fake it through dithering.  Although it too lacks reduced color fidality because it uses to shades of gray (I think dark gray and white) and thus only has 2 real colors to work with.  In the previous post, the encoder used what I call 'massive dithering.'  The new code does it's best to avoid that.

The fourth frame is really questionable.  On the one hand it looks a lot better due to less dithering, and it has better dynamic range (black to white again) but on the other hand there a few problems.  For one, it only choose 1 color and 3 shades of gray.  Which means flesh tones end up as gray.  Now she looks like a zombie.  Another problem you should notice there is no detail in the wall on the right side of the new image.  I'm pretty sure this is because white is too bright and washes out the details, similar to over exposure in photography.

Also in regards to the fourth image (actually all images I'm posting today), the image was converted using 'full contrast'.  In the first set of images I rendered, I set a global paremeter to reduce the contrast by about 20%.  This was just a hold-over from the black-and-white conversion where I found that setting to produce the best results.

So I'm think I need to 'localize' the contrast setting for use on a frame-by-frame basis (instead of globably for the whole video).  Of course I need to come up with a good algorithm for this, as I'm sure nobody would want to customize the contrast by hand for each frame of video...

Well it already does this to a limited extent, but only when it detects large amounts of both black and white.  I think a more general purpose technique would be better.  And as the new rendering of frame 4 shows, it doesn't always do a good job, even in this special case.

[Hydrophilic]

Now I don't know if it is an inherent limit of the VIC-II, or just my faulty algorithims, but I think the dark scenes produce some of the best looking images.  Here are a few of my favorites.  I think the software has improved enormously in this regard.  The first color images I posted rarely had any dark scenes because they almost always looked like crap so I didn't post them.  Now the dark scenes are usually the best looking.

If I could only get the 'white' scenes to look this good, I could almost call this phase of the project complete...

Oh about the images... the 1st is another frame from the motorcycle scene, but without the headlight forcing the use of white.  The 2nd frame emulates light gray so it has a moderate amount of dithering.  (Not the 'massive dithering' I mentioned previously.)  The 3rd frame uses real light gray.  This works a lot better than white, don't you think?  The 4th frame features a brunette, which is why the hair is not red.

[Hydrophilic]

The last 4 images are what I would judge 'really good'.  The 4 prior to that I would call 'acceptable' given the limits of the VIC-II and my own self-imposed constraint of a single palette for the entire bitmap (as opposed to optimizing the 3 colors in each cell).

Now here are some that I would judge anywhere from 'unacceptable' to 'absolutely terrible'.

The first thing you should know is that all of these were digitized from color video.  But they all ended up as gray scale.  Sometimes this is okay if the scene doesn't contain a lot of color, but these scenes contain color in at least 25% of the pixels, which means the encoder should allocate at least one hue to the palette but it does not.

In the 1st image, the flesh tones and blonde hair are missing (also in the last 2).  In the 2nd image, the orange mountains and the blue sky are missing.  The problem seems to me that obviously the saturation is too low.  This was anticipated and the encoder is designed to increase the saturation in such scenes in order to 'reveal' colors of low saturation.  But as these examples show, it isn't doing a very good job...

Disregarding the lack of color for the moment, the 1st image just doesn't look as good most other gray scale images.  I think this is due to white.  Unfortunately, I'm not sure how to improve it...

The last two images (no.3 and 4) are nearly identical frames from the same video, but I think you will agree #3 looks terrible!  Disregarding the lack of color, I think #4 looks good.  Really strange that two very similar frames would vary so drastically in quality.

If you look closely, you can see that #4 uses a dither of White and Medium Gray.  But #3 chose the 'middle value' of Light Gray.  And as it so happens in this particular image, most pixels are very close to Light Gray so you get a whole bunch of them, and consequently little detail.

Now by design, the encoder is trying to find the 'best color' in order to minimize dithering which improves compression and usually improves quality.  But in the case of #3, we see that it can completely ruin the image.  The only way I can think to prevent this is perform some type of Fourier/Laplace Transform on the bitmap.  This is a computationaly intensive task and the math/algorithim can get ugly.  Of course there are well-known Fast Fourier Transform algorithims, but they are still CPU intensive.  Many video encoders use them via way of SIMD/MMX/3DNOW instructions, AFAIK.  I have no intention of writting inline assembly, and even if this could be done acceptably in pure C, I'm thinking it is not neccessary.

I'm thinking a simple 'frequency analysis' over every 8th raster or so should suffice to determine if the encoder has choosen a 'terrible' color.  This wouldn't be a true transform into the frequency domain, merely a measure of how many transitions occur on the raster.  A quick-and-dirty method, if you will.

[Hydrophilic]

Here are some scenes where the 'saturation amplifier' actually worked to reveal some color.

Even though we get some color in image#1, I think it still looks unacceptable.  #2 is okay I guess...

Which tends to make me think White is the problem.  But the last two frames contain a lot of white (and properly contain no color), and look pretty good I think.  So the question is how to fix the white 'problem' without breaking the good job it does on occassion...

EditActually #4 should contain color... silly me... but importantly, the image turned out well despite the high amount of White... so it isn't the mere fact of a lot of white... a pesky problem...

[airship]

Is there any chance that the extremes could be averted by eliminating pure black and pure white as palette choices? I know in painting you can sometimes get better effects by limiting your palette. Just a thought, as I know NOTHING about the subject.

Keep up the great work!

[Hydrophilic]

At airship: I think there is a good chance!  It tries to eliminate either black or white if both are used, but only if they are not used 'a lot'.  I think some the of 'shock and awe' would be lost if one is dropped when they are 'important'  ... high use.  I guess I need to develop a better algorithim to determine when to keep and when to drop...

Having both black and white causes trouble most of the time.  But if it so happens that only one hue (or 2 closely related hues) are needed, then the images actually turn out pretty good, I think.  Here are a few to judge for yourself.

The 1st looks good overall, but there is a loss of detail in the face (just a blob of gray and orange pixels).  The 2nd looks really good but is 'cheating' because it actually uses Dark Gray instead of Black.  The third looks good and is really using black.  I'm guessing because the subject is close enough to the camera that facial details aren't lost.

[Hydrophilic]

I've previously described some of the problems with missing color due to low saturation and demonstrated how it sometimes 'reveals' these colors and sometimes fails to do so.

So you would think (at least I did) that using source video without a lot of saturated colors would prevent this problem.  Well then what better than an animated video.  And for Commodore fans, what would better than an animated movie from the 1980s?  Maybe you will recognize this one...

In the first image, there is nice use of color and a moderate amount of dithering.  If you think the dithering is bad, you should refer to the original color images I posted which sufferred horribly from 'massive dithering'.  So I'm just saying I think #1 looks good considering the alternative...

The same goes for #2

But in #3 where the camera pulls back revealing a vast area, most of the color is lost.  I think this because the large area has a wide variety of colors that average to 'gray' and only the highly concentrated areas of get 'recognized' for color...

Most of the frames of the video turned out pretty good but some just don't look right to me.  The 4th one is an example.

[Hydrophilic]

Here are some more images from the same video I thought you might enjoy.  I do.  I really have no comment except for image#2.  I think he should be saying,

My cannon is so powerful, it will destroy the palette of the VIC-II!

[Hydrophilic]

Well good looking images from an animated film don't really count in my opinion.  They were just test how the encoder would perform with high saturation and a lot of colors.

So here are some images from some 'real videos' that actually did a good job with color.  Or so I think, you be the judge.

The 1st ones does a mediocre job with a limited number of hues... it didn't get the flesh tones.  #2 gets the flesh tones but suffers from a moderate amount of dithering as a consequence.  #3 is a demonstration of the outfit she choose to wear... and her fingernails are really painted black, it's not a missing color...

#4 is for that Yankee, Airship.  I'm really supprised the encoder picked out blue, considering how dark a color it is... the pink blob in the background is the girl from image#3... once she gets closer, the blue is lost and gets rendered as black.  What I really like about #4 is the different shades of red in the flag... there really aren't multiple shades!  Just a very convincing dither... I think...

[Hydrophilic]

Here are some images I think are interesting.  The first 2 show the effect of blue light in the club very well.  The last two have slightly less of a blue light effect, but the wide camera angle (with many people in the scene) suffers terribly from pixelation...  Revealing the limits of the VIC-II, or at least of my encoder... I don't think there is anything that could be done except switch to hi-res mode with devastating color consequences...

There are several more real nice images and a few terribles ones that I could share, but I need to stop and take a break.  Last night I actually had a very vivid dream in VIC-II color.  Everything was a VIC-II color or a dither thereof.  Really bizare seeing shadows and lighting effects cast in VIC-II pixelation... and instead of walking from room to room, the room would disolve in a bizarre combination of VIC-II colors only for the next room to re-materialize in its place...

I wish I were experienced enough at organic chemistry and molecular biology to turn that into a pill... I would make a fortune at the clubs...

If you haven't suffered overload yet, you can download a ZIP file containg over 250 screen shots (8.2MB).

[Hydrophilic]

Gee, I was hoping somebody would say if they like the original 'extreme dither / high color' or the new 'moderate dither / moderate color' version.

The important thing (for my sanity) is the recent break from coding has resulted in no more VIC-II Color dreams   I did have another VIC-II dream a few nights ago, but without all the fantastic colors.  That's probably better from a psychological perspective, but not quite as entertaining.  Whoever develops a device to record dreams onto a DVD will make a fortune!!

I'm gonna stay off this for another week and see if you folks have any comments.  With Christmas fast approaching, I don't think I'll be able to perfect color in time.  So my demo will contain several videos, many monochrome and a few in color.  Is that okay?