Parsing GIF Format (NOT using Extension Builder ) SUCCESS!!!

[OpenXTalkPaul]

So far it's taken a few hours of reading the GIF89a File format Specification and some tinkering around with a little trial and error. It took a little extra time to realize that I had a malformed file in my test GIFs collection, which was throwing me off because my code was correct but that file wasn't.

So far this:
Reads GIF Data Header ('magic number'/version)
Reads Global screen descriptor which includes:
- over all width and height
- whether there is a global color table or not
- intended color resolution
- global index for background color if there is one
Reads Global Color Table (CLUT) if there is one, which is a list of RGB values (byte triplets)
Checks next byte to for marker that indicates the next block is an image descriptor block or an extension block (which are things like frame delay time for animated GIFs)
Reads through all Extension Blocks and Image Descriptor Blocks, Image Frames Data Blocks (no LZW decode), local color tables (skips reading for now), Comments Blocks, Plain Text Blocks (Skips because GIF renderering engines never adopted it), and winds up at (or near) the GIF stream terminator. There may spots where it's a bit or byte or two off the mark.
Still testing this for accuracy using various test GIF files.

Theres lots of binaryDecode() used and some twiddling of packed bits (1s and 0s) in this script.
I'm trying to document in the comments every step, as I go through the GIF89a Spec.

That's as far as I've gotten so far.

Code: Select all

--- WORKING CODE TO LARGE TO POST HERE!

I should probably rename the function GIFHeaderInfo() since it reads more than the header now. It's named like that because it started as a function that extracted the overall width and height of a GIF directly from it's data (without making invisible image boxes and other such nonsense just to get the image dimensions, which is the common answer given when the question has been asked in the past, but I found that to be extremely lame to even need a 'solution' for something so essential to dealing with graphics).

I suppose if we tinkered with the engine a bit we could expose more of GIFLib that the engine uses to parse GIF, or it could be added again separately as an extension wrapper (but I wonder if that would cause some conflict since the same library/symbols would be loaded 2x, in two different ways?).

I could probably write this with straight Extension Builder without GIFLib, but that would be basically the same as this xTalk script version with slightly different syntax and lots of variable declarations.

--- Leap Day (2/29/24) EDIT:
I just updated the code listing, it reads more of a GIF's meta information now, not 100% yet.

---EDIT (3/15/24)
WORKING CODE! It's too large to post the script here so I've attached the working stack.
NOTE: This is alpha or beta quality code, I've tested it on a bunch of GIF files, but there's very little error checking built into the scripts.

GIFParser_Working.oxtstack

(220.01 KiB) Downloaded 12 times

[richmond62]

One of the most vital things with an animated GIF will be to know the intervals between frame displays.

[OpenXTalkPaul]

One of the most vital things with an animated GIF will be to know the intervals between frame displays.

Working on it, almost there... unfortunately I have some real world things to take care of right now.

[richmond62]

The real world does tend to obtrude.

What amazes me is what you have achieved despite the obtrusion.

[OpenXTalkPaul]

The real world does tend to obtrude.

What amazes me is what you have achieved despite the obtrusion.

Coding is a bit escapist for me, it's much better for mental health than doom-scrolling the news...
I also drink a lot of coffee and don't have good sleep habits

The fact is this friction will only be worn by persistence!
Leave out conditions, courageous convictions will drag the dream into existence.
Neil Peart - Rush

[OpenXTalkPaul]

Been learning ( relearning some of it) about GIF89a formatted GIF files.

The official GIF89a specification doesn't include application extensions, but probably the only app extension block that really matters to us is the 'NETSCAPE2.0' extension which set the commonly used standard for providing a 'loop amount' value. https://giflib.sourceforge.net/whatsina ... sion_block

Here's an interesting article about some peculiarities of the GIF89a format:
https://medium.com/@lbudorick/weird-thi ... fed7ccce03

[OpenXTalkPaul]

An even better article about parsing GIF files, that includes a GIF89a file with no Global Color Table, which like several other features is actually GIF89a spec compliant but virtually unsupported by any modern GIF renderer and generally may cause havoc (this particular GIF crashes the OXT Engine if you try to put it into an Image Control!)

Sunflower_as_gif_89a-no-GlobalColorTable.gif (54.67 KiB) Viewed 530 times

https://blog.darrien.dev/posts/you-dont-know-gif/

[OpenXTalkPaul]

The other interesting GIF89a spec things that get ignored by pretty much all modern GIF renderers are:
Plain Text extension blocks which are ignored by most if not all modern GIF render. Intended for displaying text (with fixed width bitmap-fonts). We could use it to embed custom data (such as a 'Move to x,y' relative coordinates list) into our GIFs. Of course we could always define our own Application Extension Block (as 'NETSCAPE2.0' did) or use a Comment Block and stuff whatever data we want to keep with the GIF there.
And, the per-frame 'User Input' flag bit, which could be used to tell the rendering app to pause for user input in a Power-Point style presentation or similar use-case.

Additionally I find it rather interesting that:
Every individual image frame of a GIF can have it's own topLeft origin point and height and width as long as the Xs and Ys all fall within the global screen descriptor's bounding rectangle. So you can have a small amount of pixel data per frame, moved around on a larger canvas which should be very efficient byte size-wise for doing things like moving a small sprite around inside of the area of the GIF. This can also can be used to render a larger image in chunks in an efficient way. I'm thinking about transition effects like a checkerboard fill/wipe effect here, but GIF was designed this way because of 9600 baud or 14.4k modems were the standard at the time (1987), SLOWWWWWwwwww..., kids today don't even know!!! So with this format you could render half the image in parts, that download while also rendering (and potentially allow user to cancel the download if you didn't like what you were seeing), which is what the Progressive / Interlaced save file settings in Photoshop and the like are about. This mechanism can also be used as a trick to pack more color resolution into a GIF file because every individual image frame of a GIF can also have its own Color Look Up Table (CLUT). Interestingly we could have a image frame that only updates a single empty pixel but overrides the GIF-Global Color Table with the Frames custom color table, changing the appearance of all of the pixels in the image.

Anyway I don't need to mess with any lzw compressed packed bits of the pixel data blocks, we'll be using the Engine's GIF render and pixel editing capabilities for things like that. We just want to be able to mess with the wrappers that are the settings used with these frame images.

[OpenXTalkPaul]

Getting closer to being able to read every block in a GIF.
Once I have a fully working GIF parser, I can use it to make more targeted functionality (like handlers to do things like extract the frame delay values)

Look at this extra XMP sh!t that Adobe tacked on to this GIF!
I'm somewhat familiar with XMP (Extensible Metadata Platform)... but unless you are using some content management system that uses XMP metadata (originally from Adobe and now an ISO standard https://www.iso.org/standard/75163.html ), then this is just an extra 173 bytes in the GIF that most people would not want to be in there:

XMPinAGIF.jpg (89.83 KiB) Viewed 494 times

GIF, having been designed for dial-up BBS in 1987, is designed to be as small as possible, optimized for transmission speed and smallest possible memory footprint, so tacking on lots of metadata defeats the original intended design goals of GIF.

To be fair, Adobe is far from the only Application to stuff some extra data into GIF Application Extension Blocks, I've noticed ImageMagick can stuff some junk into GIFs it creates, but it's probably a more practical use of 14 extra bytes.

Extension Block Begin = !
Extension Block is an Application Block
Extension Block Size is 11
ImageMagick
SubBlock Length = 14
SubBlock Data:
gamma=0.454545

They probably could have packed that gamma value in a more efficient way...meh.

[OpenXTalkPaul]

Unfortunately I didn't get any coding done this weekend. I had a Covid booster shot and basically had Covid symptoms for about a day and a half, which is really what you want from a vaccination ( it's an immune response), but it wiped me out.

Anyway, I've got GIF file format sorted for the most part (I'm not doing any LZW bits decoding), I just need to fix something with the image reading loop (currently it only reads to the first image data in the GIF).

The next step after that I think should be to extract the parts of the GIF, so we can easily get at the things we're interested in, like Graphic Control Extension blocks which contain a frame's delay time. I may just put all of the data chunks into an Array, that way I can store the LZW compressed image data for every frame in its own array element, along with relevant Color Table and Graphic Control blocks.

It's easy to find byte offsets for things like, extension blocks ( marker is exclamation "!") or image data start (marker is a comma ","), so you can sort of skip everything that you aren't interested in and just find offsets for the stuff you are interested in tweaking.

Here's a very good article that helps clarify some the oddities of GIF format:
http://www.matthewflickinger.com/lab/wh ... _bytes.asp

[OpenXTalkPaul]

OK so I've updated the first post, as far as the script source.
I abandoned efforts to short-cut read the Extension Blocks by using byteOffset() with various GIF marker bytes ("!"&ExtTypeByte which is an Extension Block Marker and one of several different Ext.Block Type Marker Byte).
The problem with that is the Image Data could contain that same byte pair but those are be meant to be decoded as pixels.
So we can only parse GIF the hard way it seems, by iteratively reading through every byte in the GIF file data.

Here's a Demo Stack that just dumps the GIF's info into a Field:

GIF Parse.oxtstack

(115.33 KiB) Downloaded 12 times

Here's an (already outdated) screen shot of the script reading info about GIF (that has some historical significance).

GIFParsing.jpg (195.27 KiB) Viewed 407 times

That version didn't connect the comment block's data sub-blocks together correctly (notice the words 'exten'sion' and 'l'cation ) which has since been fixed. Most of the time there won't be any comment blocks in a GIF, if any there may be one comment, it seems it's often used like an 'Creator's Name' field. This file had two, one before the image (Software That Generated the GIF file), and one after the image.

If anyone would like to test it out on a bunch of GIFs give feedback, that would be great.

Once I'm certain that I have the correct formula for reading all data chunks sequentialy, then we will have a solid basis for our very own xTalk GIF utility library.

[OpenXTalkPaul]

Now I'm in the process of changing the script to stuff each of these data chunks into one big multi-dimensional associative array (while also keeping the ability to dump the commented info into that field as text).

For example the handler will return an array that contains elements like tGIFArray["image"][1} -- which would be the first frame if it's read an animated GIF.

To get a count of all frames in a GIF (without using the buiit-in property/syntax) , you would use something like:

Code: Select all

put the number of elements in tGIFArray["image"] into tFrameCount. 

To get the local color table for image frame 1 you would use something like:

Code: Select all

if tGIFArray["image"][1}["colorable"] is not empty then
 put tGIFArray["image"][1}["colorable"] into fld "Color Table"
end if

The way Arrays read in xTalk, is somewhat less than 'Natural Language' than I'd like, but it's great for packing up structures like these, including binary data, into a variable in memory, and arrays are faster for a lot of things than similar storage containers such as custom properties.

So after that, the next step will be allowing for editing the elements of this Array structure, and then reassembly into a new edited GIF stream data that we will be able to use like:

Code: Select all

put GIFArrayToGIFfile(tGIFArray) into tNewGIF
set the text of image "MyImage" to tNewGIF -- or write to file --  write tNewGIF to file tFile

[OpenXTalkPaul]

I've modified the read handler to pack the parts of the GIF into an associative array for editing, it can still dump a labeled log of parsing process too (for debugging purposes).
Now I'm working on the inverse handler that takes one of these GIFArrays and reassembles a GIF formatted data from it.
I hope to have this working by later on tonight.

[OpenXTalkPaul]

SOOOO close I can taste it!
I have it now that it reassembles the GIF file data from the GIFArray that I'm using to store all of cut up parts of the original GIF. The reassembled GIF is a valid GIF formatted file data, it opens in Preview.app!

The only problem is that it's a completely blank (white) canvas. There is something wrong with the way that it's pulling the frame image data out of the original. There's something I'm missing. It's likely that there's an incorrect offset in the script.
I'll get it corrected eventually. Persistence!

[OpenXTalkPaul]

After two weeks, and a little sleep deprivation .... SUCCESS!!!!!!!!!!!
I was off by two bytes on slicing up each frame's LZW image data, but now it's working, YAY!
The script is now too large too post here, but I will be creating a GitHub repo for this eventually.

The first post has been edited to include a working stack demo.

Here we see frame (1) of the original animated GIF (on the left side) and the extracted image (frame 2) put into another image control (on the right):

GIFToArrayAndBackWorking.jpg (144.72 KiB) Viewed 331 times

This may look incorrect because there's parts of the image missing, but this is actually correct!
The way GIF can work (if the App that encoded it supports it) is that it only needs to store the pixels that are actually being changed from one frame to the next. In this way GIF can be highly efficient at storing animations image data.

hillbilly.gif (8.67 KiB) Viewed 320 times

NoticeSizeDiff.jpg (55.62 KiB) Viewed 331 times

Notice the size difference between these two (single image frame GIFs), the edited image is 2k smaller, because all of the comments and plaintext blocks have not been retained. Currently the extracted image retains only minimal amount of the original GIF's parameters, for now I wanted just enough to make a new valid GIF file.

SOME NOTES:
There's plenty more to do. The script currently does very little error checking and it is slower than it could be because it builds a text log of the parsing process (for debugging purposes), and it hasn't been optimized for speed in any way.
It's memory usage isn't optimal either, it currently passes several full copies of the GIF around but instead it should pass by reference (using the @ symbol).
The script does NOT actually decode any of a GIF's image data, it just stores it, still LZW compress in an element along with a 'LZWCodeBits' element that's needed to decode the LZW image data that it's stored with.
I didn't implement the LZW decoding of the image pixel data because ( it makes my brain hurt and ) we don't need it to. We already have GIFLib included in our engine that can do that. This scope of these scripts is only extracting and editing a GIF's Parameters, although it can extract the image data (and will eventually be able to add, remove or replace image frame elements) it can not decode them to plain old RGB pixel data.

[xAction]

Good job. I just looked at this wiki-article on the GIF spec and it hurt my brain. Makes one really appreciate software that just works.

[OpenXTalkPaul]

There's some good info in that Wikipedia article, but along with the official GIF89a spec, this article from the GIFLib site has been the most helpful: https://giflib.sourceforge.net/whatsina ... bytes.html

[OpenXTalkPaul]

Since this script uses plain old script and not some external or extension thing, and because GIF format hasn't changed since 1996 (when Netscape2.0 added 'repeatCount'), it should work even for retro users, maybe even as far back as MetaCard 1.x!

In fact, I'm pretty sure I could modify this script to work on OpenXION or maybe even HyperCard (would need Array & 'writeNull' XCMDs).

[richmond62]

Slightly OT, but GIMP can open an animated GIF, where each frame comes up as a layer.

[OpenXTalkPaul]

Slightly OT, but GIMP can open an animated GIF, where each frame comes up as a layer.

That's the same way animation frames appear in Photoshop too. Gimp mimics Photoshop's behavior for lots of things.

With these script functions in their current state we can already use them to separate animation frames into their own image control containers and then layer them on top of each other.