How to parse D64 files, part 2: file chains

Last time, we saw how to crack open a D64 file with Perl and get at the first directory block. You may recall, two of the bytes in a directory entry point to the first block of the file proper, by way of a Track number and Sector number.

Now let's see how to read files in general.

A file in a Commodore disk image is stored in blocks. Those blocks may be scattered across the disk; in fact, due to mechanical considerations, files were almost never written in a contiguous set of blocks. Instead, a block was written, then the next block was written a few sectors away, then nthe next a few more sectors away, and so on.

When a block is read, the next block's location is found in the first two bytes. The remaining 254 bytes are file data proper.

So then, suppose a directory entry indicates that a file begins at track $t, sector $s. Using Perl, the file could be reconstructed in a manner similar to this:

my $fileData = readFile( $startTrack, $startSector );

readFile( $buffer, $t, $s )
{
return $buffer unless $t;
my $byteOffset = 256 * ($sectorOffset[ $t ] + $s);
($t, $s) = unpack "CC", substring( $diskImage, $byteOffset, 2 );
$buffer .= unpack "C*", substring( $diskImage, $byteOffset + 2, 254 );
return readFile( $buffer, $t, $s );
}


@sectorOffset will require some more explanation.

How to parse D64 files, part 1

This is a short how-to for parsing commodore 1541 diskette images, colloquially called D64 files after their 3-character suffix.

These instructions assume you know how to program in at least one C-derived programming language. I will be using Perl, but it will be very C-like.

STEP ONE: install Perl. A good and free distribution can be had over at http://activestate.org.

STEP TWO: fetch some D64 files. One likely place is http://lemon64.com.

Now it's time to write some code. D64 files are simply data laid out the way a 1541 would see it as it reads it straight from the disk, from the beginning to the end. So the code will have to navigate the structure of the 1541 diskette format. First though, let's slurp the entire image into a buffer..

my $filename = 'whatever.d64';
my $filesize = -s $filename;

open IN, $filename;
binmode IN;
my $buffer;
read( IN, $buffer, $filesize );
close IN;

Okay, we've got the disk in a buffer. Now what? Now we wrest the structure from this pile of bytes. That structure begins with the DIRECTORY. The directory offset is 18 * 21 + 1 blocks in, and a block is 256 bytes. So let's put that in a variable for later.

my $directorySector = (18-1) * 21 + 1;
my $offset = 256 * $directorySector;

Now we need to parse the actual data from the directory. The directory block consists of eight entries of 32 bytes each. Each byte has a meaning, some point to locations on the disk, some are part of a filename, some are status bytes for the file, etc. I will iterate over the directory block, 32 bytes at a time, and at each iteration unpack some of the current 32 byte structure into its component values:

for ( my $j=0; $j<256; $j+=32 )
{
   my ($dirtrack,    # unsigned Char
       $dirsector,   # unsigned Char
       $type,        # unsigned Char
       $track,       # unsigned Char
       $sector,      # unsigned Char
       $filename)    # 16-character ascii string
   = unpack 'CCCCCa16', substring( $buffer, $offset + $j );

   print "$filename [$type] is located at $track/$sector\n";
}

One very important pair of data in the above structure are $track and $sector. They tell us where to find the first block of that file.

This is a good stopping point. We've taken a D64 file, read it in, and printed out the contents of the first directory sector -- I.e. The first 256-byte block of the directory. Next time, we'll see how to read in an entire file.

Main Points from Adobe

So I attended a day-long presentation on Acrobat and Creative Suite 5.5 recently. Here are the primordial take-aways I got out of it:

* mobile is "big" for Adobe (now). Really!
* the typical information worker spends 17 hours a week *creating* content.
* rich media will be 25% of content by 2013.

* HTML5 is not a document format.

^ That's something which occurred to me during the Acrobat-portion of the presentation.

* process improvements and reducing costs are the top priorities in companies.
* process improvement leads to productivity gain.
* reduced costs are achieved via standards, best practices.

* that's what drove Adobe's work on Acrobat X. Editing content, table data, headers, OCR, sharepoint integration, highliter and sticky-notes, PPT to PDF, embedded QT and MPG, commenting tools, forms, digital signatures, document comparison, a macro system, legal and governmental electronic document standards support, et al.

That's it in a nutshell.

Oh yes: Flex/ActionScript content is maybe 10% of Adobe's business.

Modern Perl

I've been talking with an acquaintance the other day about Modern Perl -- i.e. the current way of coding Perl.

Perl's very flexibility is evident, since code written in 1994's Perl will still run on 2011's Perl, and yet you can do things today that you couldn't do back then.

This is due to at least two things:

(1) there's an active core development of Perl, which extends the language in useful but backwards-compatible ways (this is possible partly due to the power, flexibility, and dynamism of Perl itself),

and

(2) there's active module development by the community to meet the programming needs of the community. CPAN continues to astound me. There is nothing with the scope and depth of CPAN anywhere for any other programming language.


Languages like Scheme and Python will insist on a core orthogonal operating set. And with some languages (like SmallTalk), you can go a long way on that. But let's face it, you have to get work done sooner or later. Even Common LISP understood that. Perl's amazing library of modules lets you do anything you like... even emulate and embed other languages.


Say you've got a relatively clean, small (and therefore embeddable), C-like language which includes hashtables and some improved control structures, used for scripting small actions. As a loose superset of these sorts of languages, Perl is ideal for aggregating and linking libraries, finding and factoring out common code fragments, and even generating the code and acting as a test harness. In fact, Perl modules could exist today for performing most of these functions -- a Git module could double as a library front end; a text tool could find redundant code; and a template engine could generate code from a JSON structure.

You can't do that with Lua itself. But you could do it all with Perl. And since Perl's syntax descends from C, it's easy to pick up Perl as another tool in your belt.

HandyWrite - the Ultimate Shorthand System

It's time for an app that understands shorthand. It's time to bring shorthand back.

Check this out:

http://www.alysion.org/handy/handywrite.htm

This guy designed what looks like a near-optimal shorthand system:

1. He started with Gregg shorthand. Note that just using Gregg letters alone you'll speed up your note-taking significantly. The downside is that the notes themselves are often abbreviations, so you have to transcribe soon after you've taken the notes or else it becomes a bit opaque.

2. He extended it to represent the full range of English pronunciation. Now you have 1:1 sound-correspondence to English. You can read back what you wrote years later. i.e. It's a full writing system.

3. He then added a symbology for just 100 of the most common English words. Not at all as extensive as classical shorthand, but all you need is to be able to record notes as fast as they're spoken.


Ok, the best thing about it in my opinion is that it honors and re-uses prior art in simply extending Gregg shorthand. That alone is worth something in my book -- and not just for sentimental reasons: Gregg is a thoughtful and elegant system.

I've jotted down the letter forms, and will be practicing them little by little as I have time. My goal is to be able to record meetings with it.

Actually, my real goal is to write an app that will let me take notes with it.

Perl Programming

I'm a Perl fanatic. Ever since I learned of it waaay back in 1995, I've been able to do amazing things with it, primarily in my job function as a programmer.

As a programmer, I face two daily tasks. First is writing code to accomplish something for the company, i.e. business logic. Web pages for example. The other task is managing piles and piles of data that a company has to manage.

1. Programming requires writing code. Believe it or not, at every job I have used Perl at one time or another to write code for me. Perl is wonderful for code generation. The code I've generated the most of using Perl is: Java.

2. Data is strewn in multiple formats in multiple places. And Perl is supreme at data mining. Whether I'm scraping web pages or tossing binary digits, Perl is awesome.

So if you have to handle data, I suggest Perl is the best fit for the job. It's not that you can't do it with other languages (I've used, and still use, Java, Python, C, AWK, SmallTalk, JavaScript, ActionScript, Ruby, Flex, and LISP); it's just that Perl is handier.

[Interspel] "Instant English"

As none of you-all know, I am a proponent of Interspel, a proposal by Valerie Yule to standardize the subset of English spelling which even literate native speakers have perpetual problems with. I'll link to Interspel's Wikipedia entry below.

Anyway, I recently heard a comment about how Instant Message English -- I'll call it Instant English -- tends to be far too "cooked down" to be much use beyond vapid grunts. The example given was:

u r, u no (i.e. you are, you know)

It's quite terse. Too terse, too ambiguous beyond very simple replies. However, the principles are sound: remove useless letters. And if the vocabulary grows, perhaps it will grow alongside 20th-Century English Spelling, or even replace some of it.

Hopefully for the better.

Interspel entry in Wikipedia