Filenames.WTF

Publication date: 3 February 2012
Originally published, in a much shorter version, 2011 in Atomic: Maximum Power Computing
Last modified 03-Feb-2012.

 

File extensions are ridiculous.

Here we are, with gigabytes of RAM and CPUs that do tens of billions of operations per second, and we're still using three-character suffixes to tell different kinds of file apart.

"Suffix typing", identifying file type by letters on the end of the filename, was getting old when the concept of the directory was not often seen in home computing, and Bon Scott was still alive. But here suffix typing still is.

There's been some progress. We've got to the point where a.file.name.can-look-like-this.doc.exe.xxx.txt and still work (provided it really is a text file). And you can have suffixes with more than three characters.

But you usually don't, do you? My copy of Windows knows six hundred and something suffixes, and the great majority are three or fewer characters.

Why, you might fairly say, does my amazing megacomputer not just look at files to figure out what they are? It might not be able to determine every file type that way, but common things like MP3s and JPGs and Word DOCs should be easy, right?

Well, yes, that is right. It's what applications do when you tell them to open a file. Sometimes, if you've got a file of format A that the app understands, but it's suffixed as format B that the app also understands, the app will smile at you patronisingly and open the file without complaint. And pretty much every flavour of UNIX back to time immemorial (which, in this case, means the mid-Seventies) has a command inventively titled "file", whose purpose is to figure out what a file is, regardless of the file's name.

You can have "file" on a Windows system too, if you install a UNIX-alike shell like Cygwin. Mac OS X is UNIX-based and thus has "file" built in, along with numerous other standard UNIX commands.

And, if I may be allowed yet another digression into nostalgia for a long-vanished age, version 2 of AmigaDOS introduced "datatypes", which allowed it to identify any kind of file you had a datatype for, and which instantly made new kinds of file accessible to programs that dealt with that kind of data - image files, sound files, word-processor documents, et cetera.

Suppose you want to make your Amiga paint program understand a new file format, say PNG, which was created more than two years after Commodore released the last Amiga and then went bankrupt. You just find yourself a PNG datatype, and copy it into the datatypes folder. And now your paint program can load PNGs.

Stepping back into the twenty-first century, it's now normal for personal computer OSes to include some sort of background indexing service - Spotlight on the Mac, Windows Search in, uh, Windows - which tootles along through all of your data files and indexes their entire contents. Identifying the file type along with the contents wouldn't be a big deal. If the user opens a folder that has files in it that the OS hasn't yet scanned, it could just whip through them on demand, in the same way that it creates thumbnails for images and video files.

And yet, nobody does that. Even the Macintosh has moved away from "smart" file identification; suffixes are more important in Mac OS X than they ever were before.

(Before OS X, Macs didn't need filename suffixes at all, instead using a separate "resource fork" for every file. This generally worked pretty well, as long as the files were never touched by any heathen resource-fork-less foreign systems. OS X has resigned itself to having to cope with dumb, suffixed, forkless files.)

So why is all this so?

Basically, it's because smart file typing can be very slow.

There are an awful lot of file types out there. This alphabetical list of file formats has about 3500 entries, as I write this. This other Wikipedia list prunes out a ton of old and/or extremely obscure formats that most people don't care about, and still lists more than 1250 formats.

You might think that, by modern computer standards, telling four-digit numbers of file types apart doesn't sound too daunting. But this is not one of those situations where a large problem becomes tiny when you can throw billions of arithmetical operations per second at it.

The core of the problem is that files don't all conveniently identify themselves in the first kilobyte. Even if you ignore pathological examples like raw and uncompressed video from cheap webcam software and RAM dumps from program crashes, there are still plenty of file formats that can't be neatly identified by looking at the beginning, or the end, or any other particular offset within the file.

Modern "wrapper" or "container" formats like WAV or Ogg, and simple but wide-ranging file headers like Internet media types, are easy to identify. That's the whole idea of a wrapper; it communicates what format the file is, so even if you don't have any software that can make use of the data inside, you can at least tell what it is. That's basically how the Amiga datatypes work, too, and in the early days of the Amiga before datatypes had been invented, there was a vain hope that most data would come in an IFF wrapper. The only native Amiga files that had suffixes at all were the "dot info" files that contained icon data for a matching other file.

(Later, though, Amiga users really made things fun for anyone trying to use Amiga files on some other system, by inventing naming conventions that had a three-character filename prefix!)

Many file types don't have wrappers, or any "magic number" file signature conveniently provided at some consistent point in the data. Actually, it's worse than that; as that list of file signatures shows, there are often multiple file types that have the same "signature" and, furthermore, there's no guarantee that a file that has the right bytes at the right offset to be some particular file type isn't, actually, a different file type that by pure coincidence happens to have those bytes in that location!

(See also, trying to figure out what language some text is in. It's harder than you'd think. So is filename sorting.)

To positively identify a file type by "scanning" it, therefore, the only option is to do whatever quick signature scanning you can, make a tentative identification of the file type, and then try to load the file quietly in the background with some software that uses that file type, and see if the test software swallows the file happily or pukes it back up.

This is suddenly a monstrous undertaking. Either you're trying to robo-run whole commercial applications in the background just to index files and flogging your storage so hard that SSD owners start to suspect someone's swapped their old hard drive back in, or you have to reverse-engineer those applications into a lighter-weight multi-format file tester that won't get you sued by Adobe, Symantec, Intuit, Microsoft if you yourself are not Microsoft and possibly even if you are, et cetera.

An example of how badly this can suck, even when it seems simple at first: There are a zillion programs that store their configuration data in plain-text files using the suffix INI or CFG or something. Dealing with those seems easy. ID them loosely as looking not unlike plain text, or more carefully by making sure there's no high-ASCII weirdness in there that a text editor can be expected to barf on. Now send file to text editor; if your computer can't figure out which of the hundreds of different script and settings-file and data-storage formats that're all plain text it's looking at, you probably won't mind if it just shows it to you as plain text. Job done, right?

If there are weird non-text characters in there, though, you're probably looking at a config file from one of those annoying programs that stores its settings in some opaque binary format that humans are not expected to be able to read or edit. There is very probably no "right program" to view this file, anywhere.

But perhaps it's not binary at all; perhaps it's "plain text" that's encoded in a way you didn't think of.

Now enjoy explaining all this to customers who call the help line asking why they can open this.ini but not that.ini. There won't be that many of them, of course, because most users never open anything.ini, and a large subset of those who do know that sometimes such files are text and sometimes they aren't, and have no need for fancy file-ID magic at all.

So you, the programmer that implemented this cursed thing, might actually find yourself simultaneously annoyed by the complaints about un-openable .inis and .cfgs, and simultaneously annoyed at how few complaints there are.

I've got another one. You could use an auto-file-ID system as insurance against accidental idiocy or practical jokes.

Suppose you have a JPEG file that's accidentally, or as a joke, been named something.txt. You try to open it, the OS sends it to Notepad or whatever, and you get some mangled garbage in the window, and/or an error saying that the text editor can't open it because it's not a text file.

If the text editor can then say "misidentified file problem, help!" to the OS, the OS can run "file", or something similar, on the file and figure out what it really is, and then give you the option to try to open it with a program that can actually open JPEGs. If that works, the OS could then give you the option to automatically rename the file. This requires applications to be rewritten to actually make the request to the OS, but this would be an easy enough change to make to the operating system's own built-in viewers and players and editors for common file types.

Now, suppose that you have accidentally, or someone else has as a joke, set all .JPG files to be opened by Notepad. Again, the program will fail to open the file and call for help, the OS will scan the file, discover that it is what the suffix says it is but this program can't open it, and perhaps ask you if you'd like to return that file association to the default, or even roll it back to whatever it was before the most recent change.

Again, this system is not something that'd require a lot of OS-development work, provided you don't mind most third-party software never working with it.

But also again, neither of these problems are problems that most users ever have. And lack of third-party software support would seriously reduce its usefulness. And a significant subset of the people who do have either of these problems will be able to solve the problem themselves.

But wait!

You could avoid a lot of the agony by, when you do your (relatively) quick file-ID scan, also checking the filename suffix! A file that looks like a Rich Text Format document, and has the suffix RTF, is almost certainly an RTF document, after all. So there!

(RTF is a particularly well-defined format. There's a published spec for it, but experienced programmers know that they can save a lot of time by using the more accurate definition that RTF is "whatever you get when you tell Word to save in RTF format".)

If you're looking at the darn filename suffix anyway, though, you only gain two things from checking the file content too.

First, you can detect a file that has a particular suffix but which isn't actually the format indicated by that suffix, as in the accident-or-practical-joke bit, above. But, again except for accidental renamings, jokes, and malware files called nudecelebrity.jpg.exe, this is an uncommon problem.

Second, you can detect clashing suffixes, where more than one valid file format shares the same suffix. This is even less common, unless you're a digital archivist trying to tell which failed 1980s home computer a particular ".PIC" file came from.

Since these minor features are all you gain from flogging through files looking for possibly-nonexistent, possibly-misleading identity clues in addition to looking at the suffix, you might as well call the whole thing off, and only look at the suffixes in the first place.

I'm not exaggerating that "flogging" part, either. A file-identifying system may have to hunt through the entire file to find identifying "magic number", presuming there's even one there. And files, today, can be both very large and very numerous.

What all this adds up to is no way to truly auto-ID files, beyond simple suffix typing, without making compromises that'll annoy a user who just wants to open that file he just clicked (but can't, because the OS is busy hunting through some other file for ID data), or mystify a user who just paged through 20 windows worth of icon-view files and is now going to be looking at a window full of question marks for the next 30 seconds (because the OS has been avoiding the previous problem by only examining files when the user actually opens a window and looks at the icons, except that's exactly what the user just did).

This would create confusion and frustration among users, because they'd never know whether a folder's contents would ID quickly or not. Sometimes such an on-demand scan would be really fast, because the OS would, say, strike obvious JFIF (JPG) headers on every file in there and take essentially zero time on top of the standard Windows thumbnail-scan. But sometimes it would be really slow, because the files are all a format that can't be quickly identified, so the OS has to plough through all of every file hunting magic numbers, and then never find them because the format is, say, some manufacturer's brand new improvement on their previous RAW photo format, or it's some other manufacturer's slight misinterpretation of a format you'd otherwise be able to ID.

You could attack these problems by popping up notifications that say things like "gee, this file called 150_encrypted_pr0n_dvds seems to be taking a bit of a while to identify, do you want to wait?", but the user most likely to decide that long file-typing waits mean the computer is broken is also the one most likely to click "OK" without reading such a message.

Now that we're transitioning from mechanical to solid-state storage, though, the disk thrashing that makes it difficult to identify large numbers of files in a timely fashion is going away. It may never be practicable to do the final see-if-a-program-can-actually-load-it test for everything, but scanning for magic-number IDs, at least in the background when nothing important is happening, is a low-impact feature for a SSD system with tons of transfer bandwidth and near-zero latency.

I still don't know whether anybody's going to bother, given the circle of suffixes being good enough, so that's the only typing OSes use, so only crazy people make files without a suffix, so suffixes continue to be good enough. But when a mainstream desktop OS has the "file" command just sitting there doing nothing, it wouldn't be a large project to scan files with it automatically.

We've already got fast enough processors and large enough physical RAM to make complex comparisons against large file-ID databases practicable, when the computer's idle. All we need is for SSDs to replace spinning disks and we'll be all set, at least for local fixed drives.

We may even get this done before we use up all 46,656 alphanumeric three-character suffixes.

Other columns

Learning to love depreciation

Overclockers: Get in early!

Stuff I Hate

Why Macs annoy me

USB: It's worth what you pay

"Great product! Doesn't work!"

The virus I want to see

Lies, damned lies and marketing

Unconventional wisdom

How not to e-mail me

Dan's Quick Guide to Memory Effect, You Idiots

Your computer is not alive

What's the point of robot pets?

Learning from spam

Why it doesn't matter whether censorware works

The price of power

The CPU Cooler Snap Judgement Guide

Avoiding electrocution

Video memory mysteries

New ways to be wrong

Clearing the VR hurdles

Not So Super

Do you have a license for that Athlon?

Cool bananas

Getting rid of the disks

LCDs, CRTs, and geese

Filling up the laptop

IMAX computing

Digital couch potatoes, arise!

Invisible miracles

Those darn wires

Wossit cost, then?

PFC decoded

Cheap high-res TV: Forget it.

V-Pr0n

Dan Squints At The Future, Again

The programmable matter revolution

Sounding better

Reality Plus™!

I want my Tidy-Bot!

Less go, more show

In search of stupidity

It's SnitchCam time!

Power struggle

Speakers versus headphones

Getting paid to play

Hurdles on the upgrade path

Hatin' on lithium ion

Wanted: Cheap giant bit barrel

The screen you'll be using tomorrow

Cool gadget. Ten bucks.

Open Sesame!

Absolutely accurate predictions

The truth about everything

Burr walnut computing

Nothing new behind the lens

Do it yourself. Almost.

The quest for physicality

Tool time

Pretty PCs - the quest continues

The USB drive time bomb

Closer to quietness

Stuff You Should Want

The modular car

Dumb smart houses

Enough already with the megapixels

Inching toward the NAS of our dreams

Older than dirt

The Synthetics are coming

Pr0nBack!

Game Over is nigh

The Embarrassingly Easy Case Mod

Dumb then, smart now

Fuel cells - are we there yet?

A PC full of magnets

Knowledge is weakness

One Laptop Per Me

The Land of Wind, Ghosts and Minimised Windows

Things that change, things that don't

Water power

Great interface disasters

Doughnut-shaped universes

Grease and hard drive change

Save me!

Impossible antenna, only $50!

I'm ready for my upgrade

The Great Apathetic Revolution

Protect the Wi-Fi wilderness!

Wi-Fi pirate radio

The benign botnet

Meet the new DRM, same as the old DRM

Your laptop is lying to you

Welcome to super-surveillance

Lemon-fresh power supplies

A>B>C>A!

Internet washing machines, and magic rip-off boxes

GPGPU and the Law of New Features

Are you going to believe me, or your lying eyes?

We're all prisoners of game theory

I think I'm turning cyborg-ese, I really think so

Half an ounce of electrons

Next stop, clay tablets

A bold new computer metaphor

Won't someone PLEASE think of the hard drives?!

Alternate history

From aerial torpedoes to RoboCars

How fast is a hard drive? How long is a piece of string?

"In tonight's episode of Fallout 4..."

How hot is too hot?

Nerd Skill Number One

What'll be free next?

Out: Hot rods. In: Robots.

500 gig per second, if we don't get a flat

No spaceship? No sale.

The shifting goalposts of AI

Steal This Education

Next stop: Hardware piracy

A hundred years of EULAs

The triumph of niceness

The daily grind

Speed kings

Alt-tCRASH

Game crazy

Five trillion bits flying in loose formation

Cannibalise the corpses!

One-note NPCs

Big Brother is watching you play

Have you wasted enough time today?

The newt hits! You die...

Stuck in the foothills

A modest censorship proposal

In Praise of the Fisheye

Filenames.WTF

The death of the manual

Of magic lanterns, and MMORPGs

When you have eliminated the impossible...

Welcome to dream-land

Welcome to my museum

Stomp, don't sprint!

Grinding myself down

Pathfinding to everywhere

A deadly mouse trap

If it looks random, it probably isn't

Identical voices and phantom swords

Boing!

Socialised entertainment

Warfare. Aliens. Car crashes. ENTERTAINMENT!

On the h4xx0ring of p4sswordZ

Seeing past the normal

Science versus SoftRAM

Righteous bits

Random... ish... numbers

I get letters

Money for nothing



Give Dan some money!
(and no-one gets hurt)