How To: Extracting ID3 Tags from MP3 or Music files

Just the other day I was using Rhythmbox to play a few songs I had downloaded. I don’t remember what it was that struck me about the album cover image that was automatically loaded in the bottom left corner of my screen, but it made me wonder how that had happened? I mean, not all my songs had album cover images loading every now and then. Then what was special about this particular song (oh btw, the song was Amplifier by Imran Khan)? I also happened to notice that the recently downloaded music files were a little larger in size in comparison to the songs from yesteryear.

I knew about album art being put inside the same folder as the audio files (a lame image titled folder.jpg), but this was different. There was no such image present in the album directory. So I guessed that it must be part of the file header. A few Google searches later, I affirmed my hypothesis and found myself going through the entire spec for the latest version of ID3 tags at http://www.id3.org/Developer_Information

ID3 tags have been around for slightly over a decade with the first version ( a simplistic “TAG” appended towards the end of the file) released in 1997. Over the years, they have evolved along with the quality of sound they represent. The current ID3 tags (v3.2.4) are capable of storing more information about a song, than perhaps musicians and producers can even provide. What’s more interesting is that these tags allow for variable length fields. So your information is not confined (unlike the earlier versions where names and other information had to be squeezed within 30 bytes).

A review of the ID3 tags specification will tell you that all the metadata about a song is organized as “frames”. Each frame represents a unique field. For instance you have one frame (TPE1) telling you the name of the performing artist and another frame (TALB) bearing the name of the album. The album art may be found under yet another frame (titled APIC, yes.. the entire jpeg image is actually stored against this frame in the file). All frame IDs are four bytes long (TPE1, APIC, TALB, TRDC… etc).You can find a more comprehensive list of frames at http://www.id3.org/id3v2.4.0-frames

From a programmer’s perspective, what one needs to know is:

• how much part of the file must be parsed? It makes little sense to go on from the first byte to the last when the info you need is only present in the beginning of the file
• how do we know how much data to parse after each frame ID (the data is variable length right?)

Firstly, the size of the entire ID3 tag can be got by checking the size field of the ID3 tag. This info can be found in the ID3 header (first ten bytes of the file that look like this: “ID3 XXXXXXX”). Attention must be paid to the fact that the bytes that will be read in hexadecimal format, and if you’re going to read one byte at a time, you will have to clobber a few of them together to construct the size field. I had to write a small routine that would convert a hex string to an integer so that I could compute the value of the size field. So getting hold of this value will solve the first issue.

Each frame ID is followed by a size field as well. This value indicates the length of data corresponding to each frame. So we simply use that to solve the second issue.

Another point to note is that not all tags listed in the specification, will be present in the file stream. Most mp3 files will contain basic info about the performers and producers, the album and a jpeg for the album art if you’re lucky.

I created a class called TagParser, which basically creates a sliding window (4 bytes long) and rolls all the bytes in the file through this window. Whenever the window holds a valid frame id, it looks for the size field for that frame and proceeds to extract the metadata for that frame based on the value of the size field.

The extracted data is stored against the corresponding frame id in a dictionary. The parser scans the entire ID3 Tag in a file and extracts any frame that can be found. (I made an app using this module called "Appellation". Go to my project page and download the app.)

The object of this class invokes a method called ‘parser’ which accepts a path to a file as an argument and returns a dictionary containing the frame IDs as keys and a tuple of (description, metadata) as the value against each key.

Extracting audio from PWI files

My father has been using a Windows mobile phone (IMATE Jasjar) for quite some time now, and makes voice recordings using the Notes application that comes along with it.

These notes are saved as .PWI files. The good part about the notes is that they combine text and voice all in the same file. He did a lot of reading online, about how to use MS Word to manipulate .pwi files. But when nothing really worked out for him, he came to me with the problem.

So I wrote him a command line utility in python, that would help him extract all the audio from his files. In order to do this, I first opened the file in a hex editor (I use Bless Hex Editor on Ubuntu). Within a minute I realized that the format was extremely simplistic. All the audio was saved in the files as RIFF Wave (yes.. really really old). And what was even more convenient was the fact that all the audio was stored after a "RIFF" tag in the file.

Another point to note here was, that a single PWI file could store several audio snippets, each one starting out with a 'RIFF' tag.

This is what the file looked like:

So all you have to do is extract the pieces of the file starting with the 'RIFF' tags. Python, as always, would prove to be the most efficient language for a task like this. Simply read the entire file. Split the content into pieces using the 'RIFF' identifier. Ignore the first piece. Each successive piece would be an individual audio file. Take each piece and write it into an independant wave file.


########################################################################
# filename : pwi2wave.py
# utility : converts .pwi files to .wav files (handles multiple wave
# files stored in the same pwi note)
# author : Asim Mittal
#
########################################################################

import os,sys

if __name__ == '__main__':

filename = sys.argv[1]
filename.lstrip().rstrip()

if os.path.isfile(filename) is True: #check if the path provided was indeed a file
name,extension = os.path.splitext(filename)
try:
fRead = open(filename,'rb')
except:
print '\n\nError reading file!! Exiting'
sys.exit(0)

content = fRead.read() #read the entire file
lstPieces = content.split('RIFF') #split the file into pieces using the RIFF tag
fRead.close() #close the file i/o stream


#now lstpieces contains all the parts of the file separated
#by the RIFF tag. The first element of this list is not important
#so we'll take everything starting after.
#also note, when storing the wave part of the file, we will have to
#add the 'RIFF' tag again

del lstPieces[0] #not of interest to us

for index in range(0,len(lstPieces)):
newname = name + '_wave_' + str(index) + '.wav'
newContent = 'RIFF'+lstPieces[index]
fWrite = open(newname,'wb')
fWrite.write(newContent)
fWrite.close()


#the above loop creates as many wave files as there are audio segments
#in the pwi file

else:

#the filename didn't point to a file. tell the user about it
print '\n\nBad path provided. Please give the path to an actual file.'



And that is how its done!

How to: Building a front end for Python help

I'm sure most python programmers out there who have used the Python interpreter's built in help utility and regretted not being able to perform a simple "Find" operation on everything that was visible. So I decided, that it was about time, this was taken care of.

A good front end for an extremely useful help utility was lacking in Python, and so, I named my application "Venom" (something a python doesn't have, but could really do wonders with!). It is a really simple tool that consists of a single widget form that looks something like this:


Its only a text box!!! Its simple design was inspired by the Google desktop search utility. All you have to do is enter the module name (as you would have done at the 'help' utility prompt) and press the enter key.

This tool also remembers all the searches made during every run, and you can scroll through earlier queries simply by using the up/down arrow keys. Another interesting feature is that, this tool can be hidden onto the taskbar or system tray. There is an icon placed in the system tray, clicking which toggles the visibility of the tool.

When you press enter, the query is validated and if the help utility does dish something out, it is stored and displayed to you in your web browser as a simple HTML file. You can then perform simple search/find operations, which were not possible when the interpreter was executed directly from the shell.

The entire utility is built using PyQt4. The application is completely cross platform and I have tested it out on Ubuntu and Windows 7. You must have the PyQt site-packages and a Python 2.6 interpreter installed. PyQt4 is free and may be downloaded from the Riverbank Software Website.

From a programmer's perspective, this utility was a great way to understand how to create and use a system tray icon in PyQt (using the QSystemTrayIcon class). The UI was designed using Qt Designer and converted into Python code using pyuic4.

Click here to download the source!! (look for Venom)

Hope you like it... Its really made searching for help on Python a lot simpler. A funny thing I noticed was that the system tray icon was appearing in full color only when the utility was being run from the terminal. When I ran this application from anywhere else, the system tray icon didn't appear or was vague, but space was made available in the system tray for this icon. In order to prevent more confusion, I added a tooltip. Clicking on the system tray icon will toggle the visibility of this application.

Any ideas on how this may be expanded further...?? Please comment on this post or mail me.

A Small PING utility to keep the network awake

I recently set up a small home network on windows, and was using samba to access the same using Ubuntu.

I realized over a period of time that some computers on the network have a tendency to fall asleep, preventing me from being able to access shared data. A simple solution to this was a simple "ping". But the task of facing "access denied" and then running a terminal to ping the particular computer was getting too tedious. Moreover, since I have a wireless router in between, the IP addresses for all the machines keep hovering in an around my current IP address.

Today I decided that enough was enough, and I had to squash this issue once and for all. So I wrote something called "pingTEN" ('ten' being the reverse of 'net' .. heehee!)

Its a really simple utility which allows my computer to ping X number of IP addresses on either side of my current IP address. Since this operation can be a major CPU hog, you can also program Y number of seconds, for which the program will sleep once all the pings are complete. It performs a single packet quiet ping.

Since I usually have about two to three computers active on my home network at any time, I simply type this:

shell:~$python pingten.py 2 10

Now 2 (is X) and 10 (is Y). Another cute feature I've added is that everything that is put out on the console (if you're running it from the terminal), will be written down periodically into a file, with a timestamp. Why is this important?? Simply because, I can now choose to run this in the background (using the System > preferences > startup applications menu) and be able to monitor if it is still active. It runs as if it were a daemon or a background process.

If I simply open nautilus (or whatever file browser utility is available), I can actually see the file size changing from 270 bytes to 0 bytes to 270 bytes... and so on. With every set of pings, the file is overwritten with 270 bytes of data. The program then sleeps for ten seconds and rewrites the file when it wakes up, causing the size to get nullified.

Its a really simple way to keep track of my application, which I cannot see without the terminal.

Click here to download the source!

Adding your Windows partition to the GRUB

So I've been toying around with Windows 7 and decided to install Ubuntu 9.10 to go with it. Obviously Windows 7 was the first to be installed (that way, I hoped that the GRUB loader would recognise a Windows installation and automatically add it to the list of operating systems to choose from); however, to my disbelief, the GRUB tricked me and Windows was nowhere to be found in the list.

My first reaction was of frustration, but I decided to use the existing installation of Ubuntu to help me figure out how to go about getting my access to Windows back.

Here's what I did:

Firstly, when Ubuntu loads up, mount the windows partition (as if it were just another hard drive). There is a folder boot (which may/may not contain a sub-folder called grub). However, this is of no consequence to us. Then there is another folder called "Boot" (capital 'B') which contains all the boot files.Rename the former ('boot', the not so important one) to XYZ and rename the latter ("Boot", more valuable) to 'boot'. Yes all we did was change the case of 'Boot' to 'boot'.

Now open a terminal, perform a simple 'fdisk':


Shell:~$ sudo fdisk -l /dev/sda

Device Boot Start End Blocks Id System
---------------------------------------------------------------------------------------------------------------------------
/dev/sda1 * 1 4080 32771576 7 HPFS/NTFS
/dev/sda3 8585 30401 175245052+ b W95 FAT32
/dev/sda5 4081 8323 34081866 83 Linux
/dev/sda6 8324 8584 2096451 82 Linux swap / Solaris


This is what I got. Here my Windows 7 is loaded on /dev/sda1 which is NTFS. Now create a file in /etc/grub.d/ called 11_Windows

Shell:~$sudo nano /etc/grub.d/11_Windows

Nano is my text editor of choice. You can use anything you like ('gedit' etc.). In this file, write the following:

#!/bin/sh -e
echo "Adding Windows">&2
cat<
menuentry "Windows 7"{
set root=(hd0,1)
chainloader --force+1
}
EOF

Save it. Close it and return to the terminal. Now let's recap. You made the "Boot" folder of your windows partition "boot" and created a file in the /etc/grub.d/ directory.

Now simply run the utility to update the grub. You can do this by entering the following at the terminal:

Shell:~$sudo update-grub

Output (on my system):

Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.31-20-generic
Found initrd image: /boot/initrd.img-2.6.31-20-generic
Found linux image: /boot/vmlinuz-2.6.31-14-generic
Found initrd image: /boot/initrd.img-2.6.31-14-generic
Adding Windows
Found memtest86+ image: /boot/memtest86+.bin
Found Windows 7 (loader) on /dev/sda1
done

Notice how the "Adding Windows" appears. That happens to be the cute little script we just wrote tipping the odds in our favor. And voila!! Windows 7 is found on /dev/sda1 (which we verified manually a little while back)

Thats all there is to it. Hope this helps.