Over the last few weeks I have been spending some of my free time working on an application for Don McAllister of ScreenCastsOnline called the SCO DVD Downloader. This article explains the evolution of the application and why either Don would have incurred higher costs or I would have lost money had it been a commercial project. It will be followed up by an article about polishing the application (and what, unfortunately, was left unpolished) and a few articles that will cover the more technical coding subjects touched upon here.
All the way back in December 2008 I wrote a small AppleScript for Don to help him process data that was being emailed to him. Just over a year later, towards the end of February 2010, he emailed me asking if I could write him another AppleScript. This time he needed a way to help some of his subscribers download 8GB of video files; two 4GB disk image files that mirror the contents of the two DVDs he mails to members. He even devised a basic workflow for me:
- Run script
- Enter a destination folder
- Press OK
- Check to see if the folder exists and there is 8GB space
- Ask for Disk 1 or Disk 2 or Both
- Warn that there is a lot of data to download
- Check to see if any of the files already exist and if so start downloading from the next one in the predefined list
- If OK, start downloading from a list of URLs contained in the script (there will be a list of about 70 RAR* files making up the two DVD images)
- Once completed, that would be it
- If it would be possible to programatically UnRAR the downloaded files, that would be fantastic!
* RAR is an archive file format that supports data compression, error recovery and file spanning.
In addition to the above, Don also mentioned that the RAR archive would be password protected.
Four things immediately occurred to me:
- This would probably be easier to develop as a proper application rather than just being an AppleScript.
- This would never be a financially viable project. The software would never see a financial return and asking Don to pay out hundreds of pounds for it would be madness.
- Since I am on the verge of becoming an indie developer this would be a great portfolio application for a high profile client.
- Since I was not yet self-employed I could spare a few evenings and weekends working on it for free.
For some reason Don seemed delighted with the idea of me developing the application for free and so began a process where I learn some new techniques, learnt how I could improve my own project management skills and created an application that would certainly never have been financially viable to develop commercially.
Can I Even Do It?
The downloader had two major areas of functionality that I’d not done before. The first was downloading files and the second was rebuilding and extracting the contents of RAR files. Whenever I’m playing around with something new I tend to experiment in small test projects and these two areas were no different.
Playing around with UnRAR was pretty straightforward and once I’d got to grips with the Terminal commands it was just a case of wrapping them in an NSTask which allows a Cocoa application to perform Terminal commands.
The download tests were slightly more complex but not a major problem although I did have some concerns about things like resuming partial downloads. Fortunately, before I spent too much time on it all I discovered ASIHTTPRequest, which is essentially a wrapper to make downloading and uploading files simpler. After a few seconds thought I decided that using someone else’s more capable and better tested code was much a better option.
Safe in the knowledge that everything that Don wanted the application to do was possible I started planning out how the application would look and behave.
Designing The Application
Working from Don’s workflow it was obvious that the application would need to perform six core tasks:
- Import a list of files that are to be downloaded (Don and I agreed that the list of files that needed to be imported would be attached to the email he sends out notifying users about the application).
- Allow the user to select where the downloaded files are going to be saved.
- Allow the user to select which set of files are to be downloaded.
- Display the amount of free disk space.
- Handle the actual downloading of the files, skipping any that had previously been downloaded and resuming any partially downloaded ones.
- Allow the user to enter the password and re-create the disk images from the individual RAR files.
The main purpose of the application is to download files so it makes sense that the primary UI element would be some sort of table showing the list of files being downloaded and the status of the downloads, probably as a percentage of the file stored locally.
The other two main areas of functionality are importing the list of files and recreating the disk images from the individual RAR files. Both involve more than just starting a process: importing the list of files also ties in with choosing a downloads folder and displaying the available disk space; re-creating the disk images involve choosing a folder to create them in and entering a password. Therefore I decided that using sheets may be a good choice since they are modal which contains the user’s interactions and they focus the user’s attention since they would slide in over the main window.
The First Build - Too Much Optimism
With the basic design planned out and the core functionality pre-tested in small, stand-alone projects it was a relatively simple process to build an initial version of the application. To preserve my bandwidth Don had created a password protected set of RAR files that compacted and split up a small test disk image. My testing went well and the application happily downloaded the test files and created the disk image perfectly. I sent the application over to Don for him to have a look at. This was also when we started to hammer out some of the minor functional details, something that was much easier to do now that Don had a copy of the application to try out himself.
Don’s initial testing went fine and, as was the case for me, everything worked well. He had some feedback about the UI, primarily asking for dialogues to prompt the user about the next step that needed to be performed. For example, once the list of files had been imported he wanted a dialogue asking the user if they wanted to start downloading them; once the downloads were complete a dialogue asking if they wanted to create the disk image from them. This actually coincided with a group of Mac developers discussing this idea, the general consensus being that applications shouldn’t show prompt messages like this. However I conceded to Don’s request with the caveat that each dialogue would contain a ‘Don’t show me this again’ option.
The general conclusion was that everything was working well and after these changes had been made we would be able to re-test the application and get it out to the subscribers. The development process had been slightly longer than I’d anticipated but the application was looking good and was certainly more polished and usable than an AppleScript solution would have been.
Build 2 - Things Go South
After making these and some other changes I sent Don a new build of the application. After testing the application against our smaller set of test file Don tested it against a full set of DVD files. The files downloaded and were UnRARed without any problems but when he then tried to mount the disk image it failed, reporting that the disk image was an unsupported format. Assuming that there was some form of corruption going on Don decided that the best option was to move away from RAR files and use PAR since these can be validated and repaired after they have been downloaded. To simplify the process Don wanted to remove the disk image creation aspect of the application and suggested that his members should use MacPAR deLuxe to verify them and Split&Concat to merge the files and create the disk images.
Before committing to it however we exchanged a number of emails where we tried to establish whether something in my download code was causing the files to corrupt or whether the actual RAR files were corrupt. Because we both had to re-download the entire 4GB set of files again, Don using FTP and me using Safari, this process ran over overnight and into the next day.
Neither of us had any success with the downloaded file and additionally Don was getting an CRC error with his set of files when he was trying to verify them using UnRAR via the Terminal (in hindsight this may simply have been caused by Don not entering the password in the command-line version of UnRAR).
Because Don had tested the disk images before splitting them with RAR and they had been fine we assumed that RAR was the cause of the problem and we decided to go down the PAR route, using Split&Concat to split the disk images and then using PAR to create repairable archives of them.
This was our mistake and we should have done some more investigation into the disk image files at this point rather than assume that the RAR process had gone wrong somewhere. In our case this mistake was not critical and lead to us both learning some new things but in a commercial project this may well have been the start of the whole thing unravelling.
I had done some playing around with MacPAR deLuxe and Split&Concat and arrived at the conclusion that the functionality they were offering was not terribly hard to replicate and could be performed within the application itself. The added incentive was that Don’s subscribers would have to use only one application to download the files, verify them amd merge them back into a disk image rather than three separate ones. It would involve a little extra work but I felt that it was worth doing to make things simpler for the subscribers.
Build 3 - The First PAR Build
I sent Don a new build of the application, albeit with a few rough edges, to try the new PAR file processing. Using it to process a test set of files had gone well so I at least knew that the process worked. However, having been burnt by the RAR files I decided to do a full DVD download once I’d sent it. A few hours later my worst fears were confirmed, the disk image still wouldn’t mount.
It appears that disk images are fragile beasts and for some reason the disk images that Don had created, one for each DVD, were somehow corrupt. It hadn’t been the RAR process causing problems at all, the original files were the cause. I suppose that we should have been able to determine this once we’d established that the RAR files were broken but we didn’t and we now faced the task of finding a solution, the simplest being to zip up the files, split the zip file using Split&Concat and then PAR the results.
Involving Other Testers
After modifying the application to handle zip files which was in itself a fairly significant process, if only because of the user interface changes needed, and performing various other fixes and changes we were finally ready to send the application out to a handful of external testers. We had checked that the full-sized as well as our test zip-based files were handled properly and all seemed well.
The feedback from the testers was interesting, both because they found some issues and because they came up with an idea that had completely escaped me and Don but would simplify things from the subscribers’ point-of-view no end. The idea was simply that the list of files that needed to be downloaded should not be distributed by Don and imported into the application but rather that it should be stored online and the application could read it from a web server. This not only meant that the subscribers didn’t have to deal with files and folders but it also would allow Don to add additional file sets quickly and easily.
The Final Steps
The application was altered to look for the list of files on a web server and the user interface altered to remove the file selection control and replace them with fields to allow subscribers to enter their ScreenCastsOnline username and password (the file list is stored in the protected area of the ScreenCastsOnline web server) Further test builds were sent out, bugs fixed and the process repeated until we were happy with the result.
On 15 March Don sent the application out to the relevant subscribers and after three weeks of work the project was completed.
In addition to using some new areas of the Cocoa frameworks I learnt a few new things.
Had this been a chargeable project I think that a lot of the responsibility for the additional work involved would have been laid at the client’s feet. The corrupt disk image files weren’t the easiest things for either of us to diagnose but they were the cause of two fairly significant sets of changes to the project and a significant amount of development time. Had zip files been used initially then the project would probably have only taken a week or two rather than three. The element of liability would have worked in my favour in that my hours would probably have been covered but I imagine that the client would be less than pleased to have faced a bill that would probably have been double that of the original estimate. I’m not sure if there is ever a simple solution to this sort of situation other than to discuss the potential of it arising in advance and to make it clear that if the client’s actions cause additional work then they will be billed for it.
The idea of hosting the list of files on a web server rather than distributing a file for people to import seems obvious in hindsight but was not something that either Don or I thought of. This, much more than anything else, brought home the importance of planning and designing the application before you begin coding to me. The Mac developer community includes a certain Matt Gemmell who has an excellent segment on the MDN Show and in the most recent one, episode 21, he was discussing exactly this. As a developer it is incredibly important to consider all of the usability aspects of the application and whilst I now feel a bit like an errant pupil waiting to be admonished by an, albeit swearing, Scottish headmaster. Even if I didn’t have Matt’s words ringing in my ears I’d certainly be doing more to improve in this area.
There were a couple of occasions where some additional research would have helped. For example, the switch from RAR to PAR files and Split&Concat was probably completely unnecessary and had I know more about RAR’s abilities to self-heal which was the primary purpose of moving to PAR files then we could have saved an awful lot of time and code changes. There would still have been the necessary switch from disk image files to zip files but I think we could have saved around a week. The next time I am dealing with something I only have a little knowledge of I’ll be sure to learn a little more about it than I probably need just to be safe.
Working on a commercially unfeasible project for the sake of a portfolio application may seem like madness but it also has added benefits. In my previous job my Mac applications were used by a limited number of people in a very tightly defined way. I had never released a Mac application publicly and, although this isn’t quite the same as getting an application out for sale, it is a valid alternative. I now have something that I can point potential clients to as an example of my work and I will have a series of blog posts explaining how it was developed, something I couldn’t do with a paying client. In a strange way it has also given me added confidence in my capabilities as a developer. Developing for a wider audience and getting feedback from it that your work is good is a huge confidence booster.
Working with Don has been a lot of fun. There were definitely times when we were both stressed by what was going on and by effectively insisting that I created a full-blown application rather than a simple AppleScript I know that I delayed the release of the downloadable content so I do appreciate his patience. I hope that the end result has been worth it for him and that he feels that he got as much out of it all as I did. Thank you Don.
Earlier today Don discovered the cause of the corrupt disk image files. His email is quoted below but the upshot is that disk images are not an inherently bad format to use and RAR is certainly not a bad tool to use.
Anyway, in Don’s own words:
It was the strangest thing. The source files were held on a mirrored set of drives in my disk array. I tend to keep all valuable data on mirrored drives as well as a second copy on my Drobo.
I’d been testing some beta software which caused a few problems with some of my mirrored drives. I can’t remember if this particular mirror was affected but either way, I decided to split the mirror into two independent drives, both identical to each other. Or so I thought!
It transpires that one of the drives had somehow become corrupted - no errors and could be accessed perfectly well, I could see all the files and it looked just like a normal drive.
I’d tested the disk images before I split the drive so was confident they were OK.
As luck would have it, I split the DMGs from the corrupted drive and uploaded them. If I’d had used the other drive, the initial problems we had would not have occurred and saved us both (well you mainly) all that extra work.