New Year Outlook

No, I’m not going to talk about the way I think 2012 will go. Nothing that exciting. Instead, I’m going to talk about how I create a new Outlook Data file every year to contain e-items for that year. I did this originally because the PST file format had a 2GB limit. This limit has since been lifted, but I still practice this annual ritual of creating a new pst file for the year and cleaning out my inbox. I went and compressed each pst file and checked the sizes. Here they are:

Outlook Archive 2000.pst – 11,921 KB
Outlook Archive 2001.pst – 532,721 KB
Outlook Archive 2002.pst – 1,218,193 KB
Outlook Archive 2003.pst – 631,921 KB
Outlook Archive 2004.pst – 799,817 KB
Outlook Archive 2005.pst – 889,345 KB
Outlook Archive 2006.pst – 1,017,065 KB
Outlook Archive 2007.pst – 1,412,129 KB
Outlook Archive 2008.pst – 1,904,657 KB
Outlook Archive 2009.pst – 3,220,297 KB
Outlook Archive 2010.pst – 5,073,353 KB
Outlook Archive 2011.pst – 3,666,449 KB

I also have a 2010 Alerts folder… mostly automated e-mails I had on for a while, weighing in at 2,694,289.

How’s that for New Year Minutiae!

iPhoto and iMovie optimization on Mac OS X

If your iPhoto and iMovie libraries on your Mac grow to anything beyond a few gigs, these programs can begin to slow down. In this post, I discuss how I resolved many of these performance issues. If your video archive is small, you won’t notice any of these problems and you probably don’t need any of these solutions. Most of what you’ll see below are ways to make iPhoto and iMovie see only the important stuff; divide and conquer.

I hope you find this post useful for helping to manage your growing iPhoto and iMovie libraries. If you have your own techniques, I’d love to hear them.

iPhoto and iMovie take a long time to load

For some reason, iPhoto and iMovie are very sensitive to how big your library is. Very sensitive.

iMovie in particular has a nasty habit of looking through all of your events and making sure they are all “optimized” when it starts up. Having a big library makes this very slow, particularly for certain kinds of videos like 3g phone videos iMovie doesn’t like to deal with. The solution here is to remove the content from the UI so it’s not scanned.

One way to accomplish this is to simply delete the event. Create a project, make a movie, export it to iTunes, and delete the event. If you delete the clips, though, you may as well delete the project that used the footage because it’s useless. If you’re like me, you like to keep around the original clips so you can make multiple movies from a single event. I wish I could purge like this, my data would be a lot easier to manage.

What I did to solve this problem without losing the original clips was to move all of my events to a “special” folder. By “special” I mean a folder not managed by iMovie. Then, I created symlinks into the standard ~/Movies/iMovie Events folder and iMovie Events folders at the root of each hard drive. These folders is where iMovie looks for events. This allows me to instantly filter what iMovie can see without moving gigs and gigs around my hard drive.

lrwxr-xr-x    1 nic  staff    58 Jun  5 18:06 Antonio Math Bee Grade 5 -> ../Movies/iMovie Events.localized/Antonio Math Bee Grade 5
drwxr-xr-x  157 nic  staff  5338 Sep 11 23:03 Broncos 2011 Week 1
drwxr-xr-x   44 nic  staff  1496 Jul 23 09:57 July 2011
lrwxr-xr-x    1 nic  staff    44 Jun 21 16:22 June 2011 -> ../Movies/iMovie Events.localized/June 2011a
lrwxr-xr-x    1 nic  staff    44 Jun 21 08:55 March 2011 -> ../Movies/iMovie Events.localized/March 2011
lrwxr-xr-x    1 nic  staff    41 Jun  5 18:06 Ohio TC -> ../Movies/iMovie Events.localized/Ohio TC
lrwxr-xr-x    1 nic  staff    45 Jun 21 08:54 Shananigans -> ../Movies/iMovie Events.localized/Shananigans
drwxr-xr-x   24 nic  staff   816 Sep 11 23:03 The Fight
lrwxr-xr-x    1 nic  staff    44 Jun 21 16:25 Track 2011 -> ../Movies/iMovie Events.localized/Track 2011

The entries with -> after them are symlinks and the path to the right is the actual location of the clip directories. iMovie doesn’t know the difference so it just works. Notice how some events are not symlinked. This is because iMovie by default writes real directories. It would be great if iMovie used symlinks, more on this soon.

To create symlinks, move to the directory where you want the symlink and type this at the Terminal:

ln -s /path/to/original/file

Time Machine backups are a bit of a problem, though. By default, the movies are imported into ~/Movies/iMovie Events and iMovie Events folders at the root of each hard drive. These obviously aren’t symlinked as I mentioned. So, when I have accumulated enough Events that I want to start hiding some, I’ll need to physically move these to my special folder. Not such a big deal, but this kind of move prompts Time Machine to backup the files in the new location again. This was a big problem since my backup drive is not much larger than my regular storage. I had to wipe the backup and backup clean to avoid out of backup space warnings.

As I mentioned earlier, this whole mess of managing movies would be best solved by a future version of iMovie and not by hacking the file system. For instance, it would be nice for iMovie to facilitate hiding events and projects from the main UI so the main UI doesn’t spend so much time accessing these older events.

Clicking around iMovie cases a lot of access, so the UI is largely unresponsive in many cases

This is a ridiculous feature, IMHO. The point of the feature is you can click on a project name in the browser and play the project without actually opening the project. The problem is that to move a project into a folder, you need to first click on it. This is an EPIC FAIL in UI as clicking causes enormous delays while the preview feature is launched.

The “iPhoto Videos” event just gets bigger and bigger and takes longer and longer to load

Most cameras can take pictures and video. iPhoto allows you to import videos taken on your camera and places them in events. This is not such a bad thing. What’s bad is that the videos all funnel into a single event in iMovie called “iPhoto Videos”. Clicking on “iPhoto Videos” causes iMovie to load previews of every video in your entire iPhoto library and as your library grows, so does your wait time.

It would be better, IMHO, to have the iPhoto Videos folder expand and show you the events where your movies lay. This would make things much faster. Since this isn’t the way it works, I have banned myself from importing videos into iPhoto. This is kind of a pain as I need to introduce a second step when grabbing content off my camera. I use the Image Capture application to grab the videos as a first step and drop them onto my hard drive. Then, I drag the videos into iMovie. Finally, I import the rest of the pictures into iPhoto.

Meh.

Moving clips from event to event takes forever, a lot longer than manually moving the underlying files

It’s possible that this problem is specific to my system. At least the crashing part. Yes, in many cases moves never finish and eventually result in a crash. Your milage may vary. Moving clips is generally a complex operation since I believe all projects referencing the clips are scanned and updated with the new location if necessary. The net effect is that moving clips from event to event is massively time consuming. Again, this is a basic problem that can probably be solved behind the scenes with symlinks or pointers to clips somehow. Moving clips on the same hard drive should be more or less instantaneous.

Still, I try to move clips often to make sure Events aren’t too large and clicking on a event doesn’t need to scan too many clips.

Projects can easily lose access to the clips they use if you go mucking around with moving events

In the original iMovie HD, clips were stored internally in the movie project file (a package) and things were very reliable. The advantage to the new UI, of course, is more flexibility. You can use event footage in more than one project. If you use the symlink technique I spoke about earlier, you may start getting these kinds of errors when referencing clips that have no symlinks from projects. All you need to do is create new symlinks as described above.

iPhoto slows to a crawl

iPhoto performance began to get intolerable when my library started to exceed 50GB or so. At my library’s peak, it was almost 90GB, over 30k photos, and 10 years of events. I am sure this is not large by professional standards, but it’s plenty too large for iPhoto to deal with.

The easiest way to bring sanity to performance is to simply create a second iPhoto library: divide and conquer. This is easy as all you need to do is to a) close iPhoto, b) rename the default library, and c) restart iPhoto. When you do this, a new library will be created automatically for you. You can get the old library back by clicking on it in the finder. Once you do that, the library you clicked on will be the default o ne. So, to get the new library back, you need to exit iPhoto and double-clic on the new library.

One caveat to this is that you lose your Faces database. But I do not use that feature.

There are some good iPhoto library managers out there but IMHO, this is the kind of thing that should be built into iPhoto. Or, better yet, Apple could spend time optimizing iPhoto and make this whole technique unnecessary. After all, Google does fine with Picasa.

Conclusion

I’ll conclude with a bunch of tips and links you may find useful:

  • Other than creating symlinks, don’t muck with event or project folders in the finder or the terminal. The names of the event and project folders is essential to keeping projects properly linked to the underlying clips.
  • You can rebuild your iPhoto library, thumbnail caches, etc., by holding down the Command and Option keys on the keyboard after launching iPhoto before the main window shows up. IMHO, this is stupid. These features should be available via preferences. Perhaps they can fuck things up for the user so Apple would rather they be hidden. Meh.

Primordial Ooze WordPress Theme is up on GitHub

Although the  theme is still not ready for prime-time, here is the link to GitHub where it’s ready for forking. https://github.com/NickCody/PrimordialOozeWpTheme. Upcoming changes will be around:

  • An article summary main index page (possibly)
  • Links to archives, rss
  • Links to the static files that make up the legacy for the original Movable Type Ooze (from 2003 to 2008)
  • Sleeker comment threads
  • More contact information in the footer
I’ll also try and add +1, tweet this, reddit this, etc. to my posts though I’m not so bold as to think anyone will ever actually click on these.

 

My first hand-coded WordPress theme

I decided to roll my own WordPress theme since I just couldn’t find any theme simple enough or which satisfied the ideals I had imagined for a simple no-nonsense blog. My friend Kevin would probably say what I was looking for was a Kotan (枯淡コード) theme inspired by the same principles he bases his code upon in his well-named blog. As for the look itself, I was inspired by the style used by Armin Ronacher’s blog, Thoughts and Writings so I more or less stole it. Most of this theft involved re-verse engineering the look and feel and then implementing php and css via WordPress standards. I had always wanted to write my own theme, but the task was always daunting to me. I thought it would all be too complex to get right. In the past few months, my CSS skills have been getting better. My ability to inspect the CSS and debug how it’s working has gotten better. So, suddenly, I just knew what to do.

The theme is far from complete, I am basically leveraging WordPress’s ability to provide default functionality when you don’t do anything. So, I implemented the most  important bits. After a few weeks when I implement links to my archives, stylize comments, etc., I’ll see if I can submit the theme to the WordPress theme gallery (with due credit fo Mr. Ronacher) and see how the whole theme submission process works.

I called  the  theme “Nick Cody” but I may change it. There are a total of 204 lines of php and css, most of the php was simple modifications to the defaults. Some php files I didn’t bother editing at all. I did very little work and the whole effort took about 4 hours from start to finish.

Pour your files from your phone onto your tablet

http://www.cultofmac.com/youll-pour-files-between-future-ios-devices-to-transfer-data-patent/103838

A new Apple patent reveals an interesting new method of exchanging information between your devices through “metaphorical gestures” that would, for example, allow you to “pour” your files into your iPad by tipping your iPhone over it.

Construder

http://j.mp/owHcBh, cool OpenGL game written in Perl/C; via @newsycombinator

More storage

I have been importing video at a dizzying rate at home. My Mac Pro has 2.5TB of local storage and one of my 1TB drives is approaching capacity. This overloaded by 1TB time machine disk I have tucked away in the living room inside my Time Capsule / Airport Base Station. This is not surprising since that time machine disk backs up all of the other macs in the house: 3 iMacs and a laptop. The time has come for another investment. I found a cheap 2TB 5400 rpm drive that simply hooks up via USB and directly connected that to my desktop and tucked it away behind my monitor. I’ll use that for the Mac Pro backup and leave the Time Capsule to back up all of the other machines. Space issues should be solved for now. If I actually fill up the second 1TB drive in my desktop, then the 2TB will need tob e replaced. However, I’m more than 70% done transferring DV/HDV footage from tape to my machine so I think the first 1TB drive will be enough for what I have now and probably what I’ll need for 2011 new footage.

One note on my HDV footage. The HDV camcorder I have shoots 1080i video, the “i” being interlaced video. Interlaced video doesn’t look so good on a computer screen and 1 hour of this video chews up 45GB of space! Having all of this raw data would be a good source to create lower-resolution videos like 720p, 540p, 480p, but the massive space requirement did not seem worth it. On my current budget, I just didn’t want to spend hundreds more dollars on 4-8TB of space to accommodate all of the footage I have and backing it up. So, I import the video as non-interlaced 540p, which is H.264 encoded and still looks great.

If I’m ever creating a video and having this mildly degraded source causes the resulting work to be sub-optimal, I can consider re-importing the original footage at full native resolution. However, seems like in most cases I’m OKwith the quality of final projects based on the 540p clips.