Since the process to run and debug code in PyCharm is similar, we will look how to run code first (jump to the debugging section if you already know how to run code in PyCharm).

In this article shortcuts are written with the first letter of keyboard modifiers followed by the actual keys in lower case. The keys are presented for both OS X and Windows/Linux in the format (mac, win). For example, the shortcut (⌘-S-⌥-t, S-A-C-t) means to press the keys ⌘, Shift, Option, and t at the same time on the Mac, and Shift, Alt, Control, and t on Windows and Linux. When only one shortcut is presented, it will be the same on all operating systems.

Running Python Code

If we haven’t run any code in our project before, the icons to run and debug will be disabled:

Some items will be disabled in the Run menu as well:

To run a script for the first time, we need to either create a configuration (Run→Edit Configurations…) or ask PyCharm to run the current file by going to Run→Run… (⌥-C-r, A-C-r), where PyCharm will automatically offer to run the current file. In the following example I’m editing a file named analysis.py. Notice how PyCharm creates a new configuration named analysis:

PyCharm will run the file and show a tool window with the results, if any.

The Run tool window has many buttons to help us deal with the running script. We can do things like pause, stop, and rerun the script and move up and down the stack trace.

The following table summarizes the main actions. The manual has a complete description.

We can hide the Run tool window with (⌘-4, A-4).

Once a script is run for the first time, it becomes available in the Run menu. PyCharm will show the most recently run command in Run→Run '<script name>' (C-r, S-F10), where <script name> is the name of the Python file without the extension (analysis, in our previous example).

This command will run the original file even if we are working on a different one. In our example, if we are editing a different file (say chorales.py) and type C-r on the Mac or S-F10 on Windows or Linux, PyCharm will run the file analysis.py, since this is the last file we ran. This is useful when we have a master script and are making modifications to a sub-module, for instance.

After we run the script for the first time, the related commands in the menubar will be available, and we can choose the script to run there:

PyCharm will create a new tab every time we run a script. If we don’t close the previous tab we may end up with many different tabs, so make sure you close each tab after running, and are, in fact, on the right tab.

New Configuration

When we want to run things with more control we need to create a configuration by going to Run→Edit Configurations… and clicking on the + button (⌘-n, A-Insert). PyCharm will ask what type of configuration we want (that is, if we want to run a Python script, a Django or Pyramid server, and so on) and will show the appropriate options for each type.

We can choose the script to run, passing parameters if necessary, the Python interpreter, and the working directory. We can also launch an external tool before running the script, which is great if we need to do things like data pre-processing.

In the Logs tab we can add log files to be shown in PyCharm’s console during the script execution or save the console output to a file.

Naturally, we can have multiple configurations; we can switch between them by going to Run→Run… (⌥-C-r, A-C-r) and picking the one we want.

The Python Debugger

To debug code in PyCharm, we need to create a new configuration in Run→Edit Configurations… or debug an existing script by going to Run→Debug… (⌥-C-d, S-A-F9). After that it will be available in Run→Debug '<script name>' (C-d, S-F9), where <script name> is the name of the Python file without the extension (analysis, in our example). As you can see, it works just like running code.

Similarly to the Run command, the menubar will now be enabled and we can choose the script to debug by clicking on the insect icon.

One of the main differences between running and debugging a script is that in debugging we want to define breakpoints. In PyCharm, a breakpoint must be on a line with working code; it can’t be on a line that is blank or starts with a comment. We create breakpoints by clicking on the gutter on the left.

We can also create a temporary breakpoint by Alt-clicking on the gutter. As the name suggests, a temporary breakpoint will be removed after being hit. Notice that the temporary breakpoint icon is different from the regular one. Check the manual for more information about breakpoint icons.

PyCharm’s debug tool window has everything we expect in a graphical debugger. From here we can navigate the call stack and inspect the data available in each stack frame. We can keep an eye on some variables by adding them to the watch list.

The debug tool window has many of the buttons we have seen in the Run tool window, allowing us to pause, stop, and rerun the script. In addition, we can view the breakpoints—also available in Run→View Breakpoints… (⌘-S-F8, S-C-F8)—and mute (that is, disable) all breakpoints.

The icons in the top right corner, allowing us to step in and out of the program, are the ones we will use the most:

In the following image, we can see each icon with its name. The table below shows the name, shortcut, and description for each icon.

The variables pane is a great way to explore complex data. It was invaluable when I was learning Django and getting acquainted with the HttpRequest object, for instance. In the following example, we can see some music21 data, especially the complex Score class. The Variables pane helps me to inspect attributes such as duration and elements.

We can see all breakpoints in Run→View Breakpoints… (⌘-S-F8, S-C-F8). From here we can add, remove, or edit breakpoints; and set conditional breakpoints and log messages or expressions.

One way to quickly edit a breakpoint is to right-click on the breakpoint on the gutter. A popup will allow us to enable the breakpoint and add a condition.

We can also see the breakpoints by selecting the Favorites Tool (⌘-2, A-2).

A quick way to disable a breakpoint is by clicking on it with the Alt key. The breakpoint icon will change.

Another way to disable a breakpoint is by right-clicking on it and unchecking the first option (“Line 93 in analysis.py enabled” in the following example).

Besides defining breakpoints when a line of code is reached, we can define a breakpoint that is triggered when an exception is thrown. To add an exception breakpoint, we go to Run→View Breakpoints… (⌘-S-F8, S-C-F8) and select Python Exception Breakpoint after clicking the + button.

We need to include the exception class by searching for its name. The search field works with both built-in and custom exception classes.

Our new exception now appears on the list, where we can customize it.

In most cases, we want to activate the breakpoint when the exception is raised (the default is to activate the breakpoint when the exception terminates):

Having configured an exception breakpoint, PyCharm will create the breakpoint automatically when the exception is raised (ChoralException, in our example). Notice that the exception breakpoint icon is different from the regular one.

Python Prompt

Sometimes it’s useful to be able to access a Python prompt while debugging. We can do this by clicking on the Console tab when the debugger is running and selecting the Show Python Prompt icon.

Now we can use the prompt to inspect and change the variables in the current scope.

Evaluating Expressions

Although it’s super-useful to have a full Python prompt available while debugging, the Evaluate Expression feature may be sufficient if we just want to quickly evaluate an expression.

As expected, we can use any variable in the current scope:

If we select the Code Fragment Mode button, we can add multiple lines of code. Completion works here as usual.

Inspecting and Watching Variables

The fastest way to inspect the value of variables while debugging is by hovering the cursor over a variable name. PyCharm will show the variable name and its value.

And, as expected, we can inspect and set a variable value by choosing the correct option in the popup menu on the Variables pane.

The same apply to watched variables; we can edit and inspect them by using the popup menu.

Remote Debugging

PyCharm has support to debug code remotely. Probably, the most popular and useful way is to debug code in a Vagrant box.

Conclusion

PyCharm has a debugger that is second to none. If you want to learn more about PyCharm and how to use it to debug JavaScript and Django templates, check my ebook, Modern Python Development with PyCharm.

Powerful editors like Emacs and Vim are fantastic. I find it puzzling that some editors or IDEs perform basic operations such as search and replace clunkily while Vim and Emacs get it just right.

And yet a search for terms like “Vim as a Python IDE” or “Emacs as a Python IDE” returns hundreds of thousands of links, which shows that people want features such as smart completion, code navigation, and project support. To give you one data point, one of my most popular posts is “how to configure Emacs as a Python IDE”, even if it’s six years old and out-of-date.

PyCharm is one of the most popular IDEs for Python and it’s packed with great features. In this post I will show how to get started with PyCharm quickly to have a productive Python development environment. I’m using PyCharm 3.0.2 Professional Edition on the Mac (they have an open-source version as well). I use a heavily modified default keymap (it also has Emacs and Vim keymaps). I suggest you check the official Mac and the Windows/Linux reference cards for the default shortcuts.

Initial Customization

When you start PyCharm for the first time it asks what keymap and theme you want to use. I don’t like its Emacs keymap, so I use the default Mac OS X keymap and customize the shortcuts I want:

If this is the first time you are using PyCharm and don’t have a configuration to import you may want to click on “Configure” in the welcome screen to set some basic things:

I like to show the line numbers and method separators (Editor→Appearance→Show method separators, Show line numbers):

Also, I like the Solarized theme. I install it by following the instructions and select it as the default theme in Editor→Colors & Fonts. Solarized light for IntelliJ has some weird colors choices (for instance, it uses gray for the default text instead of a crisper black), so I change it on Editor→Colors & Fonts→General. My advice is not to be afraid to change the colors to something you like. You can export your settings in case you need to re-install PyCharm.

Compared to some IDEs, I find PyCharm visually clean, and if you want an even cleaner UI, you can hide the toolbar, the tool buttons, the navigation bar, and the status bar (on the View menu):

The Python Interpreter

You can create new projects with File→New Project or open an existing project with File→Open Directory. PyCharm supports many types of Python projects such as Django or Flask out-of-the-box. For this post I’ll create a simple project to test the pyquery library (I’ll choose the “Empty project”).

You need to tell PyCharm what Python interpreter you want to use since it can use a different interpreter for each project. It will use this information to index all available libraries. A good practice is to have one virtualenv for each project. You can either use an existing virtualenv or create a new one from PyCharm (Settings→Project Interpreter→Python Interpreters).

For this project I’ll create a new virtualenv and make it available to all projects:

You can quickly search, read the description, and install packages from PyCharm. This is nothing you couldn’t do on the terminal, but it’s very convenient to be able to search for packages without leaving your IDE.

Some packages may take a while to install, especially if they need to be compiled. PyCharm runs the installation in background and you can see what is happening by clicking on the status bar:

As we can see, it installed not only pyquery, but its dependencies as well, such as lxml (as we would expect):

Shortcuts

PyCharm is no Vim, but you can do a lot by using only the keyboard. You can assign shortcuts to dozens (if not hundreds) of actions in Settings→Keymap. I change the default keymap pretty heavily since I like to have shortcuts like Control-a, Control-e, Control-k, and Control-y that are default in any Cocoa text box but are not setup by default in the “Mac OS X” keymap. You can also define mouse shortcuts. For instance, I changed Quick Documentation to use Option-Click. If you are a Vim user you may want tot try IdeaVim, a Vim emulator plugin. I heard good things about it, but I didn’t have the chance to try it yet.

I use the Dvorak keyboard layout, and annoyingly, there’s a bug where the wrong keys are used. People have reported this problem with French Canadian and German layouts as well. This means that I have to rebind quite a few shortcuts just to avoid this problem.

PyCharm uses function keys such as F1 and F2 a lot. As you probably know, on the Mac the top keys work as “multimedia” keys by default and as function keys by pressing the Fn key. We can swap this behavior in System Preferences→Keyboard.

I like to be able control the sound volume and change the brightness without having to reach for the Fn key, but I also like to use the function keys while programming without reaching for the Fn key (I’m lazy, what can I say?). I use Palua to switch between the “multimedia” and function keys. You can switch with a global key or you can configure Palua to switch automatically when using a specific application. I use the multimedia keys for all applications (the default) and configure Palua to use the Function keys for Xcode and PyCharm.

External Editor

PyCharm can launch any external tool, so I configure it to open Emacs with the current file. This is useful when you want to do some quick editing that is faster with your favorite editor (Emacs, Vim, TextMate, etc).

Go to Settings→External Tools, make sure “Open console” is unchecked and insert the path to the Program, the Parameters (in this case we can get the full file name with $FilePath$) and the Working directory ($ProjectFileDir$/$FileRelativeDir$/).

You can assign it to a shortcut:

And now it’s available from the menu as well.

Running Code and the REPL

There are many ways to run our code with PyCharm. We can use the venerable print function (or statement, depending on your Python version), import code on the REPL, send code to the console, use the debugger, or use IPython on a terminal.

The Venerable Print

Using print to display values is a useful and widely used technique, but it can lead to messy code if not used carefully.

We can run the code by clicking on the “play” button or with Control-R if the file is already selected to run.

If the file is not selected to run, with can select and run it with Control-Option-R or by right clicking and selecting from the context menu. PyCharm will remember your choice and you can use Control-R the next time. It sounds complicated but it is really simple. Check the manual for more information.

Console and REPL

My favorite way is to run the code on the REPL. You can open a Python console inside PyCharm on Tools→Run Python Console (I’ve assigned it to Control-C Control-C). It’ll use IPython if available (I recommend you install IPython inside the virtualenv) and will add the path of the current file to Python’s path. On the console you can import the functions you want to execute as usual.

Completion is available in the console as well:

Having imported the function, we can run it as usual. And we can toggle the console visibility easily with ⌘–4.

Execute Selection in Console

If the code I want to run is more than a couple of one-liners and not formal enough to write unit tests (I may be playing with data, for instance) I may save it in a scratch file. We can select the code we want to run (usually I just select the whole file with ⌘-A) and pick “Execute Selection in Console” from the context menu (or, even better, use the keyboard shortcut). [I omitted some menu items in the screenshot to make it shorter].

In the following example the headlines variable is available in the console since we have selected the whole scratch file to be executed in the console:

Macros

It would be nice if we could send the whole file to the console in one operation. Fortunately, we can record a macro. Select Edit→Macros→Start Macro Recording and perform the operations as before (select the whole file, send the selection to console, switch to the console). As you can see, my macro has only four actions (I press the down arrow key to remove the selection):

After you record the macro it’s available in the Macros menu and you can assign a shortcut to it (I use Control-C Control-E, but for some reason PyCharm only shows the first half of shortcuts with two strokes).

Debugger

Many people like to use the debugger to inspect data. Even if I prefer to use the REPL, sometimes the debugger is just more efficient, especially when inspecting complex objects. PyCharm doesn’t allow you to set a breakpoint on an empty line, so in this short example I had to add an extra line (I used print, but I could have used pass) since I wanted the debug to stop after setting the headlines variable. In real-life code this is seldom necessary.

Vanilla IPython

Finally, there’s nothing wrong with running an external terminal with IPython and using features such as %run and autoreload.

Finding Commands

It’s easy to get overwhelmed with so many commands. If you use PyCharm on the Mac you can use Help→Search as usual, or you can use Help→Find Action in any platform. It allows you to search for any PyCharm command, including the ones not available from the menu. (Emacs addicts will note this is somewhat similar to Emacs’ M-x). This is very useful and powerful and I use it all the time.

Code Completion

Completion in PyCharm is top notch. By default PyCharm will suggest things while you are typing:

Since this feature can be a little power hungry, you can disable it by selecting File→Power Save Mode. You can still use completion by calling it explicitly with Control-Space.

When completing, you can narrow the suggestion list by typing a substring:

Or you can type only the first letters of CammelCaseClasses or function_names_with_underscores:

If you type Control-Space once PyCharm will try to list the most related items:

If you type Control-Space again it will list every name it knows. This can be overwhelming, but notice it is listing the function name_of_mother from the unimported bar.py file.

But often you just want to complete the name of a local variable in one of the opened buffers. This feature has been available in Emacs and Vim for ages and now it’s available in PyCharm as well. The manual calls it Hippie Completion, but the actual command name (that is, the name you will find in Find Action) is “Cyclic Expand Word” and in my tests it’s working even inside docstrings.

Completion may not work in some cases when a library doesn’t have type hints. This StackOverflow page suggests to set a breakpoint in pdb and list the possible attributes with dir. One similar solution is to create a breakpoint in PyCharm and evaluate an expression at the breakpoint by clicking on the last icon in the debug toolbar (or using the appropriate keyboard shortcut). In this case I evaluated the expression “dir(r)”.

Access to Documentation

PyCharm has three ways to access documentation: Quick Definition, Quick Documentation, External Documentation, and Parameter Info. You can access them from the View menu or from the respective shortcuts.

Quick definition will show the whole definition (duh!) of a symbol (class, method, function, etc), including the documentation, of course. It’s useful when you just want to take a quick look at the definition without jumping to it.

Quick documentation will show the symbol’s documentation and signature.

Finally, External Documentation, and Parameter Info are pretty straightforward. The first opens the documentation in the default browser and the second shows the parameter information for a function or method (useful to quick check the name of keyword arguments, for instance).

The external documentation works out-of-the-box with Python (of course), PyQt4, PySide, gtk, wx, numpy, scipy, and kivy and you can add the path for external documentation in Settings→Python External Documentation.

Code Quality

PyCharm checks if your code conforms to PEP8 while you are typing. It’ll let you know if you have too many spaces or blank lines, if you are using underscores instead of CamelCase for classes names, and so on. If you want, you can configure PyCharm to run pylint as an external tool, as well.

Navigation Commands

PyCharm’s power starts when you master its navigation commands. It can jump to classes, functions, and so on. You can jump to the definition of a class, method, function, or variable quickly with ⌘-B or ⌘-Click on a Mac or Ctrl-B or Control-Click on Windows and Linux.

The mechanism to navigate to a class, file, or symbol is pretty much the same. You are presented with a dialog where you can type a substring, include items outside your project, and filter the results if necessary. Go to the Navigate menu or use the appropriate keyboard shortcut:

This is a typical dialog to navigate to a Class definition:

You can enter part of the name, including the file extension. For instance, if you want to open a JavaScript in a Django project but don’t remember its name, you can search for “.js”.

These features allow us to navigate the source code very quickly. Let’s suppose we are studying the Django source code and we want to see how the function render is implemented. We don’t know where it’s defined, so we go to Navigate→Symbol (⌥-⌘-O), type “render” and pick the first option (we can see it’s defined in django.shortcuts). If we have hidden the navigation bar we can show it quickly with Navigate→Jump to the Navigation Bar (⌘-↑) and see where the file is located (django→django→shortcuts.py):

Sometimes it’s useful to filter the completion results to take out things you are not interested. In the following example I remove the results related to JavaScript to have a much cleaner completion list:

PyCharm allows you to fold blocks such as classes, methods, and functions, but you can create your own foldable regions. This is useful when it makes sense to group things (classes, methods, functions, etc) in one bigger block. These blocks can be folded allowing us to focus on particular regions of the code. You can list all regions in a file by going to Navigate→Custom Region or using the appropriate keyboard shortcut. Sadly there’s no way to list all regions in a project.

Unit tests

Before we can run a test we need to add a new Run/Debug configuration. We go to Run→Edit Configurations and click on the plus button on the top left corner. Then, we select “Python tests” and the test runner. For this example I’ll use Unittests:

Finally, we need to add the folder where the tests are and pick the right Python interpreter:

Now we can run our tests just like we run code: by clicking in the toolbar or by selecting Run→Run (Control-⌥-R). This will open a dialog where you can choose what you want to run (you may have other things to run). You can run all tests or only the test on the cursor.

After you run the tests, PyCharm will remember your last choice, so you can repeat it with Control-R. This is useful if you are fixing one particular test case and want to run it multiple times without running the other tests. You don’t need to have the cursor on the test file you want to run; you can run the tests from any file. You can go from the code to its test by selecting Navigate→Test (or appropriate hotkeys).

Dealing with multiple files

As you noticed, PyCharm uses one tab per file:

To switch to tabs we can use the commands Select Next Tab and Select Previous Tab. They are bound to Control-→ and Control-← by default, but OS X uses these keys, so I rebind them to Control-⌘-N and Control-⌘-P (I use the same shortcuts to switch tabs in the Terminal).

One more direct approach to go to a tab is to use Navigate→File as we’ve seen, or View→Recent Files (⌘-E) and View→Recently Changed Files (⇧-⌘-E). These last two commands will present a list of files where you can click on type a substring to narrow the selection. In the following example I only had to type “⌘-E”, “s”, “enter” to go to the tab with the scratch.py file:

Split Windows

PyCharm has support for split windows. It’s not as advanced as in Emacs and Vim but it’s useful enough.

No command to deal with split windows have shortcuts assigned to them by default, so you’ll have to do it yourself. These are the keys that I use:

• Split Vertically, Control-S
• Split Horizontally, Control-H
• Unsplit, Control-C Control-U
• Unsplit All, Control-C Control-A
• Goto Next Splitter, Control-C Control-N
• Goto Previous Splitter, Control-C Control-P
• Move to Opposite Group, Control-C Control-M

Move to Opposite Group will move a tab to another group, but it only works if we have only two panes. In this case it’s just easier to drag and drop the tab to the desired place.

Conclusion

PyCharm can do a lot for us out-of-the-box. We saw how to use virtualenv, define shortcuts, use an external editor, run code, define macros, find commands with Find Action, be productive with code completion, access documentation, navigate the source code, run unit tests, and deal with multiple files and split windows.

And we are just scratching the surface. We didn’t talk about editing, version control, plugins, debugging (local and remote), templates, using TextMate bundles, web development, and much more.

I suggest you check PyCharm’s official documentation, including the Quickstart Guide and the Online Help. Also, Andrew Brookins wrote a very good PyCharm review from a Vim’s user perspective. I recommend it even if you don’t use Vim.

Fluidsynth

The easiest way to generate a MP3 from a MIDI on the Mac is by using fluidsynth. You can install it easily with homebrew:

brew install libsndfile lame
brew install --with-libsndfile fluidsynth

If everything is working properly, when you run fluidsynth -T help you should see something like the following:

kroger@orestes $ fluidsynth -T help
FluidSynth version 1.1.6
Copyright (C) 2000-2012 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

-T options (audio file type):
'aiff','au','auto','avr','caf','htk','iff','mat','mpc','paf','pvf','raw','rf64','sd2','sds','sf','voc','w64','wav','wve','xi'

auto: Determine type from file name extension, defaults to "wav"

With fluidsynth working, the conversion is simple. Just pass a SoundFont and a MIDI file as parameters and you should be good to go:

fluidsynth -F output.wav ~/Soundfonts/my-soundfont.sf2 myfile.midi
lame output.wav

If you are lazy, check this bash script. It accepts multiple MIDI files and removes the intermediate WAV file.

Timidity++

Timidity++ is my go to program to convert midi files to wav. To install just use homebrew:

brew install timidity

You may need to create a configuration file at /usr/local/Cellar/timidity/2.14.0/share/timidity/timidity.cfg and list where your SoundFonts are. For instance, in my timidity.cfg file I have:

soundfont /Users/kroger/Dropbox/Sfonts/Steinway-Grand-Piano1.2.sf2

You can read more about the configuration format by using its man page: man timidity.cfg.

There you go. We have two ways to convert MIDI files to MP3 on the Mac using free software.

Hackers turned founders rightfully track many things about their products (downloads, page visits, conversions, etc) but, surprisingly, many of them don’t track where they spend their time.

Of course it’s easy to fool yourself thinking that working more hours means accomplishing more. Working 10 hours on something doesn’t

necessarily mean accomplishing twice of working 5 hours. But I believe that time is a valuable metric that can provide you with significant insights. I track what I call “focused time”. If I stop working on my task to read my email, I’ll stop the stopwatch. If I’m working and someone or something interrupts me, I’ll pause the stopwatch.

The Basic Questions

I like to be able to answer the following questions:

1. How many hours did it take me to finish project X?
2. How many hours to finish task T?
3. For how long have I been working on task T today?
4. How many hours have I worked on my product this week?

Advantages

To me, the main advantage of tracking your time is focus. I trained myself to focus on the task at hand when I start the stopwatch. Also, it makes me think for a second what I’m trying to accomplish. Instead of just thinking “I’m going to work on my app”, I’ll be more specific, like “I’m going to close bug #23”. It sounds obvious but it’s not uncommon for me to start one task and jump to another without even realizing (and without finishing the first task). Since many tasks are connected the jump will feel fluid, but I may end up with many half-finished tasks at the end of the day. This alone would be a compelling enough reason to track my time. Even if I didn’t save the data, I’d still do it just to help me in keeping focused. But since I do have the data, there are other advantages as well.

I can use past data to help me to make decisions. For instance, if a project or task is taking longer than it should, I can decide to kill it. I do this often with scripts that I think I can finish in half hour but end up taking hours instead. The flip side is that I’m becoming much better at predicting how long something is going to take.

Knowing how long things can take helps me to sort my priorities and say no when necessary. For instance, I know that it takes me around 15 hours to write a basic project proposal. So when someone asks me to write something I can say “I wish I could, but this is going to take about 15 to 20 hours and I need to work on project X instead”.

I’m not big on motivation. Although I think it’s important, it’s hard to stay “motivated” all the time. But it’s very motivating to see how many hours you have been putting on your projects.

What I Need

There are many different type of programs to log time. Some concentrate on tracking the time of tasks, others on tracking billable hours and generating billing reports. I don’t need any complicated features, just Projects (such as “Write post ‘Time tracking for founders’), Tasks (such as “Write first draft”), and Categories (such as “Blog”). Unfortunately I’ve found few programs that can help me to answer the questions that I proposed above.

Before I switched to the Mac, I used a very nice Gnome applet called Hamster. I haven’t been able to find anything similar for the Mac. I tried I few but it seems I’ll have to write my own.

On the menubar

To me, the most important feature is be able to see the task name and stopwatch on the menubar. I like to be able to glance at the menubar and see what I’m supposed to be working and for how long I’ve been working. In the example below (using Hamster) you can see that I’m working on a task called “make examples” and I’ve been working on it for 27 minutes. You can configure Hamster to show a notification from time to time with the task name (the black thing in the screenshot). Some will find this annoying, but it helps me to keep focused. You can, of course, disable it.

Pause and Stop

I like to be able to pause and stop the timer. This is a simple feature but a few applications don’t implement it. Extra points to have a global hotkey for these actions. It’s handy to deal with interruptions.

Input and edit past events

It’s important to be able to rename, delete, and change the duration of an entry. It’s also useful to be able to input old entries, specially for tasks I have performed outside the computer. Believe it or not, some apps don’t have this feature.

Work Offline

There are a good number of online apps to track time (I have even used one for a few months). The obvious advantage is that you can access them in different computers and operating systems. But since I work mainly from one computer, I prefer to use a program that works offline.

Quick Statistics

One thing that I like about Hamster is that it shows some basic information on the main screen without the need to go to the separate reports window. In the following image we can see the total time for a category (Blog: 0.7h), the time for the current task (27 min), and the project name (“Time tracking for founders”).

Manual vs. Automatic

Automatic time tracking programs such as RescueTime are great, but they don’t help me to answer the questions I want. These programs work by recording how long you stay in one program or website. This is great to find your major distractions, but one major problem is that one program or website may have multiple uses. For example, watching videos on Youtube can be a productive task (lectures, screencasts, tutorials) that is part of a project or a plain distraction (kittens videos FTW!). However, applications like RescueTime will mark both as distractions.

Also, automatic time tracking tools are passive (duh! That’s their main selling point). I like to see in real time how long things are taking and, as I mentioned, it help me to focus.

Analysis

After more than one year collecting data, I was able to analyze it and gather some interesting results. In the following image we can see the amount of time I worked in different categories (startup is green, day job is red) over six months. In the first weeks I was trying to work in too many different things at once, therefore not focusing on my startup. Also, we can see I used my vacation time to do some serious startup work. (The original image has the number of hours on the Y axis, but I’m not comfortable sharing this information.)

I was able to find my most productive day, that is, the day I worked consistently more hours on my startup over a period of n months. Surprisingly, it was Monday:

Mon: 22.72%
Thu: 17.95%
Sun: 16.33%
Tue: 16.06%
Fri: 10.37%
Wed:  8.74%
Sat:  7.83%

Also, after analyzing this data, I gave up trying to be productive on Saturdays. Now it’s my day off.

But to me the biggest payoff is when I analyze the time I’ve spent on individual projects. For instance, I realized I spent a huge amount of time fiddling with tools when I was writing my book (I tend to do that). The next time I write a book I’ll be careful not to do that (as much).

Energy Level

Of course time is only one metric. Energy level is as import as time. If you have time and no energy you are not going to accomplish much. On the other hand, given enough time and energy you can do anything you want. Unfortunately, tracking energy levels is difficult and invasive. Out of curiosity, I recorded my perceived energy level during the day for almost two weeks. Every half-hour I’d rate my energy level as:

• 0: sleeping
• 1: need a nap
• 2: low energy
• 3: very good
• 4: very energetic

Originally I was going to show the following graphic that has the average levels for each week:

But once I reviewed the data, I realized the graphic above is useless. Look how wild my energy levels were during the two weeks:

I collected this data one year ago and, frankly, I don’t know what to make of it today. I’m still curious about energy levels, though. Maybe I’ll do this experiment again.

What I Use Now

The setup I have now it’s not perfect but it’s working for me. To collect the time I use StopWatchMenu, a simple Mac app that I developed myself. It sits on the menubar and allows me to enter and pause tasks:

At the end of the day I enter the data in a simple Django app I hacked together (I run it locally). I use this app to generate reports and analyze the data I have:

Naturally, I’d like to have one single app for all my time tracking purposes. I couldn’t find one Mac application that works the way I want and generates the reports I care about. Even Hamster doesn’t generate the reports I want; when I was using it I’d export the data and use a Python script to generate reports.

Conclusion

I’m fully aware that time is not the only metric; what counts is results. But tracking my time has been beneficial, not intrusive and it helps with procrastination and focus. Usually I don’t like to say how much time I’ve spent on things because, in my experience, people who don’t track time don’t have the slightest idea of how long things may take (specially for long projects) and may misinterpret the data (if I tell you how long I’ve spent building my first product you’d flip out 😉

Finally, I don’t necessarily care about the precise duration of a project, down to the minute. I like to know the order of time a project took to be accomplished. Some projects will need the order of a couple dozen hours while others the order of one hundred hours. So even if I didn’t track every single minute I was working on a project (I don’t track the time I’m having ideas on the shower or on a long walk) I still get a decent estimate of how much time was necessary to complete a project and how much time I may need for similar projects.

Let me know in the comments how tracking your time has helped you and if you have a good solution to track energy levels.

Please keep in mind that I came up with these techniques while writing my book. My main goal was to finish the book and to spend as little time as possible fiddling with my tools. Therefore, while some of these solutions are ok, others are kind of hacky. I’m posting them hoping they’ll be useful, but they are not a polished product.

Themes

Sphinx comes with a few themes out-of-the-box and it’s very easy to create new ones. I created three themes: a minimalist html theme for previewing the book while I was writing (called ‘book’), an Epub theme (‘epub2’), and a mobi theme (‘mobi’). Here is a screenshot of the html theme:

Besides removing things from the base theme such as the sidebar, I used @font-face to define Anonymous Pro as the font for the code examples. I defined the following in my themes/book/static/default.css_t (with similar code for for bold, italic, and bold-italic variants):

@font-face {
font-family: AnonymousPro;
font-weight: normal;
font-style: normal;
src: url("fonts/Anonymous Pro.ttf");
}

In themes/book/theme.conf I defined Palatino as the body font:

bodyfont = 'Palatino', serif
headfont = Tahoma, Geneva, sans-serif;

The ePub theme is very simple. Sphinx comes with a default theme for ePub, but it has some rough edges. For instance, it shows a copyright notice at the end of each chapter:

Most Sphinx themes extends a basic theme. Since the html for the ePub file is very simple, I decided to create the layout.html file from scratch, without inheriting from the base theme. As you can see here, my Epub theme is super simple. In the css file I use the same @font-face trick I used in the html theme. I also changed small things, like not showing bullets in the Table of Contents:

In the following image you can see the difference between the out-of-the box ePub style and my style using pygments and AnonymousPro:

The mobi theme is very similar to the epub2 style, except it doesn’t use @font-face and the highlighted source code is black and white, since most kindle readers don’t support these features. However, there’s a bug in iBooks that prevents it from showing the right font when one uses the span tag, as pigments does to generate the highlighted source code. Kindle seems to have similar restrictions. To fix this I created custom builders.

The ePub Builder

The builder epub2 is a subclass of the built-in ePub builder. It disables visible links and replaces the span tag with samp due to the bug I mentioned earlier. By default the built-in ePub builder will generate links like this:

but I prefer not to show the url:

You can see the full builder here. A much nicer solution would be to create a new writer by subclassing writers.html.HTMLWriter and have it emit samp directly instead of span. However, I could not find a way to make my builder use the new writer without copying a lot of code from the original HTMLWriter (and, therefore, negating the benefits of subclassing).

The mobi Builder

I created a mobi builder by copying the epub builder from Sphinx and making the necessary changes. Maybe I could have subclassed it (I’ve seen a mobi builder on github that did that), but I wanted to have separate configuration options for the mobi file, such as mobi_cover. It uses Amazon’s kindlegen to convert the html pages to the mobi format.

Code examples

Sphinx makes it really easy to show code examples with literalinclude, specially with the :pyobject: option, which shows only a selected Python class, function, or method in a file. In the following example I want to show the definition of the function note_name defined in pyknon/simplemusic.py:

.. literalinclude:: pyknon/simplemusic.py
    :pyobject: note_name

This is very simple and straightforward, but I also wanted an easy way to show examples displaying the usage of a function and the result of its computation, like in the following image:

I could just type the code in the python REPL and copy and paste the result, but if the function changes I might need to update the examples manually, which could lead to some examples being outdated or wrong. And code examples that won’t run or with mistakes can be a big source of frustration in programming books. To solve this I hacked an extension called code-example that behaves like literalinclude, but it adds the code and the result of its computation in a Python REPL. Its usage is the same as literalinclude:

.. code-example:: note_name.py

Following is the content of note_name.py. Notice note_name is called four times, but note_name.py doesn’t have the result of the function calls. The result will be computed and displayed by code-example. Also, import lines won’t show in the result:

from pyknon.simplemusic import note_name

note_name(0)
note_name(1)
note_name(13)
note_name(3)

You can find code-example here. It’s hackish, but it saved me a lot of time checking if my code examples where updated and gave more confidence in the final result.

Autoimage

One of the most annoying things about working with a source file that is intended to have multiple outputs such as PDF and HTML is how to deal with images. Often images will have to be scaled differently depending on the output. For instance, I used to have code like the following:

.. only:: not latex

.. image:: figs/notation3.png
    :scale: 40

.. only:: latex

.. image:: figs/notation3.png
    :scale: 80

It has quite a bit of repetition and sometimes I’d type an extra space or the wrong number of blank lines and the output would be wrong since Sphinx is fastidious about spacing and blank lines (for instance, it needs the blank like after the line with ..only). I wrote the extension autoimage to simplify this. With it the previous example becomes:

.. autoimage:: notation3.png
    :scale-html: 40
    :scale-latex: 80

Autoimage is somewhat smart. It tries to use a pdf image if it’s available and the backend is LaTeX, and looks for a black-and-white image if the configuration option black_and_white is true. After I finished the extension and converted the whole book I discovered that Sphinx caches things and share values among builds. It means that, in the previous example, if I built a pdf and a html version of my book in that order, notation3.png would be scaled 80% in both cases, instead of 80 and 40 percent. To solve this I just run sphinx-build with the -E option.

Configuration

My Sphinx configuration file is the most boring part of this blog post, but I’m including it here in case you’re curious. I unset almost all LaTeX options. This allows me to use these options in any LaTeX style I want. I use two styles, one for the color version and another for B&W:

latex_elements = {
    'papersize': '',
    'fontpkg': '',
    'fncychap': '',
    'maketitle': '\cover',
    'pointsize': '',
    'preamble': '',
    'releasename': "",
    'babel': '',
    'printindex': '',
    'fontenc': '',
    'inputenc': '',
    'classoptions': '',
    'utf8extra': '',
}

latex_additional_files = ["mfgan-bw.sty", "mfgan.sty", "_static/cover.png"]

I also map a few unicode characters to their LaTeX equivalents:

def setup(app):
    from sphinx.util.texescape import tex_replacements
    tex_replacements += [(u'♮', u'$\natural$'),
                         (u'ē', u'=e'),
                         (u'♩', u'quarternote'),
                         (u'↑', u'$\uparrow$')]

Compiling

Finally, I use a custom Makefile to run Sphinx. It manages to be even more boring than my configuration file, but it’s what allows me to generate multiple outputs. The secret is to use the -D option in sphinx-build to set or override a setting in the configuration file. For instance, these are the options I use to generate a B&W PDF to be printed and a color PDF to be read on the screen:

PAPEROPTS= -D latex_elements.pointsize=11pt -D latex_elements.preamble=\usepackage{mfgan-bw} \
-D pygments_style=bw -D black_and_white=True -D code_example_wrap=67 \
-D latex_show_pagerefs=True

SCREENOPTS= -D latex_elements.pointsize=12pt -D latex_elements.classoptions=,openany,oneside \
-D latex_elements.preamble=\usepackage{mfgan} -D pygments_style=my_pygment_style.BookStyle \
-D code_example_wrap=67

And these are the related targets that use those options:

pdf:
    $(SPHINXBUILD) -b latex $(SCREENOPTS) $(ALLSPHINXOPTS) $(BUILDDIR)/latex
    sed -i .bak -f process-latex $(BUILDDIR)/latex/$(LATEX-NAME).tex
    rsync -a latex/ $(BUILDDIR)/latex/
    $(MAKE) -C $(BUILDDIR)/latex pdf

pdf-bw:
    $(SPHINXBUILD) -b latex -t black_and_white $(PAPEROPTS) $(ALLSPHINXOPTS) $(LATEX-BW)
    sed -i .bak -f process-latex $(LATEX-BW)/$(LATEX-NAME).tex
    sed -i .bak '/\setcounter{page}{1}/d' $(LATEX-BW)/sphinxmanual.cls
    rsync -a latex/ $(LATEX-BW)/
    $(MAKE) -C $(LATEX-BW)/ pdf

As we can see, I use sed to do some pre-processing and cleanup. Also, I use latex-mk to compile the LaTeX files. The Makefile that comes with Sphinx will always compile the .tex file three times, even if it’s not necessary, while latex-mk will only run LaTeX if necessary, resulting in shorter build times.

Conclusion

I’m pretty happy with the final result. People have complemented me on how good the pdf looks. I think the Epub and Mobi versions are good enough, although not perfect. However, they are as good as many commercial ebooks I’ve purchased. Although Docutils and Sphinx can be extended, I wish they were even easier to extend, specially to change the generated HTML and LaTeX outputs. The worst part in this process was dealing with Epub and mobi readers. I love these devices as an user, but generating good looking technical books proved to be a challenge due to bugs and lack of precise documentation. After I launched my book I was horrified to discover that older Kindles don’t display tables. I had to replace the tables with images in the mobi file.

I’m sorry I don’t have a simple one-click-install plugin to make Sphinx generate beautiful books automatically. I’ll mature the ideas in this post and submit patches to these projects. Let me know in the comments of similar features you need or things I could have done better.

Update: I forgot to mention that I’ve used the XeLaTex engine to be able to use TrueType fonts. That’s another reason I unset most LaTeX variables on conf.py.

Edit: All (my) code linked is this post is released under the MIT license.

We can see music in three layers:

1. natural: acoustics and psychoacoustics.
2. logical: the math for things like transposition and inversion.
3. social: the result of social conventions and usage over hundreds of years such as note and interval names.

I find that hackers and programmers can learn the first two layers very quickly and, as a result, they can have a better understanding of the social layer. In the book I explain musical concepts in English with a corresponding implementation in Python, using the Pyknon music library. By implementing musical concepts in a programming language we can have a more precise way to describe and understand them.

For instance, some music students have problems to transpose notes mentally. However, it becomes easy once you realize that transposition is nothing more than the sum of a note and a transposition index (in the following example we use an integer notation for notes where C = 0, C# = 1, …, B = 11):

I wrote the book using the reStructuredText markup and I used Sphinx to generate the following formats: black & white pdf (for the printed version), color pdf, mobi for kindle, epub, and html (for previewing). My favorite feature in Sphinx is the literalinclude directive with the pyobject option to include the source code of functions and methods.

My second favorite feature is the possibility of creating custom directives. For example, I created a directive to insert the source code and the result of its execution in a Python REPL, as we can see in the transposition example above. I also created custom layouts for epub, mobi, and latex, and did some bending to use XeLaTeX (to be able to use nicer fonts). I’ll write about the details of using Sphinx to write books in an upcoming blog post.

Finally, I used the services of Proofreadingpal to professionally proofread and edit the manuscript.

I’m very pleased with the result. The idea for this book has been in the back of my head for a long time and it’s a joy to finally see it materialize. If you have suggestions or questions, I’d love to hear from you.

The harmonization for the major scale is pretty simple; you stack notes vertically following every other note in the scale:

In modern music it’s fairly common to use non-traditional scales to generate new and interesting harmonies. For instance, in the example bellow I have a 7-note scale that is very different from the good ol’ major scale. Right after the scale we have harmonizations for every 3, 4, and 5 notes of the scale. Also notice that we break the thing about having the interval in which we pick notes be the same of the horizontal notes (that is, having chords in thirds, fourths, and so on). This leads to very interesting harmonies.

By the way, this is one of the reasons I got into programming in the first place; to generate material to use in my compositions (using Pascal back in the 90’s, if you must know).

So, in order to harmonize every scale we need to know how many scales exist and the answer will vary depending to whom you ask. In a way a scale is an ordered set of notes, so I used the 129 pitch class sets with cardinality greater than 5. Here is a complete list of all pitch class sets. Some people will find that using prime forms is too condensed and doesn’t take in account a bunch of scales, but it’ll do for the purpose of this post.

The code to generate the harmonizations is actually very simple (with the help of my in-progress python library). We harmonize the first note of a scale with harmonize_first_note and use it in a list comprehension to harmonize every note of the scale. The helper function rotate_set will generate every rotation of a list.

def harmonize_first_note(scale, interval, size):
    i = (interval - 1)
    indexes = [x % len(scale) for x in range(0, size * i, i)]
    return [scale[index] for index in indexes]

def harmonize_scale(scale, interval, size=3):
    scales = musiclib.rotate_set(sorted(scale))
    return [harmonize_first_note(scale, interval, size) for scale in scales]

Here’s how we use it: A scale is defined as a list of integers (we are using integer notation). Let’s put the major scale in a variable. The result of harmonize_first_note will be, as expected, just the first chord (C major) as a list of integers:

scale = [0, 2, 4, 5, 7, 9, 11]

harmonize_first_note(scale, 3, 3)
>>> [0, 4, 7]

And the result of harmonize_scale will be a list with the harmonization of every chord in the scale. In the next example I’m using 4 as the chord size to generate tetrads:

harmonize_scale(scale, 3, 4)
>>> [[0, 4, 7, 11], [2, 5, 9, 0], [4, 7, 11, 2], [5, 9, 0, 4],
     [7, 11, 2, 5], [9, 0, 4, 7], [11, 2, 5, 9]]

Finally, lets generate tetrads separated by fourths instead of thirds:

harmonize_scale(scale, 4, 4)
>>> [[0, 5, 11, 4], [2, 7, 0, 5], [4, 9, 2, 7], [5, 11, 4, 9],
     [7, 0, 5, 11], [9, 2, 7, 0], [11, 4, 9, 2]]

The code to generate the LilyPond file is boring, so I won’t talk about it here.

Generating harmonizations for every scale is interesting and all, but it start to get interesting when we can filter things. For instance, I may want to generate harmonizations for the scales that have more than 4 notes and have consecutive intervals greater than 1 semitone. This can be accomplished with filter_sets. This is how I’d express the previous example:

filter_sets(lambda intervals, size: size > 4 and 1 not in intervals)

The function filter_sets accepts an anonymous function with a list of the consecutive intervals in a set and the size of a set as parameters. If the condition in the body of the anonymous function is met the set is returned:

def filter_sets(condition):
    sets = {}
    for forte, pc_set in musiclib.PC_SETS.items():
        intervals = musiclib.intervals(pc_set)
        size = len(pc_set)
        if condition(intervals, size):
            sets[forte] = pc_set
    return sets

This is an example of how high-order functions can be used to abstract code. Also, the built-in function all is very handy to chain conditions. Let’s say I want to get scales that have more than 4 notes and have only 1 consecutive halftone and 1 consecutive whole tone:

filter_sets(lambda i, s: all([s > 4, i.count(1) == 1, i.count(2) == 1]))

Since I want to filter scales by interval content, it makes sense to write a small function to help me. cond_interval_count abstracts the previous code:

def cond_interval_count(count_list):
    return lambda i, s: all([i.count(interval) == count for
                             count, interval in count_list])

So if I want the scales that have no consecutive halftones and only 1 consecutive whole tone I can write it as:

cond = cond_interval_count(((0, 1), (1, 2)))
filter_sets(cond)

There are a few things that can be improved:

• Use some pitch spelling algorithm, like (Meredith, 2003) so we won’t have chords like C, D#, G
• Have better functions for filtering. The function I’m using looks for consecutive intervals but it would be better to search for every interval in the Pitch Class Set.
• Handle octaves better in the LilyPond generation. Some notes would be better on a different octave:

This post only scratches the surface of all things we can do. For anything more complicated than that I’d use music21, a nice and professional Python library for computer-aided musicology. But it’s interesting too see how far we can go with a few dozen lines of pure Python code and the use of nice abstractions. If you have an interest in this kind of thing, there are lots of Python packages for music programming and Notes from the Metalevel: An Introduction to Computer Composition is a nice book about the topic (but it uses Scheme, not Python for the code).

The first chapter is about building abstractions with functions. I think it’s remarkable that a book for beginners (pretty smart beginners, but still) introduces assignment only in the third chapter (on page 220). I really think this is the way to start a programming course. Probably all students know about mathematical functions and with functions we can talk about things like bound variables, scope, abstraction, composition, and recursion.

A powerful language needs to have the following things to allow the combination of simple ideas to form complex ideas:

• primitive expressions: the simplest entities in a language. Things like numbers and arithmetic operations and functions.

• means of combination: “by which compound elements are built from simpler ones”;. Nesting combinations, such as square(2 * square(3 + 7)) are simple means of combination.

• means of abstraction: “by which compound elements can be named and manipulated as units”.

def is the simplest mean of abstraction. The following code creates a function and associates it with a name:

def square(x):
    return x * x

It’s important to make the distinction of the act of creating a function and naming it. We can create a function without a name (an anonymous function) with lambda:

lambda x: x * x

And we can assign it to a variable, giving it a name:

square2 = lambda x: x * x

And, in fact, we can see (with the help of the bytecode disassembler module) that Python will generate the same bytecode for both square and square2:

>>> import dis
>>> dis.dis(square)
1 0 LOAD_FAST 0 (x)
3 LOAD_FAST 0 (x)
6 BINARY_MULTIPLY
7 RETURN_VALUE

>>> dis.dis(square2)
1 0 LOAD_FAST 0 (x)
3 LOAD_FAST 0 (x)
6 BINARY_MULTIPLY
7 RETURN_VALUE

Having defined square, we can use it in combinations:

square(2 + 5)
square(square(7 + square (3)))

And, naturally, we can use square as a building block:

def sum_of_squares(x, y):
    return square(x) + square(y)

sum_of_squares(3, 4)

def f(a):
    return sum_of_squares(a + 1, a * 2)

The association between names and values, such as the name of a function, is saved in a place called the environment. Chapter 3 will talk about the environment in greater detail.

Applicative and Normal Order

To evaluate combinations we follow a recursive rule (quoted verbatim):

1. Evaluate the subexpressions of the combination.
2. Apply the procedure that is the value of the leftmost subexpression (the operator) to the arguments that are the values of the other subexpressions (the operands).

The substitution model is a simple model to help us understand what happens during evaluation. To evaluate procedures we have the following rule:

To apply a compound procedure to arguments, evaluate the body of the procedure with each formal parameter replaced by the corresponding argument.

Keep in mind that “the purpose of the substitution is to help us think about procedure application, not to provide a description of how the interpreter really works” and SICP presents “a sequence of increasingly elaborate models of how interpreters work, culminating with a complete implementation of an interpreter and compiler in chapter 5”.

In the example below we can see two ways to evaluate the function f we defined previously. The function f is defined in terms of sum_of_squares, which is defined in terms of square, which is defined as the multiplication of a number by itself. In the evaluation method on the left, an expression such as 5+1 is evaluated and applied immediately. This method of evaluation is known as applicative order evaluation (a kind of strict evaluation, used in most programming languages, inclusive Python and Scheme). The evaluation method on the right only evaluates an expression when needed. It’ll fully expand all function calls first, and then evaluate what’s left. This is known as normal order evaluation. Compare the result of both methods in line 5:

 # applicative order normal order
f(5) f(5)
sum_of_squares(5+1, 5*2) sum_of_squares(5+1, 5*2)
sum_of_squares(6, 10) square(5+1) + square(5*2)
square(6) + square(10) ((5+1) * (5+1)) + ((5*2) * (5*2))
(6 * 6) + (10 * 10) (6 * 6) + (10 * 10)
36 + 100 36 + 100
136 136

Of course both methods yield the same answer, but this will not always be the case, as we’ll see latter in this post (exercise 1.5) and in this series.

Conditional Expressions

The main point of this sub-section is to show how to write conditional expressions in Scheme by implementing a function named abs to calculate the absolute value of a number. Since abs is a built-in function in Python, I’ll use myabs (it’s pretty hard not to think about cheesy late-night infomercials with a function named like this ;-)). The first implementation follows the mathematical definition and uses multiple predicates:

def myabs(x):
    if x > 0:
        return x
    elif x == 0:
        return x
    elif x < 0:
        return x

This is unnecessarily long and can be shortened as:

def myabs(x):
    if x < 0:
        return -x
    else:
        return x

And it can be even shorter by using the ternary operator (new in Python 2.5):

def myabs(x):
    return -x if x < 0 else x

This sub-section also shows logical operators such as and, or, and not. For instance, in Scheme the expression 5 < x < 10 would be written as (and (> x 5) (< x 10)) but the same thing in Python is just 5 < x < 10. Pretty cool, huh?

Exercise 1.4

This exercise asks the reader to describe the behavior of the following procedure. I’ll use the original Scheme code, then I’ll show the equivalent in Python.

(define (a-plus-abs-b a b)
  ((if (> b 0) + -) a b))

This code may look weird at first (and I’m not talking about the parenthesis), but we can just apply the substitution model to understand how it works. Let’s copy the function’s body:

((if (> b 0) + -) a b)

If b is greater than 0, the conditional expression will return the + (addition) function, otherwise it’ll return the – (subtraction) function. In Scheme + and – are functions, just like sqrt. Let’s suppose that b is greater than 0 and the conditional expression will return +. We substitute + for the conditional expression:

(+ a b)

The resulting expression is just the sum of a and b. This kind of thing is possible because in Scheme functions are first-class; we can create them at runtime, pass them as arguments to other functions, and return them as values.

Functions are also first-class in Python, but + and – are not functions. We can access Python’s basic operators with the operator module. For instance, operator.add(2, 2) is equivalent to the expression 2 + 2. So, we can write the Scheme code above in Python as:

from operator import add, sub

def a_plus_abs_b(a, b):
    return (add if b > 0 else sub)(a, b)

Exercise 1.5

This exercise asks the reader to describe the behavior of the following code if the interpreter uses applicative order evaluation and normal order evaluation.

def p():
    return p()

def test(x, y):
    return 0 if x == 0 else y

test(0, p())

In applicative order evaluation, the interpreter will enter in an infinite loop, regardless of the value of test‘s first argument, because both operands will be evaluated before the function is called (line 7). With normal order evaluation, the interpreter will evaluate only what is necessary, so if the first argument of test is 0, it’ll return 0 because the second argument will not be evaluated.

Square Roots By Newton’s Method

One way to calculate square roots is by using Newton’s method of successive approximations. We start with a guess g for the square root of a number x and calculate a better guess by averaging g with x/g, or:

The following procedure tests if the guess we have is good enough for the number x (the radicand) we want to compute the square root. If not it’ll keep trying to improve the guess until it’s good enough:

def sqrt_iter(guess, x):
    if is_good_enough(guess, x):
        return guess
    else:
        return sqrt_iter(improve(guess, x), x)

To improve a guess we average it by the number x divided by the guess:

def improve(guess, x):
    return average(guess, x/guess)

Average is easy enough to define:

def average(x, y):
    return (x + y)/2

Of course, we need to define is_good_enough. A basic test is to see if the square of the guess minus the original number x is smaller than some threshold (we use 0.001). This is not a good test for very small and large numbers (see exercise 1.7 in the book) but will do for a first try:

def is_good_enough(guess, x):
    return abs(square(guess) - x) < 0.001

Finally, we need to start at some point. We begin with 1.0 as a guess:

def sqrt(x):
    return sqrt_iter(1.0, x)

Procedures as Abstractions

One problem with our implementation of sqrt is that functions like is_good_enough, sqrt_iter and improve are cluttering the global namespace. It’s very important to decompose a problem in sub-parts like we did, were each function does only one thing, but it’s also important to be able to group things that are not going to be used in other contexts (like improve and sqrt_iter). One solution is to nest the procedures in one block structure:

def sqrt(x):
    def is_good_enough(guess, x):
        return abs(square(guess) - x) < 0.001

    def improve(guess, x):
        return average(guess, x/guess)

    def sqrt_iter(guess, x):
        if is_good_enough(guess, x):
            return guess
        else:
            return sqrt_iter(improve(guess, x), x)

    return sqrt_iter(1.0, x)

Now the functions sqrt_iter, is_good_enough, and improve are internal to sqrt and are not exposed to other programmers.

But there’s something else. The variable x is bound in the scope of sqrt and since improve, is_good_enough, and sqrt_iter are in the scope of sqrt we don’t need to pass x explicitly as as argument to these functions. We can rewrite the code to make x a free variable inside these functions. In this case the interpreter will get the value of x from the enclosing scope_._ This is an example of lexical scoping:

def sqrt(x):
    def is_good_enough(guess):
        return abs(square(guess) - x) < 0.001

    def improve(guess):
        return average(guess, x/guess)

    def sqrt_iter(guess):
        if is_good_enough(guess):
            return guess
        else:
            return sqrt_iter(improve(guess))

    return sqrt_iter(1.0)

Summary

In the first section of SICP the authors introduce the notion of the environment and lexical scoping (both will be explored in more detail latter in the book), evaluation methods, and, above all, functional abstraction. I think this is a pretty sophisticated introduction for beginners and shows why SICP is considered one of the classics of computer science.

We can find more about the book’s goals in the Preface to the First Edition:

Our design of this introductory computer-science subject reflects two major concerns. First, we want to establish the idea that a computer language is not just a way of getting a computer to perform operations but rather that it is a novel formal medium for expressing ideas about methodology. Thus, programs must be written for people to read, and only incidentally for machines to execute. Second, we believe that the essential material to be addressed by a subject at this level is not the syntax of particular programming-language constructs, nor clever algorithms for computing particular functions efficiently, nor even the mathematical analysis of algorithms and the foundations of computing, but rather the techniques used to control the intellectual complexity of large software systems. Our goal is that students who complete this subject should have a good feel for the elements of style and the aesthetics of programming.

The emphasis in bold is mine to show a couple of my favorite quotes (my favorite quote of all times must be Alan Perlis’ “Syntactic sugar causes cancer of the semicolon”, but I digress).

The whole book is available for free and the book’s page at MIT has lots of extra material. Also, there are some highly recommended video lectures by the authors of the book.

SICP uses the Lisp dialect scheme, but since SICP concentrates on programming ideas rather than languages, there are some efforts to translate the code in the book to other programming languages. I think this is a good thing, because even if scheme is a beautiful language (it really is) and I think everyone should learn it, it’s good to see how the ideas presented in SICP can be translated to other languages.

I have read a good chunk of SICP and watched most video lectures before, but I didn’t do the exercises. Now, I want to read it again and do as many exercises as I can. Since I’ve been programming a lot in Python these days, I am curious to see how the ideas in SICP will translate to Python. So, my plan is to read the book and do the exercises using Python instead of Scheme. There are good notes and solutions for the exercises on the web and I may consult those as needed (giving fully credit, of course).

I’ll post my notes with the (Python) code. I’ll try to capture the essence of each section, instead of trying to post every single thing and exercise, but we’ll see how it works. I hope this will be a good resource for the Python community. Let me know in the comments what you think.

Why don’t you go ahead and read the post about section 1.1?

The advantage of reviewing code on paper instead of the monitor is that I can give my eyes a break and I can annotate more freely, connect things with arrows, draw boxes, etc. I don’t print my code everyday, after all this is not the 80’s, but I like to print the code when I feel it could benefit from some refactoring and re-structuring and my brain could benefit from same change in the medium and even location (I can grab the printouts and go to a park or cafe).

The best way I found to print code is by using the LaTaX listings package (it should be included in any modern LaTeX distribution). This package is intended to include code snippets in papers and books, but it works to generate code listings as well.

These are the options I have in my LaTeX document header:

documentclass[10pt]{article}
usepackage[T1]{fontenc}
usepackage[scaled]{beramono}
renewcommand*familydefault{ttdefault}
usepackage{listings}

The beramono package is a version of Bitstream Vera Mono modified to work with TeX. Without it we’ll get the default LaTeX mono font:

But I like Bitstream Vera better (it looks even better on paper):

Listings has support for Python out-of-the-box and has lots of options and features (check the manual to see what listings can do). These are the options I like to customize:

\lstset{
  language=Python,
  showstringspaces=false,
  formfeed=newpage,
  tabsize=4,
  commentstyle=itshape,
  basicstyle=ttfamily,
  morekeywords={models, lambda, forms}
}

Options like tabsize and showstringspace are self-explanatory. The option formfeed=newpage will replace the formfeed character by the LaTeX command newpage, creating a new page at that point. Sometimes I put some line feed characters in my code so I can do things like narrowing to a page in Emacs. Emacs shows line feed characters as ^L, as you can see here:

The option morekeywords will add the keywords in the list of keywords recognized by Listings. In my example I added lambda, models and forms (I’ve been working a lot with Django lately), so these keywords will be pretty-printed.

Finally, I use the command lstinputlisting to include a source code file in the LaTeX file (instead of having to type the whole thing in the LaTeX file). I like to separate each section by a horizontal line and add some space at the end. I define a command to abstract this:

\newcommand{code}[2]{
  hrulefill
  subsection*{#1}
  lstinputlisting{#2}
  vspace{2em}
}

So in the end this is how I insert a Python file:

\code{Models}{../testapp/models.py}

And this is the result:

I believe that reviewing my code on paper has save me hours of work and I find listings to be a very good option to print source code. I added a full LaTeX template here.

Do you print your source code? If so, what tool do you use? Let me know in the comments.