Notebook Workflow

In this course I set things up using Jupyter Notebooks. I will use these for lectures and assignments. I find that this is a good workflow for scientific computing. For a big project I will use notebooks in several different ways.

Development notebooks

These are like lab notebooks. You use them as scratch space to try out ideas. Once you have working code, you encapsulate it in functions. You then leave a record of testing the functions.

After the development is complete, and you are ready for production, you either copy the code into a new production notebook, or you copy it into a text file “whatever.jl”. You can load that code into your production notebook with include("whatever.jl"). If it is a more sophisticated set of tools, you can turn it into a package and import using typing whatever.

You can extract the code from a development notebook by clicking on File, and selecting Download as .jl.

Some people like to have both a text editor and a notebook open at one time. They make edits to the .jl file, and lode it in with include. They then do all of their interactive debugging in the notebook.

Every once and a while you need to clean up your development notebook – or even copy it over to a completely new notebook to get rid of all the dead ends. It is good practice, however, to leave in place the work you did to verify that everything is working correctly. That way if you make a change you can run the same tests. This is an informal version of the “unit test” paradigm.

Presentation notebooks

You often want to share your code and results with collaborators. One strategy is to use presentation notebooks. This is similar to my lectures. You put fully debugged code (or include it from text files), and run the relevant parts of it to illustrate the physics. One can tweak it on the fly to do further explorations.

Static views of the notebook can be shared by using nbviewer. Dynamic views can be shared using web services like binder.

Production notebooks

Once you have working code, you often want to keep the production runs separate from the development work. For code that runs relatively quickly (minutes to hours), you can comfortably do this from notebooks. Make sure that any data which takes more than a couple minutes to produce is saved to disk. Don’t rely on the notebook’s record as a way to save data.

Either copy your development code to the production notebook, or load it in with include.

Scripts

If you are going to be running code overnight, or you might be running on multiple machines, then it is convenient to use scripts instead of a notebook. Put all of the code in a whatever.jl file, and run it with julia whatever.jl. Make sure that any data is saved to a file. If you are using a Unix environment (including linux or Mac OSX), you can redirect standard output and standard error into text files by julia code.jl >code.out 2>code.err. I would recommend against using standard output for results.

Jupyterlab

We will often work with our notebooks within the environment called “Jupyterlab”. It provides a graphical interface with Jupyter notebooks, text files, interactive sessions and output windows.