Section author: Robert Nikutta <nikutta@noao.edu>

1.9. Jupyter Notebooks

1.9.1. Introduction

A very convenient way to do data science is interactively in your browser, for instance through Jupyter notebooks.

Jupyter notebooks are self-contained programs (for Data Lab they are written in Python), which can be loaded and run right inside your browser. This is very convenient, as you can use the entire power of Data Lab and of the Python language, without giving much thought to the installation and configuration of software.

You can recognize Python notebooks on their file name suffix .ipynb, i.e. my_first_notebook.ipynb. This file stores all the input command you use in the notebook, as well as the output that these commands generate (for instance, tables and graphs).

Data Lab was built from the start with Python notebooks in mind, and provides a containerized notebook server for registered users, as well as a more restricted public notebook server (if you want to try it out before signing up).

1.9.2. Using Data Lab through a notebook

There are three ways of using Data Lab through a Jupyter notebook:

  1. Data Lab’s public notebook server
  • No user account required
  • Your work will only persist for the duration of the session. Download the notebook if you want to preserve it.
  • No virtual storage allocation
  • Access is limited to some databases
  1. Data Lab’s registered users notebook server
  • User account registration required
  • Saved notebooks will persist as long as you want to keep them
  • Your account comes with a generous allocation of virtual storage space. You can use it to store query results, upload/download files to/from, share files with collaborators, perform deeper analysis on data, etc.
  • Access to all database holdings at Data Lab
  1. Run a notebook server on your own computer

1.9.3. Launch a notebook server

Launch any of these notebook servers (as described in the above section):

  1. Data Lab’s public notebook server: http://datalab.noao.edu/notebooks
  2. Data Lab’s notebook server for registered users: http://datalab.noao.edu/devbooks
  3. Launch a notebook server on your own computer: type jupyter notebook in a terminal; a browser window pops up

Data Lab’s notebook server for registered users (2nd option above) will prompt you for your registered user name and password.

Data Lab registered users notebook server

If you don’t have a Data Lab account yet, you can register for one right away.

In all three cases you should see a notebook “dashboard” that looks similar to this:

Data Lab notebook server dashboard

If you’re using the Data Lab notebook servers (public or registered), the dashboard will be pre-populated with several example notebooks. You can either load any of them by clicking on its name, or create a new notebook to start with a clean slate.

1.9.4. Basic notebook usage

1.9.4.1. Keyboard shortcut cheat sheet

Many of the notebook functions can be accessed through quick and convenient keyboard shortcuts. Most shortcuts are invoked by pressing the ESC key (release it after pressing), which enters the command mode of the notebook, followed by one or two additional keys, which are pressed simultaneously or in sequence (releasing the previous key before pressing the next one), depending on the function you wish to invoke.

A very useful keyboard shortcut combination is ESC,h (i.e. press the ESC key, release it, then press the h key) , which brings up a window with all major keyboard shortcuts… how convenient!

List of keyboard shortcuts

1.9.4.2. Load an existing notebook

If you want to load a notebook that is visible in the dashboard, simply click on its name. A new browser tab will open and the notebook loaded.

If your notebook file is located under one or several sub-directories in the dashboard, simply click your way through to it by traversing the directories in the dashboard view.

You cannot access .ipynb files that are outside (i.e. above) the root of the directory tree in the dashboard. You can always jump back by clicking on the “home” icon in the dashboard view:

root directory / house icon

Upon opening a notebook file, the upper right corner of the new tab will (very) briefly display status notifications, including a Kernel ready status message.

Kernel ready

This means that the Python kernel is ready for your input.

1.9.4.3. Create a new notebook

To create a new notebook, click on the New button in the upper right corner of the dashboard, and select a Python kernel (here Python 2).

Create new notebook

A new browser tab will open with an empty notebook awaiting your input:

New notebook

If you want to experiment, you may want to create a copy - select the box to the left of a notebook name, and a “duplicate” option will appear.

1.9.4.4. Set notebook title (and .ipynb file name)

When you create a new notebook, it’s title will be Untitled (possibly with an auto-incrementing counter appended to it, e.g. Untitled3, etc.).

The title of a notebook also serves as the root of the notebook file name to be saved. It is a good idea to give the notebook a descriptive title, e.g. one that helps you recognize later what this notebook was all about.

To change the notebook title, click on the current title. A small window will pop up:

Change title

You can now edit the highlighted text (here Untitled), then click on the OK button to confirm your change

Title changed

Note the changed title (here Pythagoras). On the next save, the file name of this notebook will be Pythagoras.ipynb. (The standard extension is automatically forced.)

1.9.4.5. Edit and run a cell

Commands and text are entered into so-called cells inside a notebook.

Each cell can contain one or more (or many!) lines of input. The content of the cell will be only executed when you explicit tell it to do so (via keyboard shortcut Shift+ENTER).

An edited cell.

In the example above we have instructed the notebook to import the Data Lab authentication client, under a shortcut name ac. Once the input was typed in the cell, a Shift+Enter (pressed simultaneously) executed the content of the cell. The import was performed, and a new empty cell appeared beneath the first cell, awaiting new input.

When the execution of a cell does not return immediately, the cell number at the left changes temporarily to [*] during processing - during a long task this might be your only evidence that something is happening.

1.9.4.6. Set cell type

Each cell can be set to one of several input modes (e.g. code, markdown, etc.). The default is code (meaning valid Python commands). In the example below we switch to markdown which is good for formatting, headers, descriptive text, etc.

Change cell mode to markdown.

After changing the cell type to markdown we enter a large title (H1 size):

Edited markdown cell.

Finally, a Shift+ENTER keyboard shortcut executes this cell as well, rendering the markdown content:

Executed markdown cell.

As before, a new empty cell is created right underneath the rendered one.

You can also use keyboard shortcuts to switch the cell type. E.g., to change a cell type to markdown, press ESC,m. To change it to code, type ESC,y.

1.9.4.7. Run the entire notebook

To run all cells in a notebook, one after another, starting at the top, click on the Cell entry in the menu bar, then on Run all, as shown below:

Run all cells

The entire notebook will be executed.

1.9.4.8. Save and export

Your notebook is automatically saved after a set period of time (typically 120 seconds). If you wish to save the file manually at any given time, either type ESC,s or click on the Save and Checkpoint icon (left-most icon that looks like a floppy disk).

Save a notebook

A notebook .ipynb file also saves all output generated by the notebook cells. This makes a notebook very portable, and you can easily share it with others, email it to yourself, etc.

1.9.5. Logout from the notebook server

To log out from any of the Data Lab notebook servers, click on the Logout button in the upper-right corner (either from a notebook browser tab, or from the notebook dashboard tab).

Logout from the notebook server

The public server does not have a logout option - logout is autmoatic when the last open window is closed.

1.9.6. Troubleshooting

1.9.6.1. Restart the kernel

Sometimes you may wish to restart a kernel, i.e. restart the Python notebook you are currently running.

One reason for doing so is when you have changed some external .py file that your notebook is importing. If you try to import that file again, the new changes won’t be picked up (unless you explicitly reload that file). Another common reason to restart a running kernel is when there is some technical hiccup with the notebook; a restart of the kernel can help resolve the issues.

To restart a kernel, either type ESC,0,0 (Escape,zero,zero keys), or click in the menu bar on Kernel, then on Restart:

Restart kernel menu

Whether you used the mouse or the keyboard shortcut, a confirmation dialog will pop up:

Restart kernel dialog

Confirm your wish to indeed restart the notebook kernel by either clicking on the Restart button, or by hitting the ENTER key.

The kernel will be restarted, and you can re-run all relevant cells in it. (You must re-run the cells, since a kernel restart forgets the current state of memory variables).

1.9.6.2. Restart the notebook server

If you experience technical difficulties in executing your notebook, and you have already tried to remedy them by restarting the kernel (to no effect), then the problem might be on the Data Lab notebook server side.

if you are logged in as a registered user, then you can then try to restart the notebook server. For this, click on the Control Panel button in the upper-right corner (either from a notebook browser tab, or from the dashboard tab):

Control Panel button

You should see two buttons. Click on the red one saying Stop My Server:

Stop notebook server

Wait a few seconds until the red button disappeared. The notebook server has been shut down, and you can restart it my clicking on the remaining green button My Server:

Start notebook server

On the public server, quitting all windows has the effect of restarting the server - at the price of losing unsaved work!