Don't Look! I'm changing!

URL Copied

Charles Karpati | Nbdev

In this chapter, we will learn about publishing python notebooks with [Nbdev](http://nbdev.fast.ai/tutorial/).

(Non-Technical) Introduction

Fastai, the organization that created this tool, helps empower others by lowering the barrier of entry when breaking into software (AI) development.

In their own words:

Nbdev is a tool that allows you to fully develop a library in Jupyter Notebooks, putting all your code, tests, and documentation in one place. -Nbdev

The Nbdev library and tutorial is actually written using a notebook. They can be created and published by running the notebook on itself!

If you check, you can see that their tutorial is actually being hosted on GitHub pages.

Some of the tools that Fastai provides includes:

• Nbdev - For software development and documentation

• Fastpages - Works like Nbdev but for blogging

• Fastscript - Converts a python function to a CLI script

• Fastprogress - Reveals a progress bar for long processes

• FastiAi - Data convenience tools and AI models

• FastAi Course - V3; a practical deep-learning course

Are you using one of these products? Tweet @jeremyphoward and staff or @HamelHusain for all their work!

With Nbdev, you can:

1. Publish notebooks as a module on PyPI.

• A) "Flag" cells to export them as part of the Python library

2. Install it anywhere: undefined

• A) This can be used in conjunction with Fastscript to create CLI scripts

• B) The CLI help interface is reachble by typing undefined

3. Convert notebooks to documenation in HTML and Markdown.

• A) Use the index.ipynb to auto-generate a homepage and a Github and PyPI README.md file.

• C) Have the backticked function auto-link to a functioning GitHub source code.

• D) Host the documentation on GitHub pages so they are always up to date.

• E) "Flag" cells to either collapse or hide the cells or the output from documentation.

• F) Personalize the Jekyll website templates used to make the site.

• G) Store photos in an images folder for use in your notebook and have it copied to your website.

4. Use created modules across notebooks in realtime.

• A) Auto-reload modules across notebooks during development if a their corresponding .py script is updated

While we were on-topic!

If this is more than you need, consider the following:

• Sharing the Colab links directly.

• Nbconvert notes into HTML, LaTeX, PDF, Markdown, etc.

• Using mybinder

  Have a repository full of Jupyter notebooks? With Binder, open those notebooks in an executable environment, making your code immediately reproducible by anyone, anywhere.

(Technical) Overview

1. Nbdev is a Python library that is executed directly from the terminal.

2. At its heart, a user simply enters a directory and executes one of the Nbdev terminal commands.

3. Python notebooks are automatically located and processed for publication and deployment.

4. Specific "Flags" (either the Comment or "Magic" variety) tell Nbdev how to treat cells.

5. Photos stored in the 'images' folder will be copied over to your publication folder.

Nbdev Terminal Commands

Entering these commands into the terminal at the project's root directory will handle most of everything.

Pip and Git specific steps are not listed, but are included in sections below.

Important: Information in this section is ever-changing! Quotes found here came from Nbdev's official documentation, from which you will derive much more insight with respect to using these functions.

Create

• undefined: creates a new Nbdev project.

• undefined: propagates any change in the library back to the notebooks.

Compile

• undefined: converts a notebook to a Markdown file.

• undefined: builds the documentation from the notebooks.

• undefined: builds the library from the notebooks.

• undefined/ undefined: increments version in settings.py by one.

Nbdev ignores any file that starts with an underscore. Version bumping must be performed each time prior to uploading to PyPI.

Combine commands with with regular expressions for more power.

• A commands like undefined will run all files except ones with filenames that start with 'test'

Publish

• undefined: removes all superfluous metadata form the notebooks to avoid merge conflicts.

• undefined: installs the git hooks that use ( undefined and undefined) automatically on each commit/merge.

GitHooks must run once on project creation for them is complete so that can be set up permanently.

Check

• undefined: reads all notebooks to make sure none are broken.

• undefined: trusts all notebooks so that the HTML content is shown.

• undefined: updates an existing Nbdev project to use the latest features.

nbdev_upgrade can run any time there are new updates, through it only needs to run once per update.

Nbdev Flags (Mark Up/Cell Magic)

Nbdev scripts rely on special comments or magics, depending on your choice, for cell-level handling of notebooks.

Whereas Nbdev's terminal commands are executed at a project's root directory, content covered in this section is placed in the notebooks' cells.

Most Nbdev flags define design instructions for when we create the website/module (Ex. undefined, undefined).

1. Basic Comment and Magics

Warning: Content in this section is pulled from the Fastpages blog posts and the Nbdev docs. If you have any issues, you will have to scoure these sources to resolve them!

To gain access to Nbdev's new 'magics', you my find it neccesary to execute the command undefined only once; add undefined to the top of a notebook in a code-cell (this article covers that); and put Flags at the top of a cell for it to work.

Important: The list shown below was obtained from here. As always, please refer to the official source for up to date information.

2: Autoreloading

Put undefined undefined in ipynbs. If you are working the notebook and importing the local version of your library, the imported modules will auto update whenever the corresponding .py file gets updated.

Note: This opens new doors in your programming experience.

Provided all the notebooks have access to the same file directory:

1. You can edit .py's and use undefined to push the exported code's edits back into their original ipynbs (ran anywhere)

2. You can edit .IPYNB's and use undefined to push the exported code's edits back into their .pys (ran anywhere)

The autoreload feature will let you immediately test the new modules from any desired notebook.

3. Console Scripts are made by marking up functions using the fastscript tool and editing the settings.ini file (see this example).

4. Anchor Links

Anchor links are web links that allow users to leapfrog to a specific point on a website page. They save them the need to scroll and skim-read, making navigation easier. - Telegraph

For example, Both landmark and hyperlink have been placed on this line. Click the link for this line to zip up to the top of your screen.

Code:

5: Jekyll Automated Document Hyperlinking

Mentioning an exported class with backticks undefined will create a hyperlink to its sourcode on GitHub. This will only work if it is an exported function.

• Functions and Classes have are automatically documented.

• Class methods are not shown by default, so use undefined) to do this.

6. Jekyll Metadata

The cell undefined located at the top of a cell can also be used for YAML, a data-serialization standard. This is used as Front Matter that can be used with Jekyll to render templates.

In the markdown cell with the title, you can add the summary as a block quote by putting an empty block quote for an empty summary. Next, add a list with any additional metadata you would like to add undefined.

7. Jekyll Images

The 'images' folder can be used to conjure up pictures, with an example provided below:

undefined

These images will be added to and displayed in the docs.

8. Jekyll Notes

The following section was written using the following markups:

undefined, undefined, undefined and undefined

Note: This is a note.

Warning: This is a warning.

Tip: This is a tip.

Important: This is an Important doc link to undefined, which should also work fine.

9. Jekyll Search

This is functionality not covered here, but may be added in the future.

10. Jekyll Custom Sidebar

This is used for your HTML documentation, which can be configured using meta-markup JSON.

The default sidebar lists all html pages with their respective title, except the index that is named "Overview". To build a custom sidebar, set the flag custom_sidebar in your settings.ini to True then change the sidebar.json file in the doc_folder to your liking. Otherwise, the sidebar is updated at each doc build.

Getting Started

To get started, install the Nbdev libarary.

Connect to Folder

Give Colabs access to your Google Drive

Navigate to the Google Drive directory where you store your projects folder.

Creating a New Project

The Nbdev library will only work with projects that have the proper file structure and files.

Once configured, you may simply write in a notebook (placing Nbdev 'FLAGS' where needed) and publish using an Nbdev terminal commands. The Nbdev command will process the notebook and each of its cells according to rules denoted by the 'FLAGS'.

Getting a template

You can grab a free project template containing all the needed files and proper structure. With this, you can get started in one of two methods shown below.

Method 1) Grabbing a template using GitHub and the browser.

Now, create a new repository.

Method 2) Grabbing a template using the Nbdev library and the terminal.

An Nbdev terminal command can be used to create a template project directory.

Use -h to view positional args.

We will need to register your GitHub credentials appropraitely configured for it to work.

Now, you can just run the command to create a new project/directory by a name of your choosing.

If you enter it, you will see it comes all set up.

As you can see, it comes all set up.

Configuring the Template

There are some one-offs to perform on the template.

GitHooks

Run undefined

To set up GitHooks that will remove metadata from your notebooks when you commit, greatly reducing the chance you have a conflict.

But if you do get a conflict later, simply run nbdev_fix_merge filename.ipynb. This will replace any conflicts in cell outputs with your version. If there are conflicts in input cells, then both cells will be included in the merged file, along with standard conflict markers (e.g. =====). Then, you can open the notebook in Jupyter and choose which version to keep. ~ nbdev tutorial

Settings.ini

Everything is included in the template for the library to be packaged.

Your settings.ini is where all parts of nbdev look for any required configuration information.

Complete the Python form below to update the undefined file.

Really, only the 'folder_name' and 'github_username' fields need to be edited for this to work.

The form values entered here will be inserted into the settings.ini doc.

Edit the Notebooks

To start, notice how there are two undefined's.

These are two notebooks used for the template.

1. undefined - When index.ipynb gets converted to 'index.html' it becomes the documentations homepage since browsers understand 'index.html' to be a website's homepage. When published, this notebook will also be made into the README.md document that can be shown on GitHub repositories pages and PyPI. Once published to PyPI, the index notebook (i.e. the homepage) can be used to show others how to install and use the library!

2. undefined - This is the 'core' component of the python library. You don't have to keep the 'core' part. The left two digits in the filename help the website navigation so you can increment your notebooks by the order you want them displayed. The index.ipynb is the only notebook that does not have this logic apply to it. Inside the notebook, you will see a template near ready for deployment. If the lib_name was to be replaced and a function was declared with an undefined at the top of its cell; this would be ready for deployment and publishing to PyPI!

Create the library and documentation.

Install libraries used in your exported module here, or else the next spet will not work.

For now, since we are working with the start template, this should not be a problem.

Once ready, run undefined

• You will now find that you have a new directory, with the name of whatever you set lib_name to.

Congratulations! You have now successfully created a Python library.

The library itself is still empty. In order to export code, be sure to use the undefined Flag on cells.

Be sure to explore the Flags and Commands!

Configuring Git-Github

In order to add an existing project to GitHub:

• Create a new repository with the repositories name the same as the new library.

• Ensure the repository is absolutely empty so we can make a clean push.

One you do create the GitHub repository through the website, you can connect the your Git repository to the GitHub repository using this command:

You will now be able to commit this local repository to GitHub.

Create a PyPI account

In order for you to be able to publish to PyPI, you must have an account. Click "register" on the top right corner of their website to build an account.

Your username should be the undefined value from your settings.ini file.

Publishing!

Refer to the Build Proccess, PyPI, and Git sections below.

Pipeline 1/3: NBDEV

(Run once Modules, Docs, and the README are created using NBDEV and are ready to be published onto the web.)

You need to install the libraries in this nb so that they are used in your other nb.

Enter the project if it exists. Otherwise, go to the folder you want it to exist at.

As long as you are wherever you are developing the library in your folder, both of these commands will work:

Pipeline 2/3: GIT

Before you push, run undefined to ensure that the push will work. Of course, that is ensured by undefined.

Run once the Modules, Docs and README have been created using NBDEV but not published.

If you get the following error "fatal: could not read Username for 'https://github.com': No such device or address", you will need to re-establish your Git-GitHub connection.

Start by running undefined, then re-add like so undefined.

Be sure to make the replacements whereever needed and run this. Once this is running, be sure to remove the password.

If GitHub pages does not show, consult this guide. Typically, this is a problem with the Markdown.

Pipeline 3/3: PyPI

Run once the Modules, Docs and README have been created using NBDEV.

Be sure you have a PyPI account that has the same username as your GitHub account before continuing.

You could save your credentials into a .pypirc file, but this is not recommended.

To publish to Pypi, install twine.

Be sure to run this final nbdev command undefined prior to publishing. Otherwise, you might have issues.

Other than what has been discussed above, Nbdev has everything else all set up! Simply run make PyPI and enter your credentials when prompted at the bottom of the terminal output. Your password will be censored. Therefore, it is safe to post this code online without clearing it.

Misc Tests

These things don't work... yet?

Thank's for visiting! - Terms of use