Home Software 01 Github 02 Colabs 03 Shell Basics 04 Apis 05 Webscraping 06 Nbdev 07 Javascript Packag... 08 Cloud Functions 09 Browser Extension Css Css Animations Javascript Llm Paradigms Protocols Websites

Don't Look! I'm changing!

URL Copied

⚠ī¸ This document is now depricated.⚠ī¸

BinderBinderBinderOpen Source Love svg3

NPM LicenseActiveGitHub last commit

GitHub starsGitHub watchersGitHub forksGitHub followers

TweetTwitter Follow

!pip freeze > requirements.txt
from nbdev import *

(Non-Technical) Introduction

forthebadge made-with-python

ForTheBadge built-with-loveGitHub license

ForTheBadge powered-by-electricity

Fastai, the organization that created this tool, helps empower others by lowering the barrier of entry when breaking into software (AI) development.

In their own words:

Nbdev is a tool that allows you to fully develop a library in Jupyter Notebooks, putting all your code, tests, and documentation in one place. -Nbdev

The Nbdev library and tutorial is actually written using a notebook. They can be created and published by running the notebook on itself!

If you check, you can see that their tutorial is actually being hosted on GitHub pages.

Some of the tools that Fastai provides includes:

Are you using one of these products? Tweet @jeremyphoward and staff or @HamelHusain for all their work!

With Nbdev, you can:

  1. Publish notebooks as a module on PyPI.

  1. Install it anywhere: pip install <nameOfModule>

  1. Convert notebooks to documenation in HTML and Markdown.

  1. Use created modules across notebooks in realtime.

While we were on-topic!

If this is more than you need, consider the following:

(Technical) Overview

  1. Nbdev is a Python library that is executed directly from the terminal.

  2. At its heart, a user simply enters a directory and executes one of the Nbdev terminal commands.

  3. Python notebooks are automatically located and processed for publication and deployment.

  4. Specific "Flags" (either the Comment or "Magic" variety) tell Nbdev how to treat cells.

  5. Photos stored in the 'images' folder will be copied over to your publication folder.

Nbdev Terminal Commands

Entering these commands into the terminal at the project's root directory will handle most of everything.

Pip and Git specific steps are not listed, but are included in sections below.

Important: Information in this section is ever-changing! Quotes found here came from Nbdev's official documentation, from which you will derive much more insight with respect to using these functions.

Create

Compile

Nbdev ignores any file that starts with an underscore. Version bumping must be performed each time prior to uploading to PyPI.

Combine commands with with regular expressions for more power.

Publish

GitHooks must run once on project creation for them is complete so that can be set up permanently.

Check

nbdev_upgrade can run any time there are new updates, through it only needs to run once per update.

Nbdev Flags (Mark Up/Cell Magic)

Nbdev scripts rely on special comments or magics, depending on your choice, for cell-level handling of notebooks.

Whereas Nbdev's terminal commands are executed at a project's root directory, content covered in this section is placed in the notebooks' cells.

Most Nbdev flags define design instructions for when we create the website/module (Ex. default_exp, autoreload).

1. Basic Comment and Magics

Warning: Content in this section is pulled from the Fastpages blog posts and the Nbdev docs. If you have any issues, you will have to scoure these sources to resolve them!

To gain access to Nbdev's new 'magics', you my find it neccesary to execute the command nbdev_upgrade only once; add from nbdev import * to the top of a notebook in a code-cell (this article covers that); and put Flags at the top of a cell for it to work.

Important: The list shown below was obtained from here. As always, please refer to the official source for up to date information.

Comment flagMagic flag
default_expnbdev_default_exportDefine the name of the module everything should be exported in.
exportsnbdev_export_and_showExport and show code in the docs.
exportinbdev_export_internalExport but do not show in docs and or add to __all__.
exportnbdev_exportExport but do not show in docs.
hide_inputnbdev_hide_inputDo not show input of a test cell in docs.
hide_outputnbdev_hide_outputDo not show output of a test cell in docs.
hidenbdev_hideDo not show a test cell or markdown in docs.
default_cls_lvlnbdev_default_class_levelDefine the default toc level of classes.
collapse_output open or collapse-outputnbdev_collapse_outputInlcude output in the docs under a collapsable element.
collapse_input close or collapse-inputnbdev_collapse_outputInlcude input in the docs under a collapsable element.
collapse_show or collapse-shownbdev_collapse_input openInlcude input in the docs under a collapsable element that is open by default.
collapse_hide or collapse-hidenbdev_collapse_inputInlcude input in the docs under a collapsable element.
collapsenbdev_collapse_inputInlcude input in the docs under a collapsable element.

2: Autoreloading

Put %load_ext autoreload %autoreload 2 in ipynbs. If you are working the notebook and importing the local version of your library, the imported modules will auto update whenever the corresponding .py file gets updated.

Note: This opens new doors in your programming experience.

Provided all the notebooks have access to the same file directory:

  1. You can edit .py's and use update_lib to push the exported code's edits back into their original ipynbs (ran anywhere)

  2. You can edit .IPYNB's and use build_lib to push the exported code's edits back into their .pys (ran anywhere)

The autoreload feature will let you immediately test the new modules from any desired notebook.

3. Console Scripts are made by marking up functions using the fastscript tool and editing the settings.ini file (see this example).

4. Anchor Links

Anchor links are web links that allow users to leapfrog to a specific point on a website page. They save them the need to scroll and skim-read, making navigation easier. - Telegraph

For example, Both landmark and hyperlink have been placed on this line. Click the link for this line to zip up to the top of your screen.

Code:

undefined

5: Jekyll Automated Document Hyperlinking

Mentioning an exported class with backticks functionName will create a hyperlink to its sourcode on GitHub. This will only work if it is an exported function.

6. Jekyll Metadata

The cell #default_exp located at the top of a cell can also be used for YAML, a data-serialization standard. This is used as Front Matter that can be used with Jekyll to render templates.

In the markdown cell with the title, you can add the summary as a block quote by putting an empty block quote for an empty summary. Next, add a list with any additional metadata you would like to add get_metadata.

7. Jekyll Images

The 'images' folder can be used to conjure up pictures, with an example provided below:

![](notebookfolder/images/company_logo.png)

These images will be added to and displayed in the docs.

8. Jekyll Notes

The following section was written using the following markups:

> Note:, > Warning:, > Tip: and > Important: text goes here

Note: This is a note.

Warning: This is a warning.

Tip: This is a tip.

Important: This is an Important doc link to add_jekyll_notes, which should also work fine.

9. Jekyll Search

This is functionality not covered here, but may be added in the future.

10. Jekyll Custom Sidebar

This is used for your HTML documentation, which can be configured using meta-markup JSON.

The default sidebar lists all html pages with their respective title, except the index that is named "Overview". To build a custom sidebar, set the flag custom_sidebar in your settings.ini to True then change the sidebar.json file in the doc_folder to your liking. Otherwise, the sidebar is updated at each doc build.

Getting Started

To get started, install the Nbdev libarary.

Connect to Folder

Give Colabs access to your Google Drive

from google.colab import drive
 drive.mount('/content/drive')

Navigate to the Google Drive directory where you store your projects folder.

Creating a New Project

The Nbdev library will only work with projects that have the proper file structure and files.

Once configured, you may simply write in a notebook (placing Nbdev 'FLAGS' where needed) and publish using an Nbdev terminal commands. The Nbdev command will process the notebook and each of its cells according to rules denoted by the 'FLAGS'.

Getting a template

You can grab a free project template containing all the needed files and proper structure. With this, you can get started in one of two methods shown below.

Method 1) Grabbing a template using GitHub and the browser.

Now, create a new repository.

! git clone https://github.com/fastai/nbdev_template.git
ls
!mv nbdev_template VitalSigns
 !mv VitalSigns ../z
!cp RBIntel.ipynb ../VitalSigns
cd ../VitalSigns
/content/drive/My Drive/VitalSigns
!ls
00_core.ipynb docs MANIFEST.in settings.ini CONTRIBUTING.md index.ipynb RBIntel.ipynb setup.py docker-compose.yml LICENSE README.md

Method 2) Grabbing a template using the Nbdev library and the terminal.

!git init
Initialized empty Git repository in /content/drive/MyDrive/vitalSigns/.git/

An Nbdev terminal command can be used to create a template project directory.

Use -h to view positional args.

cd ../
! nbdev_new -h
Click to toggleusage: nbdev_new [-h] Create a new nbdev project from the current git repo optional arguments: -h, --help show this help message and exit

We will need to register your GitHub credentials appropraitely configured for it to work.

Click to toggle
! git config --global user.email "charles.karpati@gmail.com"
 ! git config --global user.name "bnia"

Now, you can just run the command to create a new project/directory by a name of your choosing.

! nbdev_new 'test123'

If you enter it, you will see it comes all set up.

!cd VitalSigns
/bin/bash: line 0: cd: VitalSigns: No such file or directory

As you can see, it comes all set up.

ls

Configuring the Template

There are some one-offs to perform on the template.

GitHooks

Run ! nbdev_install_git_hooks

To set up GitHooks that will remove metadata from your notebooks when you commit, greatly reducing the chance you have a conflict.

But if you do get a conflict later, simply run nbdev_fix_merge filename.ipynb. This will replace any conflicts in cell outputs with your version. If there are conflicts in input cells, then both cells will be included in the merged file, along with standard conflict markers (e.g. =====). Then, you can open the notebook in Jupyter and choose which version to keep. ~ nbdev tutorial

Settings.ini

Everything is included in the template for the library to be packaged.

Your settings.ini is where all parts of nbdev look for any required configuration information.

Complete the Python form below to update the settings.ini file.

Really, only the 'folder_name' and 'github_username' fields need to be edited for this to work.

The form values entered here will be inserted into the settings.ini doc.

Click to toggle
#@title Example form fields
 #@markdown Forms support many types of fields.
 
 # Name of the project
 folder_name = "VitalSigns"  #@param {type:"string"}
 company_name = "BNIA-JFI" #@param {type:"string"}
 # GitHub Username
 github_username = 'bniajfi'  #@param {type:"string"}
 description = "Python Scripts for BNIA-JFI's Vital Signs Data"  #@param {type:"string"}
 keywords = "Community Data"  #@param {type:"string"}
 # Who are you?
 author = "Charles Karpati"  #@param {type:"string"}
 author_email = 'charles.karpati@gmail.com' #@param {type:"string"}
 # Where are your notebooks? They are currently at the basepath.
 path_to_locate_notebooks = "."  #@param {type:"string"}
 # Where should your documentation be put? {type:"string"}
 path_to_place_documentation = "docs" #@param {type:"string"}
 #@markdown ---
 
 user=github_username
 nbs_path=path_to_locate_notebooks
 doc_path=path_to_place_documentation
 lib_name=folder_name
 
 # Now let's rewrite the settings.ini file.
 
 innertext = """
 [DEFAULT]
 # All sections below are required unless otherwise specified
 host = github
 lib_name = """+lib_name+"""
 company_name = """+company_name+"""
 
 user = """+user+"""
 description = """+description+"""
 keywords = """+keywords+"""
 author = """+author+"""
 author_email = """+author_email+"""
 copyright = MIT
 branch = master
 version = 0.0.1
 min_python = 3.6
 audience = Developers
 language = English
 # Set to True if you want to create a more fancy sidebar.json than the default
 custom_sidebar = False
 # Add licenses and see current list in `setup.py`
 license = apache2
 # From 1-7: Planning Pre-Alpha Alpha Beta Production Mature Inactive
 status = 2
 
 # Optional. Same format as setuptools requirements
 # requirements = 
 # Optional. Same format as setuptools console_scripts
 # console_scripts = 
 # Optional. Same format as setuptools dependency-links
 # dep_links = 
 
 ###
 # You probably won't need to change anything under here,
 #   unless you have some special requirements
 ###
 
 # Change to, e.g. "nbs", to put your notebooks in nbs dir instead of repo root
 nbs_path = """+nbs_path+"""
 doc_path = """+doc_path+"""
 
 # Whether to look for library notebooks recursively in the `nbs_path` dir
 recursive = False
 
 # Anything shown as '%(...)s' is substituted with that setting automatically
 doc_host =  https://%(user)s.github.io
 #For Enterprise Git pages use:  
 #doc_host = https://pages.github.%(company_name)s.com.  
 
 
 doc_baseurl = /%(lib_name)s/
 # For Enterprise Github pages docs use:
 # doc_baseurl = /%(repo_name)s/%(lib_name)s/
 
 git_url = https://github.com/%(user)s/%(lib_name)s/tree/%(branch)s/
 # For Enterprise Github use:
 #git_url = https://github.%(company_name)s.com/%(repo_name)s/%(lib_name)s/tree/%(branch)s/
 
 
 
 lib_path = %(lib_name)s
 title = %(lib_name)s
 
 #Optional advanced parameters
 #Monospace docstings: adds 
 tags around the doc strings, preserving newlines/indentation.
 #monospace_docstrings = False
 #Test flags: introduce here the test flags you want to use separated by |
 #tst_flags = 
 #Custom sidebar: customize sidebar.json yourself for advanced sidebars (False/True)
 #custom_sidebar = 
 #Cell spacing: if you want cell blocks in code separated by more than one new line
 #cell_spacing = 
 #Custom jekyll styles: if you want more jekyll styles than tip/important/warning, set them here
 #jekyll_styles = note,warning,tip,important
 """
 
 # Write-Overwrites 
 file1 = open("settings.ini", "w")  # write mode 
 file1.write(innertext) 
 file1.close() 

Edit the Notebooks

To start, notice how there are two .ipynb's.

These are two notebooks used for the template.

  1. index.ipynb - When index.ipynb gets converted to 'index.html' it becomes the documentations homepage since browsers understand 'index.html' to be a website's homepage. When published, this notebook will also be made into the README.md document that can be shown on GitHub repositories pages and PyPI. Once published to PyPI, the index notebook (i.e. the homepage) can be used to show others how to install and use the library!

  2. 00_core.ipynb - This is the 'core' component of the python library. You don't have to keep the 'core' part. The left two digits in the filename help the website navigation so you can increment your notebooks by the order you want them displayed. The index.ipynb is the only notebook that does not have this logic apply to it. Inside the notebook, you will see a template near ready for deployment. If the lib_name was to be replaced and a function was declared with an # export flag at the top of its cell; this would be ready for deployment and publishing to PyPI!

! mkdir ./notebooks
 ! mv ./00_core.ipynb ./notebooks/
 ! mv index.ipynb ./notebooks/
!ls
CONTRIBUTING.md docs LICENSE RBIntel.ipynb settings.ini docker-compose.yml index.ipynb MANIFEST.in README.md setup.py

Create the library and documentation.

Install libraries used in your exported module here, or else the next spet will not work.

For now, since we are working with the start template, this should not be a problem.

Once ready, run ! nbdev_build_lib

!ls
CONTRIBUTING.md docs LICENSE RBIntel.ipynb settings.ini docker-compose.yml index.ipynb MANIFEST.in README.md setup.py
! nbdev_build_lib --fname RBIntel.ipynb
!ls
CONTRIBUTING.md docs LICENSE RBIntel.ipynb settings.ini docker-compose.yml index.ipynb MANIFEST.in README.md setup.py

Congratulations! You have now successfully created a Python library.

The library itself is still empty. In order to export code, be sure to use the #export Flag on cells.

Be sure to explore the Flags and Commands!

cd ../dataguide
!ls

Configuring Git-Github

! git init

In order to add an existing project to GitHub:

One you do create the GitHub repository through the website, you can connect the your Git repository to the GitHub repository using this command:

# Make replacements whereever needed and run this! Be sure to remove our password once this is run.
 
 # ! git remote add ORIGIN https://:@github.com//.git

You will now be able to commit this local repository to GitHub.

Create a PyPI account

In order for you to be able to publish to PyPI, you must have an account. Click "register" on the top right corner of their website to build an account.

Your username should be the user value from your settings.ini file.

Publishing!

Refer to the Build Proccess, PyPI, and Git sections below.

Pipeline 1/3: NBDEV

(Run once Modules, Docs, and the README are created using NBDEV and are ready to be published onto the web.)

You need to install the libraries in this nb so that they are used in your other nb.

Enter the project if it exists. Otherwise, go to the folder you want it to exist at.

cd ../
/content/drive/MyDrive/Sites
!ls
%cd dataplay
/content/drive/MyDrive/Sites/dataplay
%cd dataguide
%cd notebooks
%cd ../
!ls
build dist LICENSE py2html CONTRIBUTING.md docs Makefile README.md dataplay index.html MANIFEST.in settings.ini dataplay.egg-info index.ipynb notebooks setup.py

As long as you are wherever you are developing the library in your folder, both of these commands will work:

# first. builds the .py files from from .ipynbs
 !nbdev_build_lib
# second. Push .pu changes back to their original .ipynbs
 !nbdev_update_lib 
# nbdev_build_docs builds the documentation from the notebooks
 !nbdev_build_docs --force_all True --mk_readme True 
# sometimes. Update .ipynb import statements if the .py filename.classname changes. 
 # !relimport2name
# this will reload imported modules whenever the .py file changes. 
 # whenever the .py file changes via nbdev_build_lib or _update_lib. 
 # %load_ext autoreload
 # %autoreload 2
%cd notebooks
! nbdev_clean_nbs
 ! nbdev_fix_merge 00_github.ipynb
 ! nbdev_fix_merge 01_colabs.ipynb
 ! nbdev_fix_merge 01_5_Explore_and_Download.ipynb
 ! nbdev_fix_merge 02_scooterExploration.ipynb
 ! nbdev_fix_merge 03_nbdev.ipynb
 ! nbdev_fix_merge index.ipynb
 ! nbdev_clean_nbs
 ! find . -name "*.bak" -type f -delete
# nbdev_nb2md(fname:"A notebook file name to convert", dest:"The destination folder"='.', img_path:"Folder to export images to"='', jekyll:"To use jekyll metadata for your markdown file or not"=False)
 ! nbdev_nb2md 00_github.ipynb --dest "../markdown" 
 ! nbdev_nb2md 01_colabs.ipynb --dest "../markdown" 
 ! nbdev_nb2md 02_scooterExploration.ipynb --dest "../markdown" 
 ! nbdev_nb2md 03_nbdev.ipynb --dest "../markdown" 
 ! nbdev_nb2md index.ipynb --dest "../markdown" 
 ! find . -type d -name "*files" -exec rm -rf {} \;
%cd ../
!ls
build dataplay.egg-info LICENSE notebooks settings.ini CONTRIBUTING.md dist Makefile py2html setup.py dataplay docs MANIFEST.in README.md

Pipeline 2/3: GIT

Before you push, run nbdev_clean_nbs to ensure that the push will work. Of course, that is ensured by nbdev_fix_merge.

# This code was meant to do what the code above it is doing it. Kept for posterity.
 # fix_conflicts('notebooks/index.ipynb') # This is ran as a method within nbdev_fix_merge
 # from nbdev.export import *
 # from nbdev.merge import *
 # tst_nb = read_nb('notebooks/index.ipynb')
 # print(tst_nb)

Run once the Modules, Docs and README have been created using NBDEV but not published.

# cd datalab
! git add *
ls
!git config --global user.name "bnia"
!git config --global user.email "charles.karpati@gmail.com"
! git commit -m "Collapse Experiments P2"
# git push -f origin master
! git push -u ORIGIN main

If you get the following error "fatal: could not read Username for 'https://github.com': No such device or address", you will need to re-establish your Git-GitHub connection.

Start by running ! git remote rm ORIGIN, then re-add like so ! git remote add ORIGIN https://<USER NAME>:<passwrd>@github.com/<USER NAME>/<REPO NAME>.git.

Be sure to make the replacements whereever needed and run this. Once this is running, be sure to remove the password.

If GitHub pages does not show, consult this guide. Typically, this is a problem with the Markdown.

Pipeline 3/3: PyPI

Run once the Modules, Docs and README have been created using NBDEV.

Be sure you have a PyPI account that has the same username as your GitHub account before continuing.

You could save your credentials into a .pypirc file, but this is not recommended.

# ! echo "[pypi]" > .pypirc
 # ! echo "username = 'username'" >> .pypirc
 # ! echo "password = 'password'" >> .pypirc

To publish to Pypi, install twine.

Be sure to run this final nbdev command ! nbdev_bump_version prior to publishing. Otherwise, you might have issues.

!nbdev_bump_version
Old version: 0.0.20 New version: 0.0.21

Other than what has been discussed above, Nbdev has everything else all set up! Simply run make PyPI and enter your credentials when prompted at the bottom of the terminal output. Your password will be censored. Therefore, it is safe to post this code online without clearing it.

! make pypi
ls

Misc Tests

These things don't work... yet?

from IPython.display import display
var = "hide"
 display(var)
var = "hide_output"
 print('The input of this cell is visible as usual.\nWhile the OUTPUT of this cell is collapsed by default, you can expand it!')
var = "collapse_output"
 display(var)
#collapse_show
 var = "collapse_show"
 display(var)
#collapse_hide
 var = "collapse_hide"
 display(var)
#collapse-hide 
 print('The input of this cell is visible as usual.\While the OUTPUT of this cell is collapsed by default, you can expand it!')
Click to toggle
var = "collapse_input"
 display(var)
Click to toggle
var = "collapse_input open"
 display(var)
var = "collapse_output"
 display(var)
var = "collapse_output open"
 display(var)
print('The input of this cell is visible as usual.\nWhile the OUTPUT of this cell is collapsed by default, you can expand it!')
summary and details togetherdetails 1
summary and details split

summary and details split

Entirely Split

Entirely Split

# md within html**md within html**
print('The input of this cell is visible as usual.\nWhile the OUTPUT of this cell is collapsed by default, you can expand it!')
%%html
 
code html collapse output code html collapse output