ddobrinskiy’s Blog - nbdev formating and linting

Introduction

nbdev is a great tool.

However some of you may not be comfortable with the default fastai codestyle, or maybe your team is already using a different one.

This post will show you how to integrate your nbdev workflows with industry standard tools like:

black , The uncompromising code formatter
isort , A Python utility / library to sort imports.
mypy , Optional static typing for Python
pre-commit , A framework for managing and maintaining multi-language pre-commit hooks.

Note

An astute reader may notice that one can enable black_formatting = True in settings.ini

However,

this will only format the code in the module, not the code in the notebook.
this does not allow you to use other tools like isort or mypy

nbqa

use nbqa to format and verify code in your notebooks

nbqa is a tool that allows you to run any tool (formatter/linter) on your notebooks, without having to worry about the details.

black

Black is a python code formatter that is becoming the de-facto standard

pip install nbqa

nbqa black nbs/

isort and mypy

isort will sort your imports, and mypy will check your type annotations

pip install isort mypy

nbqa isort nbs/
nbqa mypy nbs/

Automating your flows with Makefile

But you are already using nbdev command-line arguments like nbdev_export and nbdev_test, who wants to remember more commands?

Make is a great tool for automating your workflows, usually comes pre-installed on most unix systems.

Create a Makefile in the root of your project and add the following lines:

Makefile

# Makefile
format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

Now you can run make format to format your notebooks.

Let’s add more commands to our Makefile:

Makefile

format:
    nbqa black nbs/
    nbqa isort nbs/
    nbqa mypy nbs/

export:
    nbdev_export

mypy:
    nbqa mypy nbs/ --ignore-missing-imports --check-untyped-defs

test:
    nbdev_test --n_workers 4

prepare:
    # Export, test, and clean notebooks, and render README if needed
    nbdev_prepare

docs:
    nbdev_docs

And now the cherry on top: Makefile supports dependecies, so we can chain commands together.

add this to your Makefile:

Makefile

all: format prepare mypy

Running make all will format your notebooks, export them, run mypy, run tests and update README.md

pre-commit

Want to automate your formatting even more?

Force your team to format their notebooks before committing?

Use pre-commit

install and configure pre-commit

pip install pre-commit # install 

pre-commit install # enable for this repo

create .pre-commit-config.yaml

.pre-commit-config.yaml

repos:
  - repo: https://github.com/fastai/nbdev
    rev: 2.2.10
    hooks:
      - id: nbdev_clean
      - id: nbdev_export

Add nbqa hooks

.pre-commit-config.yaml

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"]
        additional_dependencies: ["mypy", "types-toml", "types-requests"]
      - id: nbqa-black
      - id: nbqa-isort

control your hooks behaviour

.pre-commit-config.yaml

  - repo: https://github.com/nbQA-dev/nbQA
    rev: 1.5.3
    hooks:
      - id: nbqa-mypy
        args: ["--ignore-missing-imports", "--check-untyped-defs"] # pass args to mypy
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
        additional_dependencies: ["mypy", "types-toml", "types-requests"] # install additional dependencies
      - id: nbqa-black
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting
      - id: nbqa-isort
        exclude: "directory_to_exclude/.*" # eclude any directory/files from formatting

reproducible environments with Pipenv/Poetry/virtualenv

Sometimes you may run into reproducibility issues, where your notebook works on your machine, but not on your team’s.

The best way to avoid it is to use a dependency manager, such as:

But how do you integrate it with nbdev, since it uses settings.ini to manage dependencies?

In this blog post I show an example of how to use Pipenv with nbdev.

Concusion

I hope you find this post useful.

These changes may make nbdev more friendly to a team environment.

Please don’t hesitate to post any questions or comments below.