Version Controlling Your Data Science Projects

Easy Virtual Environments with Poetry

Poetry makes creating new environments easy with its poetry init command. This will ask a series of questions about your project which the tool will use to generate a pyproject.toml file. You can also choose to add any package dependencies when creating the environment. It's simple and often faster to add dependencies later, so I typically skip this step.

If there's not a console at the bottom of your VSCode window, open it with Ctrl+Shift+`, or Terminal -> New Terminal and run the initialization.

$ poetry init

This command will guide you through creating your pyproject.toml config.

Package name [data_analysis]:  my_awesome_analysis_repository
Version [0.1.0]:
Description []:
Author [Ryan Frazier <ryan@fotonixx.com>, n to skip]:
License []:
Compatible Python versions [^3.9]:  >3.8,<3.9

Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no

Once it's done, pyproject.toml will be generated in the root directory with the information you entered. By default Poetry will create the environment in a separate directory, but you can access it with poetry run. Test that your environment is running by executing poetry run py --version, and making sure it matches the version you specified.

$ poetry add jupyterlab numpy scipy matplotlib pandas ipykernel

Registering Our Environment as an IPython Kernel

Jupyter uses IPython Kernels running as a separate process to evaluate cells. In order for Jupyter to use our newly created virtual environment, we need to register it with a kernel using the ipykernel package. Run the below command, replacing name and display-name with appropriate values for the project.

$ poetry run python -m ipykernel install --user --name project_x_env --display-name "My Awesome Data Science Environment"

Now you can Launch Jupyter from any environment, including the global environment, and still access this environment's packages. If you don't have Jupyter installed globally, you can run the installation in our environment by executing poetry run jupyter lab. Alternatively, creating and opening an *.ipynb file in VSCode will enable the data science view.

Our newly created Kernel as a selectable option

Making a New Repository

To keep things simple we'll use VSCode's Git plugins to create, commit to, and push our repository. On the left side of the VSCode window you should see the directory structure with the pyproject and poetry.lock files. Depending on your Poetry settings the virtual environment may also be in this base directory.

Open the Command Pallette (Ctrl+Shift+P or f1 on Windows) and type "git init". There should be only one option that reads "Git: Initialize Repository". Press Enter and select the current folder to initialize the repository.

Notice that pyproject.toml and poetry.lock have turned green in the Explorer (Ctrl+Shift+E)! This is because VSCode knows you have a repository in the directory and that those files have not been staged for a commit. If you open the Source Control Panel (Ctrl+Shift+G) you'll see both those files listed as changes with a "U" on the right meaning they're unstaged. Before staging and making our initial commit, however, we want to add a few more files that will help flesh out the repository. These include:

• a readme.md file in the base directory

• a .gitignore file in the base directory

• a new directory /notebooks/ with a readme.md inside of it.

• a /data/ directory with an empty .gitkeep file inside of it.

When you've added the files your project directory should look similar to below:

Choosing which Files to Ignore

Git uses the .gitignore file to black-list specific files or even entire directories from being captured into version control. This helps keep the repository size small and only commit files that are necessary to reproduce the environment. As a rule of thumb the following should be excluded from your commits:

• Any IDE settings (the ./.vscode/ directory )

• Auto-generated files (*.pyc, file backups etc.)

• User specific environment files

• Large files that don't often change (downloaded datasets, third-party libraries)

It might seem alarming that datasets should not be part of version control, after all you have nothing to analyze without data! Further down we'll talk about how to synchronize data with the repository, but for now we'll ignore all files in the /data/ directory except for our whitelisted .gitkeep file.

# Ignore data directories
/data/*
!/data/.gitkeep

The rest of our .gitignore file is built off of GitHub's python.gitignore with the above additions to ignore our data directory, as well as VSCode settings, and Jupyter backup files. The entire file can be found here.

Creating a Git Hook to Prevent Messy Commits

While I love Jupyter for exploration and data analysis, one thing that always bothers me is how the code lives in the same file as evaluated evaluated output. When version controlling notebooks this can cause issues for a couple of reasons:

1. If the data you're working on is private but the code is public, the private data could end up in an output and committed, available for anybody to see.

2. Git works by logging differences in your file. This includes things like cell number, cell output, and picture metadata, if you version control an evaluated notebook you'll have unstaged changes as soon as you evaluate a cell, even though none of the written code actually changed!

We want a way to scrub our notebooks of all evaluated output before committing them. To do so we'll use githooks which are custom scripts that run when you perform a Git command. Flipping through the Githook documentation, the pre-commit hook is exactly what we need. Unfortunately, installing a Githook is a manual process that requires you to add a file to your /.git/ directory.

In order to make it as easy as possible for anybody to use this template, as well as make writing the githook simple, we'll instead use the pre-commit python package and write our hook with a YAML file that will live in the root of our version control. The pre-commit config we'll be using comes from Yuri Zhauniarovich's blog and uses nbconvert to scrub the output in-place.

Let's add pre-commit and nbconvert to our poetry environment. Since it's not needed to actually run the notebooks, and will only be used by people contributing to the codebase we'll install them as developer packages.

$ poetry add pre-commit nbconvert --dev

To define the hook, a new file in the base directory called .pre-commit-config.yaml and add the following text:

repos:
  - repo: local
    hooks:
      - id: jupyter-nb-clear-output
        name: jupyter-nb-clear-output
        files: \.ipynb$
        stages: [commit]
        language: system
        entry: poetry run jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace

The last piece is to install the githook so that it's run before every commit. Pre-commit makes this easy for us with it's install argument.

$ poetry run pre-commit install
pre-commit installed at .git\hooks\pre-commit

This is a good spot for our first commit! We've fleshed out the repository, filled our .gitignore and added most of our template files (even if they are empty). To commit using VSCode we'll again use the Source Control Panel (Ctrl+Shift+G). You can manually stage files by pressing the "+" icon to the right of the files, or you can stage all changes by clicking the "+" to the right of the "Changes" drop-down.

All commits should have a meaningful message so that you can look back and quickly understand what was changed. Chris Beam's Blog has a great post on the importance of a good commit message and how to write one, but for our first commit message we'll keep it simple with "initial commit".

With the files staged and commit message filled out, press the check-mark at the top of the panel to commit the changes.

Why Don't We Store Everything in the Git Repository?

To understand why we wouldn't want to store our large data files in a Git repository, lets peel back what happens when you clone an existing repository onto your local system. From Atlassian's Git-LFS tutorial:

The solution to this problem is to not version control the large files at all, but instead version control small reference files that tell you where the data lives so you only check-out the version you want, instead of the entire history. Git-LFS (Large File System) is one implementation of this strategy, but we're going to instead use the Data Version Control (DVC) Python package, which is specifically designed with data-analysis in mind.

Setting up DVC

Since DVC is a Python package, we can install it with Pip just like our other dependencies. The assumption is anybody using our repository will want access to the data, not just developers, so we won't both with the --dev flag either.

$ poetry add dvc

DVC is designed to mimic a Git workflow, so many of the commands you'd use for Git have DVC parallels. For example, to initialize DVC in our repository and add the data directory we use the init and add commands respectively.

$ poetry run dvc init
$ poetry run dvc add ./data/
~~~
To track the changes with git, run:
        git add data.dvc .gitignore

Notice the final output after adding the data directory, DVC is telling us that it's created a new file called data.dvc that's tracking changes to our data directory. It's also updated .gitignore automatically for us to ignore /data/ (Our previously committed .gitkeep will not be ignored since it's already tracked). DVC's offering a convenience in case we had forgotten to ignore the directory ourselves, but since we already had, let's revert our .gitignore to its previous state. In the Source Control Panel, right-click .gitignore and select "Discard Changes". This will reset the file to its state in the previous commit.

If you add new data to the directory, you can update data.dvc by running dvc add ./data/ again. Make sure to commit data.dvc as soon as you add new data, otherwise committed notebooks might lose sync with the data changes.

Adding a Remote

DVC allows you to back-up version-controlled data to remote servers, perfect for enabling multiple users to access the same version of a dataset. From the documentation:

I recommend setting up a default remote even if you're the only one looking at the dataset. Mine are usually directories on an internal network drive, but DVC has support for multiple storage types, so use whichever structure works best for you. The remote can even be a separate directory on you computer.

Once the remote is set-up, pushing to it is as simple as running dvc push <remote>. To pull from your remote you similarly run dvc pull <remote>.