Building Python Applications with Nix

Posted on May 9, 2018 by Rebecca Skinner

Introduction

Nix is a package general purpose package manager available for nix systems, including Linux (including NixOS), bsd, and MacOS. Nix provides several novel capabilities, specifically immutability and reproducability. A nix package defines exact versions of all of it’s dependencies, and a nix installation manages each individual application independently. In this post we’ll look at using Nix during the development process, comparing it to virtualenv and pip for developing a python application.

Background

Most software developers these days find ourselves working across a variety of applications written in different languages, with different dependencies, and targeting different platforms. The net result of this is a proliferation of tools designed to address the problem of how to manage our development environments. Most of these tools are specialized in one way or another. Tools like homebrew, apt, and chocolatey may help us install applications, but then we must turn to separate tools, often language specific, like pip, gem, and hex to install language dependencies. Tools like virtualenv, rbenv, and pkg-config help us manage separate tool and dependency versions. We may even use tools like Docker to make it easier to distribute our applications across different environments.

Some languages and toolchains, like go dep and haskell’s stack and cabal-install, manage versioning and dependencies within the scope of a project’s source tree, others like virtualenv, manage virtual environments with through environment variables or symlinks.

The nix package manager addresses several many of the problems that developers might otherwise need to use multiple tools to manage. Nix provides a mechanism to create immutable environments for building and running applications by combining package installation, dependency management, and environment setup all in a single tool. Nix can be used in place of tools like homebrew or apt to distribute and install packages, it can be used to manage local user-wide environment options like shell and editor configs, and it can be used to manage the build environment for specific applications. These configurations can be composed and combined into unique reproducible environments.

The rest of this article will focus on one particular use-case of nix: as a replacement for tools that manage virtual environments. In our example we will be developing a python application, and using nix in place of virtualenv to manage our applications dependencies.

Setting Up the Environment

Install Nix

Nix can be installed on most *nix systems from source or via a curl-bash command.

curl https://nixos.org/nix/install | sh

Once you’ve installed nix, you’ll want to configure your local environment to use nix. Add the following into your ${HOME}/.bashrc:

source /home/rebecca/.nix-profile/etc/profile.d/nix.sh

You can check that everything is working by running:

rebecca@debian:~$ which hello
rebecca@debian:~$ echo $? # demonstrate that "hello" wasn't installed
rebecca@debian:~$ nix-shell -p hello # start nix-shell including hello
[nix-shell:~]$ which hello # now it's found!
/nix/store/6mab2znnw7j96k3vsdw9vyckady29r46-hello-2.10/bin/hello
[nix-shell:~]$ exit

Create a development Environment

The sample project we’ll be using is a small Python 3 web appliation built using Flask. We’ll have minimal dependencies, in this case just python3 and the flask library.

We’ll start by creating a nixpkg called default.nix. The first thing we’ll want to do is import the list of nixpkgs that we know about in our environment:

with import<nixpkgs> {};

The with keyword allows us to bring a set of variables into scope in a statement. The import allows us to import another nix package. In our case, <nixpkgs> is an nix variable defined in the $NIX_PATH environment variable:

[nix-shell:~/projects/py-todo]$ echo ${NIX_PATH}
nixpkgs=/home/rebecca/.nix-defexpr/channels/nixpkgs

The import statement allows us to specify parameters that will be passed into the nixpkgs that we import. In our case, we don’t want to pass anything in so we simply use {} in our import statement.

Derivations

Derivations are the core of building nixpkgs. A derivation is, in short, a nix statement that tells us how to derive the environment for a given package. In our case, we want to derive our environment starting with the default standard environment, which will give us things like a bash shell and standard tools like make and gcc:

stdenv.mkDerivation rec {
  name = "env";
}

The rec keyword allows us to have recrusively defined references. This means that individual fields in our derivation can reference on another. Allowing these greatly simplifies the process of writing our derivations, at the cost of having nix expressions that may not terminate.

Declaring Dependencies

To declare our dependencies, we’ll first create a list of nix package names that we need for our environment. We can find the names of the packages that we want to install with nix-env.

Looking up the available python 3 packages is straightforward:

rebecca@debian:~/projects/py-todo$ nix-env -qaP python3
nixpkgs.python34      python3-3.4.8
nixpkgs.python34Full  python3-3.4.8
nixpkgs.python35      python3-3.5.5
nixpkgs.python35Full  python3-3.5.5
nixpkgs.python3       python3-3.6.5
nixpkgs.python36Full  python3-3.6.5
nixpkgs.python3Full   python3-3.6.5

Looking for the name of our Flask package is a bit more difficult since we’re not entirely sure what it might be called. We can use grep to broaden our search:

rebecca@debian:~/projects/py-todo$ nix-env -qaP | grep Flask | grep python3.6
nixpkgs.python36Packages.flask                                           python3.6-Flask-0.12.2
nixpkgs.python36Packages.flask_assets                                    python3.6-Flask-Assets-0.12
nixpkgs.python36Packages.flask-autoindex                                 python3.6-Flask-AutoIndex-0.6
nixpkgs.python36Packages.flaskbabel                                      python3.6-Flask-Babel-0.11.1
nixpkgs.python36Packages.flask-babel                                     python3.6-Flask-Babel-0.11.2
nixpkgs.python36Packages.flask_cache                                     python3.6-Flask-Cache-0.13.1
nixpkgs.python36Packages.flask-common                                    python3.6-Flask-Common-0.2.0
nixpkgs.python36Packages.flask-compress                                  python3.6-Flask-Compress-1.4.0
nixpkgs.python36Packages.flask-cors                                      python3.6-Flask-Cors-3.0.3
nixpkgs.python36Packages.flask_elastic                                   python3.6-Flask-Elastic-0.2
nixpkgs.python36Packages.flask-limiter                                   python3.6-Flask-Limiter-1.0.1
nixpkgs.python36Packages.flask_login                                     python3.6-Flask-Login-0.4.1
nixpkgs.python36Packages.flask_mail                                      python3.6-Flask-Mail-0.9.1
nixpkgs.python36Packages.flask_migrate                                   python3.6-Flask-Migrate-2.1.1
nixpkgs.python36Packages.flask_oauthlib                                  python3.6-Flask-OAuthlib-0.9.3
nixpkgs.python36Packages.flask_principal                                 python3.6-Flask-Principal-0.4.0
nixpkgs.python36Packages.flask-pymongo                                   python3.6-Flask-PyMongo-0.5.1
nixpkgs.python36Packages.flask-restful                                   python3.6-Flask-RESTful-0.3.6
nixpkgs.python36Packages.flask_script                                    python3.6-Flask-Script-2.0.6
nixpkgs.python36Packages.flask-silk                                      python3.6-Flask-Silk-0.2
nixpkgs.python36Packages.flask_sqlalchemy                                python3.6-Flask-SQLAlchemy-2.1
nixpkgs.python36Packages.flask_testing                                   python3.6-Flask-Testing-0.7.1
nixpkgs.python36Packages.flask_wtf                                       python3.6-Flask-WTF-0.14.2

Now that we know what packages we need to import, we can declare them in our list of dependencies, which We’ll call dependencies:

dependencies = [
  python3
  python36Packages.flask
]

Notice the pattern of our package names. python3 refers to the python3 package, which we can see is pinned to python3-3.6.5. Likewise, our Flask package, python36Packages.flask is pinned to python3.6-Flask-0.12.2. For python packages, the general pattern for package names will be pythonXYPackages.PackageName

Configuring our environment

The stdenv derivation will import several additional nix packages for us. Among them is the buildenv package, which allows us to setup a build environment. This build environment is what we’ll use for interactive development in our application.

The buildEnv package has several variables that we can set to fine-tune our build environment. For our purposes we are concerned with setting the name of our environment and the paths that we want to import into our working environment.

We set variables inside of a nix package using braces. In the snippet below we are bringing in the buildEnv package with our custom name and import paths:

env = buildEnv {
  name = name;
  paths = dependencies;
};

Our Finished nixpkg

Putting all of the above together, we find ourselves with a rather small nixpkg that brings in everything we need for our application:

with import<nixpkgs> {};
stdenv.mkDerivation rec {
  name = "env";
  dependencies = [
    python3
    python36Packages.Flask
  ];
  env = buildEnv {
    name = name;
    paths = dependencies;
  };
}

Running Our Application

Once we’ve set up our default.nix file, running our application inside of our nix environment is as simple as calling nix-shell. This will set up a local environment with our requested packages available in our $PATH so that all of our dependencies are available.

rebecca@debian:~/projects/py-todo$ nix-shell
[nix-shell:~/projects/py-todo]$ which python3
/nix/store/96wn2gz3mwi71gwcrvpfg39bsymd7gqx-python3-3.6.5/bin/python3
[nix-shell:~/projects/py-todo]$ which flask
/nix/store/swamiw3ygj9wws3spa44wgmjxhlldj3y-python3.6-Flask-0.12.2/bin/flask
[nix-shell:~/projects/py-todo]$ ./run_server
 * Serving Flask app "todo_server"
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Credits and Further Reading