Python Development Environment 2020

Introduction

The objective of this post is to demonstrate my python development environment and describe the various tools I use to manage Python versions/packages.

I will not go into detail and behind scenes to understand each tool but I will try to instead summarize in 1 or 2 sentences the problem and the solution for each puzzle.

Installing Python with Homebrew

Installing Python using Homebrew brew install python3 means you will have only a single version of python and you have no control over what that version is.

Homebrew installs all packages into its own directory at /usr/local/Cellar. To see the current (active) version installed run the following command:

Copycopy code to clipboard
1brew --cellar python

In some cases you might find multiple versions installed under /usr/local/Cellar/python@3.8 and /usr/local/Cellar/python@3.9 that's because Homebrew updated the Python version. However, the python3 command will be overwritten each time.

Since I am on OS X Homebrew is my default package manager and many other packages have Python3 as a dependency I have an installation of Python via Homebrew on my system as well.

For example, installing python bindings for QT brew install pyqt will install python@3.9 as its dependency.

pyenv

Allows you to install many different versions of Python on the same system. To install pyenv follow the official installation guideline.

Note you can easily install pyenv with Homebrew - pyenv does not depend on Python itself.

Copycopy code to clipboard
1brew install pyenv
2brew install pyenv-virtualenv
3
4echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
5echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~.zshrc
6echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.zshrc
7exec $SHELL

Python for Neovim

Neovim requires a package to be installed for Python plugins to work. Since I'm using virtualenv for all of my work, and I like to keep my environments organized and clean, I have a dedicated environment for Neovim, and only Neovim. This ensures that I don't need to install the pynvim package to each virtual environment.

Copycopy code to clipboard
1pyenv install 2.8.6
2pyenv install 3.9.0
3
4pyenv virtualenv 2.8.6 neovim2
5pyenv virtualenv 3.9.0 neovim3
6
7pyenv activate neovim2
8pip install neovim
9pyenv which python  # note the path
10
11pyenv activate neovim3
12pip install neovim
13pyenv which python  # note the path

Note the interpreters paths and add them to your init.lua config file:

Copycopy code to clipboard
1vim.g.python_host_prog = '/full/path/to/neovim2/bin/python'
2vim.g.python3_host_prog = '/full/path/to/neovim3/bin/python'

Lastly, within my neovim3 virtual environment, I install a bunch of other python packages that are required by Neovim plugins:

Copycopy code to clipboard
1pip install 'python_language_server[all]' pyls-black flake8 pep8-naming jupyter_qtconsole_colorschemes

To check that the python environment is set up correctly run the Ex command :checkhealth inside Neovim:

pipenv

Pipenv is a dependency manager for Python projects. It is my main driver for every python project which allows me to keep packages of different Python projects isolated from one another as well as specifying the required Python version for each project.

Copycopy code to clipboard
1# Preferred installation method via `pipx`
2pip install --user pipx
3pipx install pipenv
4# create a project directory
5mkdir ~/py-project && cd py-project
6# init a python project using Python 3.9, specifically:
7pipenv --python 3.9

Under the hood pipenv utilizes tools such as:

1. virtualenv - makes sure that packages are isolated
2. pip - fetches and installes packages
3. pyenv - installs the specified python version

Avoiding reinstalling Jupyter for all virtualenvs

In many of my projects, I work with Jupyter notebooks. It becomes really annoying when I need to install pipenv install jupyter and all its dependencies to each virtual environment. The solution for this problem is fairly easy, based on Max Horn blog post:

The trick is, that we only need to install the JuPyter kernel in the individual virtual environments, and register these kernels in the global installation.

I modified a bit his script (added an argument for specifying the python version) which allows initializing a python environment with ipykernel via pipenv.

Copycopy code to clipboard
1init_pipenv () {
2echo -e "Setting up* pipenv environment"
3    pipenv --python $1
4    echo -e "Installing ipython kernel"
5    pipenv install --dev ipykernel
6    # get name of environment and remove checksum for pretty name
7    venv_name=$(basename -- $(pipenv --venv))
8    venv_prettyname=$(echo $venv_name | cut -d '-' -f 1)
9    echo -e "Adding ipython kernel to list of jupyter kernels"
10    $(pipenv --py) -m ipykernel install --name $venv_name \
11    --display-name "Python3 ($venv_prettyname)"
12}

Note: I have JuPyter installed in the global Python 3.9.0 environment

Typical workflow including Jupyter and ipykernel via pipenv:

Copycopy code to clipboard
1# create project directory
2mkdir ~/py-project & cd py-project
3# init python environment with Python 3.8.6
4init_pipenv 3.8.6
5# run ipykernel
6pipenv run python -m ipykernel
7# open a new pane or terminal
8jupyter notebook

pipenv + neovim + tmux

In order for LSP to resolve your imports and indicate Neovim that you're inside a virtualenv, you have to spawn Neovim within the virtualenv: pipenv run neovim or spawn a subshell in the virtualenv and run Neovim inside:

Copycopy code to clipboard
1# spawn subshell in virtualenv
2pipenv shell
3# run neovim
4neovim

However, if you're using tmux and vim-tmux-navigator in your workflow there is a known issue when you try to navigate between your Neovim and tmux panes.