Managing your computational environment

Kyle Niemeyer

Oregon State University

Introduction to virtual environments

Power of Python for research comes from rich ecosystem of third-party packages

These include general numerical and scientific tools like NumPy, SciPy, and Pandas, and domain-specific software like Cantera. You should rely on mature packages that provide useful features when possible—don’t reinvent the wheel!

But, each software package has its own set of dependencies, and might require a specific version of some other package (like when a newer version breaks or changes some functionality).

Your results may depend on the specific versions and combination of packages you have installed.

This is particularly important for research analysis.

🚨 As a rule, don’t install packages into your global environment!

… even if the documentation says pip install x

Best practice: use virtual environments

  • Virtual environments are lightweight installations of packages in a particular location (like in your working directory or project project) that build on top of your system Python installation.

  • Virtual environments isolate the set of dependencies for a project or package from each other, and from your system configuration.

  • These are intended to be easy to set up, disposable, and replaceable.

  • On a Mac or Linux machine, I recommend using the built-in venv module to create a virtual environment for your work, then install inside that using the pip installer.

  • On Windows, your best bet is probably the Anaconda Distribution. Anaconda comes with hundreds of scientific packages already, and also lets you set up system-wide virtual environments.

Using venv to manage virtual environments

Creating and activating a virtual environment

python3 -m venv .venv

This creates a new directory .venv with:

  • .venv/bin with link to Python, pip, and activation scripts; and
  • .venv/lib/site-packages where packages will be installed.

To activate the environment, source the activation script:

source .venv/bin/activate

Tip

Try this yourself. .venv/bin will be added to your PATH, and your shell will usually indicate you are “in” a virtual environment.

To leave the virtual environment, run the deactivate function:

deactivate

Note

Closing/terminating your shell will automatically deactivate the virtual environment, and you’ll need to manually activate whenever you start a new session.

Installing packages

pip install <package>

Recording what you installed

An environment nobody else can recreate isn’t much better than no environment at all. Write down what is in it:

pip freeze > requirements.txt   # exact versions, from an active environment
pip install -r requirements.txt # recreate it somewhere else

For the conda family, the equivalent pair is:

conda env export > environment.yml
conda env create -f environment.yml

Note

Keep these files in version control alongside the analysis they belong to.

Loose ranges vs. exact pins

A library should support a range of dependency versions, so it can be installed alongside other people’s packages: numpy >= 1.24.

An analysis you want to reproduce exactly wants the opposite: every dependency pinned, down to the transitive ones. That is what a lock file gives you (pip-tools, uv, conda-lock, pixi).

We come back to this in the packaging module.

Alternatives for managing virtual environments

pip family

Tools that all work similarly to venv:

  • venv: comes with Python, simple
  • virtualenv: installable package with same interface as venv, but faster and with more options
  • uv: written in Rust and extremely fast

In addition, there are project management tools that both handle environments, packaging, production, etc.:

These work a bit differently, but can be quite powerful particularly for locking specific versions of dependencies. Consider returning to these when we are thinking about packaging and distribution.

Conda family

Conda is an open-source tool for managing virtual environments across your system, originally designed for Python package management but now supports a wide range of software (e.g., compiled C++ dependencies).

Virtual environments created with Conda are not location-dependent like venv, but the same approach applies: do not install in the base/default environment.

Conda is recommended if you expect to need non-Python dependencies.

conda config --set auto_activate_base false  # turn off the default environment
conda env create -n some_name  # or use paths with `-p`
conda activate some_name
conda deactivate

Conda-based package managers

Some alternatives to conda exist and offer some advantages, including speed:

  • miniforge: minimal Conda installation without all the packages that come with a full Anaconda distribution
  • Pixi: fast, Rust-based package manager and workflow tool
  • Micromamba: tiny version of Mamba, statically linked C++ binary

A closer look at uv

uv (from the team behind Ruff) has become a common default because one fast tool covers the whole lifecycle:

uv venv                # create a virtual environment
uv add numpy           # add a dependency to pyproject.toml, and lock it
uv run pytest          # run a command in the project environment, auto-synced
uvx ruff check .       # run a tool without installing it (like pipx)

You don’t have to adopt it. But if you find yourself juggling venv + pip + pip-tools, this is the tool that replaces all three.

The one rule to carry forward

One environment per project, and always work inside an activated one.

Everything in the rest of the school — installing your own package in editable mode, running your tests, building your docs — assumes you are inside an active environment.

References and further reading

Schreiner, H., Niemeyer, K., Holland, J. G., Dang, G., & Restrepo, M. I. (2024). “INTERSECT: Packaging” [Data set]. https://github.com/INTERSECT-training/packaging

Scientific Python Library Development Guide