Package Better with Conda Build 3

Handling version compatibility is one of the hardest challenges in building software. Until now, conda-build provided helpful tools in terms of the ability to constrain or pin versions in recipes. The limiting thing about this capability was that it entailed editing a lot of recipes.

Conda-build 3 introduces a new scheme for controlling version constraints, which enhances behavior in two ways. This document is intended as a quick overview of new features in conda-build 3. For more information, see the docs.

Handling version compatibility is one of the hardest challenges in building software. Until now, conda-build provided helpful tools in terms of the ability to constrain or pin versions in recipes. The limiting thing about this capability was that it entailed editing a lot of recipes.

Conda-build 3 introduces a new scheme for controlling version constraints, which enhances behavior in two ways. First, you can now set versions in an external file and provide lists of versions for conda-build to loop over. Matrix builds are now much simpler and no longer require an external tool, such as conda-build-all. Second, there have been several new jinja2 functions added, which allow recipe authors to express their constraints relative to the versions of packages installed at build time. This dynamic expression greatly cuts down on the need for editing recipes.

Each of these developments have enabled interesting new capabilities for cross-compiling, as well as improving package compatibility by adding more intelligent constraints.

This document is intended as a quick overview of new features in conda-build 3. For more information, see the docs.

These demos use conda-build’s python API to render and build recipes. That API currently does not have a docs page, but is pretty self explanatory. See the source at: https://github.com/conda/conda-build/blob/master/conda_build/api.py

This jupyter notebook itself is included in conda-build’s tests folder. If you’re interested in running this notebook yourself, see the tests/test-recipes/variants folder in a git checkout of the conda-build source. Tests are not included with conda packages of conda-build.

from conda_build import api 
import os 
from pprint import pprint

First, set up some helper functions that will output recipe contents in a nice-to-read way:

def print_yamls(recipe, **kwargs): 
 yamls = [api.output_yaml(m[0]) 
 for m in api.render(recipe, verbose=False, permit_unsatisfiable_variants=True, **kwargs)] 
 for yaml in yamls: 
 print(yaml) 
 print('-' * 50)

def print_outputs(recipe, **kwargs): 
 pprint(api.get_output_file_paths(recipe, verbose=False, **kwargs))

Most of the new functionality revolves around much more powerful use of jinja2 templates. The core idea is that there is now a separate configuration file that can be used to insert many different entries into your meta.yaml files.

!cat 01_basic_templating/meta.yaml

 package:
 name: abc
 version: 1.0

 requirements:
 build:
 - something {{ something }}
 run:
 - something {{ something }}

# The configuration is hierarchical - it can draw from many config files. One place they can live is alongside meta.yaml:
!cat 01_basic_templating/conda_build_config.yaml
 

 something:
 - 1.0
 - 2.0

Since we have one slot in meta.yaml, and two values for that one slot, we should end up with two output packages:

print_outputs('01_basic_templating/')
 
 Returning non-final recipe for abc-1.0-0; one or more dependencies was unsatisfiable:
 Build: something
 Host: None
 
 ['/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h1332e90_0.tar.bz2',
 '/Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-h9f70ef6_0.tar.bz2']

print_yamls('01_basic_templating/')
 package:
 name: abc
 version: '1.0'
 build:
 string: '0'
 requirements:
 build:
 - something 1.0
 run:
 - something 1.0
 extra:
 final: true
 
--------------------------------------------------
 package:
 name: abc
 version: '1.0'
 build:
 string: '0'
 requirements:
 build:
 - something 2.0
 run:
 - something 2.0
 extra:
 final: true
 
--------------------------------------------------

OK, that’s fun already. But wait, there’s more!

We saw a warning about “finalization.” That’s conda-build trying to figure out exactly what packages are going to be installed for the build process. This is all determined before the build. Doing so allows us to tell you the actual output filenames before you build anything. Conda-build will still render recipes if some dependencies are unavailable, but you obviously won’t be able to actually build that recipe.

!cat 02_python_version/meta.yaml
 
 package:
 name: abc
 version: 1.0
 
 requirements:
 build:
 - python
 run:
 - python
 
!cat 02_python_version/conda_build_config.yaml
 
 python:
 - 2.7
 - 3.5
 
print_yamls('02_python_version/')
 
 package:
 name: abc
 version: '1.0'
 
 build:
 string: py27hef4ac7c_0
 requirements:
 build:
 - readline 6.2 2
 - tk 8.5.18 0
 - pip 9.0.1 py27_1
 - setuptools 27.2.0 py27_0
 - openssl 1.0.2l 0
 - sqlite 3.13.0 0
 - python 2.7.13 0
 - wheel 0.29.0 py27_0
 - zlib 1.2.8 3
 run:
 - python >=2.7,<2.8
 
 extra:
 final: true
 
--------------------------------------------------
 package:
 name: abc
 version: '1.0'
 build:
 string: py35h6785551_0
 requirements:
 build:
 - readline 6.2 2
 - setuptools 27.2.0 py35_0
 - tk 8.5.18 0
 - openssl 1.0.2l 0
 - sqlite 3.13.0 0
 - python 3.5.3 1
 - pip 9.0.1 py35_1
 - xz 5.2.2 1
 - wheel 0.29.0 py35_0
 - zlib 1.2.8 3
 run:
 - python >=3.5,<3.6
 extra:
 final: true
 
--------------------------------------------------

Here you see that we have many more dependencies than we specified, and we have much more detailed pinning. This is a finalized recipe. It represents exactly the state that would be present for building (at least on the current platform).

So, this new way to pass versions is very fun, but there’s a lot of code out there that uses the older way of doing things—environment variables and CLI arguments. Those still work. They override any conda_build_config.yaml settings.

# Setting environment variables overrides the conda_build_config.yaml. This preserves older, well-established behavior.
os.environ["CONDA_PY"] = "3.4"
print_yamls('02_python_version/')
del os.environ['CONDA_PY']
 
 package:
 name: abc
 version: '1.0'
 build:
 string: py34h31af026_0
 requirements: build:
 - readline 6.2 2
 - python 3.4.5 0
 - setuptools 27.2.0 py34_0
 - tk 8.5.18 0
 - openssl 1.0.2l 0
 - sqlite 3.13.0 0
 - pip 9.0.1 py34_1
 - xz 5.2.2 1
 - wheel 0.29.0 py34_0
 - zlib 1.2.8 3
 run:
 - python >=3.4,<3.5
 extra:
 final: true
 
--------------------------------------------------

# Passing python as an argument (CLI or to the API) also overrides conda_build_config.yaml
print_yamls('02_python_version/', python="3.6")
 
 
 package:
 name: abc
 version: '1.0'
 build:
 string: py36hd0a5620_0
 requirements:
 build:
 - readline 6.2 2
 - wheel 0.29.0 py36_0
 - tk 8.5.18 0
 - python 3.6.1 2
 - openssl 1.0.2l 0
 - sqlite 3.13.0 0
 - pip 9.0.1 py36_1
 - xz 5.2.2 1
 - setuptools 27.2.0 py36_0
 - zlib 1.2.8 3
 run:
 - python >=3.6,<3.7
 
 extra:
 final: true
 
--------------------------------------------------

Wait a minute—what is that h7d013e7 gobbledygook in the build/string field?

Conda-build 3 aims to generalize pinning/constraints. Such constraints differentiate a package. For example, in the past, we have had things like py27np111 in filenames. This is the same idea, just generalized. Since we can’t readily put every possible constraint into the filename, we have kept the old ones, but added the hash as a general solution.

There’s more information about what goes into a hash at: https://conda.io/docs/building/variants.html#differentiating-packages-built-with-different-variants

Let’s take a look at how to inspect the hash contents of a built package.

outputs = api.build('02_python_version/', python="3.6",
 anaconda_upload=False)
pkg_file = outputs[0]
print(pkg_file)
 
 The following NEW packages will be INSTALLED:
 openssl: 1.0.2l-0
 pip: 9.0.1-py36_1
 python: 3.6.1-2
 readline: 6.2-2
 setuptools: 27.2.0-py36_0
 sqlite: 3.13.0-0 tk: 8.5.18-0
 wheel: 0.29.0-py36_0
 xz: 5.2.2-1
 zlib: 1.2.8-3
 
 source tree in: /Users/msarahan/miniconda3/conda-bld/abc_1498787283909/work
 Attempting to finalize metadata for abc
 
 INFO:conda_build.metadata:Attempting to finalize metadata for abc
 
 BUILD START: ['abc-1.0-py36hd0a5620_0.tar.bz2']
 Packaging abc
 
 INFO:conda_build.build:Packaging abc
 
 The following NEW packages will be INSTALLED:
 openssl: 1.0.2l-0
 pip: 9.0.1-py36_1
 python: 3.6.1-2
 readline: 6.2-2
 setuptools: 27.2.0-py36_0
 sqlite: 3.13.0-0 tk: 8.5.18-0
 wheel: 0.29.0-py36_0
 xz: 5.2.2-1
 zlib: 1.2.8-3
 
 Packaging abc-1.0-py36hd0a5620_0
 
 INFO:conda_build.build:Packaging abc-1.0-py36hd0a5620_0
 
 number of files: 0
 Fixing permissions
 Fixing permissions
 updating: abc-1.0-py36hd0a5620_0.tar.bz2
 Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
 # Automatic uploading is disabled
 # If you want to upload package(s) to anaconda.org later, type:
 
 anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
 
 # To have conda build upload to anaconda.org automatically, use
 # $ conda config --set anaconda_upload yes
 
 anaconda_upload is not set. Not uploading wheels: []
 /Users/msarahan/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2

# Using command line here just to show you that this command exists.
!conda inspect hash-inputs ~/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
 
 Package abc-1.0-py36hd0a5620_0 does not include recipe. Full hash information is not reproducible.
 WARNING:conda_build.inspect:Package abc-1.0-py36hd0a5620_0 does not include recipe. Full hash information is not reproducible.
 {'abc-1.0-py36hd0a5620_0': {'files': [],
 'recipe': {'requirements': {'build': ['openssl '
 '1.0.2l 0',
 'pip 9.0.1 '
 'py36_1',
 'python '
 '3.6.1 2',
 'readline '
 '6.2 2',
 'setuptools '
 '27.2.0 '
 'py36_0',
 'sqlite '
 '3.13.0 0',
 'tk 8.5.18 0',
 'wheel '
 '0.29.0 '
 'py36_0',
 'xz 5.2.2 1',
 'zlib 1.2.8 '
 '3'],
 'run': ['python '
 '>=3.6,<3.7']}}}}

pin_run_as_build is a special extra key in the config file. It is a generalization of the x.x concept that existed for numpy since 2015. There’s more information at: https://conda.io/docs/building/variants.html#customizing-compatibility

Each x indicates another level of pinning in the output recipe. Let’s take a look at how we can control the relationship of these constraints. Before now you could certainly accomplish pinning, it just took more work. Now you can define your pinning expressions, and then change your target versions in just one config file.

!cat 05_compatible/meta.yaml
 
 package:
 name: compatible
 version: 1.0
 
 requirements:
 build:
 - libpng
 run:
 - {{ pin_compatible('libpng') }}

This is effectively saying “add a runtime libpng constraint that follows conda-build’s default behavior, relative to the version of libpng that was used at build time.”

pin_compatible is a new helper function available to you in meta.yaml. The default behavior is: exact version match lower bound (“x.x.x.x.x.x.x”), next major version upper bound (“x”).

print_yamls('05_compatible/')
 
 package:
 name: compatible
 version: '1.0'
 build:
 string: h3d53989_0
 requirements:
 build:
 - libpng 1.6.27 0
 - zlib 1.2.8 3
 run:
 - libpng >=1.6.27,<2
 extra:
 final: true
 
--------------------------------------------------

These constraints are completely customizable with pinning expressions:

!cat 06_compatible_custom/meta.yaml
 
 package:
 name: compatible
 version: 1.0
 
 requirements:
 build:
 - libpng
 run:
 - {{ pin_compatible('libpng', max_pin='x.x') }}
 
print_yamls('06_compatible_custom/')
 
 package:
 name: compatible
 version: '1.0'
 build:
 string: ha6c6d66_0
 requirements:
 build:
 - libpng 1.6.27 0
 - zlib 1.2.8 3
 run:
 - libpng >=1.6.27,<1.7
 extra:
 final: true
 
--------------------------------------------------

Finally, you can also manually specify version bounds. These supersede any relative constraints.

!cat 07_compatible_custom_lower_upper/meta.yaml
 
 package:
 name: compatible
 version: 1.0
 
 requirements:
 build:
 - libpng
 run:
 - {{ pin_compatible('libpng', min_pin=None, upper_bound='5.0') }}
 
print_yamls('07_compatible_custom_lower_upper/')
 
 package:
 name: compatible
 version: '1.0'
 build:
 string: heb31dda_0
 requirements:
 build:
 - libpng 1.6.27 0
 - zlib 1.2.8 3
 run:
 - libpng <5.0
 extra:
 final: true
 
--------------------------------------------------

Much of the development of conda-build 3 has been inspired by improving the compiler toolchain situation. Conda-build 3 adds special support for more dynamic specification of compilers.

!cat 08_compiler/meta.yaml
 
 package:
 name: cross
 version: 1.0
 
 requirements:
 build:
 - {{ compiler('c') }}

By replacing any actual compiler with this jinja2 function, we’re free to swap in different compilers based on the contents of the conda_build_config.yaml file (or other variant configuration). Rather than saying “I need gcc,” we are saying “I need a C compiler.”

By doing so, recipes are much more dynamic, and conda-build also helps to keep your recipes in line with respect to runtimes. We’re also free to keep compilation and linking flags associated with specific “compiler” packages—allowing us to build against potentially multiple configurations (Release, Debug?). With cross compilers, we could also build for other platforms.

!cat 09_cross/meta.yaml
 
 package:
 name: cross
 version: 1.0
 
 requirements:
 build:
 - {{ compiler('c') }}

# But, by adding in a base compiler name, and target platforms, we can make a build matrix
# This is not magic, the compiler packages must already exist. Conda-build is only following a naming scheme.
!cat 09_cross/conda_build_config.yaml
 
 
 c_compiler:
 - gcc
 target_platform:
 - linux-64
 - linux-cos5-64
 - linux-aarch64
 
print_yamls('09_cross/')
 
 Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-64
 Host: None
 WARNING:conda_build.render:
 Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-64
 Host: None
 Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-cos5-64
 Host: None
 WARNING:conda_build.render:Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-cos5-64
 Host: None
 Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-aarch64
 Host: None
 WARNING:conda_build.render:Returning non-final recipe for cross-1.0-0; one or more dependencies was unsatisfiable:
 Build: gcc_linux-aarch64
 Host: None
 
 
 package:
 name: cross
 version: '1.0'
 build:
 string: '0'
 requirements:
 build:
 - gcc_linux-64
 extra:
 final: true
 
--------------------------------------------------
 package:
 name: cross
 version: '1.0'
 build:
 string: '0'
 requirements:
 build:
 - gcc_linux-cos5-64
 extra:
 final: true
 
--------------------------------------------------
 package:
 name: cross
 version: '1.0'
 build:
 string: '0'
 requirements:
 build:
 - gcc_linux-aarch64
 extra:
 final: true
 
--------------------------------------------------

Finally, it is frequently a problem to remember to add runtime dependencies. Sometimes the recipe author is not entirely familiar with the lower level code and has no idea about runtime dependencies. Other times, it’s just a pain to keep versions of runtime dependencies in line. Conda-build 3 introduces a way of storing the required runtime dependencies on the package providing the dependency at build time.

For example, using g++ in a non-static configuration will require that the end-user have a sufficiently new libstdc++ runtime library available at runtime. Many people don’t currently include this in their recipes. Sometimes the system libstdc++ is adequate, but often not. By imposing the downstream dependency, we can make sure that people don’t forget the runtime dependency.

# First, a package that provides some library.
# When anyone uses this library, they need to include the appropriate runtime.
 
!cat 10_runtimes/uses_run_exports/meta.yaml
 
 
 package:
 name: package_has_run_exports
 version: 1.0
 build:
 run_exports:
 - {{ pin_compatible('bzip2') }}
 requirements:
 build:
 - bzip2

# This is the simple downstream package that uses the library provided in the previous recipe.
!cat 10_runtimes/consumes_exports/meta.yaml
 
 package:
 name: package_consuming_run_exports
 version: 1.0
 requirements:
 build:
 - package_has_run_exports

# Let's build the former package first.
api.build('10_runtimes/uses_run_exports', anaconda_upload=False)
 
 
 The following NEW packages will be INSTALLED:
 bzip2: 1.0.6-3
 source tree in: /Users/msarahan/miniconda3/conda-bld/package_has_run_exports_1498787302719/work
 Attempting to finalize metadata for package_has_run_exports
 
 INFO:conda_build.metadata:Attempting to finalize metadata for package_has_run_exports
 
 BUILD START: ['package_has_run_exports-1.0-hcc78ab3_0.tar.bz2']
 Packaging package_has_run_exports
 
 INFO:conda_build.build:Packaging package_has_run_exports
 
 The following NEW packages will be INSTALLED:
 bzip2: 1.0.6-3
 number of files: 0
 Fixing permissions
 Fixing permissions
 updating: package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
 Nothing to test for: /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
 # Automatic uploading is disabled
 # If you want to upload package(s) to anaconda.org later, type:
 
 anaconda upload /Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2
 
 # To have conda build upload to anaconda.org automatically, use
 # $ conda config --set anaconda_upload yes
 
 anaconda_upload is not set. Not uploading wheels: []
 
 
 ['/Users/msarahan/miniconda3/conda-bld/osx-64/package_has_run_exports-1.0-hcc78ab3_0.tar.bz2']
 
 
print_yamls('10_runtimes/consumes_exports')
 
 package:
 name: package_consuming_run_exports
 version: '1.0'
 build:
 string: h8346d2f_0
 requirements:
 build:
 - package_has_run_exports 1.0 hcc78ab3_0
 run:
 - bzip2 >=1.0.6,<2
 extra:
 final: true
 
--------------------------------------------------

In the above recipe, note that bzip2 has been added as a runtime dependency, and is pinned according to conda-build’s default pin_compatible scheme. This behavior can be overridden in recipes if necessary, but we hope it will prove useful.