We’re excited to announce that the Pyston team has joined Anaconda to continue the open-source development of their optimizing Python interpreter. Kevin Modzelewski and Marius Wachtler will join Anaconda’s team of funded OSS developers, alongside our contributors to Dask, Numba, Bokeh, and many other projects.
What is Pyston?
Pyston is a fork of the standard CPython interpreter (Python 3.8, currently) which has been extensively enhanced to improve the execution performance of most Python programs. It does this through various techniques, including attribute caching, Quickening, a bytecode JIT using DynASM (of LuaJIT fame), among other things. You may have heard of the earlier Pyston 1.0 project (by the same developers) at Dropbox, which ended in 2017. The new Pyston 2.x series is a complete rewrite of the codebase from scratch, starting from a fork of CPython 3.8.
Pyston’s goal is to be a general-purpose accelerator of all Python applications and currently achieves speedups between 1.15x and 2x on a wide range of Python use cases.
Why is Anaconda hiring the Pyston team?
Anaconda’s involvement in Python optimization, scalability, and performance projects goes back nearly a decade. One of Anaconda’s oldest open-source projects is the Numba compiler, an LLVM-based JIT compiler for numerical Python functions running on the CPU or GPU. As a result, we’ve been thinking about Python compilers for a long time, and we see the potential for Pyston to quickly bring faster Python to a mainstream audience.
In fact, Pyston is very complementary to Numba. Numba addresses many numerical use cases very well but cannot optimize entire programs, and it does not address the wider world of Python use cases. Pyston comes at the Python compilation problem from a different direction. Still, common ancestry with the CPython interpreter means that Numba “just works” with it, and the two systems can be used in tandem within the same program. Numba can speed up individual functions by 2-10x (or more), and Pyston can improve the performance of everything else. We are excited to see what is possible in the future with the Numba and Pyston teams working together more closely.
We also think that some of Pyston’s improvements can be upstreamed back into CPython, and that the Pyston developers’ experience will be beneficial to the efforts at Microsoft to speed up CPython by 5x in the next four years. Similarly, Pyston will incorporate improvements from upstream CPython as well. That said, we think Pyston has a unique opportunity to push the boundaries of performance and explore what is possible, independent of the (completely reasonable) constraints that a large project like CPython has.
What does this mean for Pyston?
From Anaconda’s perspective, we are now a stakeholder and sponsor of Pyston, a community open-source project. The maintainers of that project (Kevin and Marius) will have the freedom to make independent decisions about their project, and grow their user, contributor, and maintainer community independent of Anaconda. Anaconda may incorporate Pyston into current and future products, but that work will be separate from the Pyston open-source project.
In the short term, we agree with the Pyston team that the most important thing right now is to get Pyston into the hands of users so they can see what it is capable of and give feedback to the project. Anaconda has significant engineering expertise and infrastructure for building, testing, and distributing software, making it even easier for more people to try Pyston in the future.
FAQs
Q: Is the license of Pyston changing?
A: No. Pyston will continue to use the same PSF license as CPython.
Q: Does this mean installing Pyston will require installing Anaconda or using conda?
A: No. We want Pyston to be distributed in as many ways as possible. Conda has some significant benefits, such as the ability to download and install new Python interpreters. Still, we want to reach all users with the tools they are already familiar with, conda or otherwise.
Q: Is Pyston becoming an Anaconda product?
A: No. Pyston is and will continue to be an open-source project. We may incorporate Pyston into Anaconda products in the future but have no specific plans to announce right now. These products would be clearly distinguished from the Pyston project.
Q: Will Anaconda offer commercial support for Pyston?
A: We have nothing to announce right now.
Q: Will Pyston be added to the Anaconda package distribution?
A: This is something we want to do, but we have no timeline for this at the moment.
The Future
We’re optimistic about the potential for Pyston to improve the Python experience for all users and reduce the costs of deploying Python applications at scale. Keep your eyes on this space for future announcements about the Pyston roadmap and other Anaconda initiatives to advance scalable computing in Python.
