Anaconda Recognized in insideBIGDATA’s IMPACT 50 List
![Blog Header template (3)](https://www.anaconda.com/wp-content/uploads/2024/06/Blog-Header-template-3.png)
Daniel Rodriguez
Daniel Rodriguez
At Anaconda, we’ve developed a rigorous new approach to AI development called Evaluations Driven Development (EDD). By continuously testing AI models using real-world cases and user feedback, EDD enables us to create AI assistants that are reliable, relevant, and truly impactful for users.
Our Anaconda Assistant, an AI coding companion for data scientists, exemplifies the power of EDD. Trained on real code samples, errors, and fixes, it offers in-context suggestions and debugging help to supercharge your Python workflow. And thanks to EDD, it keeps getting smarter with each update.
We believe EDD is the future of AI development, ensuring that AI tools don’t just demo well, but deliver real value. If you’re excited by AI’s potential but wary of the hype, read on to learn how EDD works and why it’s a game-changer for building AI applications that make a difference.
As a data scientist working with Python, you know the frustration of hitting roadblocks in your code. That’s where the Anaconda Assistant comes in.
Leveraging state-of-the-art language models trained on real-world Python code, errors, and solutions, the Assistant offers a range of features to streamline your workflow:
But the Assistant’s most popular feature by far is intelligent debugging. Telemetry data shows that 60% of user interactions involve asking for help with pesky errors.
Just describe your error to the Assistant and within seconds, you’ll get a plain-English explanation of the problem and suggestions for fixing it. No more hours lost scouring Stack Overflow!
Best of all, the Assistant is always getting smarter thanks to our Evaluations Driven Development (EDD) process. Every interaction with users who have consented to data collection is an opportunity to refine the prompts and queries we use to elicit relevant, reliable support from the underlying language models.
In the next section, we’ll explore how EDD enables us to continuously improve Anaconda Assistant without the need for costly and time-consuming model retraining.
Our Evaluations Driven Development (EDD) methodology, powered by our in-house “llm-eval” framework, consists of rigorously testing and refining the prompts and queries we use to elicit relevant, reliable outputs from the underlying language models. Rather than just optimizing for abstract benchmarks, we evaluate the Anaconda Assistant on its ability to handle the actual challenges faced by data scientists in their daily work.
This is where “llm-eval” comes in. It’s a comprehensive testing framework that allows us to simulate thousands of realistic user interactions and assess the Assistant’s responses across a wide range of scenarios, from debugging complex errors to generating readable, well-documented code.
Here’s a high-level overview of how it works:
To make this a bit more concrete, let’s take a closer look at how we’ve applied EDD to one of the Anaconda Assistant’s most important capabilities: error handling.
Error handling is a critical aspect of the Assistant’s functionality, as data scientists rely on clear, actionable guidance when encountering bugs in their code. To assess the Assistant’s performance in this area, we used the “llm-eval” framework to systematically test its ability to diagnose and fix errors across a wide range of real-world debugging scenarios.
One particularly illuminating case study involved the following code, which raises a `ValueError` when an invalid age is passed to the `create_person` function:
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def __str__(self):
return f"Name: {self.name}, Age: {self.age}"
def create_person(name, age):
if age < 0:
raise ValueError("Age cannot be negative.")
return Person(name, age)
def print_person_details(person):
print(f"Person Details: {person}")
# Example usage
people = [
create_person("Alice", 25),
create_person("Bob", -5), # Raises ValueError
create_person("Charlie", 30),
]
for person in people:
print_person_details(person)
We provided the Assistant with this code snippet and the corresponding error message, and evaluated its responses based on accuracy: Does it correctly identify the root cause and provide a fix that resolves the issue?
To evaluate the Assistant’s performance, we tested its responses using different language models, versions, and temperature settings. In AI models like the one behind the Anaconda Assistant, the temperature setting controls the randomness and creativity of the generated text. Lower temperatures lead to more precise but conservative responses, while higher temperatures allow for greater diversity but potentially at the cost of coherence or factual accuracy. Finding the right balance is key to achieving reliable, high-quality results.
The initial evaluation results were eye-opening:
Model | Temperature | Iterations | Success Rate |
GPT-3.5-Turbo (v0125) | 0 | 500 | 12% |
GPT-3.5-Turbo (v0125) | 1 | 500 | 13% |
Mistral 7B Instruct v0.2 | 0 | 500 | 0% |
Mistral 7B Instruct v0.2 | 1 | 500 | 2% |
Under these conditions, the Anaconda Assistant was only able to correctly identify the bug and provide a working solution in at most 13% of the test executions. However, rather than being discouraged by these results, we saw them as a valuable opportunity to leverage the insights gained from our “llm-eval” framework to drive meaningful improvements in the Assistant’s performance. By carefully crafting the prompts and queries used to elicit error explanations and fixes, and by iterating on these prompts based on the evaluation results, we believed we could significantly enhance the Assistant’s error handling capabilities. In the next section, we’ll explore how we are applying these prompt engineering techniques to transform the Assistant into a robust and reliable debugging companion.
Prompt engineering is the process of carefully designing the text input (the “prompt”) given to an AI model to guide it towards producing the desired output. Just as a well-worded question can help a human give a more relevant and useful answer, an effectively engineered prompt can significantly improve the quality of an AI’s responses. This might involve techniques like providing examples, breaking down complex tasks into steps, or specifying format requirements.
To improve the Anaconda Assistant’s accuracy in error handling, we employed several prompt engineering techniques that have proven effective in guiding language models towards more accurate and relevant responses:
Here’s a simplified version of the original prompt used:
system:
You are an AI Assistant called 'Anaconda Assistant'. As a Jupyter Notebook expert, your main task is to help debug code and briefly explain errors. Ask before providing corrected code. If providing corrected code, only provide the corrected code, nothing else.
[...]
user:
```python
def fibonacci(n):
fib = [0, 1]
for i in range(2, n+1):
fib.append(fib[i] + fib[i-1])
return fib fibonacci(10)
```
<output>
IndexError Traceback
----> 1 fibonacci(10)
----> 4 fib.append(fib[i] + fib[i-1])
</output>
Explain this error.
assistant:
The error occurs because the code is trying to access an index that is not yet in the list. To fix this error, you need to change the indexing in the loop.
The error occurs in the following line:
```python
fib.append(fib[i-1] + fib[i-2])
```
Explanation:
The code tries to access `fib[i]` and `fib[i-1]` to calculate the next Fibonacci number. \nHowever, `fib[i]` is not yet in the list, causing the IndexError. By changing the line to `fib.append(fib[i-1] + fib[i-2])`, \nwe correctly access the two previous Fibonacci numbers to calculate the next one.
[...]
```python
{error_cell_code}
```
<output>
{error_output}
</output>
Explain this error.
To rigorously evaluate the accuracy of the generated responses, we developed a comprehensive testing framework within “llm-eval” that executes the generated code snippets in a controlled environment. This framework captures detailed information about the execution process, including any errors or exceptions encountered, and compares the output to the expected results.
By running this evaluation process across hundreds of generated code snippets, we are systematically measuring the Anaconda Assistant’s accuracy in explaining and fixing real-world Python errors. The resulting evaluation data provided a wealth of insights into the Assistant’s performance, including:
These insights are becoming invaluable guides for our prompt engineering efforts. In the next section, we’ll take a closer look at the results of this evaluation-driven optimization process and explore some of the specific improvements we were able to achieve.
As part of our Evaluations Driven Development (EDD) framework, we are employing an innovative technique called Agentic Feedback Iteration to further refine the prompts and queries used in the Anaconda Assistant. This process leverages the advanced capabilities of large language models to provide targeted feedback and suggestions for improvement based on the evaluation results.
Here’s how it works:
By leveraging the advanced language understanding and generation capabilities of large language models, Agentic Feedback Iteration allows us to rapidly optimize the prompts and queries in a data-driven, targeted manner.
The specific changes made to the prompts and queries through this process directly address the key challenges and opportunities identified in our initial evaluations. For example:
User prompt modification:
System prompt changes:
Through multiple rounds of Agentic Feedback Iteration, we are significantly improving the accuracy of the Anaconda Assistant’s responses, particularly on the most challenging and nuanced error scenarios. In the next section, we’ll dive into the detailed results of this optimization process and explore some of the most impressive improvements achieved.
To assess the impact of our prompt engineering efforts and the Agentic Feedback Iteration process, we conducted a new round of evaluations using the refined prompts. We focused on the same error handling scenario, running it hundreds of times to check whether the generated code still included the original bug or provided a successful fix.
The results showed a remarkable improvement across all models and settings compared to our initial evaluations. The most significant gains were observed with the Mistral 7B model, which achieved a perfect 100% success rate when using a temperature setting of 1. This means that, under these conditions, the Anaconda Assistant was able to correctly identify the bug and generate a working solution in every single test execution.
These results provide compelling evidence for the power of Evaluations Driven Development and the transformative potential of advanced language models combined with rigorous testing, prompt engineering, and techniques like Agentic Feedback Iteration. By continuously refining our approach and expanding its applications, we believe we can unlock even greater levels of performance and versatility, not just for the Anaconda Assistant but for AI-powered tools across a wide range of domains.
In the next section, we’ll explore some of the exciting future directions for EDD and the Anaconda Assistant, and how you can get involved in shaping the future of AI-assisted data science.
Our Evaluations Driven Development (EDD) process represents a paradigm shift that is revolutionizing how we build, test, and deploy AI-powered tools across industries. With EDD, we’re building trust, reliability, and unprecedented capabilities into the very fabric of AI development.
The success of EDD in dramatically improving the performance and reliability of the Anaconda Assistant is just the beginning. As we look to the future, we are excited to explore new opportunities to build upon this foundation and drive even greater innovation and impact.
Our top priorities for the future of EDD include:
Whether you’re a data scientist, developer, or AI enthusiast, there are many ways to get involved and contribute to the future of EDD and the Anaconda Assistant. From trying out the Assistant in your own projects and providing feedback, to contributing to the development of the “llm-eval” framework once we release it. Your input and expertise are invaluable in shaping the future of AI-powered innovation.
Imagine a future where AI-powered tools like the Anaconda Assistant are not just nice-to-haves, but indispensable partners in data science workflows, enabling users to focus on high-level problem-solving while AI handles the repetitive tasks. Where AI-driven code analysis and generation tools help developers write more reliable, efficient, and secure software.
This is the future that Evaluations Driven Development is meant to enable. By combining rigorous, data-driven testing with targeted prompt engineering and techniques like Agentic Feedback Iteration, we’re not just improving AI models—we aim to fundamentally change the way we develop and deploy AI.
The future of AI-driven development is bright, and with Evaluations Driven Development as our guiding light, we’re excited to see where this journey takes us. Let’s embrace this new paradigm together, and unlock the full potential of AI to transform our world for the better.
Talk to one of our experts to find solutions for your AI journey.