зеркало из https://github.com/microsoft/FLAML.git
329 строки
15 KiB
Plaintext
329 строки
15 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a href=\"https://colab.research.google.com/github/microsoft/FLAML/blob/main/notebook/autogen_agentchat_MathChat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Auto Generated Agent Chat: Using MathChat to Solve Math Problems\n",
|
|
"\n",
|
|
"`flaml.autogen` offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation. Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",
|
|
"\n",
|
|
"MathChat is an experimental convesational framework for math problem solving. In this notebook, we demonstrate how to use MathChat to solve math problems. MathChat uses the `AssistantAgent` and `MathUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/FLAML/blob/main/notebook/autogen_agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `MathUserProxyAgent` implements a different auto reply mechanism corresponding to the MathChat prompts. You can find more details in the paper [An Empirical Study on Challenging Math Problem Solving with GPT-4](https://arxiv.org/abs/2306.01337) or the [blogpost](https://microsoft.github.io/FLAML/blog/2023/06/28/MathChat).\n",
|
|
"\n",
|
|
"## Requirements\n",
|
|
"\n",
|
|
"FLAML requires `Python>=3.8`. To run this notebook example, please install flaml with the [mathchat] option.\n",
|
|
"```bash\n",
|
|
"pip install flaml[mathchat]\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# %pip install flaml[mathchat]~=2.0.0"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Set your API Endpoint\n",
|
|
"\n",
|
|
"The [`config_list_from_json`](https://microsoft.github.io/FLAML/docs/reference/autogen/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from flaml import autogen\n",
|
|
"\n",
|
|
"config_list = autogen.config_list_from_json(\n",
|
|
" \"OAI_CONFIG_LIST\",\n",
|
|
" filter_dict={\n",
|
|
" \"model\": {\n",
|
|
" \"gpt-4\",\n",
|
|
" \"gpt4\",\n",
|
|
" \"gpt-4-32k\",\n",
|
|
" \"gpt-4-32k-0314\",\n",
|
|
" \"gpt-4-32k-v0314\",\n",
|
|
" \"gpt-3.5-turbo\",\n",
|
|
" \"gpt-3.5-turbo-16k\",\n",
|
|
" \"gpt-3.5-turbo-0301\",\n",
|
|
" \"chatgpt-35-turbo-0301\",\n",
|
|
" \"gpt-35-turbo-v0301\",\n",
|
|
" \"gpt\",\n",
|
|
" }\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well).\n",
|
|
"\n",
|
|
"The config list looks like the following:\n",
|
|
"```python\n",
|
|
"config_list = [\n",
|
|
" {\n",
|
|
" 'model': 'gpt-4',\n",
|
|
" 'api_key': '<your OpenAI API key here>',\n",
|
|
" },\n",
|
|
" {\n",
|
|
" 'model': 'gpt-4',\n",
|
|
" 'api_key': '<your Azure OpenAI API key here>',\n",
|
|
" 'api_base': '<your Azure OpenAI API base here>',\n",
|
|
" 'api_type': 'azure',\n",
|
|
" 'api_version': '2023-06-01-preview',\n",
|
|
" },\n",
|
|
" {\n",
|
|
" 'model': 'gpt-3.5-turbo',\n",
|
|
" 'api_key': '<your Azure OpenAI API key here>',\n",
|
|
" 'api_base': '<your Azure OpenAI API base here>',\n",
|
|
" 'api_type': 'azure',\n",
|
|
" 'api_version': '2023-06-01-preview',\n",
|
|
" },\n",
|
|
"]\n",
|
|
"```\n",
|
|
"\n",
|
|
"If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",
|
|
"\n",
|
|
"You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Construct agents for MathChat\n",
|
|
"\n",
|
|
"We start by initialzing the `AssistantAgent` and `MathUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for MathChat. The detailed instructions are given in the user message. Later we will use the `MathUserProxyAgent.generate_init_message` to combine the instructions and a math problem for an initial message to be sent to the LLM assistant."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from flaml.autogen.agentchat.contrib.math_user_proxy_agent import MathUserProxyAgent\n",
|
|
"\n",
|
|
"autogen.ChatCompletion.start_logging()\n",
|
|
"\n",
|
|
"# 1. create an AssistantAgent instance named \"assistant\"\n",
|
|
"assistant = autogen.AssistantAgent(\n",
|
|
" name=\"assistant\", \n",
|
|
" system_message=\"You are a helpful assistant.\",\n",
|
|
" llm_config={\n",
|
|
" \"request_timeout\": 600,\n",
|
|
" \"seed\": 42,\n",
|
|
" \"config_list\": config_list,\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"# 2. create the MathUserProxyAgent instance named \"mathproxyagent\"\n",
|
|
"# By default, the human_input_mode is \"NEVER\", which means the agent will not ask for human input.\n",
|
|
"mathproxyagent = MathUserProxyAgent(\n",
|
|
" name=\"mathproxyagent\", \n",
|
|
" human_input_mode=\"NEVER\",\n",
|
|
" code_execution_config={\"use_docker\": False},\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Example 1\n",
|
|
"\n",
|
|
"Problem: Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.\n",
|
|
"\n",
|
|
"Correct Solution: \n",
|
|
"We have \\begin{align*} (2x+10)(x+3)&<(3x+9)(x+8) \\quad \\Rightarrow\n",
|
|
"\\\\ 2(x+5)(x+3)&<3(x+3)(x+8) \\quad \\Rightarrow\n",
|
|
"\\\\ 2(x+5)(x+3)-3(x+3)(x+8)&<0 \\quad \\Rightarrow\n",
|
|
"\\\\ (2x+10-(3x+24))(x+3)&<0 \\quad \\Rightarrow\n",
|
|
"\\\\ (-x-14)(x+3)&<0 \\quad \\Rightarrow\n",
|
|
"\\\\ (x+14)(x+3)&>0.\n",
|
|
"\\end{align*} This inequality is satisfied if and only if $(x+14)$ and $(x+3)$ are either both positive or both negative. Both factors are positive for $x>-3$ and both factors are negative for $x<-14$. When $-14<x<-3$, one factor is positive and the other negative, so their product is negative. Therefore, the range of $x$ that satisfies the inequality is $ \\boxed{(-\\infty, -14)\\cup(-3,\\infty)} $."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# given a math problem, we use the mathproxyagent to generate a prompt to be sent to the assistant as the initial message.\n",
|
|
"# the assistant receives the message and generates a response. The response will be sent back to the mathproxyagent for processing.\n",
|
|
"# The conversation continues until the termination condition is met, in MathChat, the termination condition is the detect of \"\\boxed{}\" in the response.\n",
|
|
"math_problem = \"Find all $x$ that satisfy the inequality $(2x+10)(x+3)<(3x+9)(x+8)$. Express your answer in interval notation.\"\n",
|
|
"mathproxyagent.initiate_chat(assistant, problem=math_problem)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Example 2\n",
|
|
"Problem: For what negative value of $k$ is there exactly one solution to the system of equations \\begin{align*}\n",
|
|
"y &= 2x^2 + kx + 6 \\\\\n",
|
|
"y &= -x + 4?\n",
|
|
"\\end{align*}\n",
|
|
"\n",
|
|
"Correct Solution: Setting the two expressions for $y$ equal to each other, it follows that $2x^2 + kx + 6 = -x + 4$. Re-arranging, $2x^2 + (k+1)x + 2 = 0$. For there to be exactly one solution for $x$, then the discriminant of the given quadratic must be equal to zero. Thus, $(k+1)^2 - 4 \\cdot 2 \\cdot 2 = (k+1)^2 - 16 = 0$, so $k+1 = \\pm 4$. Taking the negative value, $k = \\boxed{-5}$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"math_problem = \"For what negative value of $k$ is there exactly one solution to the system of equations \\\\begin{align*}\\ny &= 2x^2 + kx + 6 \\\\\\\\\\ny &= -x + 4?\\n\\\\end{align*}\"\n",
|
|
"mathproxyagent.initiate_chat(assistant, problem=math_problem)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Example 3\n",
|
|
"Problem: Find all positive integer values of $c$ such that the equation $x^2-7x+c=0$ only has roots that are real and rational. Express them in decreasing order, separated by commas.\n",
|
|
"\n",
|
|
"Correct Solution: For the roots to be real and rational, the discriminant must be a perfect square. Therefore, $(-7)^2-4 \\cdot 1 \\cdot c = 49-4c$ must be a perfect square. The only positive perfect squares less than 49 are $1$, $4$, $9$, $16$, $25$, and $36$. The perfect squares that give a integer value of $c$ are $1$, $9$, and $25$. Thus, we have the equations $49-4c=1$, $49-4c=9$, and $49-4c=25$. Solving, we get that the positive integer values of c are $\\boxed{12, 10, 6}$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"math_problem = \"Find all positive integer values of $c$ such that the equation $x^2-7x+c=0$ only has roots that are real and rational. Express them in decreasing order, separated by commas.\"\n",
|
|
"mathproxyagent.initiate_chat(assistant, problem=math_problem)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"------------------------------------\n",
|
|
"### Using other prompts\n",
|
|
"\n",
|
|
"MathChat allows different prompts that instruct assistant to solve the problem.\n",
|
|
"\n",
|
|
"Check out `MathUserProxyAgent.generate_init_message(problem, prompt_type='default', customized_prompt=None)`:\n",
|
|
"- You may choose from `['default', 'python', 'two_tools']` for parameter `prompt_type`. We include two more prompts in the paper: \n",
|
|
" 1. `'python'` is a simplified prompt from the default prompt that uses Python only. \n",
|
|
" 2. `'two_tools'` further allows the selection of Python or Wolfram Alpha based on this simplified `python` prompt. Note that this option requries a Wolfram Alpha API key and put it in `wolfram.txt`.\n",
|
|
"\n",
|
|
"- You can also input your customized prompt if needed: `mathproxyagent.generate_init_message(problem, customized_prompt=\"Your customized prompt\")`. Since this mathproxyagent detects '\\boxed{}' as termination, you need to have a similar termination sentence in the prompt: \"If you get the answer, put the answer in \\\\boxed{}.\". If the customized is provided, the `prompt_type` will be ignored.\n",
|
|
"\n",
|
|
"\n",
|
|
"### Example 4 (Use the \"python\" prompt):\n",
|
|
"\n",
|
|
"Problem: If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ?\n",
|
|
"\n",
|
|
"Correct Solution: Subtracting the two equations gives: \n",
|
|
"\\begin{align*}\n",
|
|
"(729x+731y)-(725x+727y) &= 1508-1500\\\\\n",
|
|
"\\Rightarrow\\qquad 4x+4y &= 8\\\\\n",
|
|
"\\Rightarrow\\qquad x+y &= 2.\n",
|
|
"\\end{align*}\n",
|
|
"\n",
|
|
"Multiplying this equation by 725 and subtracting this equation from $725x+727y=1500$ gives \\begin{align*}\n",
|
|
"(725x+727y) - 725(x+y) &= 1500-725(x+y) \\implies \\\\\n",
|
|
"2y &= 50.\n",
|
|
"\\end{align*}So we can write $x-y$ as $(x+y) - 2y$, which equals $2 - 50 = \\boxed{-48}$.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# we set the prompt_type to \"python\", which is a simplied version of the default prompt.\n",
|
|
"math_problem = \"Problem: If $725x + 727y = 1500$ and $729x+ 731y = 1508$, what is the value of $x - y$ ?\"\n",
|
|
"mathproxyagent.initiate_chat(assistant, problem=math_problem, prompt_type=\"python\")"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Example 5 (Use the \"two_tools\" prompt)\n",
|
|
"\n",
|
|
"Problem: Find all numbers $a$ for which the graph of $y=x^2+a$ and the graph of $y=ax$ intersect. Express your answer in interval notation.\n",
|
|
"\n",
|
|
"\n",
|
|
"Correct Solution: If these two graphs intersect then the points of intersection occur when \\[x^2+a=ax,\\] or \\[x^2-ax+a=0.\\] This quadratic has solutions exactly when the discriminant is nonnegative: \\[(-a)^2-4\\cdot1\\cdot a\\geq0.\\] This simplifies to \\[a(a-4)\\geq0.\\] This quadratic (in $a$) is nonnegative when $a$ and $a-4$ are either both $\\ge 0$ or both $\\le 0$. This is true for $a$ in $$(-\\infty,0]\\cup[4,\\infty).$$ Therefore the line and quadratic intersect exactly when $a$ is in $\\boxed{(-\\infty,0]\\cup[4,\\infty)}$.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# The wolfram alpha appid is required for this example (the assistant may choose to query Wolfram Alpha).\n",
|
|
"import os\n",
|
|
"if \"WOLFRAM_ALPHA_APPID\" not in os.environ:\n",
|
|
" os.environ[\"WOLFRAM_ALPHA_APPID\"] = open(\"wolfram.txt\").read().strip()\n",
|
|
"\n",
|
|
"# we set the prompt_type to \"two_tools\", which allows the assistant to select wolfram alpha when necessary.\n",
|
|
"math_problem = \"Find all numbers $a$ for which the graph of $y=x^2+a$ and the graph of $y=ax$ intersect. Express your answer in interval notation.\"\n",
|
|
"mathproxyagent.initiate_chat(assistant, problem=math_problem, prompt_type=\"two_tools\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "flaml_dev",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.17"
|
|
},
|
|
"orig_nbformat": 4
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|