利用特定任务系统

可信思维链¹ 通过将其分解为两个阶段，提高了语言模型生成的推理链的可信度

1. 翻译：我们首先将用户查询翻译成一系列推理步骤。这些是我们可以确定性地执行的一组特定任务步骤。
2. 问题解决：我们执行步骤并得出可以推导出的最终答案。这确保了我们的思维链能够得出与推理步骤一致的答案。

他们在论文中列举了一些这些特定任务步骤的例子

1. 数学应用题：可由解释器执行的 Python 代码，用于得出最终答案
2. 多跳问答：这是一个多步骤推理过程。为了解决这个问题，他们使用 Python 和 Datalog（一种关系和逻辑编程语言）的组合来得出最终答案
3. 规划：在尝试生成解决用户查询的计划时，他们会用编程语言生成一系列符号目标，然后调用 PDDL 规划器来获得解决用户查询的计划

在下面的示例中，我们展示了如何使用大型语言模型生成可由解释器执行的 Python 代码以得出最终答案。

我们可以在 instructor 中实现它，如下所示

import instructor
from openai import OpenAI
from pydantic import BaseModel, Field

client = instructor.from_openai(OpenAI())


class ReasoningStep(BaseModel):
    id: int = Field(description="Unique ID")
    rationale: list[str] = Field(
        description="""Specific sections from prior reasoning
        steps or the context that ground this reasoning step"""
    )
    dependencies: list[int] = Field(
        description="""IDs of prior reasoning steps that this
        reasoning step depends on"""
    )
    eval_string: str = Field(
        description="""Python Code to execute to generate the
        final evaluation"""
    )


def generate_reasoning_steps(query: str) -> list[ReasoningStep]:
    return client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": """
                You are a world class AI who excels at
                generating reasoning steps to answer a
                question. You will be given a question
                and you will generate a list of reasoning
                steps that are needed to answer the
                question.

                At each point you should either
                - declare a variable to be referenced
                later on
                - combine multiple variables together to
                generate a new result that you should
                store in another variable

                The final answer should be stored in a
                variable called `answer`.
                """,
            },
            {"role": "user", "content": query},
        ],
        model="gpt-4o",
        response_model=list[ReasoningStep],
    )


if __name__ == "__main__":
    steps = generate_reasoning_steps(
        """If there are 3 cars in the parking lot and 2 more
        cars arrive, how many cars are in the parking lot
        after another 2 more arrive?"""
    )

    code = "\n".join([step.eval_string for step in steps])
    print(code)
    """
    initial_cars = 3
    arriving_cars = 2
    cars_after_first_arrival = initial_cars + arriving_cars
    final_car_count = cars_after_first_arrival + 2
    answer = final_car_count
    """
    exec(code)

    local_vars = {}
    exec(code, {}, local_vars)
    print(local_vars.get("answer"))
    #> 7

参考资料

¹: 可信思维链推理