基于一致性的自适应提示 (COSP)¶

COSP 是一种旨在通过基于模型响应的一致性和置信度选择高质量示例来改进少样本学习的技术。这种方法通过识别模型能够可靠处理的示例，有助于创建更有效的提示。

概述¶

COSP 过程包括两个主要阶段

示例生成：为潜在示例生成多个响应
通过模型多次运行每个示例
收集响应和置信度分数
示例选择：基于熵和重复性选择最佳示例
计算响应的熵以衡量一致性
评估重复性以确保可靠性

COSP 工作原理¶

阶段 1：示例生成¶

对于数据集中的每个潜在示例

生成多个响应（通常为 3-5 个）
计算这些响应的熵
衡量响应间的重复性

from typing import List
from pydantic import BaseModel, Field
import instructor
from openai import OpenAI

class Response(BaseModel):
    content: str = Field(description="The model's response to the prompt")
    confidence: float = Field(description="Confidence score between 0 and 1")

client = instructor.from_openai(OpenAI())

def generate_responses(prompt: str, n: int = 3) -> List[Response]:
    responses = []
    for _ in range(n):
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            response_model=Response
        )
        responses.append(response)
    return responses

阶段 2：示例选择¶

计算每个示例的指标

熵：衡量响应变异性
重复性：检查响应一致性

import numpy as np
from scipy.stats import entropy

def calculate_metrics(responses: List[Response]) -> tuple[float, float]:
    # Calculate entropy
    confidences = [r.confidence for r in responses]
    entropy_score = entropy(confidences)

    # Calculate repetitiveness
    unique_responses = len(set(r.content for r in responses))
    repetitiveness = 1 - (unique_responses / len(responses))

    return entropy_score, repetitiveness

实现示例¶

以下是 COSP 实现的完整示例

from typing import List, Tuple
from pydantic import BaseModel, Field
import instructor
from openai import OpenAI
import numpy as np
from scipy.stats import entropy

class Example(BaseModel):
    text: str
    score: float = Field(description="Combined quality score")
    entropy: float = Field(description="Entropy of responses")
    repetitiveness: float = Field(description="Repetitiveness of responses")

class COSPSelector:
    def __init__(self, client: OpenAI, n_samples: int = 3):
        self.client = instructor.from_openai(client)
        self.n_samples = n_samples

    def generate_responses(self, prompt: str) -> List[Response]:
        return [
            self.client.chat.completions.create(
                model="gpt-4",
                messages=[{"role": "user", "content": prompt}],
                response_model=Response
            )
            for _ in range(self.n_samples)
        ]

    def calculate_metrics(self, responses: List[Response]) -> Tuple[float, float]:
        confidences = [r.confidence for r in responses]
        entropy_score = entropy(confidences)

        unique_responses = len(set(r.content for r in responses))
        repetitiveness = 1 - (unique_responses / len(responses))

        return entropy_score, repetitiveness

    def select_examples(self, candidates: List[str], k: int) -> List[Example]:
        examples = []

        for text in candidates:
            responses = self.generate_responses(text)
            entropy_score, repetitiveness = self.calculate_metrics(responses)

            # Combined score (lower is better)
            score = entropy_score - repetitiveness

            examples.append(Example(
                text=text,
                score=score,
                entropy=entropy_score,
                repetitiveness=repetitiveness
            ))

        # Sort by score (lower is better) and select top k
        return sorted(examples, key=lambda x: x.score)[:k]

使用示例¶

# Initialize COSP selector
client = OpenAI()
selector = COSPSelector(client)

# Candidate examples
candidates = [
    "The quick brown fox jumps over the lazy dog",
    "Machine learning is a subset of artificial intelligence",
    "Python is a high-level programming language",
    # ... more examples
]

# Select best examples
best_examples = selector.select_examples(candidates, k=3)

# Use selected examples in your prompt
selected_texts = [ex.text for ex in best_examples]
prompt = f"""Use these examples to guide your response:

Examples:
{chr(10).join(f'- {text}' for text in selected_texts)}

Now, please respond to: [your query here]
"""

COSP 的优势¶

改进一致性：通过选择低熵和高重复性的示例
更好的性能：更可靠的少样本学习
自动化选择：无需手动筛选示例
质量指标：示例质量的可量化衡量标准

局限性¶

计算成本：每个示例需要多次 API 调用
时间开销：对于大型候选集，选择过程可能很慢
模型依赖性：性能可能因模型而异

通用自适应提示 (USP)
思维链提示
自我一致性

参考资料¶

原始 COSP 论文：arXiv:2305.14121
相关工作：自我一致性提高了语言模型中的思维链推理能力