API Reference¶
This reference documents the public API for ProofOfThought.
ProofOfThought¶
The main entry point for the reasoning system.
Location: z3adapter.reasoning.proof_of_thought.ProofOfThought
Constructor¶
def __init__(
self,
llm_client: Any,
model: str = "gpt-5",
backend: Literal["agentic", "json", "smt2"] = "agentic",
max_attempts: int = 3,
max_iterations: int = 10,
verify_timeout: int = 10000,
optimize_timeout: int = 100000,
cache_dir: str | None = None,
z3_path: str = "z3",
postprocessors: Sequence[str | Postprocessor] | None = None,
postprocessor_configs: dict[str, dict] | None = None,
agentic_config: AgenticConfig | None = None,
) -> None
Parameters:
llm_client: OpenAI/AzureOpenAI client instance (any OpenAI-compatible client)model: Deployment/model name (default:"gpt-5")backend:"agentic","smt2", or"json"(default:"agentic"— see Agentic Reasoning)max_attempts: Retry limit for single-shot generation, json/smt2 only (default:3)max_iterations: Tool-loop turn limit, agentic only (default:10)verify_timeout: Z3 timeout in milliseconds — governs all backends, including the agentic loop's in-process Z3 (default:10000)optimize_timeout: Optimization timeout in ms, JSON only (default:100000)cache_dir: Program cache directory (default:tempfile.gettempdir())z3_path: Z3 executable path for the SMT2 backend (default:"z3"; the agentic backend needs no CLI binary)postprocessors: Postprocessor names/instances — json/smt2 only; combining with the agentic backend raisesValueErrorat constructionagentic_config: Optionalz3adapter.agentic.AgenticConfigwith full control over the loop. When provided it is authoritative — set the model on it (a conflict withmodellogs a warning)
query()¶
def query(
self,
question: str,
temperature: float | None = None,
max_tokens: int = 16384,
save_program: bool = False,
program_path: str | None = None,
enable_postprocessing: bool = True,
) -> QueryResult
Parameters:
question: Natural language questiontemperature: Sampling temperature, honored on every backend.None(default) sends nothing and uses the provider default — required for models like GPT-5 that reject non-default temperaturesmax_tokens: Max completion tokens per LLM response (default:16384)save_program: Save the generated program to disk — for the agentic backend, the final trajectory program (default:False)program_path: Custom save path (default: auto-generated, uniquified, incache_dir)enable_postprocessing: Apply configured postprocessors, json/smt2 only (default:True)
Returns: QueryResult
On the agentic backend, query() runs the iterative z3_solve ⇄ finish tool loop (see Agentic Reasoning). On json/smt2 it runs the classic generate → execute → feedback retry loop bounded by max_attempts.
QueryResult¶
Contains the results of a reasoning query.
@dataclass
class QueryResult:
question: str # Input question
answer: bool | None # Boolean view of answer_text; None when
# non-boolean OR no answer (check success)
json_program: dict[str, Any] | None # Generated program if JSON backend
sat_count: int # SAT occurrences (agentic: last verdict)
unsat_count: int # UNSAT occurrences (agentic: last verdict)
output: str # Raw Z3 output (agentic: last verdict's)
success: bool # An answer was produced
num_attempts: int # Attempts (json/smt2) / iterations (agentic)
error: str | None # Error message if failed
answer_text: str | None # CANONICAL answer, every backend
smt_history: list | None # Agentic trajectory: every program + verdict
iterations: int # Agentic tool-loop turns
verified: bool # Agentic: finish backed by a decisive verdict
proof_status: str | None # Agentic: "proof_by_contradiction" /
# "sat_witness" / "unverified"
Field contract (uniform across backends):
answer_textis the canonical answer — always populated when an answer was produced ("True"/"False"for json/smt2, the rawfinish()value for agentic). Prefer it for anything that isn't strictly boolean.answeris the boolean coercion ofanswer_textvia a small unambiguous vocabulary (yes/no/true/false/valid/invalid/verified/violated). Multiple-choice letters, numbers, and raw Z3 verdicts coerce toNone.successmeans "an answer was produced" — it does not imply formal verification. Verification strength lives inverifiedandproof_status.
ProofStatus¶
Location: z3adapter.agentic.ProofStatus (a StrEnum; values serialize as plain strings)
| value | meaning |
|---|---|
proof_by_contradiction |
last decisive verdict was a clean unsat of the negated candidate — a proof |
sat_witness |
last decisive verdict was a clean sat — a consistency witness, weaker |
unverified |
answer recorded without a usable verdict — an ordinary LLM guess |
See Agentic Reasoning for the full protocol, including finish rejection.
AgenticSolver / AgenticConfig / AgenticResult¶
The low-level agentic API, usable without ProofOfThought.
Location: z3adapter.agentic
from z3adapter.agentic import AgenticSolver, AgenticConfig
solver = AgenticSolver(client, AgenticConfig(model="gpt-5"))
result = solver.solve(question, answer_format="Answer with one of: Yes, No.",
temperature=None, max_tokens=None) # per-call overrides
AgenticConfig fields: model, max_tokens, temperature (None = don't send), max_iterations, max_consecutive_nudges, max_finish_rejections, z3_timeout_ms, lenient_extraction, system_prompt.
AgenticResult fields: question, answer, explanation, verified, proof_status, smt_history, messages (full transcript incl. reasoning channel), iterations, token_usage, error, extraction_method (tool_call_finish / tool_call_finish_unverified / text_pattern_extracted / none).
EvaluationPipeline¶
Facilitates batch evaluation of reasoning questions on datasets.
Location: z3adapter.reasoning.evaluation.EvaluationPipeline
Constructor¶
def __init__(
self,
proof_of_thought: ProofOfThought,
output_dir: str = "evaluation_results",
num_workers: int = 1,
) -> None
Parameters:
proof_of_thought: Configured ProofOfThought instanceoutput_dir: Results directory (default:"evaluation_results")num_workers: Parallel workers (default:1, usesThreadPoolExecutorif> 1)
evaluate()¶
def evaluate(
self,
dataset: list[dict[str, Any]] | str,
question_field: str = "question",
answer_field: str = "answer",
id_field: str | None = None,
max_samples: int | None = None,
skip_existing: bool = True,
) -> EvaluationResult
Parameters:
dataset: JSON file path or list of dictsquestion_field: Field name for question text (default:"question")answer_field: Field name for ground truth (default:"answer")id_field: Field for sample ID (default:None, auto-generatessample_{idx})max_samples: Limit samples (default:None, all)skip_existing: Skip cached results (default:True)
Returns: EvaluationResult
Scoring: boolean-like ground truths (bool, 0/1, "true"/"yes"/...) compare via the boolean answer; everything else — multiple choice, free form — compares whitespace/case-normalized answer_text. A sample is failed only when no answer was produced.
Caching behavior:
Results are cached by saving {sample_id}_result.json and {sample_id}_program{ext} files to output_dir. Each result file records answer, answer_text, verified, and proof_status, so verified-accuracy can be computed separately from raw accuracy.
EvaluationMetrics¶
Provides comprehensive metrics for evaluation results.
@dataclass
class EvaluationMetrics:
accuracy: float # correct / (correct + wrong)
precision: float # binary metrics: boolean-coercible pairs only
recall: float
f1_score: float
specificity: float
false_positive_rate: float
false_negative_rate: float
tp: int
fp: int
tn: int
fn: int
total_samples: int # correct + wrong + failed
correct_answers: int
wrong_answers: int
failed_answers: int # no answer produced
Binary classification metrics (precision/recall/f1_score/the confusion matrix) accumulate only for samples where both the answer and the ground truth coerce to booleans; non-boolean datasets still get accuracy via text comparison.
Backend¶
Defines the abstract interface for execution backends.
Location: z3adapter.backends.abstract.Backend
Interface Methods¶
class Backend(ABC):
@abstractmethod
def execute(self, program_path: str) -> VerificationResult:
pass
@abstractmethod
def get_file_extension(self) -> str:
pass
@abstractmethod
def get_prompt_template(self) -> str:
pass
def determine_answer(self, sat_count: int, unsat_count: int) -> bool | None:
if sat_count > 0 and unsat_count == 0:
return True
elif unsat_count > 0 and sat_count == 0:
return False
else:
return None
Concrete implementations: AgenticBackend, SMT2Backend, and JSONBackend.
AgenticBackend.reverify()¶
def reverify(self, program_path: str, proof_status: ProofStatus | str | None) -> bool
Re-runs a saved trajectory program and returns True when it still yields the verdict that backed the original answer (unsat for proof_by_contradiction, sat for sat_witness; always False for unverified/None). This encapsulates the negation polarity — execute()'s answer field reports program-level satisfiability (sat → True), which for a contradiction proof is not the question's answer.
VerificationResult¶
Encapsulates the results of Z3 verification execution.
@dataclass
class VerificationResult:
answer: bool | None # Program-level: True (SAT), False (UNSAT), None (ambiguous/error)
sat_count: int
unsat_count: int
output: str # Raw execution output
success: bool # Execution completed without exception
error: str | None # Error message if failed
Z3ProgramGenerator¶
Handles LLM-based single-shot program generation with error recovery (json/smt2 backends).
Location: z3adapter.reasoning.program_generator.Z3ProgramGenerator
generate()¶
def generate(
self,
question: str,
temperature: float | None = None,
max_tokens: int = 16384,
) -> GenerationResult
temperature is only sent to the API when explicitly set; None uses the provider default (GPT-5-safe).
generate_with_feedback()¶
Enables multi-turn conversation with error feedback:
messages=[
{"role": "user", "content": prompt},
{"role": "assistant", "content": previous_response},
{"role": "user", "content": feedback_message},
]
Utility: Azure Config¶
Provides convenient configuration for Azure OpenAI deployments.
Location: utils.azure_config.get_client_config()
Returns:
{
"llm_client": AzureOpenAI(...),
"model": str # Deployment name from env
}
Required environment variables:
AZURE_OPENAI_API_KEYAZURE_OPENAI_ENDPOINTAZURE_OPENAI_API_VERSIONAZURE_GPT5_DEPLOYMENT_NAMEorAZURE_GPT4O_DEPLOYMENT_NAME