Please solve the following questions
Case study · environment design for models
PIHF-MCP
An investigating model studies a fixed model’s failures and rewrites its instructions; the harness tests each change before keeping it.
How it works
- 01 / InvestigateExplain the failure
Inspect model trajectories and probe competing explanations.
- 02 / ReviseChange the policy
Scope a written instruction change. The executor’s weights stay fixed.
- 03 / TestKeep or roll back
Compare measurements, check regressions, and record the decision.
Expert investigation, written as a procedure. An investigating model carries it out using tools, reviewers, and evidence memory.
Explore the full harness, baseline comparison, and worked example
Prompt optimizers
The same small batches, cycled until the budget runs out
repeated passes · full re-scoring on validation
PIHF-MCP
Failure modes set the agenda; the Causal Investigator drives the work
↓ emulated by the pretrained modelNo training: the procedure is a written guide the investigator reads in context. With a generic prompt instead, on the same model, the investigator found no improvement and stopped after 102 of 600 calls.
Development cases
Correct: 3 of 23 under the starting policy, 12 of 23 under the selected one
Counts from the run; dot positions illustrative.
Failure modes
called as needed
One held-out case the investigator never saw · Dyck languages
Find the first mistaken step in a 52-step bracket-matching trace.
Input: ( [ ( ) ] < [ [ ] ] > ( …
Thought 4: [ ; stack: ( [ Thought 5: ( ; stack: ( [ ( Thought 6: ) ; stack: ( [ Thought 7: ] ; stack: empty … through Thought 52
Full case, verbatim
You are an expert in a language called dyck where you must complete the language sequence of unclosed brackets of all types (e.g., [], {}, <>). You are given an initial initial dyck language sequence and the steps, provided as
thoughts, that were used to arrive at the closing bracket sequence in the dyck language. Your job is to identify the first step that was a mistake in reasoning about the closing bracket sequence in dyck language, if any.
This can be forgetting to close a bracket or closing with the wrong bracket or incorrectly copying the prior subsequence of closing brackets to the next step.
Task: Complete the rest of the sequence, making sure that the parentheses are closed properly. Input: ( [ ( ) ] < [ [ ] ] > ( { { [ [ ( [ ] [ ] ) ] ] { [ ( [ ] ) ] } } ( < < { [ < > ] } > > ) } { { [ { { } } ] } } ) < >
Thought 1: We should process each input one by one and keep track of the stack configuration.
Thought 2: stack: empty
Thought 3: ( ; stack: (
Thought 4: [ ; stack: ( [
Thought 5: ( ; stack: ( [ (
Thought 6: ) ; stack: ( [
Thought 7: ] ; stack: empty
Thought 8: < ; stack: <
Thought 9: [ ; stack: < [
Thought 10: [ ; stack: < [ [
Thought 11: ] ; stack: < [
Thought 12: ] ; stack: <
Thought 13: > ; stack: empty
Thought 14: ( ; stack: (
Thought 15: { ; stack: ( {
Thought 16: { ; stack: ( { {
Thought 17: [ ; stack: ( { { [
Thought 18: [ ; stack: ( { { [ [
Thought 19: ( ; stack: ( { { [ [ (
Thought 20: [ ; stack: ( { { [ [ ( [
Thought 21: ] ; stack: ( { { [ [ (
Thought 22: ] ; stack: ( { { [ [
Thought 23: ] ; stack: ( { { [
Thought 24: { ; stack: ( { {
Thought 25: [ ; stack: ( { { [
Thought 26: ( ; stack: ( { { [ (
Thought 27: [ ; stack: ( { { [ ( [
Thought 28: ] ; stack: ( { { [ (
Thought 29: ] ; stack: ( { { [
Thought 30: } ; stack: ( {
Thought 31: ( ; stack: (
Thought 32: < ; stack: ( <
Thought 33: < ; stack: ( < <
Thought 34: { ; stack: ( < < {
Thought 35: [ ; stack: ( < < { [
Thought 36: < ; stack: ( < < { [ <
Thought 37: > ; stack: ( < < { [
Thought 38: > ; stack: ( < < {
Thought 39: } ; stack: ( <
Thought 40: } ; stack: (
Thought 41: { ; stack: {
Thought 42: { ; stack: { {
Thought 43: [ ; stack: { { [
Thought 44: { ; stack: { { [ {
Thought 45: } ; stack: { { [
Thought 46: } ; stack: { {
Thought 47: } ; stack: {
Thought 48: ) ; stack: empty
Thought 49: < ; stack: <
Thought 50: > ; stack: empty
Thought 51: Now, we have reached the end. The final stack is "<".
Thought 52: We will need to pop out "<". So the answer is >
Q: Is there a mistake in this sequence? Write "No" if there are no mistakes, or the number N if there is a mistake in Thought N.Comparable or better scores at a fraction of the cost
PIHF-MCP vs GEPA with the same GPT-4.1-mini executor, pooled over HoVer and LiveBench-Math
Held-out scoreSame GPT-4.1-mini executor. PIHF-MCP: mean ± SD over three test runs; GEPA: published single scores.: HoVer 59.9 ± 0.4 vs 51.7 · LiveBench-Math 63.8 ± 2.9 vs 64.1
GEPA examples, rollouts and scores are its published values. GEPA time and cost are estimates that apply PIHF-MCP’s measured per-rollout cost and latency to GEPA’s reported rollouts. PIHF-MCP scores are means ± SD over three test runs.
Read the policies and compare development rounds
What the harness wrote
The executor’s instructions, verbatim. Step through the rounds to see what the investigator changed and what each version scored.
Solve the benchmark problem carefully and completely. Use the reasoning field as scratch work; put only the final requested answer in the answer field. Core procedure 1. Read the whole input, including definitions, exceptions, examples, and the exact answer format. Identify the task and the precise object requested. 2. Build an explicit representation suited to the task: stack, entity ledger, constraint grid, expression tree, timeline, table, graph, or coordinate list. Do not solve from a quick impression. 3. Execute all relevant steps. Enumerate every qualifying row/item before aggregating; update every affected entity after each state change; evaluate choices separately; and treat supplied task rules as authoritative. Do not claim missing information until checking whether it is stated or derivable. 4. Audit independently. Replay the operations from the source, check constraints against the proposed result, and verify arithmetic signs, indexing, units, date rollover, order, and rounding. When feasible use a checksum such as totals before/after, inverse replay, or a second calculation. Use these safeguards only when relevant: - If judging a supplied chain of thoughts/steps, independently recompute each step in order and compare it with the immediately preceding valid state. Return the earliest wrong step, not a later consequence. Synthetic pattern: start x=1, rule x←x+2; “step 1: x=3; step 2: x=6” has its first error at step 2. - For Boolean or arithmetic expressions, preserve parentheses and evaluate recursively from innermost subexpressions; record intermediate truth/numeric values. - For merged/corrupted tables, first reconstruct every logical row column-by-column, then apply the filter, then compute. Keep separate lists of included values for each statistic. - For swaps, possessions, schedules, or changing properties, maintain a complete ledger after every operation and replay it once to catch omissions. - For dates/times, apply one offset at a time to an explicit date, clock, and weekday, checking modulo-7 and midnight crossings. - For causal or social-judgment questions, distinguish the abnormal/intervening cause from a compliant background or enabling condition and answer the majority-reading question actually asked. Finally, format exactly as requested. The answer field must contain only the required token, value, option, or ordered list—never an explanation, equation, option text, preface, or trailing comment. Use delimiters only when the requested answer format requires them; preserve required spelling, order, separators, and rounding.
added in this roundremoved in this round
Gains over the starting policy
Fixed GPT-5.6-Luna executor at low effort, GPT-5.6-Sol investigator; mean ± SD of three held-out runs.
| Benchmark | Cases | Starting | Selected |
|---|---|---|---|
| HoVer | 300 | 46.0% | 68.2% ± 2.2 |
| BIG-Bench Extra Hard Mini | 177 | 25.4% | 42.4% ± 2.5 |
| LiveBench-Math | 126 | 73.9% | 78.6% ± 1.0 |
Notes
Contribution. I designed PIHF and implemented PIHF-MCP.
Go deeper. Read the paper, From Answers to Policies: Efficient In-Context Learning System through Emulating Expert Investigation.