Case study · environment design for models

PIHF-MCP

An investigating model studies a fixed model’s failures and rewrites its instructions; the harness tests each change before keeping it.

How it works

  1. 01 / InvestigateExplain the failure

    Inspect model trajectories and probe competing explanations.

  2. 02 / ReviseChange the policy

    Scope a written instruction change. The executor’s weights stay fixed.

  3. 03 / TestKeep or roll back

    Compare measurements, check regressions, and record the decision.

Expert investigation, written as a procedure. An investigating model carries it out using tools, reviewers, and evidence memory.

Explore the full harness, baseline comparison, and worked example
A

Prompt optimizers

The same small batches, cycled until the budget runs out

repeated passes · full re-scoring on validation

B

PIHF-MCP

Failure modes set the agenda; the Causal Investigator drives the work

Expert scientific method distilled from feedback on a rare-disease diagnostic agentThe investigator's procedure was distilled from expert feedback on a rare-disease diagnostic agent: 82 feedback episodes across 9 development sessions, reduced to six moves: review trajectories, explain the failure, probe before revising, scope the change, gate it on tests, roll back regressions.

↓ emulated by the pretrained modelNo training: the procedure is a written guide the investigator reads in context. With a generic prompt instead, on the same model, the investigator found no improvement and stopped after 102 of 600 calls.

Development cases

Correct: 3 of 23 under the starting policy, 12 of 23 under the selected one

Counts from the run; dot positions illustrative.

Failure modes

Procedural under-execution target
Exact-answer pollution target
Missing knowledge bound
A model (here GPT-5.6-Sol) that runs the inquiry: it weighs competing explanations, picks one informative change, runs it, compares measurements and decides. It works through the harness's 32 MCP tools, which record each step.Causal Investigatorpropose · test · review · decideGPT-5.6-Sol
Toolsrun and probe casesTyped actions to inspect evidence, probe a hypothesis, propose a change, execute cases on an isolated worker, compare measurements and select. The worker never sees reference answers or research history.
Reviewerscritic + verifiers
Curatorevidence memory

called as needed

✓ Effort follows the evidence 134 of 600 calls, then stop

One held-out case the investigator never saw · Dyck languages

Find the first mistaken step in a 52-step bracket-matching trace.
Input: ( [ ( ) ] < [ [ ] ] > ( …

Thought 4: [ ; stack: ( [
Thought 5: ( ; stack: ( [ (
Thought 6: ) ; stack: ( [
Thought 7: ] ; stack: empty
…  through Thought 52
Full case, verbatim
You are an expert in a language called dyck where you must complete the language sequence of unclosed brackets of all types (e.g., [], {}, <>). You are given an initial initial dyck language sequence and the steps, provided as
thoughts, that were used to arrive at the closing bracket sequence in the dyck language. Your job is to identify the first step that was a mistake in reasoning about the closing bracket sequence in dyck language, if any.
This can be forgetting to close a bracket or closing with the wrong bracket or incorrectly copying the prior subsequence of closing brackets to the next step.
Task: Complete the rest of the sequence, making sure that the parentheses are closed properly. Input: ( [ ( ) ] < [ [ ] ] > ( { { [ [ ( [ ] [ ] ) ] ] { [ ( [ ] ) ] } } ( < < { [ < > ] } > > ) } { { [ { { } } ] } } ) < >
Thought 1: We should process each input one by one and keep track of the stack configuration.
Thought 2: stack: empty
Thought 3: ( ; stack: (
Thought 4: [ ; stack: ( [
Thought 5: ( ; stack: ( [ (
Thought 6: ) ; stack: ( [
Thought 7: ] ; stack: empty
Thought 8: < ; stack: <
Thought 9: [ ; stack: < [
Thought 10: [ ; stack: < [ [
Thought 11: ] ; stack: < [
Thought 12: ] ; stack: <
Thought 13: > ; stack: empty
Thought 14: ( ; stack: (
Thought 15: { ; stack: ( {
Thought 16: { ; stack: ( { {
Thought 17: [ ; stack: ( { { [
Thought 18: [ ; stack: ( { { [ [
Thought 19: ( ; stack: ( { { [ [ (
Thought 20: [ ; stack: ( { { [ [ ( [
Thought 21: ] ; stack: ( { { [ [ (
Thought 22: ] ; stack: ( { { [ [
Thought 23: ] ; stack: ( { { [
Thought 24: { ; stack: ( { {
Thought 25: [ ; stack: ( { { [
Thought 26: ( ; stack: ( { { [ (
Thought 27: [ ; stack: ( { { [ ( [
Thought 28: ] ; stack: ( { { [ (
Thought 29: ] ; stack: ( { { [
Thought 30: } ; stack: ( {
Thought 31: ( ; stack: (
Thought 32: < ; stack: ( <
Thought 33: < ; stack: ( < <
Thought 34: { ; stack: ( < < {
Thought 35: [ ; stack: ( < < { [
Thought 36: < ; stack: ( < < { [ <
Thought 37: > ; stack: ( < < { [
Thought 38: > ; stack: ( < < {
Thought 39: } ; stack: ( <
Thought 40: } ; stack: (
Thought 41: { ; stack: {
Thought 42: { ; stack: { {
Thought 43: [ ; stack: { { [
Thought 44: { ; stack: { { [ {
Thought 45: } ; stack: { { [
Thought 46: } ; stack: { {
Thought 47: } ; stack: {
Thought 48: ) ; stack: empty
Thought 49: < ; stack: <
Thought 50: > ; stack: empty
Thought 51: Now, we have reached the end. The final stack is "<".
Thought 52: We will need to pop out "<". So the answer is >
Q: Is there a mistake in this sequence? Write "No" if there are no mistakes, or the number N if there is a mistake in Thought N.
✗ Starting policy · answer 22“At Thought 21, the top of the stack is `(` after closing the `[` opened at Thought 20. Thought 22 closes with `]`, which mismatches the top `(`. Thus Thought 22 is the first mistake.”
Policy line the investigator wroteIf judging a supplied chain of thoughts/steps, independently recompute each step in order and compare it with the immediately preceding valid state. Return the earliest wrong step, not a later consequence.
✓ Selected policy · answer 7 · 3 of 3 runs“After Thought 6, the stack is `( [`. Processing `]` closes only `[`, leaving `(` on the stack. Thought 7 incorrectly states the stack is empty, so it is the first mistake.”

Comparable or better scores at a fraction of the cost

PIHF-MCP vs GEPA with the same GPT-4.1-mini executor, pooled over HoVer and LiveBench-Math

GEPAPIHF-MCP
Examples
1/5
692
143
Rollouts
1/25
8,890
355
TimeGEPA est.
1/9
18.3 h
2.0 h
CostGEPA est.
1/3
$38.45
$11.98

Held-out scoreSame GPT-4.1-mini executor. PIHF-MCP: mean ± SD over three test runs; GEPA: published single scores.: HoVer 59.9 ± 0.4 vs 51.7 · LiveBench-Math 63.8 ± 2.9 vs 64.1

GEPA examples, rollouts and scores are its published values. GEPA time and cost are estimates that apply PIHF-MCP’s measured per-rollout cost and latency to GEPA’s reported rollouts. PIHF-MCP scores are means ± SD over three test runs.

Read the policies and compare development rounds

What the harness wrote

The executor’s instructions, verbatim. Step through the rounds to see what the investigator changed and what each version scored.

Starting policy
Please solve the following questions
Selected policy (v2)
Solve the benchmark problem carefully and completely. Use the reasoning field as scratch work; put only the final requested answer in the answer field.

Core procedure
1. Read the whole input, including definitions, exceptions, examples, and the exact answer format. Identify the task and the precise object requested.
2. Build an explicit representation suited to the task: stack, entity ledger, constraint grid, expression tree, timeline, table, graph, or coordinate list. Do not solve from a quick impression.
3. Execute all relevant steps. Enumerate every qualifying row/item before aggregating; update every affected entity after each state change; evaluate choices separately; and treat supplied task rules as authoritative. Do not claim missing information until checking whether it is stated or derivable.
4. Audit independently. Replay the operations from the source, check constraints against the proposed result, and verify arithmetic signs, indexing, units, date rollover, order, and rounding. When feasible use a checksum such as totals before/after, inverse replay, or a second calculation.

Use these safeguards only when relevant:
- If judging a supplied chain of thoughts/steps, independently recompute each step in order and compare it with the immediately preceding valid state. Return the earliest wrong step, not a later consequence. Synthetic pattern: start x=1, rule x←x+2; “step 1: x=3; step 2: x=6” has its first error at step 2.
- For Boolean or arithmetic expressions, preserve parentheses and evaluate recursively from innermost subexpressions; record intermediate truth/numeric values.
- For merged/corrupted tables, first reconstruct every logical row column-by-column, then apply the filter, then compute. Keep separate lists of included values for each statistic.
- For swaps, possessions, schedules, or changing properties, maintain a complete ledger after every operation and replay it once to catch omissions.
- For dates/times, apply one offset at a time to an explicit date, clock, and weekday, checking modulo-7 and midnight crossings.
- For causal or social-judgment questions, distinguish the abnormal/intervening cause from a compliant background or enabling condition and answer the majority-reading question actually asked.

Finally, format exactly as requested. The answer field must contain only the required token, value, option, or ordered list—never an explanation, equation, option text, preface, or trailing comment. Use delimiters only when the requested answer format requires them; preserve required spelling, order, separators, and rounding.

Gains over the starting policy

Fixed GPT-5.6-Luna executor at low effort, GPT-5.6-Sol investigator; mean ± SD of three held-out runs.

BenchmarkCasesStartingSelected
HoVer30046.0%68.2% ± 2.2
BIG-Bench Extra Hard Mini17725.4%42.4% ± 2.5
LiveBench-Math12673.9%78.6% ± 1.0

Notes

Contribution. I designed PIHF and implemented PIHF-MCP.

Go deeper. Read the paper, From Answers to Policies: Efficient In-Context Learning System through Emulating Expert Investigation.