From “Help Me Optimize Food Delivery” to Computable Decision Space: A Six-Module Transformation Framework

Why do most RL projects fail? The problem is not the algorithm, but the first step.

1. A Common Scenario

The product manager says: “Help me optimize food delivery.”

The engineer hears: riders, orders, routes, time.

The RL algorithm sees: a high-dimensional state vector—order distribution, rider locations, traffic flow, restaurant preparation times, user patience thresholds—and a search space that may never find the optimal solution.

The most common failure mode in transformation: mapping natural language requirements directly to training objectives while skipping structured modeling. Result: the Agent performs well in the training environment but collapses rapidly after deployment to reality.

2. Six-Module Framework Overview

This framework progresses layer by layer from problem abstraction to complexity compression, with actionable self-check checklists at each layer.

3. Module Details: Key Insights and Pitfalls

3.1 Problem Abstraction: Finding the True Objective

User requirements are often proxy metrics, not true objectives.

Case: Increasing Click-Through Rate (CTR) vs. User Long-Term Retention (LTV). Agent optimizes CTR → clickbait floods → short-term CTR rises, but long-term LTV drops.

Self-Check Checklist:

• If the Agent successfully optimizes the metric proposed by the user, does the business objective necessarily achieve?
• Are there ignored constraints (compliance, brand safety)?
• When the objective is not quantifiable, a human-in-the-loop mechanism needs to be introduced.

3.2 State Space: The Goldilocks Principle

Cannot be too “fat” (redundancy leads to dimension disaster), nor too “thin” (missing key variables).

Dialogue Agent Case: State does not contain “user emotion” → Agent continues to sell when the user is already angry → satisfaction drops sharply. Solution: introduce emotional state dimension.

The brutal reality of dimension explosion: 10 variables, each with 5 states, combination count is about 10 million. 100 variables? 5^100, although less than the number of atoms in the universe (10^80), is still an astronomical number that any computational method cannot enumerate.

State abstraction must be done: clustering, embedding dimension reduction, hierarchical abstraction.

3.3 Action Space: Don’t Forget “Inaction”

A common but fatal error: forgetting that “waiting” or “taking no action” is also an action.

Scenarios: Hold positions when financial markets are uncertain; observe rather than immediately intervene when medical diagnosis test results are unclear; silently listen when user emotions are intense.

Action Space Completeness Check: After listing all actions, ask yourself: “Does it include ‘maintain status quo’ and ‘wait’?“

3.4 Transition Function: Reality Is Not Markovian

Markov Assumption: The next state only depends on the current state. Reality almost never holds. The user’s next sentence depends not only on the current state but also on the previous 5 rounds of dialogue history.

Delayed feedback is another killer: advertising today, conversions may occur 7 days later. Modeling methods: introduce time steps, state caching, credit assignment.

Simulator is infrastructure: Without a simulator, training costs are extremely high. Simulators must be high-fidelity (key metric error <5%), fast (single inference <100ms), parallelizable, and configurable for extreme scenarios.

3.5 Reward Design: The Most Insidious Failure Mode

Reward Hacking: Agent finds “loopholes” in the reward function, maximizing rewards in unexpected ways.

Classic Cases:

• Robot grasping: reward includes “hand close to object” → Agent learns to place hand above object but never grasps (grasping has failure risk).
• Customer service dialogue: reward based on “dialogue duration” → Agent deliberately delays dialogue.

Solutions: Counterfactual verification (“Is this strange high-reward behavior really what we want?”), Human Preference Reward Model (RLHF), Multi-objective Pareto optimization.

Multi-objective Trade-off: Cost vs. Quality vs. Speed. Weight selection itself is a value judgment, which should be explicitly specified by business decision-makers rather than implicitly decided by the algorithm.

3.6 Complexity Compression: From Exponential to Linear

State-action space scale calculation:

• Single step: S × A
• T-step trajectory: S × A^T (exponential growth)

If S × A > 10^6, traditional tabular methods fail, neural networks must be used. If > 10^12, state abstraction and action compression are needed.

Hierarchical Modeling: High level selects “strategy options” (e.g., “go to airport”), low level executes atomic actions (“turn left → go straight → turn right”). Total complexity drops from multiplication to addition.

4. Complete Case: Dialogue Agent Intent Understanding

Real-world Problem: “Make the Agent better understand user intent and solve problems.”

Transformation Process:

Transformation Result: Original problem (vague) → {S(608D vector), A(6 action types + mask), T(dialogue simulator), R(satisfaction + resolution rate + cost)} → standard decision space that can be processed by PPO or MCTS.

5. When to Use This Framework? When Not To?

Applicable Scenarios

• High-dimensional sequential decision-making: path planning, scheduling, dialogue systems
• Strategic games: Go, poker, bidding
• Physical control: robotic arms, autonomous driving (requires high-fidelity simulator)

Inapplicable Scenarios

• Single-step classification/prediction: spam detection (degrades to supervised learning)
• Pure creative generation: poetry, painting (reward function difficult to define, “good/bad” subjective)
• Problems themselves not quantifiable: open-ended exploration, art evaluation

Key Prerequisites

• Requires domain expert participation in state/action definition
• High-fidelity simulator construction cost may account for 50%+ of budget
• Reward function design requires repeated iteration and manual validation

6. Core Conclusions

1. The framework is a checklist, not a silver bullet. It helps experienced RL teams think systematically, but cannot replace domain knowledge.
2. Algorithm selection contributes <20% to success or failure. State space definition, reward function alignment, and simulator fidelity are the decisive factors.
3. First do simple solution comparison. Is a rule-based system sufficient? If a rule-based system can achieve 90% effectiveness, the marginal benefit of RL may not be worth the investment.

7. Six-Module Quick Reference Table