Behavioral Analysis in Zero-Shot Vision-and-Language
Navigation in Continuous Environments
9 scenes · R2R-CE val-unseen
Success rate
SPL
nDTW
Complete-system results on 50 episodes selected from the 100-episode evaluation pool used by Open-Nav.
Three findings
We follow model responses through observations, executed actions, and the end of each trajectory.
Recorded responses distinguish the “second on the left” from neighboring openings and use later views to confirm an earlier turn’s instruction match. These judgments use observations and workflow-supplied history.
Follow the evidenceAdditional views reveal a hidden passage or support rejecting an uncertain hallway candidate. Across the run, 36 of 66 adjustment processes show local relief, and 48 of 75 verification processes clarify a judgment with supporting evidence.
See an adjustmentAn unfinished crossing is recognized while rotation continues. Although 52.0% of episodes meet the endpoint success criterion, only 36.0% both succeed and end with a workflow-accepted STOP; another 16.0% succeed at the step limit.
Inspect the outcomesRecorded navigation
Four complete observation replays from the evaluated run. Follow a useful adjustment, a successful stop, or the moment progress stalls.
Action adjustment
Instruction excerptGo through the door and turn left.
At Step 23, GPT-6-Astra proposes moving forward to see past a door leaf. The next observation exposes the left-side floor; subsequent turns align the view with the passage.
Jump to an observation
Original saved RGB frames, shown at 2 observations per second. Step t is the observation after t actions; model-call delays are omitted. Descriptions also use the saved decisions and execution receipts. These selected cases illustrate behavior, not its frequency.
The study
We study GPT-6-Astra in a zero-shot Vision-and-Language Navigation in Continuous Environments (VLN-CE) system, where it interprets instructions, assesses its surroundings, and proposes actions. The system uses a common observation–decision–execution workflow with direct model API calls, without a packaged agent harness or navigation-specific fine-tuning. In this workflow, each request receives selected observations, execution feedback, and retained progress records. Evaluation covers the complete system, including context management and action control.
We evaluate the system on 50 of the 100 R2R-CE val-unseen episodes used by Open-Nav. It achieves a success rate of 52.0%, an SPL of 48.9%, and an nDTW of 70.8%. Our analysis highlights three findings. First, recorded responses link landmarks and earlier actions to instructions using observations and supplied history. Second, reviews include requests for additional views and revisions of uncertain judgments. Third, the results suggest a gap between task understanding and autonomous completion: an unfinished crossing is recognized while rotation continues.
At termination, 36.0% of episodes succeed with a workflow-accepted STOP, while another 16.0% meet the distance criterion at the step limit. These results highlight a central challenge: translating correct local judgments into sustained progress and appropriate stopping.
The navigation workflow
Direct model API calls, without a packaged agent harness or navigation-specific fine-tuning.
Interprets the instruction, assesses progress, proposes actions, and reviews arrival evidence.
Selects context, retains records, schedules reviews, and applies action and stopping constraints.
Links model judgments to actual motion. Results describe the complete system under these conditions.
Inside the trajectories
Explore the analyses behind the three findings. Each perspective connects an interpretation to what the system actually did.
Visual–language grounding
Identifying a bathroom does not establish which bathroom the instruction means. Qualifiers such as “second on the left” must agree with the observed route.
One terminal destination reference per episode. These counts measure supported evidence; missing or ambiguous evidence is not automatically an error.
Read Section 5.1 ↗
Spatial judgments and executed motion
Position, heading, passage clearance, and route connectivity determine what a visible opening means for the next action.
The bedroom is visible, but recorded restrictions block forward movement. Correctly identifying the pending crossing does not make the crossing happen.
Cases combine model judgments, controller constraints, and execution receipts. They do not establish that bypassing a restriction would be safe.
Read Section 5.2 ↗
Temporal reasoning with supplied history
In EP423, later views confirm that an earlier turn matched the instruction. The time of the action remains distinct from the time of confirmation.
Counts are episodes with support for every applicable check. Judgments use workflow-supplied history; different eligibility rules prevent ranking these as independent abilities.
Read Section 5.3 ↗
Action adjustment and local outcomes
In EP244, GPT-6-Astra proposes moving past a nearby door leaf. The executed movement reveals the hidden continuation.
adjustment processes show local relief
42 episodes contain eligible processes. Local relief is distinct from final navigation success.
Read Section 5.4 ↗
Evidence-guided verification
In EP308, additional views reveal shelving and a back wall. GPT-6-Astra rejects the uncertain hallway candidate, while the correct route remains unresolved.
verification processes clarify an uncertain judgment
Reviews are workflow-scheduled. The inventory does not measure an error-correction rate.
Read Section 5.5 ↗
Arrival and stopping
Endpoint success and accepted stopping tell different parts of the story.
Endpoint success
Success + accepted STOP
Success at step limit
Counts cover all 50 episodes. Success uses a 3 m endpoint distance criterion; accepted STOP combines model assessment with workflow checks.
| Termination | Success | Failure | Total |
|---|---|---|---|
| Workflow-accepted STOP | 18 | 3 | 21 |
| Budget termination | 8 | 21 | 29 |
| Total | 26 | 24 | 50 |
Interpreting the findings
These findings characterize GPT-6-Astra within the evaluated workflow and subset. Context selection, retained records, review scheduling, and action control all shape the observed behavior.
One run on a fixed 50-episode subset does not isolate the model’s contribution or establish full-benchmark superiority. The subset was not held out from all workflow development, and the retrospective, assistant-assisted annotations have not undergone independent human validation.
Discussion and limitations ↗