BobFlow makes AI data analysis a shared, inspectable workflow

A shared map of the analysis

In an arXiv preprint, Jiadong Bai, Yicong Huang and Chen Li present BobFlow, a system for human–agent collaboration on data science. Their central argument concerns the representation of an analysis: a visual dataflow can give analysts and agents a shared working structure, rather than leaving people to reconstruct the process from generated scripts and chat history.

A dataflow is a directed acyclic graph. Its nodes describe operations such as loading a file, filtering rows, joining tables or computing an aggregate; its edges show data dependencies. Analysts can inspect individual operations and their outputs, then request targeted changes. For example, they might change a comparison of movie budgets from release decade to genre while preserving other parts of the analysis.

BobFlow does not eliminate code. Operations outside its predefined set use custom operators with natural-language descriptions and separately registered implementations. The abstraction makes analytical structure visible, but the correctness of those implementations still matters.

Working, stopping and branching

The system combines dataflows with ReAct, an approach in which an agent alternates reasoning, actions and observations. BobFlow assembles context from the interaction history, asks a language model for actions, executes them and records the results. The agent can edit the graph, run selected operations and retrieve outputs or errors.

A graphical interface exposes the same dataflow to the analyst, alongside a chat interface. A context tracker records incremental changes to both the conversation and graph. Analysts can interrupt an ongoing analysis or return to an earlier snapshot and explore a new branch without deleting the original path.

The evaluated implementation builds on Apache Texera, using its Amber execution engine. Operator outputs are stored as Apache Iceberg tables, while PostgreSQL holds system metadata.

What the human study found

The researchers recruited 16 participants with experience ranging from none to advanced; 14 knew at least one programming language. Participants used BobFlow and a Python-based Script ReAct baseline across two scenarios, switching systems between scenarios. Both systems used Claude Haiku 5.5.

The first scenario asked participants to identify the continents with the most Olympic gold medals. Everyone assigned BobFlow reached the correct answer with their first request, whereas one third using Script ReAct needed additional requests. The second scenario involved finding and correcting a deliberately introduced price-parsing error in a rental-data analysis. Bug-identification rates were comparable, but BobFlow required fewer agent iterations and less model-inference time.

Questionnaire responses favored BobFlow most clearly for following data lineage: its average rating was 4.2 out of five, compared with 2.3 for Script ReAct. These findings concern a small study with two scenarios, not a demonstration that all analysts or tasks would benefit equally.

Benchmark gains, with exceptions

The automated evaluation compared BobFlow with Script ReAct using eight proprietary and open-weight language models. PrepBench supplied 306 table-preparation tasks; KramaBench supplied 104 broader data-analysis tasks. Accuracy was measured using each benchmark’s evaluator on a 0–100 scale.

BobFlow improved accuracy in 13 of the 16 model–benchmark combinations and reduced inference cost in 15. The largest accuracy gain occurred with Gemini 3.8 Flash on PrepBench: 18.6 points, accompanied by a 64.3% reduction in inference cost.

The results were not uniformly better. Three settings showed accuracy declines of 0.5–1.5 points, and DeepSeek V4.1-Flash on KramaBench cost 6.5% more. These monetary comparisons cover model inference calculated from token usage and provider prices, not total infrastructure or operating costs.

Smaller context is not always cheaper

BobFlow uses samples and table statistics to summarize intermediate results. On PrepBench, this optimization reduced inference cost by 14.2–31.1% across the eight models, with accuracy changes between −0.7 and +2.2 points. Computing statistics added about 10% to engine execution time for evaluated tasks processing more than one gigabyte and producing large results.

A second optimization removes historical result observations unrelated to the operations an analyst selects. On LongDS-Bench v1.1 Lite, covering 777 interactions, it reduced input-token usage by 31.7–56.9%. But cost depended on prompt caching: estimated cold-start costs fell by 18.9–44.2%, while measured warm-start costs rose by 37.0–182.9%. Editing earlier context reduced cache reuse.

The paper also documents a revealing failure: a custom parser skipped monetary amounts containing commas, and BobFlow accepted the plausible but wrong total without inspecting parsed values. The system offers more opportunities to check an analysis; it does not guarantee that the agent will use them.