An extended application to create charts with commands in natural language using CodeX
In a previous article, I presented a prototype of a method that allows the creation of information graphs and plots using instructions in natural language. In this article, I would like to give an extended and revised version and show the results of a study with users alongside some examples.
The work was presented at the 26th International Conference Information Visualisation (IV) on 19–22 July 2022 at the “Technische Universität Wien, Austria” and the paper appears in the conference publications.
Natural language interfaces have already found their way into software products for visual data analysis. They are designed to help people analyze and visualize data using various analytical methods. The market leaders in commercial software ”Tableau” and Microsoft with ”Power BI” have integrated corresponding components in their current releases.
With Tableau’s ”Ask Data”, you can enter a question in a common language and immediately receive an answer directly in Tableau. The answers come in the form of automatic data visualizations, without you having to manually drag and drop fields, invoke menus, or understand the intricacies of the data structure.
Microsoft’s counterpart is called ”Power BI Q&A”. However, getting the instructions right is still a major hurdle for users. More powerful language models may be able to help here.
In recent years, powerful new language models based on the Transformer architecture have emerged through pretraining on large text datasets, dominating all current NLP benchmarks. The aim of this work is to explore whether, and in what way, NLIs for data visualization benefit from these models. For this purpose, I have created a prototype of visualization software based on the OpenAI Codex model.
A second prototype implements the baseline pipeline for NL-based data visualization. It uses the open-source ”Natural Language for Data Visualization”(NL4DV) toolkit to interpret NL utterances. The toolkit takes as input a dataset and an utterance corresponding to that dataset, returning a JSON object that includes an ordered list of Vega-Lite specifications that can be presented as output.
In order to test the possibilities of chart generation with the help of a deep learning language model, a prototype of a web application was created that allows data visualizations to be created using text inputs and then refined with context-related palettes. The text commands are used to define the basic structure of the visualizations, and the palettes can then be used to set additional specifications such as line widths or colors (Figure above). The prototype is a mixed-initiative visualization system that allows both NL input and direct manipulation interactions through a graphical user interface (GUI).
For data handling, the prototype offers the possibility to upload CSV files or to load already existing ones via a select box. Below this, the data fields are displayed together with their data type. As export options, PNG images and an HTML export are provided in the prototype.
In addition to the possibility of watching a short tutorial video, the central area of the screen offers a multi-line text input field in which the instructions are entered in natural language. Below this, the generated visualization is displayed. In one prototype variant, this is a Matplotlib graphic, and in the other a Vega-Lite chart.
The right column offers in addition to the text commands some GUI elements. A title can be assigned, the axes can be labeled and the limits can be set. Other elements, such as line width, color palettes, or colors of bars are displayed depending on the chart type created.
Both versions of the prototype were realized as a web application with Python. The framework ”Streamlit” was used in each case. Data can be processed in CSV format and exported as PNG or HTML. The two variants differ in terms of NLP and visualization. The following chart shows the system architecture of the version using CodeX for code generation.
Details on the different parts can be found in the paper.
I tested the application with data from Gapminder.org on the evidence-based worldview for illustration and evaluation. The test persons had to solve the following exercises:
- Create histograms for the numerical characteristics
- Plot the numerical characteristics per continent using bar charts.
- Display the GDP per country with the help of a bar chart. The top 15 countries should be displayed in descending order.
- Create a pie chart that shows the distribution of the world’s population by continent.
- Create a graph that shows the time course of mean life expectancy averaged over all countries. Using a line chart for this.
- Show the relationship between GDP per inhabitant of a country and life expectancy with the help of a scatter diagram. In addition, use a color code that reflects the population size.
- Show the relationship between GDP per inhabitant of a country and life expectancy and population size using a bubble chart.
Details on the experimental procedure can be found in the paper.
Participants gave the prototype with the deep learning language model an average SUS score of 71.1 and the version with NL4DV an average score of 68.2. A score of ≥ 68 is considered an indicator of good usability.
The evaluation showed that the deep learning model has the potential for expressive text input, but does not always produce entirely predictable results and can therefore confuse users.
The Codex language model I use is a model for generating general source code and is not optimized for our task. At present, it is possible to fine-tune natural language variants of GPT3 via the API and thus the models can be better adapted to tasks. This is not yet possible for the Codex model. As soon as this is possible, I expect great progress for my approach, because here it will be possible to work specifically with source code to create infographics, and thus more targeted charts.
The paper will be soon available at IEEE Xplore Digital Library https://ieeexplore.ieee.org/xpl/conhome/1000370/all-proceedings