Engelhardt, Y. (2007). Syntactic structures in graphics. Computational Visualistics and Picture Morphology, 5, 23-35.

Engelhardt, Y. (2007). Syntactic structures in graphics. Computational Visualistics and Picture Morphology, 5, 23-35.

圖形的目的是用來說明某種的資訊，使得原本無法看見的訊息能被看見 (visualizing the nonvisual)。依據過去的文獻，本研究建議將圖形的建構(building blocks)分為三個部分： a) 顯示的圖形物件 (graphic objects)、b) 配置這些物件使其具有意義來說明訊息的圖形空間 (graphic spaces)以及c) 這些物件的圖形性質 (graphic properties)。在此，圖形物件為一個遞迴的結構 (recursive structure)，也就是：在一個圖形空間上配置的一組圖形物件能夠共同形成一個更高階的圖形物件。

本研究也提出圖形物件的語法類型 (syntactic categories)，用來解釋物件彼此間可以允許的空間關係並且區分圖形的基本構成 (basic constituents)。所有的圖形都是建立在不同語法類型的圖形物件之組合的可能性上，不同語法類型的圖形物件在圖形呈現 (graphic representation)上有不同的行為，此一限制產生它們有不同的空間定位。所有圖形物件的語法類型可分為兩大群組：1) 依附於圖形空間位置上的物件；2) 依附於其他物件上的物件。前者包括節點 (node)、線標示 (line locator)、面標示 (surface locator)以及格標記 (grid marker)；後者則有標籤 (label)、連結線 (connector)、比例區段 (proportional segment)和框架 (frame)。以地圖為例，節點 、線標示、面標示可分別代表地圖上城市、河流與湖泊或國家的標示，格標記則用於表示地圖上的經緯度，標籤則是標明節點代表的城市名稱。各種圖形物件的類型、依附類型與例子，如下表：




根據語言的語法學概念，圖形的語法學 (the syntactics of graphics)研究不同語法類型的圖形物件之間的關係，研究圖形物件與圖形空間之間以規則與限制為基礎的關係，也研究圖形物件如何組合成複合的圖形物件以及複合的圖形物件如何能以更簡單的圖形物件分析。圖形可分為實體場景和物件的影像以及抽象的圖形，前者如照片與地圖等呈現實體的空間，後者則如家族樹 (family trees)、統計圖表 (statistical charts) 等呈現概念性的空間。照片與地圖等利用影像中的空間配置呈現出真實的空間配置；家族樹和圓餅圖 (pie charts) 則以影像中的空間配置呈現非空間性的資訊。然而，實體空間的呈現並不一定表達出被呈現物件的真實座標比例(the true coordinate proportions)，許多圖形也同時由實體和概念性的空間組合而成，例如在地圖上以高度呈現國家的人口密度分布。下表是若干圖形空間的類型與代表。

Building upon the existing literature, we are suggesting to regard the building blocks of all graphics as falling into three main categories: a) the graphic objects that are shown (e.g., a dot, a pictogram, an arrow), b) the meaningful graphic spaces into which these objects are arranged (e.g., a geographic coordinate system, a timeline), and c) the graphic properties of these objects (e.g., their colors, their sizes).

We suggest that graphic objects come in different syntactic categories, such as nodes, labels, frames, links, etc. Such syntactic categories of graphic objects can explain the permissible spatial relationships between objects in a graphic representation.

In addition, syntactic categories provide a criterion for distinguishing meaningful basic constituents of graphics.

It is about images that can be regarded as ‘visualizing the nonvisual’ in an attempt to clarify information of some sort. Such images are often collectively referred to as “graphics”.

In 1914, Willard Brinton writes in his book Graphic methods for presenting facts that “The principles for a grammar of graphic presentation are so simple that a remarkably small number of rules would be sufficient to give a universal language”.

In 1967, Jacques Bertin publishes his classic Sémiologie graphique, in which he analyses the “language” of graphic representations and the “visual variables of the image”.

In 1976, linguist Ann Harleman Stewart examines the properties of diagrams and claims that “Like any language, graphic representation has a vocabulary and a grammar”.

In 1984, Clive Richards proposes a “grammatically-based analysis” of diagrams in his Ph.D. thesis Diagrammatics.

In 1986, Jock Mackinlay suggests that “graphical presentations are actually sentences of graphical languages that have precise syntactic and semantic definitions”. In Mackinlay’s approach, “the syntax of a graphical language is defined to be a set of well-formed graphical sentences”.

In 1987, Fred Lakin publishes his paper “Visual grammars for visual languages”, in which he describes his approach to the “spatial parsing” of graphics, which he defines as “the process of recovering the underlying syntactic structure of a visual communication object from its spatial arrangement”.

Kress and van Leeuwen publish their book Reading images: the grammar of visual design (1996). Unfortunately, it is difficult to extract a systematic approach to a syntactic analysis of graphics from their book.

A paper titled “The visual grammar of information graphics” (1996) by Engelhardt et al., suggests “syntactic categories of visual components”.

Robert Horn, in his book Visual Language (1998), proposes a morphology and a syntax of visual language based partly on the work of Jacques Bertin and on the Gestalt principles of perception.

In his book The grammar of graphics (1999), Leland Wilkinson describes an approach to graphics that is related to object-oriented design in computer science. However, he uses grammatical terminology “metaphorically”, and not in a linguistic sense.

Colin Ware (2000) writes about the “perceptual syntax of diagrams”, describing “the grammar of node-link diagrams” and “the grammar of maps”.

Engelhardt, in his Ph.D. thesis The language of graphics (2002) provides a detailed proposal for the analysis of syntactic structure, which he applies to a broad spectrum of graphic representations.

We propose a notion of graphic objects that will allow for recursive structures: Any graphic representation – and any meaningful visible component of a graphic representation – may be referred to as a graphic object. This means that graphic objects can be distinguished at various levels of a graphic representation. For example, a map or a chart in its entirety is a graphic object. In addition, the various symbols or components that are positioned within that map or chart are graphic objects as well.

A bottom-up description of this principle was given above: a set of graphic objects can be arranged into a graphic space, together forming a single graphic object at a higher level. This “nesting” or “embedding” (Engelhardt 2002) of graphic structures can be referred to as “recursive composition” (Card 2003).

In technical terms, a meaningful graphic space could be defined as a graphic space that involves an interpretation function from spatial positions to one or more domains of information values.

In graphics, not only the possible constituents themselves (graphic objects), and the diverse possible ways of arranging these constituents (in meaningful graphic spaces), but also the possible visual appearances of these constituents (graphic properties such as size, color), could be considered as being part of the graphic “vocabulary”. In this sense we can say that the building blocks of graphics fall into three main categories: graphic objects, meaningful graphic spaces, and graphic properties.



To make a more general statement, we claim that all graphics are based on the possibility of combining graphic constituents (graphic objects) of different syntactic categories (Engelhardt et al. 1996, Engelhardt 2002, 2006).

Graphic objects of different syntactic categories “behave” differently in a graphic representation. The constraints that govern their spatial positioning are different.




All syntactic categories of graphic objects can be divided into two main groups: 1) objects that are attached to locations in graphic space (e.g., node, line locator, surface locator, grid marker are all attached to locations in graphic space), and 2) objects that are attached to other objects (label, connector, proportional segment, frame are all attached to other objects).

Richards (1984) believes that “there seems to be little profit in using such items as an individual dot or line as a unit of analysis. If we are going to use linguistics as a model, then what is needed for present purposes is not the pictorial equivalent of a phoneme or morpheme but something closer to a noun phrase”.

The basic graphic objects in a particular graphic representation are those that can be regarded as functioning in some syntactic category within that particular graphic representation (e.g., as a label, as a node, as a connector, as a proportional segment, etc.).

The distinction between syntactics, semantics, and pragmatics was introduced by Charles Morris (1938, 1946). Morris conceives of syntactics as the investigation of the relationships between signs, of the ways in which complex signs can be constructed from simple ones, as well as the ways in which complex signs can be analyzed into more simple ones (Morris 1946/1971).

The syntactics of graphics investigates the relationships between graphic objects of different syntactic categories. It investigates the rule- and constraint-based relationships between graphic objects (of different syntactic categories) and graphic spaces.

And syntactics investigates how graphic objects can be combined into composite graphic objects, and how composite graphic objects can be analyzed into more simple ones.

Looking at the broad spectrum of graphics we can say that images of physical scenes and objects, such as pictures and maps, represent physical spaces, while many abstract graphics, such as family trees and statistical charts, represent conceptual spaces (Engelhardt 1999, 2002).

In other words, pictures and maps use spatial arrangement in the image to represent spatial arrangement in the world, while family trees and pie charts use spatial arrangement in the image to represent non-spatial information.

Representations of physical spaces do, by the way, not always have to express the true co-ordinate proportions of the represented objects.

Many graphics combine physical and conceptual spaces.

As an example of a true hybrid space (Engelhardt 1999, 2002), think of a three-dimensional landscape drawing of a country in which the drawn “mountains” do not represent physical mountains, but – for example - population density, peaking in the cities and flat in the countryside. In this case, the horizontal plane represents the physical space of the country’s geography, while the vertical dimension represents the conceptual space of population density.




We claim that all types of graphic representation of information can be analyzed in terms of their composition from graphic spaces of different sorts.

We have tried to show that specifying such a visual language means a) specifying the syntactic categories of its graphic objects, plus b) specifying the graphic space in which these graphic objects are positioned, plus c) specifying the visual coding rules that determine the graphic properties of these graphic objects (see table 1).

The syntactic structure of a graphic representation is determined by the rules of attachment for each of the involved syntactic categories (see table 2) and by the structure of the meaningful graphic space that is involved (see table 3).

With this analysis we have attempted to demonstrate that Morris’ original notion of syntactics applies well to the structure of graphics.

Wanner, F., Stoffel, A., Jäckle, D., Kwon, B. C., Weiler, A., Keim, D. A., ... & Pfister, H. (2014). State-of-the-art report of visual analysis for event detection in text data streams. In Computer Graphics Forum (Vol. 33, No. 3).

Wanner, F., Stoffel, A., Jäckle, D., Kwon, B. C., Weiler, A., Keim, D. A., ... & Pfister, H. (2014). State-of-the-art report of visual analysis for event detection in text data streams. In Computer Graphics Forum (Vol. 33, No. 3).

近年來從文本串流中偵測事件已成為熱門的研究領域，然而由於對事件的概念沒有妥善的定義以及文本資料的種類繁多，對資料分析與視覺化是一個重大的挑戰，因此能夠處理特定事件類型與多樣性文字來源的視覺分析工具的建立準則十分缺乏。在本研究中，將事件視為是從文本資料中抽取出的對使用者有價值的非預期而獨特的樣式(unexpected and unique patterns)，並且建議由新聞標準(news criteria)或新聞價值(news values)[GR65]來界定事件的價值。

在從文本資料串流利用視覺分析進行事件偵測的研究中，資料的來源從有限而書寫良好的新聞文章到社交媒體上由使用者書寫、快速產生甚至有時沒結構的文字資料；而分析任務則可依其目的分為新事件偵測(new event detection)、事件追蹤(event tracking)、事件摘要(event summarization)以及事件關聯(event associations) (Dou et al., DWRZ12)。Becker [Bec11]對於社交媒體上的事件偵測研究，將事件依據3個面向區分：1) 計畫內 (planned) vs. 無計畫 (unplanned)、 2) 趨勢 (trending) vs 非趨勢 (non-trending)、 3) 外源 (exogenous) vs. 內源 (endogenous)。

本研究以Figure 1上的流程圖表示事件偵測與探索的處理過程，首先在輸入文件資料的前處理 。在前處理之後的方法，則可分為兩類：一類首先應用自動化方法偵測資料裡的事件，然後再利用這些資訊做為視覺分析(visual analysis)的介面；另一類則是直接對前處理後的結果進行視覺化，不進行事件的自動化分析。






以下分別說明文本資料來源、文本處理方法等技術分析的面向。

文字資料來源包括：新聞、電子郵件、部落格、RSS feed、微網誌 (microblogging) 訊息、論壇(forum)上的發文(post)、客服表單、影像與視頻串流上附註的文字。目前有大半的研究是針對微網誌資料，例如Twitter，而除了文字以外，微網誌上的地理位置和作者等後設資料也是許多研究會加以利用的。

在文句偵測(sentence detection)、(tokenizing)、詞幹化(stemming)和(lemmatizing)等文本處理方法之後，進行較深入的詞類標示(part-of-speech tagging)、語法剖析(syntactic parsing)、文句中詞語關係的類型剖析 (Typed-dependency parsing)、相互指涉解析(coreference resolution)、專有名詞辦認 (named entity recognition)、極性抽取 (polarity extraction)、歧義消除 (word sense disambiguation)等。自2000到2011年，33篇事件偵測的相關論文只有17篇利用文本處理方法，但在2012年後，這個情形改變了，18篇論文中便有14篇論文使用文本處理方法，主要是詞類標示和極性抽取。

事件偵測的自動化方法中常用的技術可分為1)群集為基礎、2)以分類為基礎、3)以統計為基礎、4)以預測為基礎、5)本體論為基礎、6)模式探勘(pattern mining)、7)資料串流重複特徵的模型、8)規則式等類型。
以群集為基礎的方法將文件依據內容的不同特性分群，當群組改變時便是事件產生。
分類則是根據事件建立分類器，當文件的分類結果為事件相關，便視為是事件產生。
統計方法中，相關分析類的方法測量文件集合在詞語或詞語與時間上的相關性改變來偵測事件，另一類則是從稀少或獨特的詞語出現來發現事件。
預測為基礎的方法根據過去的歷史預測接下來文件的出現情形。
本體論(ontologies)為基礎的方法適合單一領域的事件偵測，以全自動或半自動方法產生特定本體，當偵測到活躍概念(activated concepts)中的改變時便是可能的事件。

模式探勘(pattern mining)利用A-priori 演算法抽取文件串流上的常見連續模式。
資料串流重複特徵的模型可以用來偵測事件，當一個串流明顯偏離它預期的特徵時便是偵測到一個事件。
規則為基礎的方法以人工編寫的規則偵測事件，例如根據詞語或詞頻為規則來偵測特定的事件。

在視覺化呈現上，以時間為基礎的視覺化呈現佔有大多數，共21篇論文，利用包括河流 (river)、時間線 (timeline)與圓形 (circular)等時間為基礎的呈現凸顯資料在時間上的演變。時間線的呈現會將符號放置在一或多條時間線上，表現資料項目、密度與數量，能夠表現單一或稀少的事件。河流通常用來做為群集演算法結果的視覺化，提供各群集的分布與整體的數量，較著重在高頻率的事件上。以微網誌為資料來源的應用，通常會利用微網誌上附加的地理參考資料，以地圖的方式來呈現。折線圖或長條圖等基本的視覺化呈現方式通常運用來表現事件有關的資料在時間上的數量與頻率。在各種視覺化的應用中，文字資料的呈現通常選用具有意義的關鍵詞。

在支援的分析任務上，包括 1) 提供文件集合的概觀，描述集合內發現的主題，利於進一步的分析。2) 關鍵詞語搜尋以及利用後設資訊過濾資料，在分析或視覺化時減少資料或抽取的事件數量。3) 監測資料來源中事件在時間上的發展。4) 將偵測到的不同事件間的關係視覺化。

質性的評估方法包括案例研究(case study)、使用性評估(usability evaluation)、使用案例(use case)以及軼事評估(anecdotal evaluation)，其中使用案例最為盛行。較常見的量化評估方式比較偵測到的事件與真實的資料。

Event detection from text data streams has been a popular research area in the past decade.

However, data analysts and visualization experts often face grand challenges stemming out of the ill-defined concept of event and various kinds of textual data. As a result, we have few guidelines on how to build successful visual analysis tools that can handle specific event types and diverse textual data sources.

Within this paper, events are regarded as unexpected and unique patterns extracted from text data streams, valuable to users.

In particular, data sources evolved from a relatively limited amount of well-written news articles to rapidly generated, user written, and in some cases unstructured textual data from social media services.

Dou et al. [DWRZ12] defined task according to “New Event Detection”, “Event Tracking”, “Event Summarization”, and “Event Associations”, but we expect that tasks can be even more diversified including geographic dimension which were not explored yet.

Another challenge represents the unstructured, diverse textual data. It mandates extensive processing and preparation in order to properly employ it.

Becker [Bec11] shows interesting work about event detection in social media. She divides an event using three dimensions: 1) “planned” vs. “unplanned”; 2) “trending” vs “non-trending”; 3) “exogenous” vs. “endogenous”. The last dimension aims to detect events within the data in a real-life context.

Some examples of visual social media analysis is shown in Schreck and Keim [SK13]. With screenshots of the different visualizations, the authors explain the underlying data, analysis methods, and functionality of various applications in visual social media analysis.

There exists a survey on semantic sensemaking by Bontcheva and Rout [BR12]. Though their focus was on the semantic aspects, a subsection refers to visualization approaches.

Rohrdantz et al. [ROKF11] mention tasks for the “RealTime Visualization of Streaming Text Data”. They call tasks that are relevant in terms of the scope of our paper “monitoring”, “change and trend detection” and “situational awareness”.

In the first step of the pipeline, the documents are prepared for the analysis. In this step the documents are parsed to get the plain texts and standard text preprocessing methods, such as sentence detection, tokenizing, and stemming and lemmatizing are applied. In addition to these standard methods, methods from the computer linguistic field can be used in the preprocessing step to annotate the texts with additional information. For instance, part-of-speech tagging, named entity extraction, or syntactic parsing can be used to identify types of words, persons and places, or structure of sentences.

After the preprocessing step different approaches are used to detect events (see two branches in Processing in Figure 1).
The first group of approaches applies automatic methods to detect patterns in the data. The detected patterns are then used to create a visual analysis interface for the data set, what we call visual analysis. The interaction between visualization and the automatic part shapes a visual analytics approach.
The second group of approaches skips the automatic analysis and directly visualizes the outcome of the preprocessing, what also is only visual analysis because of the lack of interaction possibilities of a certain extent.

Text Data Sources

1. News is a well-known text data source. News captures information of a real world event or happening. It consists of a title, often followed by a short summary and the body containing details about the event. News goes through a professional gatekeeping process which in the end forms the agenda of media.

2. A typical electronic document is email. ... Emails are used for personal conversations, advertisement or business information exchange. They consist of a header and a body. The header contains information about transaction: sender, receiver, timestamp, and other meta data. The body contains the textual content of the email. An email body can be of arbitrary length which is one of its characteristics.

3. Weblogs, shortly named blogs are used for information purposes of a more or less undefined audience. ... A blog can have a specific topic or can be open for various topics.

4. RSS feeds are a standardized format to broadcast short news snippets. They consist of a title and a description. RSS feeds can be used by news agencies, newspapers and blogs. ... The standardized format allows the easy integration into other applications.

5. Recently, microblogging providers are becoming more and more popular. The messages are limited with respect to their length of 140 characters. So-called “hashtags” are used in order to characterize the membership of a tweet to a certain topic. In addition, more meta data is provided, e.g. geolocation, author, place etc.

6. User forums often have hierarchical structure. A message within the forum is a post and is not strictly restricted with respect to its length. Posts which belong to the same topic shape a so called thread. On the other hand several threads often belong to a sub-forum within the main forum. The purpose of a forum is the discussion on specific issues and topics regarding the its main topic.

7. Modern customer-care systems often ask each customer to fill out a feedback form after a purchase. This form (often digital, reachable through the internet) gives the customer the opportunity to provide issues directly to the vendor. The information is a valuable source which allows the seller to react fast and adequately to issues being raised by customers. ... Often these forms are semi-structured, which means they have checkboxes for predefined questions and provide a free text field for further comments.

8. Images and video sequences can be uploaded on sharing sites such as Flickr (https://www.flickr.com/). Users can tag their content with text. These tags and little text snippets typically describe the content in a short manner or express an emotional state being associated with the photo.

Almost half of the papers use microblogging data namely Twitter. It is obvious in Table 1 that in 2010 a shift towards microblogging happened. It is also noticeable that meta data (geolocations, author information) is often used in conjunction with microblogs.

Text Processing Methods

1. Part-of-speech (POS) tagging detects the word type of tokens.

2. Syntactic parsing determines the grammatical structure of sentences. ... Full syntactic parsing uses grammars and build up a complete parse tree for a sentence. ... Shallow parsing creates meaningful chunks and avoids the complexity of full parsing.

3. Typed-dependency parsing determines the type of relations between words in a sentence.

4. Coreference resolution creates connection between referring expression, such as pronouns, and subjects in a text. A correct resolution of referring expressions could improve text mining results, e.g., polarity extraction would benefit from correctly resolved referring expressions.

5. Named entity recognition (NER) detects and labels names of, e.g., persons, locations, events, or dates in texts.

6. Polarity extraction or determines the attitude (positive vs. negative) of the writer about a subject.

7. Word-sense disambiguation techniques use the context of words to determine the correct sense of tokens. ... We only observed one paper using word sense disambiguation.

We confirm that text processing methods are used very sparingly. ... Since 2000 until the end of 2011, only 17 out of 33 papers utilized any of the methods. The 16 papers with no text processing methods solved the event detection tasks with visualization. In the year of 2012, the trend changed dramatically; 14 out of 18 papers have used text processing methods in the papers published since then.

It is also noticeable that part-of-speech tagging and polarity extraction have gained popularity since 2012 as well.

Thus, we believe that many research papers started absorbing more natural language processing techniques to further generate their event metrics.

Automatic Methods for Text Event Detection

1. Clusters are generated for different time windows based different properties in the document, e.g., co-occurrence of terms, frequency in time, or metadata. Events are generated when the set of clusters changes, e.g., a new cluster arise or two existing clusters merge.

2. Users provide a set of example documents and classifiers learn to detect the annotated events. Classifier-based techniques are used in similar cases with rule-based ones, but have the advantage that users do not need to create rules by themselves.

3. Statistical methods such as correlation or detection of outliers and significant difference are used to identify events. Correlation based methods examine collection between terms or between terms and time and detect events by changes in the correlation measures. A different type of statistical methods calculate term-wise deviation from an expected value or use other measures to identify rare or unique occurrence of terms.

4. Prediction-based methods predict the occurrence of following documents based upon past history.

5. Methods based on ontologies [HHSW09] are suitable for event analysis in single domains. Specific ontologies are generated with full- or semi-automatic methods. ... Using this type of methods, events can then be detected from changes in activated concepts.

6. Pattern mining algorithms, such as the A-priori algorithm of Wu and Chen [WC09] applied to text in [WSJ∗ 14], are used to extract common sequential patterns in document streams. Patterns can be found based on documents themselves or time intervals. In both cases, features extracted from documents are then used to define patterns.

7. Models of the recurring characteristics of data steams can be used to detect events. An event is detected when a stream deviates significantly from its expected characteristics.

8. Rule-based approaches detect events with manually created rules. For instance, users specify rules based on terms and/or frequency to detect a particular event.

Visualization of Events in Text Data

In total, 21 papers use a time-oriented visualization (river, timeline, circular) to visualize the evolution of the data over time. Time-oriented visualizations are often combined with additional visualizations to show non-time dependent information.

Maps visualization came up with microblogging data and use mainly geographic references in the meta information of the microblogs for visualization.

A problem for all visualizations is the question how to visually represent text data. This problem is usually solved by selecting meaningful keywords that are either generated by frequency or by another scoring technique such as topic models.

Basic visualizations (e.g. line or bar charts) are mainly used to give an overview of the data set by showing the time dependent relations of events. They are used to visualize the data volumes or frequencies over time, for instance, of detected topics, named entities, or keywords.

Timeline visualization use one or multiple timelines and place glyphs or shapes on these timelines to indicate single data items, densities, or volumes. Timeline visualizations are therefore preferred over river visualizations when single or rare items should be tracked, because a river visualization put the focus on high frequent events.

River metaphors are often used to visualize outcomes of cluster algorithms. Although timeline techniques could be used, rivers provide a space saving overview and give a better visual impression of the distributions of the clusters and the overall amount of data.

 Supported Analysis Tasks

1. Overview visualization give users a summary of the document collection. Common are textual summaries based on frequent terms or topic models that describe the topics found in the collection. These summaries serve as navigation support and are often used as starting point for further analysis.

2. It is also common to provide users with abilities to search for keywords or allow filtering of the data by meta information. Both tasks reduce the number of item or extracted events in the analysis or visualization.

3. Monitoring tasks are the second most frequent tasks supported by the surveyed systems. Users monitoring a data source are interested in the evolution of events in a changing data source. Time-based visualizations (e.g., timeline, river, circular) are often used for monitoring task, because they show the temporal development of events in data sources.

4. In many cases relations between different detected events are visualized. The most frequent shown relations are relation in content, time, and volume. For instance, a river visualization shows time and volume relations between different streams and with additional annotations also relations in content can be shown.

The statistical methods are combined with any type of visualizations.

Interestingly, clustering methods are often visualized by river visualizations.

Exceptionally, topic modeling techniques are not only used with time dependent visualizations but also with other visualizations such as treemaps or geographic visualizations. This pattern appears because topic models are clustering methods that return a ranked list of terms representing single topics, which are often used in visualizations to label data and find names for clusters.

Evaluation

We subdivide qualitative methods into the following categories: case study, usability evaluation, use case, and anecdotal evaluation.

Table 7 accentuates the popular usage of use cases; except for 16 of all considered papers the authors make use of this method. Typically, a use case validates through the description of a fictitious scenario that pinpoints main features whereas a case study involves a domain expert and therefore is more time-consuming [DNKS10,MBB∗ 11].

Anecdotal evaluation describes how the suggested system could be used, but do not provide sufficient evidence to judge the general efficacy of the presented technique.

Usability evaluations involve users performing particular tasks with the given system and asks for comments on usability.

The most prominent quantitative evaluation methods are comparisons of the detected events with a ground truth set. Often event databases are used as ground truth that are enriched by the authors with missing entries.

A different evaluation form of algorithms are comparison with existing algorithms and reporting quality measures. In some cases not the results of the algorithms are evaluated but the performance in the sense of runtime or memory consumption is assessed, which is important for systems working in near real-time scenarios.

We also found only four papers using a user study for evaluation. We expected more papers using user studies, because many systems present novel visualization techniques and user studies can verify the strength and weakness of the application [HHN00,LYK∗ 12,RHD∗ 12].

One thing we noticed was that data sources have dramatically changed from news to social media since 2010. Mainly due to the burst of social media, many research studies used text data streams generated out of Facebook or Twitter.

Some data sources – like for instance discussion forums – are underused than others. Discussion forums are traditional methods to collect opinions from many people, but few research topics investigate data because they are asynchronous and slow to build up in nature. Despite these limitations, they also have a strength: archival history of some topics. Several discussion forums include years of textual conversation between multiple users on a single topic. For instance, this longitudinal conversation can be used to detect certain noticeable shifts in a specific user group’s opinions on political issues over some months or years.

More importantly, visualizations were primarily used as presentation, but had no interaction possible to steer the underlying data processing algorithm in order to further analyze data in a different angle. This limitation can prevent users from providing their insights back into the visualizations.

Especially for news, news criteria (also known as news values) [GR65, HO01] can help find and develop new features for content-based feature detection. They are only mentioned once in our whole bulk of surveyed news analysis research papers [DNKS10].

According to [GR65], news criteria are: frequency, threshold, unambiguity, meaningfulness, consonance, unexpectedness, continuity, composition, reference to elite nations, reference to elite people, reference to persons, and reference to something negative.

In general, news and selection criteria could be merged into one concept we call event values. Event values are a concept including the text data producer’s and user’s perspectives. They could be implemented in the data analysis process by means of new features (feature engineering) and interactive elements, which comes along with the call for more visual analytics functionality.

Borkin, M., Vo, A., Bylinskii, Z., Isola, P., Sunkavalli, S., Oliva, A., & Pfister, H. (2013). What makes a visualization memorable?. Visualization and Computer Graphics, IEEE Transactions on, 19(12), 2306-2315.

Borkin, M., Vo, A., Bylinskii, Z., Isola, P., Sunkavalli, S., Oliva, A., & Pfister, H. (2013). What makes a visualization memorable?. Visualization and Computer Graphics, IEEE Transactions on, 19(12), 2306-2315.

過去許多視覺化專家認為視覺化的結果不應該包括「圖表上的廢話」(chart junk)，而應該越清楚地顯示資料越好，例如Edward Tufte 與Stephen Few[13, 14, 37, 38]，心理學的實驗結果也證實了簡單而清楚的視覺化較容易閱讀[11, 24]。但近來也有一些研究者發表「圖表上的廢話」也許能夠增進閱讀者的持久力，讓他們更能努力在了解圖表上，增加對資料的認識與知識[4, 8, 19]。Bateman et al. [4] 的研究指出有裝飾的圖表比素樸的圖表有較佳的記憶，並且在理解上也不會較差，神經生物學則有加上視覺困難 (visual difficulties) 可以增強觀看者理解的假說[8, 19]。除了「圖表上的廢話」之外，圖表的類型、顏色與其他美學因素也會影響人們的觀看、解讀與記憶。本研究利用410張具有圖表的媒體，對261位受試者進行圖表記憶的研究，探討圖表是否與自然景象的影像[20]一樣在不同人之間具有一致性，並且測試什麼樣的視覺化類型與屬性具有較好的可記憶性。

為了進行本研究對各種圖表類型與屬性是否影響圖表的可記憶性，本研究首先參考前人的研究制訂視覺化的分類架構 (taxonomy)。過去的分類架構有根據圖形的知覺模型 (graphical perception models)、視覺與組織配置 (visual and organizational layout) 以及圖形資料的編碼 (graphical data encodings) [6, 12, 35, 33]，或是根據視覺化的演算法 [32, 36] ，近年也有以互動型視覺化和其提供的任務作為分類架構 [16, 17, 33, 35]。本研究的分類架構以Harris [15]提出的詞彙 (vocabulary) 為主，強調圖形表示的語法結構與資訊類型 (Englehardt [12]) 以及人類對圖形的知覺 (Cleveland and McGill [11])，將視覺化分為12類，每一類又分為若干次分類，如下表所示：

在分類架構上，另外對這些圖表附加上若干性質(properties)與屬性(attributes)，性質有維度數目 (2D或3D)、多重性 (單一、群體、多板塊、組合)、插圖(pictorial)以及時間序列，屬性尚包括是否是黑白、顏色的數目、資料與非資料間的比例 (data-ink ratio)、視覺元素面積佔整個影像的比例、是否有插圖以及是否有人像等。

不同的視覺化影像來源中，科學出版品 (scientific publications) 和資訊圖表 (infographics) 在視覺化上具有的多重性非當高，顯示這兩種影像來源的製作上往往需要組合多種圖表來表示概念和理論，而另一個考量則在於節省篇幅；反之，政府與組織出版品則大多是單一視覺化。

以視覺化的類型而言，為了解釋文章中的概念或是理論的結果，科學出版品有極高的比例使用diagram，此外科學文章中也大量使用基本的視覺編碼技術(visual encoding techniques)，諸如折線圖、長條圖和散布圖，某些領域內會使用特定的視覺編碼，例如網格與矩陣圖 (grid and matrix plots)以及樹狀與網路圖 (tree and networks)。因此，相較於其他類型的來源，科學出版品中較'常使用樹狀與網路圖、網格與矩陣圖和散布圖。

資訊圖表中也使用大量的diagrams，主要包括流程圖(flow charts)和時間表(timelines)，此外表格(table)也經常被使用在資訊圖表中，這些表格通常會加上插畫的裝飾。但是資訊圖表相較於其他來源，較少使用折線圖。

新聞媒體和政府出版品主要使用長條圖、折線圖地圖和表格，折線圖通常用來表示時間序列的資料。兩種來源的差異，是政府的報告會大量使用圓形圖，例如圓餅圖。

本研究在評估圖表具有可記憶性的高低時，使用命中率 (hit rate, HR) 與誤警率 (false alarm rate, FAR)，並且用敏感指數(sensitivity index) d-prime metric整合兩個數值進行排序， d' = Z(HR) − Z(FAR) ，Z 是高斯分布(Gaussian distribution)的累積分配函數(cumulative distribution function, CDF)的倒數(inverse)。愈高的d'值表示HR較高而FAR較低，此時該視覺化圖形較容易被記憶，反之則否。本研究得到視覺化圖形的可記憶性平均測量結果為HR = 55.36%和SD = 16.51%以及FAR = 13.17%, SD = 10.73%。相較於先前的圖像可記憶性的研究結果，較自然景象的可記憶性 (HR = 67.5%, SD = 13.6%以及FAR = 10.7%, SD = 7.6%) [21] 差，但與人臉的可記憶性 (HR = 53.6%, SD = 14.3%以及FAR = 14.5%, SD = 9.9%) [2]差不多。而將本研究的測試結果隨機分為兩群，經過25次的測量，HR、FAR和d-prime在兩群間的結量結果的Spearman等級相關係數分別為0.83、0.78和0.81。由此可知，視覺化圖形在可記憶性的不同受試者測量上也有一致性，可記憶性是視覺化圖形的一種特性。

以下針對視覺化圖形的種類、性質與屬性進行可記憶性的探討；

具有插畫的視覺化圖形明顯比沒有插畫的視覺化圖形有較高的可記憶性，

愈多顏色的視覺化圖形的可記憶性愈佳，甚至在沒有插畫的圖形上，較多顏色的圖形也比只有一種顏色的圖形可記憶性來得高。

視覺元素面積佔整個影像的比例較高的視覺化圖形也比較容易被記憶。

資料與非資料間的比例較低的視覺化圖形的可記憶性較佳。

diagram、網格與矩陣圖以及樹狀與網路圖等類型較具有獨特(unique)呈現的圖形比長條圖、折線圖、表格等較一般(common)呈現的圖形較具有可記憶性，特別是圖形上沒有插圖時，因為這類圖形的呈現相近，彼此干擾，使得命中率較低而誤警率較高。

在各種來源的視覺化圖形裡，資訊圖表類具有最高的可記憶性，其次是科學出版品，最不具可記憶性的圖形來源則是政府與世界組織類。

另外，具有圓形或圓的邊的圖形的可記憶性也較高，作者認為插畫與圓形都是較自然的視覺化作品，因此較容易被記憶。


We ran the largest scale visualization study to date using 2,070 single-panel visualizations, categorized with visualization type (e.g., bar chart, line graph, etc.), collected from news media sites, government reports, scientific journals, and infographic sources. Each visualization was annotated with additional attributes, including ratings for data-ink ratios and visual densities.

Altogether our findings suggest that quantifying memorability is a general metric of the utility of information, an essential step towards determining how to design effective visualizations.

The conventional view, promoted by visualization experts such as Edward Tufte and Stephen Few, holds that visualizations should not include chart junk and should show the data as clearly as possible without any distractors [13, 14, 37, 38]. This view has also been supported by psychology lab studies, which show that simple and clear visualizations are easier to understand [11, 24].

At the other end of the spectrum, researchers have published that chart junk can possibly improve retention and force a viewer to expend more cognitive effort to understand the graph, thus increasing their knowledge and understanding of the data [4, 8, 19].

What researchers agree on is that chart junk is not the only factor that influences how a person sees, interprets, and remembers a visualization. Other aspects of the visualization, such as graph type, color, or aesthetics, also influence a visualization’s cognitive workload and retention [8, 19, 39].

Recent work has shown that memorability of images of natural scenes is consistent across people, suggesting that some images are intrinsically more memorable than others, independent of an individual’s contexts and biases [20].

This is because given limited cognitive resources and time to process novel information, capitalizing on memorable displays is an effective strategy.

Recent large-scale visual memory work has shown that existing categorical knowledge supports memorability for item-specific details [9, 22, 23]. In other words, many additional visual details of the image come for free when retrieving memorable items.

For our research, we first built a new broad taxonomy of static visualizations that covers the large variety of visualizations used across social and scientific domains. These visualization types range from area charts, bar charts, line graphs, and maps to diagrams, point plots, and tables.

We then used these 2,070 visualizations in an online memorability study we launched via Amazon’s Mechanical Turk with 261 participants. This study allowed us to gather memorability scores for hundreds of these visualizations, and determine which visualization types and attributes were more memorable.

Perception Theory and the Chart Junk Debate:

Bateman et al. conducted a study to test the comprehension and recall of graphs using an embellished version and a plain version of each graph [4]. They showed that the embellished graphs outperformed the plain graphs with respect to recall, and the embellished versions were no less effective for comprehension than the plain versions.

There has been some support for the comprehension results from a neurobiological standpoint, as it has been hypothesized that adding “visual difficulties” may enhance comprehension by a viewer [8, 19].

Other studies have shown that the effects of stylistic choices and visual metaphors may not have such a significant effect on perception and comprehension [7, 39].

In response to the Bateman study, Stephen Few wrote a comprehensive critique of their methodology [14], most of which also applies to other studies. A number of these studies were conducted with a limited number of participants and target visualizations. Moreover, in some studies the visualization targets were designed by the experimenters, introducing inherent biases and over-simplifications [4, 8, 39].

Visualization Taxonomies:

Within the academic visualization community there have been many approaches to creating visualization taxonomies.

Traditionally many visualization taxonomies have been based on graphical perception models, the visual and organizational layout, as well as the graphical data encodings [6, 12, 35, 33].

Another approach to visualization taxonomies is based on the underlying algorithms of the visualization and not the data itself [32, 36].

There is also recent work on taxonomies for interactive visualizations and the additional tasks they enable [16, 17, 33, 35].

Outside of the academic community there is a thriving interest in visualization collections for the general public. For example, the Periodic Table for Management [26] present a classification of visualizations with a multitude of illustrated diagrams for business. The online community Visualizing.org introduces an eight-category taxonomy to organize the projects hosted on their site [25]. InfoDesignPatterns.com classifies visualization design patterns based upon visual representation and user interaction [5].

Cognitive Psychology:

These studies have demonstrated that the differences in the memorability of different images are consistent across observers, which implies that memorability is an intrinsic property of an image [21, 20].

Brady et al. [9] tested the long-term memory capacity for storing details by detecting repeat object images when shown pairs of objects, one old and one new. They found that participants were accurate in detecting repeats with minimal false alarms, indicating that human visual memory has a higher storage capacity for minute details than was previously thought.

More recently, Isola et al. have annotated natural images with attributes, measured memorability, and performed feature selection, showing that certain features are good indicators of memorability [20, 21]. Memorability was measured by launching a “Memory Game” on Amazon Mechanical Turk, in which participants were presented with a sequence of images and instructed to press a key when they saw a repeat image in the sequence. The results showed that there was consistency across the different participants, and that people and human-scale objects in the images contribute positively to the memorability of scenes. That work also showed that unusual layouts and aesthetic beauty were not overall associated with high memorability across a dataset of everyday photos [20].


The taxonomy classifies static visualizations according to the underlying data structures, the visual encoding of the data, and the perceptual tasks enabled by these encodings.

It contains twelve main visualization categories and several popular sub-types for each category. In addition, we supply a set of properties that aid in the characterization of the visualizations.

This taxonomy draws from the comprehensive vocabulary of information graphics presented in Harris [15], the emphasis on syntactic structure and information type in graphic representation by Englehardt [12], and the results of Cleveland and McGill in understanding human graphical perception [11].

Dimension represents the number of dimensions (i.e., 2D or 3D) of the visual encoding.

Multiplicity defines whether the visualization is stand-alone (single) or somehow grouped with other visualizations (multiple). We distinguish several cases of multiple visualizations. Grouped means multiple overlapping/superimposed visualizations, such as grouped bar charts; multi-panel indicates a graphic that contains multiple related visualizations as part of a single narrative; and combination indicates a graph with two or more superimposed visualization categories (e.g., a line plot over a bar graph).

The pictorial property indicates that the encoding is a pictogram (e.g., a pictorial bar chart). Pictorial unit means that the individual pictograms represent units of data, such as the Istotype (International System of Typographic Picture Education), a form of infographics based on pictograms developed by Otto Neurath at the turn of the 19th century [29].

Finally, time is included, specifically as a time series, as it is such a common feature of visualizations and dictates specific visual encoding aspects regarding data encoding and ordering.

The first two attributes, black & white and number of distinct colors give a general sense of the amount of color in a visualization.

A measure of chart junk and minimalism is encapsulated in Edward Tufte’s data-ink ratio metric [37], which approximates the ratio of data to non-data elements.

The visual density rates the overall density of visual elements in the image without distinguishing between data and non-data elements.

Finally, we have two binary attributes to identify pictograms, photos, or logos: human recognizable objects and human depiction. We explicitly chose to have a separate category for human depictions due to prior research indicating that the presence of a human in a photo has a strong effect on memorability [21].

There is also a very high percentage of multiple visualizations in the scientific publication category. There are two primary explanations for this observation. First, like infographics, multiple individual visualizations are combined in a single figure in order to visually explain scientific concepts or theories to the journal readers. Second, combining visualizations into a single figure (even if possibly not directly related) saves page count and money.

In contrast, a very high ratio of single visualizations is seen in government / world organizations. These visualizations are usually published one-at-a-time within government reports, and there are no page limits or space issues as with scientific journals.

Scientific publications, for example, have a large percentage of diagrams. These diagrams are primarily used to explain concepts from the article, or illustrate the results or theories. Also included are renderings (e.g., 3D molecular diagrams). The scientific articles also use many basic visual encoding techniques, such as line graphs, bar charts, and point plots. Domain-specific uses of certain visual encodings are evident, e.g., grid and matrix plots for biological heat maps, trees and networks for phylogenic trees, etc.

Infographics also use a large percentage of diagrams. These diagrams primarily include flow charts and timelines. Also included in infographics is a large percentage of tables. These are commonly simple tables or ranked lists that are elaborately decorated and annotated with illustrations. Unlike the other categories, there is little use of line graphs.

In contrast to the scientific and infographic sources, the news media and government sources publish a more focused range of topics, thus employing similar visualization strategies. Both sources primarily rely on bar charts and other “common” (i.e., learnt in primary school) forms of visual encodings such as line graphs, maps, and tables. The line graphs are most commonly time series, e.g., of financial data. One of the interesting differences between the categories include the greater use of circle plots (e.g., pie charts) in government reports.

Looking at specific visualization categories, tree and network diagrams only appear in scientific and infographic publications. This is probably due to the fact that the other publication venues do not publish data that is best represented as trees or networks. Similarly, grid and matrix plots are primarily used to encode appropriate data in the scientific context. Interestingly, point plots are also primarily used in scientific publications. This may be due to either the fact that the data being visualized are indeed best visualized as point plot representations, or it could be due to domain-specific visualization conventions, e.g., in statistics.

Worth noting is the absence of text visualizations from almost all publication venues. The only examples of text based visualizations were observed in the news media. Their absence may be explained by the fact that their data, i.e., text, is not relevant to the topics published by most sources. Another possible explanation is that text visualizations are not as “main stream” in any of the visualization sources we examined as compared to other visualization types.

Performance Metrics:

Workers saw each target image at most 2 times (less than twice if they prematurely exited the game). We measure an image’s hit rate (HR) as the proportion of times workers responded on the second (repeat) presentation of the image. In signal detection terms: HR = HITS/(HITS+MISSES).

We also measured how many times workers responded on the first presentation of the image. This corresponds to workers thinking they have seen the image before, even though they have not. This false alarm rate (FAR) is calculated: FAR = FA/(FA+CR) , where FA is the number of false alarms and CR is the number of correct rejections (the absense of a response).

For performing a relative sorting of our data instances we used the d-prime metric (otherwise called the sensitivity index). This is a common metric used in signal detection theory, which takes into account both the signal and noise of a data source, calculated as: d' = Z(HR) − Z(FAR) (where Z is the inverse cumulative Gaussian distribution). A higher d' corresponds to a signal being more readily detected. Thus, we can use this as a memorability score for our visualizations. A high score will require the HR to be high and the FAR to be low. This will ensure that visualizations that are easily confused for others (high FAR) will have a lower memorability score.

In other words, we have measured how visualizations would be remembered if they were images. We observed a mean HR of 55.36% (SD = 16.51%) and mean FAR of 13.17% (SD = 10.73%).

For comparison, scene memorability has a mean HR of 67.5% (SD = 13.6%) with mean FAR of 10.7% (SD = 7.6%) [21], and face memorability has a mean HR of 53.6% (SD = 14.3%) with mean FAR of 14.5%(SD = 9.9%) [2]. This possibly supports our first hypothesis that visualizations are less memorable than natural scenes.

This demonstrates that there is memorability consistency with scenes, faces, and also visualizations, thus memorability is a generic principle with possibly similar generic, abstract, features.

We also measured the consistency of our memorability scores [2, 21]. By splitting the participants into two independent groups, we can measure how well the memorability scores of one group on all the target images compare to the scores of another group (Fig. 3).

Averaging over 25 such random half-splits, we obtain Spearman’s rank correlations of 0.83 for HR, 0.78 for FAR, and 0.81 for d-prime, the latter of which is plotted in Fig. 3.

This high correlation demonstrates that the memorability of a visualization is a consistent measure across participants, and indicates real differences in memorability between visualizations.

In other words, despite the noise introduced by worker variability and by showing different image sequences to different workers, we can nevertheless show that memorability is somehow intrinsic to the visualizations.

Of our 410 target visualizations, 145 contained either photographs, cartoons, or other pictograms of human recognizable objects (from here on out referred to broadly as “pictograms”). Visualizations containing pictograms have on average a higher memorability score (Mean (M)=1.93) than visualizations without pictograms (M = 1.14,t(297) = 13.67, p < 0.001). This supports our second hypothesis.

Thus not all chart junk is created equal: annotations and representations containing pictograms are across the board more memorable.

Thus an image, or image of a visualization, containing a human recognizable object will be easily recognizable and probably memorable. Due to this strong main effect of pictograms, we examined our results for both the cases of visualizations with and without pictograms. As shown in the left-most panel of Fig. 1, all but one of the overall top most memorable images (as ranked by their d-prime scores) contain human recognizable pictograms.

As shown in Fig. 4, there is an observable trend of more colorful visualizations having a higher memorability score: visualizations with 7 or more colors have a higher memorability score (M = 1.71) than visualizations with 2-6 colors (M = 1.48,t(285) = 3.97, p < 0.001), and even more than visualizations with 1 color or black-and-white gradient (M = 1.18,t(220) = 6.38, p < 0.001).

When we remove visualizations with pictograms, the difference between visualizations with 7 or more colors (M = 1.34) and those that have only 1 color (M = 1.00) remains statistically significant (t(71) = 3.61, p < 0.001).

Considering all the visualizations together, we observed a statistically significant effect of visual density on memorability scores with a high visual density rating of “3” (M = 1.83), i.e., very dense, being greater than a low visual density rating of “1” (M = 1.28,t(115) = 6.08, p < 0.001) as shown in Fig. 5.

We also observed a statistically significant effect of the data-to-ink ratio attribute on memorability scores with a “bad” (M = 1.81), i.e., low data-to-ink ratio, being higher than a “good” rating (M = 1.23,t(208) = 6.92, p < 0.001) as shown in Fig. 6. Note that using a corrected t-test, we also arrive at the results that the 3 levels of data-ink ratio are pairwise significantly different from each-other .

Summarizing all of these attribute results: higher memorability scores were correlated with visualizations containing pictograms, more color, low data-to-ink ratios, and high visual densities.

As shown in Fig. 7, diagrams were statistically more memorable than points, bars, lines, and tables. These trends remain observable even when visualizations with pictograms are removed from the data. Other than some minor ranking differences and addition of the map category, the main difference is in the ranking of the table visualization type, which without pictograms becomes least memorable.

The middle panel of Fig. 1 displays the most memorable visualizations that do not contain pictograms. Why are these visualizations more memorable than the ones in the right-most panel?

To start with, qualitatively viewing the most memorable visualizations, most are high contrast. These images also all have more color, a trend quantitatively demonstrated in Sec. 7.2 to be correlated with higher memorability. As compared to the more subdued less memorable visualizations, the more memorable visualizations are easier to see and discriminate as images.

Another possible explanation is that “unique” types of visualizations, such as diagrams, are more memorable than “common” types of visualizations, such as bar charts. This trend is also evident in Fig. 7 in which grid/matrix, trees and networks, and diagrams have the highest memorability scores.

Examples of these unique types of visualizations are each individual and unique, whereas bar charts and line graphs are uniform with limited variability in their visual encoding methodology. Previously it has been shown that an item is more likely to interfere with another item if it has similar category or subordinate category information, but unique exemplars of objects can be encoded in memory quite well [22]. This supports our findings that show high FAR and low HR for table and bar visualizations, which both have very similar visuals within their category (i.e., all the bar charts look alike).

Another possible explanation is that visualizations like bar and line graphs are just not natural. If image memorability is correlated with the ability to recognize natural, or natural looking, objects then people may see diagrams, radial plots, or heat maps as looking more “natural”.

One common visual aspect of the most memorable visualizations is the prevalence of circles and round edges. Previous work has demonstrated that people’s emotions are more positive toward rounded corners than sharp corners [3]. This could possibly support both the trend of circular features in the memorable images as well as the concept of natural-looking visualizations being more memorable since “natural” things tend to be round.

As shown in Fig. 8, regardless of whether the visualizations did or did not include pictograms, the visualization source with the highest memorability score was the infographic category (M = 1.99,t(147) = 5.96, p < 0.001 when compared to the next highest category, scientific publications with M = 1.48), and the visualization source with the lowest memorability score was the government and world organizations category (M = 0.86,t(220) = 8.46, p < 0.001 when compared to the next lowest category, news media with M = 1.46).

This may be a contributing factor to the observed trend (see Fig. 8) that visualization sources that have non-uniform aesthetics tend to have higher memorability scores than sources with uniform aesthetics. This observation refutes our last hypothesis that visualizations from scientific publications are less memorable. This may also be due to the fact that visualizations in scientific publications have a high percentage of diagrams (Fig. 2), similar to the infographic category.

The results of our memorability experiment show that visualizations are intrinsically memorable with consistency across people.

They are less memorable than natural scenes, but similar to images of faces, which may hint at generic, abstract, features of human memory.

Not surprisingly, attributes such as color and the inclusion of a human recognizable object enhance memorability. And similar to previous studies we found that visualizations with low data-to-ink ratios and high visual densities (i.e., more chart junk and “clutter”) were more memorable than minimal, “clean” visualizations.

It appears that we are best at remembering “natural” looking visualizations, as they are similar to scenes, objects, and people, and that pictorial and rounded features help memorability.

More surprisingly, we found that unique visualization types (pictoral, grid/matrix, trees and networks, and diagrams) had significantly higher memorability scores than common graphs (circles, area, points, bars, and lines).

Gan, Q., Zhu, M., Li, M., Liang, T., Cao, Y., & Zhou, B. (2014). Document visualization: an overview of current research. Wiley Interdisciplinary Reviews: Computational Statistics, 6(1), 19-36.

Gan, Q., Zhu, M., Li, M., Liang, T., Cao, Y., & Zhou, B. (2014). Document visualization: an overview of current research. Wiley Interdisciplinary Reviews: Computational Statistics, 6(1), 19-36.

文件視覺化(document visualization)是一種資訊視覺化技術，將詞語、文句、文件或它們之間的關係等文字資訊轉換為視覺形式，使得使用者在面臨大量的文件時可以更好的了解文件、減輕他們的心理負荷。文件通常較缺乏結構(minimally structured)，但有豐富的特徵(attributes)和後設資料(metadata)，因此相較於文本視覺化(text visualization)，文件視覺化主要著重在文件以及其包含的特徵和後設資料上。以下是幾種可能的應用：(1) 詞語的頻次與分布； (2) 語意內容與重複 (semantic content and repetition)； (3) 區別文件集群的主題； (4) 文件的核心內容；(5) 文件間的相似性；(6) 文件間的連結；(7) 文件內容改變的過程；以及 (8) 社交媒體上的資訊擴散與其他模式以及做為改善文本搜尋的方式。

本研究以視覺化的對象(visualization objects)與任務對蒐集到的文件視覺化技術進行分析，視覺化的對象分為單一文件、文件集合、串流文本訊息以及檢索結果，以下對各種視覺化任務進行說明：

單一文件的視覺化目的在快速了解與吸收核心內容與文本特徵，著重在詞語、片語、語意關係和內容上，分為三種類型：
1.呈現詞語頻次、分布與語彙結構等語彙特徵的語彙為基礎 (Vocabulary-Based)視覺化：
重要的技術有Tag Clouds [6,7]與Wordle [8,9]，這類技術利用位置、顏色與大小等方式呈現單一文件中的詞語頻次，近來parallel tag clouds (PTC) [10]、 ManiWordle [11]、 context preserving dynamic word cloud[12]和visualization of internet discussion with extruded word clouds. [13]等許多研究以這類方法為基礎，並加以改善。其他屬於這類但原理不同的方法還有TextArc [14]和DocuBurst [16]。
2. 呈現實體與其間關係的語意結構視覺化：
Semantic Graphs [19]利用Penn Treebank產生的剖析樹(parse tree)，產生每一句子內的主語-動詞-受語，在解決代名詞指代(pronominal anaphors)問題，將各實體相連，產生語意圖(semantic graph)。
3.呈現文件內容(Document Content)的特性與關係為基礎的視覺化：
例如WordTree [23]以樹狀結構表現詞語的上下文脈絡，樹的根節點是使用者選取的詞語，每一個分支表示詞語在文件上的上下文，節點大小代表各詞語的頻次。Arc Diagrams [25] 以半圓形的弧連結重複的次序列，用來顯示內容上重複的複雜模式(complex patterns of repetition)。

文件集合的視覺化在於顯現文件群集上的主題、文件間的相似與差異以及內容在時間上的改變，相關技術可分為
1. 文件主題的視覺化：目的在發現特定的主題以及反應各個不同主題之間的關係，著名的研究案例有ThemeScapes [26]和 INSPIRE的 ThemeView以及 The Galaxy [27]，分別以地形圖和散佈圖表現文件在主題上的分布情形，地形圖上各「山脈」的高度表示主題的強度。TopicNets則以網路圖的節點與連線呈現文件間在主題上的關係。ThemeRiver [29]和Topic Island [30]著重在文件集合內主題在時間的變化，以ThemeRiver [29]來說，X軸表時間，Y軸上則以不同顏色的「河流」代表各主題，河流的寬度表現主題主題在相關文件上的強度。
2. 文件核心內容的視覺化：目的在提供整個文件集合的概觀，例如Document Cards [35]用來呈現大量的文件集合，每一張卡片上包含文件上重要的詞語和影像，而詞語是由文本探勘(text mining)技術由文件上的文本抽取出來，影像則從文件上抽取或然後加以組合。
3. 版本更動的視覺化：呈現各版本上的差異。例如：History Flow [36] 的設計是用來顯示維基百科上不同版本的文件內容更動情形以及相對應的作者；另外，也有許多針對軟體程式碼發展的視覺化。
4. 文件關係的視覺化：發現在不同文件上實體的連結，實體包括人、地點、日期和組織等等，提供這個功能的視覺化技術如Jigsaw [46] 。其他的技術，如ContexTour [47]和PivotPaths [49] 為視覺化被使用在論文集合的應用、FacetAtlas [48]則將Google Health文件上的病因、症狀、處方和診斷等實體相連，
5. 文件相似性的視覺化：其目的在將相似的文件置於彼此接近的位置，並且能遠離不相似的文件。過去常利用自組織映射圖 (self-organizing map, SOM) [50]，將高維度的資料映射到2維平面上呈現，並使得資料間複雜而非線性的關係能夠以距離方式表現，著名的例子有 Lin 的研究[52]和WEBSOM [53]。

在文本視覺化的應用方面，由於近年社交媒體的盛行，即時的串流文本處理的研究大為盛行，研究問題包括主題與詞語的統計分析與表現、主題相關事件的凸顯以及文本訊息本身的視覺化 [54]，Christian Rohrdantz [55]進行了串流文本資料的即時視覺化相關研究的回顧，Whisper [57] 的研究可以追蹤社交媒體上的資訊擴散過程。另一個應用是檢索結果的視覺化介面，早期的研究成果如TileBars [58]，Sparkler[59]可同時將多個查詢問句的結果以視覺化的方式呈現，RankSpiral [60] 的視覺化呈現重點在比較多個查詢問句或不同搜尋引擎的檢索結果。

各種技術所提供的取用方法、對文件的要求與主要特色可參考下表



關於主要特色說明如下：
1. 擴充性 (Extension)：此方法可適用於大量的文件集合。
2. 多功能性 (Versatility)：適用於多種的視覺化任務。
3. 互動性 (Interactivity)：提供使用者比較直覺的人機介面，讓使用者參與研究與發展過程。
4. 技術 (Techniques)
在文本處理上採用可擴充 (scalable)、高效能 (high-performance) 的演算法；採用協調多視圖 (Coordinated and Multiple Views)；即時處理技術。

本研究並且提出文件視覺化有待發展的兩個研究方向，一為文件視覺化的評鑑方法 [62]，另一為理論基礎 [63]。

This overview introduces fundamental concepts of and designs for document visualization, a number of representative methods in the field, and challenges as well as promising directions of future development.

Document visualization is a class of the information visualization techniques that transforms textual information such as words, sentences, documents, and their relationships into a visual form, enabling users to better understand textual documents and to lessen their mental workload when faced with a substantial quantity of available textual documents. [1]

And compared with text visualization that aims to visualize information on the text level, document visualization concentrates more on visualizing documents that include attributes and metadata except the core textual contents.

Document visualization has significant advantages over helping people to analyze and control big quantities of textual information in many cases. For example, we can intuitively get access to (1) word frequency or distribution; (2) semantic content and repetition; (3) the topic or topics that define document clusters; (4) the core content of document; (5) similarity among documents; (6) the connections among documents; (7) how content changes over time; and (8) information diffusion or other interesting patterns in social media, as well as improve text searches.

Generally ‘document’ is a textual record or physical form/representation of ‘information’.

The evolving notion of ‘document’ among Jonathan Priest, Otlet, Briet, Sch ¨ urmeyer, and the other documentalists increasingly emphasized whatever functioned as a document rather than traditional physical forms of documents. [2]

And with the development of digital technology, anything exists physically in a digital environment, such as a mail message or a technical report, could be considered as a document.

Documents are often minimally structured and may be rich with attributes and metadata, especially when concentrated in a specific application domain.

We may learn from the good practical guidelines to create an effective user interface for an interactive information visualization tool, as propounded by Ben Shneiderman who suggested in a form of mantra that an effective information visualization tool should follow the principle:
Overview first, zoom and filter, then details on demand. [4]

The mantra is accompanied by a task taxonomy for information visualizations that specifies seven
tasks at a high level of abstraction [4]:
• Overview. Gain an overview of the entire collection.
• Zoom. Zoom in on items of interest.
• Filter. Filter out uninteresting items.
• Details-on-demand. Select an item or group and get details when needed.
• Relate. View relationship among items.
• History. Keep a history of actions to support undo, replay, and progressive refinement.
• Extract. Allow extraction of sub-collections and of the query parameters.

We firstly divide document visualization methods into three main categories:
(1) single document visualization that has more emphasis on individual words and actual single document contents;
(2) document collection visualization that has more emphasis on large document collections, themes and concepts across collection, and how documents are relate to others;
(3) extended document visualization which often deals with comprehensive tasks, involves other attributes beyond the content of documents, and is always applied in specific field, such as social media and search.

In single document visualization, the goal is to quickly understand and absorb core content and text
features. The visualization focuses on words, phrases, semantic relations, and contents.

1. Vocabulary-Based Visualization
Vocabulary is the basic unit of a document. The visualization assists people in understanding words through visual representation of the document vocabulary features, such as word frequency, word distribution, and lexical structure, thereby providing a general idea of contents and features in a document.

Tag Clouds [6,7] and Wordle [8,9] are representative methods mainly visualizing word frequency. They are widely used in the news media and personal home pages. They provide layouts of raw tokens, colored, and sized by the corresponding word frequency within a single document. We may know the main research areas/content discussed in the text by the compact visual form of words.

Recently, some other methods have been proposed, extending the tag/word cloud, such as
parallel tag clouds (PTC), [10] ManiWordle, [11] context preserving dynamic word cloud,[12] visualization of internet discussion with extruded word clouds. [13]

Other examples: TextArc [14], DocuBurst [16].

2. Visualization Based on Semantic Structure

Visualization based on semantic structure usually use entities and their relationships to reveal the semantic content.

Semantic Graphs [19] is a visualization based on the semantic representation of a document in the form of a semantic graph. Firstly, it extracts subject–verb–object for each sentence by the Penn Treebank parse tree. Then, it links the triplets to their corresponding entity, which needs to resolve pronominal anaphors as well as to attach the associate WordNet synset. Thus, the document is summarized with the semantic graph and the list of extracted triplets.

3. Visualization Based on Document Content
Visualization based on document content is not only to search for specific words but also to obtain the characteristics and relations of the contents in the document.

The WordTree visualization provides the representation of both word frequency and context. Size is used to represent frequency of the term or phrase. The root of the tree is a user-selected word or phrase, and the branches represent the contexts in which the word or phrase is used in the document. Users can click on a branch, choose a different search term or re-center the tree. [23]

Martin Wattenberg’s Arc Diagrams [25] is a visualization method that focuses on showing complex patterns of repetition. It is suited to the analysis of highly structured data like musical compositions and less well-structured data like a web page. Repeated subsequences are identified and connected by semicircular arcs. Height of the arcs represents the distance between the subsequences; and thickness of the arcs represents the length of the subsequences.

Document Collection Visualization

Document collection visualization usually intends to reveal the topic or topics that define document clusters, the similarities and differences among documents, and how contents change over time.

1. Visualization of Document Themes

The main goal is to discover one or more specific topics and to reflect the relationships among various topics.

It may be used to find hot disciplines, evolutions, and trends.

The methods, such as ThemeScapes, [26] INSPIRE’s ThemeView, and The Galaxy, [27] all developed by the Pacific Northwest National Laboratory, having less emphasis on the time factor, focus more on characteristics of the document themes at some specific points.

ThemeView uses a 3D terrain map display to represent different themes. The height of a mountain represents the theme’s strength, and the distance between two mountains represents the similarity between the two themes. Keywords are used to distinguish each mountain. [27]

The Galaxy visualization uses a similar approach that themes are visualized as 2D clouds of document points-stars in a theme galaxy (Figure 8(b)). [27]

There are other representations for visualizing documents and topics as nodes in a node-link graph. TopicNets is a web-based system for visual and interactive analysis of large sets of documents using statistical topic models. [28] The main view is a document topic graph which can allow aggregate nodes. The time dimension is represented as a separate visualization, with documents placed chronologically around a broken circle, and connected to related topic nodes which are placed inside the circle.

The methods, such as ThemeRiver [29] and Topic Island, [30] have greater emphasis on the time factor, focusing more on visualizing thematic variations over time within a collection of documents.

ThemeRiver is in the form of axes, with the X-axis representing time and the Y-axis representing different themes. The ‘river’ flows from left to right through time, changing width to portray changes in theme strength of corresponding documents. Rivers of different colors represent different themes, and the width of river (i.e., narrow or wide) indicates the strength (decreasing or increasing) of an individual topic in the associated documents. [29]

2. Visualization of Document Core Content

Visualization of document core content mainly intends to give an overview of a collection of documents without reading them entirely.

Document Cards [35] visualizes large document collections, such as paper collections and news reports, which contain both texts and images to describe facts, methods, or stories. It represents the document’s key content as a mixture of images and important terms, similar to cards in a top trumps game. [35]

The pipeline for creating Document Cards is as follows: firstly, extract the text from the original document, and use a text mining approach to extract the key terms; then go to the phases of image extraction, including image processing and image packing; finally layout the extracted key terms and images to generate the corresponding document cards.

3. Visualization of Changes over Different Versions

Visualization of changes over different versions is used to visualize differences among multiple document versions that are generated over time.

History Flow [36] is designed to show changes between multiple document versions on Wikipedia. It can visualize the process of content changes and the corresponding authors who make the amendments. It also reveals some complex patterns of cooperation and confliction, such as vandalism and repair, anonymity versus named authorship, negotiation, and content stability.

Software visualization [37–39] focuses on visualizing the software development. SeeSoft, [40] Augur, [41] and Advizor [42] are visualizations for code documents. Xia gives visual insight into version control activities, like architectural and coding differences between two software versions. [43] Beagle visualizes changes among different released versions. [44] Spectrograph shows the time and location where changes happen in the system. [45]

4. Visualization of Document Relationships

With gradual increases in document quantity, the concepts and entities within documents become larger and larger, making the analyst’s task of evaluation and sense-making more difficult. Thus, it is quite meaningful to visualize connections among documents. The visualization focuses on the correlation among documents, like the connections among entities across different documents.

Jigsaw [46] is an interactive visualization for document exploration and sense-making, and it supports the analysis of relationships among documents. It visually shows connections between entities in the documents; where entities could be people, places, dates, organizations, and so on. It is suitable to documents describing a set of observations or facts, like news stories and case reports. It provides multiple views and each view provides a different perspective.

There are other methods for visualizing relations among multiple facets. ContexTour [47] presents the relations among conferences, authors, and topics in paper collections. FacetAtlas [48] shows relations among causes, symptoms, treatments, and diagnoses in Google Health documents. PivotPaths [49] visually explores relations of authors, keywords, and citations in academic publications.

5. Visualization of Document Similarity

In many cases of document collection visualizations, the goal is to place similar documents close to each other and dissimilar ones far apart.

The self-organizing map (SOM) [50] is a nonlinear projection method. It expresses complex, nonlinear relationships between high dimensional data items into simple geometric relationships on a 2D display.

When applying to information retrieval, it usually uses map displays. [51] Different colored areas represent different concepts in documents. Size of area indicates its relative importance in collection. Neighboring regions show commonalities in concepts. Dots in regions can represent documents. Additional information can be referred in Xia Lin’s map display [52] and WEBSOM. [53]

With the rise of social media (a textual medium), text streams, such as Twitter posts, are being generated in volumes that grow every day. A large body of research has appeared in recent years. Those works have different focuses and always involve multiple targets, such as dealing with the statistical analysis and presentation of topics or terms, focusing on the emergence of topic events, [33] and visualizing the text messages themselves. [54]

Christian Rohrdantz [55] provides an overview of real-time visualization of streaming text data.

STREAMIT [56] presents a similar visual representation of text streams which applies to news documents.

Whisper [57] fulfills the requirement for tracing information diffusion processes in social media, in a real-time manner.

Search Visualization visualizes the results of search operations. The relatively early approach is TileBars [58] that intends to minimize time and effort for deciding which documents to view in detail.

Susan Havre [59] introduces a graphical method called Sparkler for visually presenting and exploring the results of multiple queries simultaneously.

RankSpiral addresses the problem of how to enable users to visually explore and compare large sets of documents that have been retrieved by different search engines or queries. [60]

We have mainly considered the visualization objects and tasks when classifying document visualization methods. Our classification is considered more acceptable than other classifications (e.g., representations: pixel-based, map-based, tree-based graphs, node-link diagrams, circle graphs, etc.1), since visualization is usually task dependent, and users commonly begin with data and tasks. Actually, each method may belong to different category even under the same classification criteria; and we classify each method according to its key visualization focus (the visualization objects and tasks).

In Table 1, we summarize and compare those methods mainly from four aspects to give readers a brief view.

• Characteristics visualized. The characteristics of a document visualized by the method, as word frequency, semantic relations, content, changes, or connections among documents.

• Principles satisfied. The design principles satisfied, as noted in Document and Document Design section, the seven tasks: 1) Overview; 2) Zoom; 3) Filter; 4) Details-on-demand; 5) Relate; 6) History; 7) Extract.

• Requirements for a document. Document types suitable to the visualization method, i.e., whether the visualization method has special requirements for a document, like document content, structure, etc.

• Main features. Discuss the visualization method’s features, especially the versatility and interactivity.

Despite this, document visualization shares the same pipeline: get the data (a document or documents), transform it into vectors, then run algorithms based on the tasks of interest (i.e., similarity, search, clustering) and generate the visualizations.

Document visualization techniques combine human wisdom and computer graphics, allowing users to efficiently and intuitively browse, explore, and understand the increasing quantity of documents.

1. Extension: Existing methods can be extended to suit for large-scale document collections.
2. Versatility: It is significant to design relatively general visualization models for different tasks within this field, since existing methods always have narrow scope of application due to its pointed direction.
3. Interactivity: It is important to design a more intuitive man–machine interface to improve user’s experience of interaction. Also it is crucial to find some interstices to allow users to participate in researching and developing process, especially the testing period.
4. Techniques:
• Algorithms. Develop and adopt scalable, high-performance algorithms for text processing, such as text summary and clustering.
• Parallel processing technology. With the adoption and popularity of Coordinated and Multiple Views (CMV), a visualization system usually includes multi-views.
• Real-time processing technology.

1. Evaluation Many document visualization methods or even information visualization methods lack a quantitative measurement which can indicate the overall quality, novelty, uncertainly, and other evaluative metrics. More recently, there exist more and more publications that reflect upon current practices in visualization evaluation. In fact, the BELIV workshop was created as a venue for researchers and practitioners to ‘explore novel evaluation methods, and to structure the knowledge on evaluation in information visualization around a schema’. [62]

2. Theoretical Foundations The 2007 Dagstuhl Workshop identified collaborative information visualization with theory building as major directions for future development. [63]

Kucher, K., & Kerren, A. (2015). Text Visualization Techniques: Taxonomy, Visual Survey, and Community Insights. In 8th IEEE Pacific Visualization Symposium (PacificVis' 15), Hangzhou, China (pp. 117-121). IEEE Computer Society.

Kucher, K., & Kerren, A. (2015). Text Visualization Techniques: Taxonomy, Visual Survey, and Community Insights. In 8th IEEE Pacific Visualization Symposium (PacificVis' 15), Hangzhou, China (pp. 117-121). IEEE Computer Society.

近年來由於可以取得大量而多樣的文本資料和採用文本處理演算法等原因，研究人員對文本視覺化(text visualization)與視覺性的文本解析(visual text analytics)的研究興趣增加。本研究針對文本視覺化技術提出一個互動的視覺調查(visual survey)。並且利用此次調查的資料，分析文本視覺化的現況，比較研究使用的各種分析與視覺化技術，以及分析有關研究者的資訊，以提供搜尋相關研究、探索次領域(subfield)以及獲得研究趨勢的洞察等目的

本研究採納前人的研究，將文本視覺化技術，以分析任務(analytic tasks)、視覺化任務(visualization tasks)、資料領域(data domain)以及資料來源(data source)、資料性質(data property)、視覺化的維度(visualization dimensionality)、視覺化的呈現(visualization representation)、視覺化的排列方式(visualization alignment)等面向，建立分類架構(taxonomy)。

分析任務是指使用者採用文本視覺化技術預期達到的主要目的，這些分類包括：
1. 文本摘要 (Text Summarization) / 主題分析 (Topic Analysis) / 實體抽取 (Entity Extraction)

2. 言談分析 (Discourse Analysis)：文本或對話轉錄(conversation transcript)裡流動的語言學分析。
3. 情感分析 (Sentiment Analysis)
4. 事件分析 (Event Analysis)
5. 趨勢分析 (Trend Analysis) / 樣式分析 (Pattern Analysis)
6. 詞法/語法分析 (Lexical / Syntactical Analysis)
7. 關係/連結分析 (Relation / Connection Analysis)
8. 翻譯/文本比對分析 (Translation / Text Alignment Analysis)

視覺化任務則是由文本視覺化技術所支援的較基層呈現與互動任務，包括：

1. 自動凸顯/建議興趣區 (Region of Interest)
2. 群集 (Clustering) / 分類 (Classification / Categorization)
3. 比較 (Comparison)
4. 概觀 (Overview)
5. 監視 (Monitoring)
6. 瀏覽 (Navigation) / 探索 (Exploration)
7. 對於不確定的對策 (Uncertainty Tackling)

資料領域，包括

1. 線上社交媒體 (Online Social media)
2. 通訊 (Communication)
3. 專利 (Patents)
4. 評論 (Reviews) / 病歷 (Medical Records)
5. 文學作品 (Literature) / 詩 (Poems)
6. 科學文章 (Scientific Articles) / 論文 (Papers)
7. 社論媒體 (Editorial Media)

資料來源有單一文件 (Document) [33]、語料庫 (Corpora) [25]以及 串流文本 (Streams) [19]；特殊的資料性質包括地理空間 (Geospatial) [11]、時間序列 (Timeseries) [14] 以及網路 (Networks) [6]；視覺化的再現包括下列項目：折線圖 (Line Plot) / 河流圖 (River) [9, 18]、像素 (Pixel) / 面積 (Area) / 矩陣 (Matrix) [13, 7, 4]、節點-連結 (Node-Link) [32]、雲 (Clouds) / 銀河 (Galaxies) [1, 3]、地圖 (Maps) [34]、文本 (Text) [26]與形符 (Glyph) / 圖標 (Icon) [28, 10]；排列則包括了輻射狀 (Radial) [35]、線性 (Linear) / 平行線 (Parallel) [8] 以及測標依賴 (Metric-dependent) [22]。

本研究指出有超過一半(56%)的文本視覺化利用主題模型(topic modeling)技術，資料來源方面大多數支援語料庫(70%)，並且許多支援時間相關的資料(43%)，而視覺再現方面主題以二維(2-D)為主，僅有極少數的研究以三維(3-D)的方式呈現，約占所有研究的4%。

文本視覺化的前五位主要作者為Daniel A. Keim (17 筆)、Shixia Liu (12 筆)、Christian Rohrdantz (9 筆)、Daniela Oelke (7 筆)和 Huamin Qu (7 筆)。將作者依據他們的合著關係建立研究者合作網路圖後，觀察網路圖的相連成分，可以發現大部分是獨立的小群體，最大的成分上共有106位作者，並且在這個成分上的兩個主要集群為University of Konstanz和Microsoft Research Asia等兩個研究團隊，Daniel A. Keim 和 Shixia Liu分別為集群的中心，並且他們二位也是網路圖上中介中心性最高的節點。雖然在本研究蒐集的資料上，這兩位作者之間並沒有直接的合作關係，但他們都曾與中介中心性第三高的兩位作者Dongning Luo 和 Jing Yang合作。


In this paper, we present an interactive visual survey of text visualization techniques that can be used for the purposes of search for related work, introduction to the subfield and gaining insight into research trends.

The interest for text visualization and visual text analytics has been increasing for the last ten years. The reasons for this development are manifold, but for sure the availability of large amounts of heterogeneous text data (caused by the popularity of online social media) and the adoption of text processing algorithms (e.g., for topic modeling) by the InfoVis and Visual Analytics communities are two possible explanations.

Analytic Tasks
these items are critical to the main analysis goals that users expect to achieve when employing a text visualization technique.

1. Text Summarization / Topic Analysis / Entity Extraction

2. Discourse Analysis
the linguistic analysis of the flow of text or conversation transcript.

3. Sentiment Analysis
for techniques related to the analysis of sentiment, opinion, and affection.

4. Event Analysis
deal with the extraction of events from the text data or involve visualization of text in some different manner

5. Trend Analysis / Pattern Analysis
both automated trend analysis and manual investigation directed at discovering patterns in the textual data.

6. Lexical / Syntactical Analysis

7. Relation / Connection Analysis

8. Translation / Text Alignment Analysis

Visualization Tasks
lower-level representation and interaction tasks that are supported by the text visualization techniques.

1. Region of Interest
the automatic highlighting/suggestion of data items/regions that could be of interest to the user for more detailed investigation

2. Clustering / Classification / Categorization

3. Comparison

4. Overview
both techniques that provide “the big picture” by displaying a significant portion of the data set as well as techniques which use special aggregated representations to provide overview while reducing the visual complexity

5. Monitoring

6. Navigation / Exploration

7. Uncertainty Tackling


Domain

1. Online Social media

2. Communication

3. Patents

4. Reviews / (Medical) Records

5. Literature / Poems

6. Scientific Articles / Papers

7. Editorial Media

Data sources include the following self-evident items: Document [33], Corpora [25], and Streams [19].

The special data properties include Geospatial [11], Timeseries [14], and Networks [6].

Representation includes the following items: Line Plot / River [9, 18], Pixel / Area / Matrix [13, 7, 4], Node-Link [32], Clouds / Galaxies [1, 3], Maps [34], Text [26], and Glyph / Icon [28, 10].

Alignment, i.e., layout, includes Radial [35], Linear / Parallel [8], and Metric-dependent [22].

As displayed in the table, our proposed taxonomy includes most of the categories except for two: we believe that the underlying data representation (e.g., bag-of-words vs. language model [30] or whole text vs. partial text [24]) is more relevant to the underlying computational methods than to observable visualization techniques.

And the same naturally holds for data processing methods (e.g., the specification of involved MDS methods [2]) that are partially covered by other categories in our taxonomy, for instance, the analytic task of topic analysis implies the usage of corresponding computational methods.

Using the data collected for the survey, we have been able to analyze the general state of the text visualization field, to compare the usage of various analysis and visualization techniques (with regard to our taxonomy), and to analyze the information about researchers in this field.

According to our current set of entries, the trend for rapid increase of text visualization techniques started around 2007.

With regard to category statistics (cf. Fig. 4), there is an obvious interest for tasks related to topic modeling (56% of all entries).

The majority of the techniques support corpora as data sources (70% of all entries), and a lot of them support time-dependent data (43% of all entries).

Another result—which is probably expected—is that only less than 4% of all entries use 3-dimensional visual representations.

We have also taken a look at the authorship statistics for the current data set. The top five authors with regard to number of techniques are Daniel A. Keim (17 entries), Shixia Liu (12 entries), Christian Rohrdantz (9 entries), Daniela Oelke (7 entries), and Huamin Qu (7 entries).

As seen in Fig. 5, the majority of author nodes are included into isolated connected components of small sizes (less than 10 nodes) while there is a big connected component with 106 nodes present in the graph.

The two major clusters in that component represent the research groups from the University of Konstanz and Microsoft Research Asia with Daniel A. Keim and Shixia Liu as cluster center nodes.

Shixia Liu and Daniel A. Keim happen to have the 1st and the 2nd largest betweenness values in the graph, respectively. While these two researchers have no direct collaboration with regard to our data set, they both have collaborated with Dongning Luo and Jing Yang who both share the 3rd largest betweenness value.