Experience defining and building Visualization Framework with D3.js and ReactJS

Posted on May 19, 2017

Experience defining and building Visualization Framework with D3.js and ReactJS

Motivation

Data visualization definitely needs to be part of MVP (Minimal Viable Product) when building any analytics system. When I started tinkering with Elasticsearch last year, I also played around with its default UI, Kibana. I was greatly impressed by the functionality that it provided especially following:

  • Ability to understand index schema and determining properties to build metrics and aggregations on.
  • Ability to create and save visualizations and dashboard.
  • Plugin architecture; though its still WIP (Work in progress).

Tremendously useful for demos and getting the thought process clear on the data intelligence, Kibana, as I played more, I found few qualities missing.

  • Charts were not interactive.
  • Custom query to visualization mapping was missing.
  • Problems supporting cluster without a client node due to the bug in the kibana code.
  • Health of cluster could turn to RED because of a bug in the Kibana code as it always set number of shards to be 1 for .kibana index.

After learning these, I setup on defining visualization framework. I went with choosing ReactJS and D3 as a base technology to build this framework. ReactJS, over time has proven itself to be conceptually sound, and its been used as a solid foundation upon which to build user interfaces. D3js, a javascript library to manipulate documents/charts based on data is also very stable and comes with varying options of charts and graphs.

I went with fulfilling following 4 core features:

  • Flexible enough to support any custom charts – static and dynamic.
    I learnt looking at many BI tools that there are certain default charts and then there are these custom charts. One should never build a stack where you need an engineering or IT team to define and create visualizations. User should be given an ability to write a query which generates response and it should be parseable by the framework to be converted into visualization/chart of choice.
     
  • Work with multiple backends at the same time.
    Most of the systems today deal with lots of data (I don’t mean big data :)). Also, most of the systems end up storing their data in multiple store or database. A UI framework should be capable enough to work with any backend and not just one. This gives flexibility to migrate to or add any backend (data storage).
     
  • Provide configuration based approach for interactive visualization. Interactive charts are super cool. Not just that, I think it gives completeness to the workflow of parsing through your data. If one knows the relationship schema within the data, he/she should be able to configure filters to build these workflows. For example, Census data: You first see a chart with a given country and all its state. Then you can click on a state and you can go to that state’s counties’ chart. You will be able to back and forth as well. 
  • Ability to save and share queries, visualizations and dashboards.
    Very common amongst the BI/DI tools but extremely important feature to have it as MVP.

Architectural points

  • Framework is written primarily using react, redux, redux-router, react-thunk library at its core.
  • Application is built as a finite state-machine. react-redux is used as a state-container and redux-router and react-router is used to do routing in application.
  • Services and its actions are defined first and then configuration is passed along as an object from one state to the other.
  • Configuration consists of a dashboard, visualizations and queries.
  • Result of one or more query can be shown as one visualization.
  • One or more visualization can be made part of a dashboard.
  • Framework supports many D3charts such as line-chart, multiline-chart, bar-chart, chord-diagram, pie-chart, donut-chart, heat map, gauge-chart table etc. D3js library is used heavily in defining some of these graphs. Every visualization must represents one and only one chart.

Snapshots

Reference : Courtesy of Ronak Shah (Problem Solver)