Create a Cumul.io API Plugin with Air Quality Data

Alright, so as I am in the process of learning about Cumul.io myself, what better way to learn about plugins than building one. So, in this post I’ll walk through how I built a plugin with an open API. I’ll go up to how I integrated it in a simple web page. I wanted to build something that I’d actually care about and be interested in looking at so I spent quite some time looking through open APIs. I wanted one that would provide some pollution or climate change data. I’ve ended up using the AirVisual API (if you have any other suggestions for future projects please tell me in the comments!). As I’m using the community version I limited myself to a few major cities around the world and am displaying daily air quality data for them. This is what I ended up with:

Like my previous post on multi-tenancy, I am accompanying this one with open repositories so you can follow the steps exactly if you so wish. First, cumulio-pollution for the plugin itself. Second, cumulio-pollution-landingpage, a simple webpage to integrate the dashboard into. If you don’t use these repos, each of the steps should still be relevant to anyone trying to build a basic plugin from an API for their Cumul.io dashboard.

Why would you build a plugin?

Cumul.io provides great visualizations for your data, but there’s the question of connecting such data to it. One way to accomplish this is by writing your own plugin, for example for an existing API. Plugins work as a bridge, or a connector from an external source of data to Cumul.io.

Without further ado, let me jump into how I got this thing working:

  1. Building the Plugin
  2. Building the Dashboard
  3. Integrating the Dashboard
  4. Filtering the Dashboard (Extra)
  5. Useful Resources

Building the Plugin

First step is to build the plugin itself. I’ve built mine on cumulio-pollution which you can clone if you’d like to follow the next steps exactly. To begin, I followed the steps in a previous tutorial on how to build plugins for Cumul.io dashboards which I found very useful, would recommend.

Here I will be building a basic plugin with no added authorization, since I’m using an open data source. I will connect to the open API from AirVisual as an example. This contains real time and historical weather and air quality data for requested cities, as well as forecast.

Once the plugin is written and added to Cumul.io, it will appear as a new data source when you want to add data to your dashboard in your account.

To start with, you will need the following:

  1. A Cumul.io account (there are free trials available)
  2. An API Key from AirVisual (I used the community edition)
  3. An App Secret from Cumul.io which you get when you create the plugin on your profile

This plugin only needs two endpoints. The /datasets endpoint and the /query endpoint:

// > index.js
var app = require('./server.js')();

app.get('/datasets', function(req, res) {
	// code that retrieves the metadata (which datasets and columns are available). 
})
app.post('/query', function(req, res) {
	// code that retrieves the actual data upon a query 
})

The /datasets Endpoint

This endpoint gives information about the structure of the data to Cumul.io. In my example, I’m only going to be providing one dataset:

app.get('/datasets', function(req, res) {   
    var datasets = [
        {
            id: "Air Visuals Data",
            name: {en: `Air Quality for Select Cities`},
            description: {en: `Real-time air quality data for select cities`},
            columns: [
                    {id: 'city', name: {en: 'City'}, type: 'hierarchy'},
                    {id: 'update', name: {en: 'Update time'}, type: 'datetime'},
                    {id: 'latitude', name: {en: 'Latitude'}, type: 'numeric'},
                    {id: 'longitude', name: {en: 'Longitude'}, type: 'numeric'},
                    {id: 'aqius', name: {en: 'Air Quality Index (US)'}, type: 'numeric'},
                    {id: 'aqcn', name: {en: 'Air Quality Index (China)'}, type: 'numeric'}
                ]
        }];
    return res.status(200).json(datasets);
});

The code defines 5 columns which the AirVisual API will fill with city names, update time, latitude, longitude and the two air quality indexes (US and China have different measures for air quality).

Note: If you run your code at this point you can have a look on postman to test GET localhost:3030/datasets

The /query Endpoint

At this point, we’ve defined what the dataset will be but there’s no actual data coming in. This is where we use the /query endpoint. In a regular plugin the result of the /query call returns an array of arrays with values that follow the exact same structure as defined in the /datasets endpoint. For example, we could return something like this:

 // > index.js
 app.post('/query', function(req, res) {
   if (req.headers['x-secret'] !== process.env.CUMULIO_SECRET)
     return res.status(403).end('Given plugin secret does not match Cumul.io plugin secret.');
  return res.status(200).json([
    ["Istanbul", "2020-10-06T14:33:34 , "41.02", "29.10" ,"96", "75"],
    ["Los Angeles", "2020-10-06T14:33:34 , "34.07", "-118.24" ,"111", "56"]],
    ...
  ]);
})

Now, what the following code does is create this array of arrays returned by the /query endpoint with responses it gets from the AirVisual API. I have picked a few major cities so as to limit the number of requests I make to the API. Feel free to change them as you like. To receive air quality and weather data, the API expects the name of the city, its state and country in the request:

var my_cities = {
    "New York City" : { "state" : "New York", "country" : "USA"},
    "Los Angeles" : { "state" : "California", "country" : "USA"},
    "Paris" : { "state" : "Ile-de-France", "country" : "France"},
    "Istanbul" : { "state" : "Istanbul", "country" : "Turkey"},
    "Hong Kong" : { "state" : "Hong Kong", "country" : "Hong Kong"},
    "Delhi" : { "state" : "Delhi", "country" : "India"},
    "Beijing" : { "state" : "Beijing", "country" : "China"}
};

Since there are limits to the number of requests I can make to the API within a given time period, I’ve implemented a cache and am checking whether the fields are over 24 hours old. If they are, I am adding to my cache:

const got = require('got');
var moment = require('moment');

const cache = {};
var city_keys = new Set();

async function getAirQuality(city) {
    const state = my_cities[city].state;
    const country = my_cities[city].country;
    let response = await got(`http://api.airvisual.com/v2/city?city=${city}&state=${state}&country=${country}&key=${process.env.API_KEY}`).json();
    if(response !== undefined && response !== null && response.data !== undefined && response.data !== null && response.data.current !== undefined && response.data.current !== null)
        return [city, response.data.current.pollution.ts, response.data.location.coordinates[1], response.data.location.coordinates[0], response.data.current.pollution.aqius, response.data.current.pollution.aqicn];
    else
        throw new Error("mesage request limit");
};

function isCacheStale(id) {
    const curTime = moment();
    return curTime.diff(cache[id].last_update, 'hours') >= 24;
}

function createCache(id) {
    if(!cache[id])
    {
        cache[id] = {
            values : [],
            last_update : null
        }
    }
}

app.post('/query', async function(req, res) {
    const details = req.body;
    if (req.headers['x-secret'] !== process.env.CUMULIO_SECRET)
      return res.status(403).end('Given plugin secret does not match Cumul.io plugin secret.');
    
    if(cache[details.id] !== undefined && !isCacheStale(details.id)) {
        return res.status(200).json(cache[details.id].values);
    }
    else {
        createCache(details.id);  
        for(const key in my_cities){
            try {
                let city_data = await getAirQuality(key);
                let city_key = city_data[0] + city_data[1];
                if(!city_keys.has(city_key)) {
                    city_keys.add(city_key);
                    cache[details.id].values.push(city_data);
                }
                cache[details.id].last_update = moment();
            } catch(error) {
                console.error("Reached request limit, will serve from cache");
            }
        }
        return res.status(200).json(cache[details.id].values);
    }
  });

If you’ve cloned and are using the cumulio-pollution repository, you can do the following to run the plugin:

  1. npm install
  2. Create a file called ‘.env’ in the root directory. Here, fill the API_KEY with the key you create with AirVisual API, and CUMULIO_SECRET with the App Secret that is created when you create the plugin on Cumul.io:
    API_KEY=XXX
    CUMULIO_SECRET=XXX

    PORT=3030
  3. npm run start or node index.js

To test that the /query endpoint works correctly I used postman (If you do, don’t forget to add the Cumul.io secret to the header):

Now your plugin is running on localhost:3030, but we have to add it to your Cumul.io account and make it accessible to it. For this step, I used ngrok. Running this will give you a https URL that you can use to make your plugin accessible from outside your PC. If you’re using Visual Studio Code you can install the ngrok extension, run it with port number 3030 and copy link (you can also run ngrok on a separate terminal). Once you’ve done that you can now;

Create the plugin on Cumul.io

Now let’s add your plugin to Cumul.io. Go to your profile and Create Plugin on the Plugins tab. Add the URL created in the previous step to Base URL & authentication:

Once it’s been created you will see the plugin on your profile and you can copy the App Secret to the CUMULIO_SECRET field in your .env file:

Building the Dashboard

Now that you have a plugin, when it’s running, you can create a dashboard on Cumul.io and add your plugin as a data source to it. Create a new dashboard and go to add data. You should now see a new plugin in the ‘Your plugins’ tab:

Add the plugin as a data source

Since we told the plugin to return only one dataset in the /datasets endpoint, you will see the following which you can import:

Import the dataset

Now you will see the Air Quality for Select Cities dataset listed on the right as one of your datasets. If you ‘Open in Editor’, you should see the following:

The dataset with data received from the AirVisual API

You can now use this dataset to build pretty dashboards to your heart’s content ๐Ÿ™‚ I, for example, decided that I would like to display a map, and that I would like the symbol colours to represent the official air quality colouring system. In the rest of this section, I’ll walk through what I did and how I achieved it;

Adding Coordinates and Custom Colours

For those of you who are interested, here is the colour scheme for the US air quality index (which is the one I used):

0 – 50 Healthy (green)

51 – 100 Moderate (yellow)

101-150 Unhealthy for Sensitive Groups (orange)

151 – 200 Unhealthy (red)

201 – 300 Very Unhealthy (purple)

301 =< Hazardous (maroon)

So I added two new fields to the dataset: Health Status and Coordinates;

Health Status:

On the Dataset Editor, select Add Formula and you can now define a chain of if statements that checks the Air Quality Index (US) column and classifies the health status:

To add colours, in the settings for the Health Status column you’ve just created, you can Edit Hierarchy and define custom colours:

Coordinates:

For this one, I already had latitude and longitude columns in the dataset so I simply had to select one of the two and add coordinates:

Once I had these two columns ready, I added a few charts to my dashboard:

The Map

For this, I’ve used a symbol map and have added the Coordinates column to Geography, the Health Status column to Color and the Air Quality Index (US) column to Measure. For more info on how to add data to your dashboard charts you can visit this webinar on adding charts.

I also want this map and the following bar chart to display today’s figures. So I’ve added a filter that checks the Update time is within the last 1 day:

The Bar Chart

The bar chart is much the same as the map, it displays the days air quality data. So again, I’ve added Air Quality Index (US) as Measure, but this time I’ve added the City to Category and Health Status to Group By. The same filter is also used here to display only the last 1 day. In the Item Settings, you can select Stacked Mode to achieve the same visuals.

The Grouped Line Chart

The line chart aims to provide a timeline for air quality data, so it will build up as you keep your plugin running. In the image above you see mine with 2 days of data. The X-axis is Update time, the Measure again is Air Quality Index (US) and the Group By is City.

The Speedometers

This last one I couldn’t resist. The speedometer was recently introduced to the dashboard editor. So, I selected two cities, Delhi and New York, and am displaying their air quality indexes here as well. In addition to the filter selecting the last 1 day, I’ve also added to each a filter on City and selected the relevant city name:

I do not select a Target for these charts, instead, I set the ranges in the settings:

There is a useful academy article on speedometers that details how to add and customize them.

Integrating the Dashboard

Once the dashboard exists, great, you have a dashboard that you can see when you login to your Cumul.io account. But you may want to have it integrated into another webpage for others to see too. Which is what I wanted to do; for this step, I’ve created cumulio-pollution-landingpage which you can clone and use. To run it:

  1. npm install
  2. Create a file called ‘.env’ in the root directory. Here, fill the CUMULIO_API_KEY and CUMULIO_API_TOKEN fields with the ones from your Cumul.io profile. If you don’t have any yet, you can create them in the API Tokens tab in your profile:
    CUMULIO_API_KEY=XXX
    CUMULIO_API_TOKEN=XXX
  3. Replace dashboardId in public/js/app.js and server.js with your own dashboard ID which you can find on the dashboard editor in Cumul.io.
  4. npm run start or node index.js

You can visit your webpage on localhost:3000

For this section you can have a look at the Integration API Documentation. As well as a previous blog post that walks through integrating a dashboard. For the full code you can visit the repo but here are some of the key points:

As an initial step, I load the Integration API in the header of my webpage:

//index.html
<html>
  <head>
    <script type="text/javascript" src="https://cdn-a.cumul.io/js/cumulio.min.js"></script>
  </head>
  <body>
   ...
  </body>
</html>

Next up I decided to add a side bar and a main content element where I would display my dashboard, and load the js/app.js script:

//index.html
<body>
  <div id="content" class="d-flex align-items-stretch w-100">
    <!-- Sidebar Container-->
    <div id="sidebar" class="d-flex flex-column h-100 text-white p-4">
        <!-- Company logo and name -->
        <div class="company-logo px-2 w-100 mb-5 mr-4">
          <img src="/images/color_logo_white_text.png" width="50%"/>
        </div>
        <div class="mb-4">
          <div class="d-flex">
            <div>
              <h5>Welcome to Demo Air Quality Dashboards</h5>
            </div>
          </div>
        </div>
        <div class="flex-fill"></div>
      </div>
    <!-- Dashboard container -->
    <div class="main-container text-center">
      <div class="dashboard-outer-container">
        <div id="dashboard-container"></div>
      </div>
    </div>
    <div id="overlay"></div>
  </div>
  <script src="/js/app.js"></script>
</body>

The actual dashboard embedding happens in js/app.js. Here I call the addDashboard function. This function loads and renders the dashboard with the specified id in the specified HTML div, which for me is the ‘dashboard-container’:

const dashboardOptions = {
  dashboardId: dashboardId,
  container: '#dashboard-container',
  loader: {
    background: '#EEF3F6',
    spinnerColor: '#004CB7',
    spinnerBackground: '#DCDCDC',
    fontColor: '#000000'
  }
Cumulio.addDashboard(dashboardOptions);

I wrap the above addDashboard in a loadInsightsPage function that is called when the window loads:

const loadDashboard = (key, token) => {
  // use tokens if available
  if (key && token) {
    dashboardOptions.key = key;
    dashboardOptions.token = token;
  }
  // add the dashboard to the #dashboard-container element
  Cumulio.addDashboard(dashboardOptions);
}

// Function to retrieve the dashboard authorization token from the platform's backend
const getDashboardAuthorizationToken = async (city) => {
  try {
    const response = await fetch(`/authorization`, {});

    // Fetch the JSON result with the Cumul.io Authorization key & token
    const responseData = await response.json();
    return responseData;
  }
  catch (e) {
    // Display errors in the console
    console.error(e);
    return { error: 'Could not retrieve dashboard authorization token.' };
  }
};

// function to load the insight page
const loadInsightsPage = async () => {
  const authorizationToken = await getDashboardAuthorizationToken();
  if (authorizationToken.id && authorizationToken.token) {
    loadDashboard(authorizationToken.id, authorizationToken.token);
  }
}
// on page load
window.onload = async () => {
  loadInsightsPage();
}

And finally, one last thing to note is server.js where the /authorization endpoint is defined. It uses the Cumul.io Node SDK to request an authorization key & token for the dashboard. The Cumul.io API key and token are first read from the .env file:

const Cumulio = require('cumulio');

const client = new Cumulio({
    api_key: process.env.CUMULIO_API_KEY,
    api_token: process.env.CUMULIO_API_TOKEN
});

app.get('/authorization', (req, res) => {
    const options = {
        type: 'temporary',
        expiry: '1 day',
        inactivity_interval: '30 minutes',
        securables: [dashboardId]
    };
    client.create('authorization', options).then((result) => {
        return res.status(200).json(result);
    });
});

app.get('/*', (req, res) => {
    res.sendFile(join(__dirname, 'index.html'));
});

app.listen(3000, () => console.log('Application running on port 3000'));

Finally, I decided that I’d like to add some extras so as to experiment with filtering from an integrated dashboard. The next section will walk through how I achieved some simple filtering by interacting with the webpage.

Filtering the Dashboard (Extra)

For this final step, I just approached it experimentally so there’s nothing fancy going on. I simply added the list of cities on the sidebar, and filtered the dashboard on that city when clicked:

Result when filtering on Los Angeles

To achieve this, I first added a list of cities to the sidebar that calls a reloadDashboard function on click:

//index.html
<div class="pages small">
   <ul class="list-unstyled text-uppercase p-0">
      <li onclick="reloadDashboard('Los Angeles')">Los Angeles</li>
      <li onclick="reloadDashboard('Delhi')">Delhi</li>
      <li onclick="reloadDashboard('Istanbul')">Istanbul</li>
      <li onclick="reloadDashboard('New York City')">New York</li>
      <li onclick="reloadDashboard('Paris')">Paris</li>
      <li onclick="reloadDashboard('Beijng')">Beijng</li>
   </ul>
</div>

Then I had to modify the getDashboardAuthorizationToken function to accept an input (the city name) which we can add to /authorization and then refreshData with the filtering added to our authorization options:

//app.js
const getDashboardAuthorizationToken = async (city) => {
  try {
    const response = await fetch(`/authorization${city ? '?city=' + city : ''}`, {});

    // Fetch the JSON result with the Cumul.io Authorization key & token
    const responseData = await response.json();
    return responseData;
  }
  catch (e) {
    // Display errors in the console
    console.error(e);
    return { error: 'Could not retrieve dashboard authorization token.' };
  }
};

const reloadDashboard = async (city) => {
  const authorizationToken = await getDashboardAuthorizationToken(city);
  Cumulio.setAuthorization(authorizationToken.id, authorizationToken.token, {dashboardId : dashboardId, container : '#dashboard-container'});
  Cumulio.refreshData();
}

//server.js
app.get('/authorization', (req, res) => {
    const options = {
        type: 'temporary',
        expiry: '1 day',
        inactivity_interval: '30 minutes',
        securables: [dashboardId]
    };
    if(req.query.city){
        options.metadata = {city : [req.query.city]};
    }
    client.create('authorization', options).then((result) => {
        return res.status(200).json(result);
    });
});

And that’s it for coding! The only thing remained to do is to add ‘city’ as a parameter to the dashboard itself on Cumul.io. Now when the reloadDashboard is called on click, we send meta_data to the dashboard with ‘city’ name, and Cumul.io uses the value I send over as its parameter value. To add a parameter, you can go to the FILTERS tab and ‘Create Parameter’ with type Hierarchy[].

I wanted to be able to see all cities on the dashboard editor so added all my cities as default values here. Then I need to use this parameter on the dashboard so on Dashboard Filters ‘Create filter’:

And that’s it ๐Ÿ™‚ I now have my own plugin that uses an API, used on a dashboard that I integrate into a webpage and filter from there. Voila ladies and gentlemen:

Useful Resources

Add a Comment

Your email address will not be published. Required fields are marked *