Wikis > Guidelines for Data Upload module
Designed by Balasoiu / Freepik

Illustration by Balasoiu / Freepik

These guidelines of the Data Upload module are a description of the process of data input when using different modules in the the Ecodistr-ICT IDSS platform. It also describes which templates that are needed in order to upload data. The Data Upload module was developed by CSTB.

In the guidelines, there are references made to different free software solutions. It is supposed that these software are already installed by the user on its own computer.

The Data Upload module aims at providing a way to send, through the Dashboard of the IDSS, data required by the different calculation modules. Before proceeding to the upload, there is a prerequisite to prepare these data and combine together in a structured manner the various information needed by these other modules, see Figure 1.

In order to explicitly define the information required and its structure, different templates have been defined. Additionally the information carried by the instantiation of these templates need to be combined (or joined) with geographic information (location, building footprint, orientation, etc.). This is mainly the case for the so-called “Building Template” that is described later in these guidelines.

Most of the time in such project as the Ecodistrict-ICT project acting at the district scale, there are existing GIS (Geographic Information System) files containing (at least) the geographic information mentioned above. Nevertheless, a quick guide has been established also to provide solutions for the elaboration of such GIS file.

Most of the time, the need for using a platform like the one developed in Ecodistrict-ICT project means that the user belongs to the domain and thus is familiar with GIS features and already owns a digital map corresponding to the study (at least a terrain description + green areas/buildings/streets footprints). Nevertheless, there is below a simple process that describes how to start a study and generate the GIS file to be uploaded even if there is no existing explicit geographic information file for the considered project. There are several GIS systems available today. The example taken here is based on QGIS which represent one open / free software solution among others. QGIS is supported by a wide and active community and as such many tutorials and guides can be found over the internet.

data_module1

Figure 1 The process to prepare the CSV file to be uploaded via the Data Upload module.

 

Getting a digital map from an open database

The first step consists in selecting on the OpenStreetMap website the map of the study as shown on the Figure 2.

Then by selecting the “export” button, it is possible to save the map displayed on the screen as a local file which is exported as an XML file (usually called by default “maps.osm”).

Figure 2 Example of the OSM interface, where you select your project area directly on the map.

Figure 2 Example of the OSM interface, where you select your project area directly on the map.

 

In the example shown in the figure, the scale is relatively small. It means that the resulting file can be considered also small (around 1MB). The current process is particularly focusing on small files. For bigger district, it is needed to use other functionality from the OSM web site like the “API Overpass” (available from the Open Street Map web site), see Figure 3.

Figure 3 Example of the OSM interface, showing a “Big” District.

Figure 3 Example of the OSM interface, showing a “Big” District.

Converting the OSM file
This OSM file needs to be transformed in order to be opened into properly by QGIS which is an open GIS system widely used. The conversion can be done by using for instance the OpenStreetMap java editor called “JSOM” (Figure 4).

Figure 4 Example of the JSOM interface: Displaying the OSM file corresponding to the project area.

Figure 4 Example of the JSOM interface: Displaying the OSM file corresponding to the project area.

 

Opened in “Java OSM Editor”, the OSM file needs to be “converted” to be readable in QGIS (same need and same process for ArcGIS).

In JOSM the user needs to:

  • Select all “Crtl+A”
  • Copy and paste directly into a new layer.
  • Save the new layer (and only this one) as a new GeoJSon file (Figure 5)

Remark: It seems that this possibility to “save as GeoJson” is not available when the JOSM is installed on a MAC.

Figure 5 Conversion via JSOM of the initial OSM file into a GeoJSON file.

Figure 5 Conversion via JSOM of the initial OSM file into a GeoJSON file.

 

The file is now ready to be opened into a GIS editor like QGIS.

Using QGIS to map the templates with the Geographic information

In QGIS, select a new project, add a new vector layer and select the Json file you created as the source file for this layer. You have now the required shape file to start working with. Especially, you can link the geographic information to the input data templates of Districts/ Spaces/Buildings (described in section “Description of the templates” below) and import/export information via the different possibilities offered by the QGIS tools.

Figure 6 Same area displayed via the QGIS interface.

Figure 6 Same area displayed via the QGIS interface.

 

Figure 7 The “Entity table” in QGIS displaying the information about each object contained in the selected map.

Figure 7 The “Entity table” in QGIS displaying the information about each object contained in the selected map.

 

Figure 8 Example of a “big” district displayed via the QGIS interface (corresponding file around 30 MB).

Figure 8 Example of a “big” district displayed via the QGIS interface (corresponding file around 30 MB).

 

As shown in the Figure 7, there is a possibility within QGIS to edit a “table of entity”. This table contains some default information but can also be extended for specific purposes. In the case of IDSS, the work consists in editing this table by adding the fields that have been defined in the templates presented later in these guidelines.

Using QGIS to export the project as a CSV file

After editing work, in order to prepare the CSV files to be uploaded via the Data Upload module, the user needs to select the layer and save it as CSV (with Semicolon as separator) and specify that the geometry must be saved as WKT (Well-Known Text) as shown in the Figure 9.

Figure 9 The “Save as” functionality in QGIS: last step to produce the CSV file to be uploaded via the Data Upload module.

Figure 9 The “Save as” functionality in QGIS: last step to produce the CSV file to be uploaded via the Data Upload module.

 

Note: At this stage, it has been decided to use the WGS84 projection (EPSG 4326, de GSP geoide) (standard lat lon) for conversion to geoJson.

The resulting CSV file is now ready to be uploaded in the Data Upload module found in the Dashboard of the Ecodistr-ICT IDSS platform.

It is worth noticing that at this stage, that the CSV file can also be opened as a spread sheet and enriched via other “classic” office tools like MS Excel. The important point is to save the resulting work always as a CSV file (and not in the xlsx format).

Using the Dashboard to access the Data Upload module and proceed to the upload

As soon as the user is connected to the IDSS Dashboard (for Dashboard guidelines turn to link >>) and when a new case has been created, it will be possible to “Collect data” (see Figure 10) and proceed to the upload of CSV files (Figure 11).

Figure 10 Via the Dashboard, a specific button “Upload data” points directly on the upload page.

Figure 10 Via the Dashboard, a specific button “Upload data” points directly on the upload page.

 

Figure 11 The upload page.

Figure 11 The upload page.

Description of the templates

Three different input data templates have been established corresponding to the different situations we faced for the project pilots:

  • “District template” aims at covering the input data needs at the district level.
  • “Space template” aims at covering the input data needs at the “Space” level (the notion of “Space” is corresponding to a subset of the notion of District. For instance, it is well suited for the description of Green areas in a district.)
  • “Building template” aims at covering the input data needs at the building level.

These templates are containing several columns starting with headers (field names). All of these headers are corresponding to district/space/building information that need to be entered into the system before making use of dedicated modules to perform some calculation over these data. As a guide to fill in the three templates, definitions, usages and unit of the variables (input data) are defined in three PDF files, “Structure of the district template”,“Structure of the space template” and “Structure of the Building template”. One line is representing one entity description. For the building template, one line represents one building and thus the building template can contain the description of several buildings. (Note that the links between input data and modules are better described in each module page in the wiki.)

The templates are available for download on Github (https://github.com/ecodistrict under DataModule -> data -> Template).

In the explaining PDF files mentioned above, there is a systematic use of the following table header to structure the description for each template.

data12

  • The “Nb” column contains a series of identifier. It can be useful to refer to these identifiers to check what are the needed entries to make use of calculation modules.
  • The “Name” column contains a list of the variables used in the different templates.
  • The “Type” column contains the description of the type in which the variable is expressed. Possible values are:
    • T for “text”. Possible values are any chain of characters
    • D for “double”. Possible values are any figure with decimals. The separator between entire values and decimals is the dot (“.”). Example : “5.01”
    • B for “boolean”. Possible values are “0” or “1”
    • I for “integer”. Possible values are any whole number
    • G for “geometry ”: contains the WKT description of the geometry for the concerned element
  • The “Module” column allows indicating for which module this variable is useful. Possible values:
    • “L” for LCC module
    • “G” for Green module
    • “D” for Dimosim module
    • “A” for the Affordability module
  • The “Definition / Unit” column is where the user will find a definition on the variable, the units used.

More Supporting modules Calculation modules

Share on Google+Share on FacebookTweet about this on TwitterShare on LinkedIn