Mapping KML folder collections in one run

A KML file can group placemarks in <Folder> elements. Cassiopeia exposes each folder as a collection, and a manifest can send each collection to its own mapping and entity type. This example uses one KML with four mapped folders and produces four NGSI-LD entity types in one run.

The previous example had one folder, so every record shared one namespace and one mapping. Here the manifest binds several folders by name.

Get the data

Google My Maps exports each map layer as a <Folder>, which makes it a convenient source for this example. Any public map can be fetched as plain KML with forcekml=1. This map has layers for Citi Bike docks, bike-rental shops, sights, and public restrooms:

curl --fail --location --output data/nyc-cycling.kml "https://www.google.com/maps/d/kml?mid=1ouRmFjRHxcdgoI4HSXml8UBqaWkh1k8i&forcekml=1"

Replace the mid with your own public map's ID and the rest of the example works the same way.

That map belongs to somebody else, and this example leans on more of it than most examples lean on their source: the manifest binds four folder names exactly as the owner wrote them, and the counts below are the placemarks each layer held when this page was written. Nothing here can stop the owner renaming a layer or adding a stop. If the counts stop matching, that is the map having changed rather than the mapping, and the answer is to point mid at another public map.

The record shape

The file holds five folders. Four contain point placemarks that this example maps; the fifth is a driving route and is left alone:

Folder	What it holds	Mapped to
Citi Bike Stations	14 docking stations	CitiBikeStation
Where to Get Bikes	8 rental-shop branches	BikeRentalShop
Places to Stop	sights along the way	Sightseeing
Restrooms	public restrooms	PublicRestroom
Directions from the route	a route line	(unmapped)

As with any foldered KML, each record is nested under a lower-case, underscored version of its folder name. A placemark in Citi Bike Stations is under citi_bike_stations; one in Where to Get Bikes is under where_to_get_bikes. Each mapping uses its folder's prefix.

Binding folders to mappings

The manifest contains the key part of the example. One input has a mappings list that pairs each folder's exact name with the mapping for that collection:

{
    source: "data/nyc-cycling.kml",
    format: "kml",
    mappings: [
        {
            collection: "Citi Bike Stations",
            mapping: "station.json5",
        },
        {
            collection: "Where to Get Bikes",
            mapping: "rental.json5",
        },
        {
            collection: "Places to Stop",
            mapping: "sight.json5",
        },
        {
            collection: "Restrooms",
            mapping: "restroom.json5",
        },
    ],
}

Each folder's placemarks go only to its own mapping, so one file produces four entity types. The unlisted Directions folder is skipped, and an unbound collection is not an error.

Four small mappings

Each mapping reads the placemark's name and geometry through its folder namespace. Station and restroom labels include a category prefix and a non-breaking space, so an anchored strip removes the prefix while preserving the rest of the name:

// station.json5, "Citi Bike - W 15 St & 10 Ave" becomes "W 15 St & 10 Ave"
identity: {
    entityName: "{{ citi_bike_stations.properties.name | regex_replace(pattern='^Citi Bike[^0-9A-Za-z]+', rep='') | trim | clean }}",
},
attributes: {
    name: {
        source: "{{ citi_bike_stations.properties.name | regex_replace(pattern='^Citi Bike[^0-9A-Za-z]+', rep='') | trim }}",
        type: "Property",
        transformation: "string",
    },
    location: {
        source: "{{ citi_bike_stations.geometry }}",
        type: "GeoProperty",
        transformation: "geometry",
    },
},

The sight mapping reads name and location directly. The rental mapping cannot key on the name alone: the owner pinned three shops at two branches each under the same name, and a name-only identity would merge each pair into one entity at one of its two locations. It appends the geohash of the placemark's point instead, the same suffix that keeping same-named entities apart uses for cities:

// rental.json5, "Bike Rent NYC" at two branches becomes two entities
identity: {
    entityName: "{{ where_to_get_bikes.properties.name | trim | clean }}-{{ geohash(lat=where_to_get_bikes.geometry.coordinates[1], lon=where_to_get_bikes.geometry.coordinates[0]) }}",
},

A placemark's point has the GeoJSON shape, longitude first, so coordinates[1] is the latitude and coordinates[0] the longitude.

Run it

cassiopeia map \
    --manifest manifest.json5

The same run in a container mounts this directory at /data and makes it the working directory, so the paths do not change. For Podman, replace docker with podman and drop the --user line: rootless Podman already maps the container's root to your user.

docker run --rm \
    --user "$(id -u):$(id -g)" \
    --volume "$PWD:/data" \
    --workdir /data \
    ghcr.io/vela-tools/cassiopeia:v1.1.0 \
    map \
    --manifest manifest.json5

From the repository root, the runner downloads the dataset, runs the mapping, and checks the output in one step:

cargo run -- run 17
cargo run -- run 17 --runtime docker

The run writes four files, one per bound folder:

CitiBikeStation.json   14 entities
BikeRentalShop.json     8 entities
Sightseeing.json        7 entities
PublicRestroom.json     7 entities

Read the result

Here is one entity from each mapped folder, all produced from the same file:

{
    "id": "urn:ngsi-ld:CitiBikeStation:W120StClaremontAve",
    "type": "CitiBikeStation",
    "name": {
        "type": "Property",
        "value": "W 120 St & Claremont Ave"
    },
    "location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [
                -73.963,
                40.811
            ]
        }
    }
}

{
    "id": "urn:ngsi-ld:BikeRentalShop:BikeRentNYC-dr5regq39",
    "type": "BikeRentalShop",
    "name": {
        "type": "Property",
        "value": "Bike Rent NYC"
    },
    "location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [
                -74.006,
                40.711
            ]
        }
    }
}

{
    "id": "urn:ngsi-ld:Sightseeing:911Memorial",
    "type": "Sightseeing",
    "name": {
        "type": "Property",
        "value": "9/11 Memorial"
    },
    "location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [
                -74.013,
                40.711
            ]
        }
    }
}

{
    "id": "urn:ngsi-ld:PublicRestroom:Pier40",
    "type": "PublicRestroom",
    "name": {
        "type": "Property",
        "value": "Pier 40"
    },
    "location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [
                -74.011,
                40.729
            ]
        }
    }
}

The map's folders become collections in the run: one KML file in, four entity types out.