Commit 51ca95d1 authored by tobinski's avatar tobinski

Upodate readme

parent 5225ab8b
# Dodis
This app gets all the links from dodis and extracts the fetched resources.
This app gets all the geographical items from [dodis](https://dodis.ch) and extracts the fetched resources to kafka.
## Sitemap
The first step is to generate a sitemap. The links are published to magpie.
## Extractor
The extractor takes the fetched resource from magpie and normalized the data frrom dodis. The result is published to geolinker topic
## SitemapReader
The SitemapReader (node stream readable) generates a list of geographical entities in dodis.
## DodisNormalizerTransformer
The normalizer takes the fetched resource from the SitemapReader and normalized the data. The result is published to geolinker topic in kafka
## Docker
To build the image use following command. The image will fetch data from a wikidata topic and streams the result back into kafka. The container based on linux alpine.
To build the image use following command. The container based on the [histub/ node-kafka-docker-base](https://source.dodis.ch/histhub/node-kafka-docker-base).
```bash
docker build -t source.dodis.ch:4577/histhub/wikidata-normalizer .
docker build -t source.dodis.ch:4577/histhub/dodis .
# Upload to the registry
docker push source.dodis.ch:4577/histhub/wikidata-normalizer
docker push source.dodis.ch:4577/histhub/dodis
```
## CD
We hav a build pipline in gitlab. So manually building of the image is not longer necessary.
## Deploy to k8
We execute a job on k8 to stream the dump into kafka
```bash
kubectl create -f wikidata-normalizer-deployment.yaml
```
\ No newline at end of file
This streming app is part of the [data helm chart](https://source.dodis.ch/histhub/charts/tree/master/charts/data).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment