Commit d84afa1f authored by Lena Heizmann's avatar Lena Heizmann
Browse files

Update readme

parent d87e3506
# Ethz
This app fetches all data (links and data) from the [ethz api](https://daas.library.ethz.ch/rib/v2/pois). It directly normalizes it and sends the results to the geolinker and linker topic in kafka.
This app fetchs all the data (links and data) from the ethz api. It directly normalized it and send the results to the `geolinker` and `linker topic` in kafka
## Crawler
We build a [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream) to get all the content from the api.
## ETH Normalizer Transformer
The ETH Normalizer Transformer takes the fetched resources and normalizes the data. The result is published to the geolinker topic.
## ETH Analyser Transformer
Ethz contains links to [wikidata](https://www.wikidata.org) and the [HLS](https://hls-dhs-dss.ch/). We send those connections to the geolinker topic as well.
We build a ReadableStream to get all all content from the api.
## eth-analyser-transformer
Ethz contains links to [wikidata](wikidata.org) and the [hls](https://hls-dhs-dss.ch/). We send those connections to the linker topic.
## eth-normalizer-transfromer
The extractor takes the fetched resource and normalized the data. The result is published to `geolinker topic`.
## Docker
To build the image use the following command. The container based on the [histub/ node-kafka-docker-base](https://source.dodis.ch/histhub/node-kafka-docker-base).
To build the image use following command. The container based on linux alpine.
```bash
docker build -t source.dodis.ch:4577/histhub/histhub/eth .
docker build -t source.dodis.ch:4577/histhub/eth .
# Upload to the registry
docker push source.dodis.ch:4577/histhub/histhub/eth
docker push source.dodis.ch:4577/histhub/eth
```
## CD
We have a build pipeline in gitlab. So manual building of the image is no longer necessary.
## Deploy to k8
This streming app is part of the [data helm chart](https://source.dodis.ch/histhub/charts/tree/master/charts/data).
\ No newline at end of file
We hav a build pipline in gitlab. So manually building of the image is not longer necessary.
```
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment