Readme.md 1.12 KB
Newer Older
Tobias Steiner's avatar
Tobias Steiner committed
1
# Magpie
Lena Heizmann's avatar
Lena Heizmann committed
2
Magpie is a stream transformer that takes urls from a nodejs stream and requests the corresponding resource from the internet. It can handle delays between requests as well as concurrency.
Tobias Steiner's avatar
Tobias Steiner committed
3

4
## Usage
tobinski's avatar
tobinski committed
5
6
7
8
9
10
11
12
13
```js
const options = {
    objectMode: true,
    logger: coreMock.getLogger(),
    delay: 200,
    concurrent: 10
};
const magpie = new MagpieTransformer(options);
```
14

tobinski's avatar
tobinski committed
15
16
17
* logger = nconf logger instance (required)
* delay = delay between requests (default: 200)
* concurrent = concurrency (default: 10)
Lena Heizmann's avatar
Lena Heizmann committed
18

tobinski's avatar
tobinski committed
19
## Api
Lena Heizmann's avatar
Lena Heizmann committed
20
The transformer expects the following payload:
tobinski's avatar
tobinski committed
21
22
23
24
```js
const payload = JSON.stringify({url: 'http://example.com/test/1'});
```
It will return an object `Document` with the `url` (string) and the `response` (request.response) properties.
25

tobinski's avatar
tobinski committed
26
## Explanation for concurrency and delay
Lena Heizmann's avatar
Lena Heizmann committed
27
If we configure a concurrency of 10 and a delay of 200, then the first 10 requests hit the server in parallel and after the response is sent back from the server each callback will wait for 200ms before sending again. After a while the delays between the requests are a bit more randomized, because the server will not always respond in the same pace.