Commit 8efdb103 authored by Lena Heizmann's avatar Lena Heizmann
Browse files

Update readme

parent 824ed9e7
# Magpie
Magpie is a stream transformer that takes url's from a nodejs stream and requests the resource from the internet. It can handle delays between requests and concurrency.
Magpie is a stream transformer that takes urls from a nodejs stream and requests the corresponding resource from the internet. It can handle delays between requests as well as concurrency.
## Usage
```js
......@@ -15,12 +15,13 @@ const magpie = new MagpieTransformer(options);
* logger = nconf logger instance (required)
* delay = delay between requests (default: 200)
* concurrent = concurrency (default: 10)
## Api
The transformer expect the following payload:
The transformer expects the following payload:
```js
const payload = JSON.stringify({url: 'http://example.com/test/1'});
```
It will return an object `Document` with the `url` (string) and the `response` (request.response) properties.
## Explanation for concurrency and delay
If we configure a concurrency of 10 and a delay of 200 then 10 requests are send in parallel and after finishing they will make a break for 200ms and send again. The first 10 requests hits the server in parallel and after the response is send back from the server each callback will wait for 200ms before send again. After a while the delays between the requests are a bit more randomized, because the server will not response in the same pace.
If we configure a concurrency of 10 and a delay of 200, then the first 10 requests hit the server in parallel and after the response is sent back from the server each callback will wait for 200ms before sending again. After a while the delays between the requests are a bit more randomized, because the server will not always respond in the same pace.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment