Commit 824ed9e7 authored by tobinski's avatar tobinski
Browse files

Update readme

parent cdc835d6
# Magpie
Magpie is a stream transformer that takes urls from a stream and requests the resource from the internet. It can handle delays between requests and concurrency.
Magpie is a stream transformer that takes url's from a nodejs stream and requests the resource from the internet. It can handle delays between requests and concurrency.
## Usage
```js
......@@ -15,6 +15,12 @@ const magpie = new MagpieTransformer(options);
* logger = nconf logger instance (required)
* delay = delay between requests (default: 200)
* concurrent = concurrency (default: 10)
## Api
The transformer expect the following payload:
```js
const payload = JSON.stringify({url: 'http://example.com/test/1'});
```
It will return an object `Document` with the `url` (string) and the `response` (request.response) properties.
## Concurrency and delay
If we configure a concurrency of 10 and a delay of 200 then 10 requests are send in parallel and after finishing they will make a break for 200ms and send again. The first 10 requests hits the server in parallel and then after the response is send each callback will wait for 200ms before send again. After a while the delays between the requests are a bit more randomized.
## Explanation for concurrency and delay
If we configure a concurrency of 10 and a delay of 200 then 10 requests are send in parallel and after finishing they will make a break for 200ms and send again. The first 10 requests hits the server in parallel and after the response is send back from the server each callback will wait for 200ms before send again. After a while the delays between the requests are a bit more randomized, because the server will not response in the same pace.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment