Magpie is a stream transformer that takes url's from a nodejs stream and requests the resource from the internet. It can handle delays between requests and concurrency.
Magpie is a stream transformer that takes urls from a nodejs stream and requests the corresponding resource from the internet. It can handle delays between requests as well as concurrency.
## Usage
```js
...
...
@@ -15,12 +15,13 @@ const magpie = new MagpieTransformer(options);
It will return an object `Document` with the `url` (string) and the `response` (request.response) properties.
## Explanation for concurrency and delay
If we configure a concurrency of 10 and a delay of 200 then 10 requests are send in parallel and after finishing they will make a break for 200ms and send again. The first 10 requests hits the server in parallel and after the response is send back from the server each callback will wait for 200ms before send again. After a while the delays between the requests are a bit more randomized, because the server will not response in the same pace.
If we configure a concurrency of 10 and a delay of 200, then the first 10 requests hit the server in parallel and after the response is sent back from the server each callback will wait for 200ms before sending again. After a while the delays between the requests are a bit more randomized, because the server will not always respond in the same pace.