Commit 6aa66b6f authored by Tobinsk's avatar Tobinsk
Browse files

Merge branch 'readme' into 'master'

Update readme

See merge request !7
parents cdc835d6 8efdb103
# Magpie
Magpie is a stream transformer that takes urls from a stream and requests the resource from the internet. It can handle delays between requests and concurrency.
Magpie is a stream transformer that takes urls from a nodejs stream and requests the corresponding resource from the internet. It can handle delays between requests as well as concurrency.
## Usage
```js
......@@ -16,5 +16,12 @@ const magpie = new MagpieTransformer(options);
* delay = delay between requests (default: 200)
* concurrent = concurrency (default: 10)
## Concurrency and delay
If we configure a concurrency of 10 and a delay of 200 then 10 requests are send in parallel and after finishing they will make a break for 200ms and send again. The first 10 requests hits the server in parallel and then after the response is send each callback will wait for 200ms before send again. After a while the delays between the requests are a bit more randomized.
## Api
The transformer expects the following payload:
```js
const payload = JSON.stringify({url: 'http://example.com/test/1'});
```
It will return an object `Document` with the `url` (string) and the `response` (request.response) properties.
## Explanation for concurrency and delay
If we configure a concurrency of 10 and a delay of 200, then the first 10 requests hit the server in parallel and after the response is sent back from the server each callback will wait for 200ms before sending again. After a while the delays between the requests are a bit more randomized, because the server will not always respond in the same pace.
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment