Skip to content

Commit 86002e0

Browse files
author
nickpape-msft
committed
PR 27668: Merge nickpape/stream-moderator to master
- Rename to stream-moderator - Add a readme - Improve the interface
1 parent 5b88fd6 commit 86002e0

File tree

10 files changed

+529
-300
lines changed

10 files changed

+529
-300
lines changed
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
## What is Stream-Moderator?
2+
3+
Oftentimes, when working with multiple parallel asynchronous processes, it is helpful to ensure that their
4+
outputs are not mixed together, as this can cause readability issues in the console or log. The
5+
stream moderator manages the output of these streams carefully, such that no two streams are writing
6+
at the same time. At any given time, one stream registered with the moderator is the **active stream**
7+
which means that particular stream will be live streaming, while the others will wait for that stream
8+
to finish before their completion.
9+
10+
For example, if you have 3 streams (e.g. from using `child_process.spawn()`).
11+
12+
Stream A will write: `AAAAA`
13+
14+
Stream B will write: `BBBBBBBBBBBBBBBBBBBB`
15+
16+
Stream C will write: `CCCCCCCCCC`
17+
18+
If these streams are all being piped directly to stdout, you could end up with something like:
19+
20+
`ABACCCBCCCCBBABBCBBABBBBBBCCAB`
21+
22+
**Yikes!**
23+
24+
Most likely, something like the following would be much more useful to users of your application:
25+
26+
`AAAAABBBBBBBBBBBBBBBCCCCCCCCCC`
27+
28+
This is where the stream-moderator comes in handy!
29+
30+
## Installation
31+
32+
Install the stream-moderator:
33+
34+
`npm install --save @microsoft/stream-moderator`
35+
36+
Import the moderator:
37+
38+
```javascript
39+
import StreamModerator from '@microsoft/stream-moderator'; // es6
40+
```
41+
42+
```javascript
43+
const StreamModerator = require('@microsoft/stream-moderator'); // commonjs
44+
```
45+
46+
## Usage
47+
48+
A stream moderator adheres to the [NodeJS Stream API](https://nodejs.org/api/stream.html), meaning that it effectively
49+
is special type of [ReadableStream](https://nodejs.org/api/stream.html#stream_class_stream_readable). This makes
50+
working with the stream moderator very simple. Imagine we had the 3 streams from the example above:
51+
52+
```javascript
53+
const streamA = getRepeaterStream('A', 5); // fake helper function that returns a ReadableStream
54+
const streamB = getRepeaterStream('B', 15); // fake helper function that returns a ReadableStream
55+
const streamC = getRepeaterStream('C', 10); // fake helper function that returns a ReadableStream
56+
```
57+
58+
Now, instantiate a stream moderator instance and register the streams with it:
59+
60+
```javascript
61+
const moderator = new StreamModerator();
62+
63+
moderator.register(streamA);
64+
moderator.register(streamB);
65+
moderator.register(streamC);
66+
```
67+
68+
`moderator` is now a stream which can be accessed with the standard stream API's. For example, you could pass the output
69+
to process.stdout:
70+
71+
`moderator.pipe(process.stdout);`
72+
73+
Or a file:
74+
75+
```javascript
76+
var wstream = fs.createWriteStream('myOutput.txt');
77+
78+
moderator.pipe(wstream);
79+
```
80+
81+
## The active stream
82+
At any given time, a single stream is designated as the **active stream**. The output of the active stream will always be
83+
live-streamed. This is particularly useful for long-running streams. When the active stream finishes, a new stream
84+
is selected as the active stream and all of its contents up to that point will be emitted. Whenever an active stream finishes,
85+
all background streams which have been completed will be emitted.
86+
87+
## Helper streams
88+
Two additional stream classes are also exported with this package:
89+
90+
### DualTaskStream
91+
A utility string-based stream with two sub-streams, `stdout` and `stderr`. These streams can be written to, and will be emitted
92+
by this class. Anything written to `stderr` will be automatically wrapped in red coloring, unless is begins with the text `Warning -`,
93+
in which case it will write the warning to `stdout` in yellow.
94+
95+
### PersistentStream
96+
A special string-based stream with a function `readAll()` which will return the contents of everything that has been written
97+
to the stream as a string, regardless of whether the stream is open or closed.
98+
99+
## Improvements
100+
NOTE: Ending the moderator stream could be improved with an option that lets you select between the following behaviors:
101+
* Close the moderator stream when ANY registered stream has been closed
102+
* Close the moderator stream when ALL registered streams have been closed
103+
* Don't automatically close the moderator stream

libraries/stream-collator/package.json

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
2-
"name": "@ms/console-moderator",
3-
"version": "0.2.1",
4-
"description": "Manage interleaving the output of multiple simultaneous processes",
2+
"name": "@ms/stream-moderator",
3+
"version": "0.3.1",
4+
"description": "Manage interleaving of multiple streams into a single stream...",
55
"repository": {
66
"type": "git",
77
"url": "https://onedrive.visualstudio.com/DefaultCollection/SPPPlat/_git/sp-next"

libraries/stream-collator/src/ConsoleModerator.ts

Lines changed: 0 additions & 144 deletions
This file was deleted.

libraries/stream-collator/src/DualTaskStream.ts

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
/**
2+
* @file DualTaskStream.ts
3+
* @Copyright (c) Microsoft Corporation. All rights reserved.
4+
*
5+
* This is a special type of stream class which has two substreams (stderr and stdout), which you can write to.
6+
*/
7+
/* istanbul ignore next */
8+
19
import * as colors from 'colors';
210
import * as stream from 'stream';
311

@@ -54,7 +62,7 @@ export default class DualTaskStream extends stream.Readable implements NodeJS.Re
5462
}
5563

5664
/**
57-
* Closes both substreams and marks closes the readable stream
65+
* Closes both substreams and closes the readable stream
5866
*/
5967
public end(): void {
6068
if (!this._stdoutClosed) {

libraries/stream-collator/src/PersistentStream.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
/**
2+
* @file PersistentStream.ts
3+
* @Copyright (c) Microsoft Corporation. All rights reserved.
4+
*
5+
* A special type of strean which keeps track of everything written to it, which can be read with the readAll() function
6+
*/
7+
/* istanbul ignore next */
8+
19
import * as stream from 'stream';
210

311
/**

0 commit comments

Comments
 (0)