Skip to content

Commit 2fd39e8

Browse files
rohan-paulrohan-paul
committed
busboy and stream note
1 parent f894c50 commit 2fd39e8

4 files changed

Lines changed: 114 additions & 2 deletions

File tree

‎Node-Express/Streams-Node.jpeg‎

58.5 KB
Loading

‎Node-Express/Streams.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
## What are Streams?
22

3+
4+
Streams are collections of data — just like arrays or strings. The difference is that streams might not be available all at once, and they don’t have to fit in memory. This makes streams really powerful when working with large amounts of data, or data that’s coming from an external source one chunk at a time.
5+
6+
However, streams are not only about working with big data. They also give us the power of composability in our code. Just like we can compose powerful linux commands by piping other smaller Linux commands, we can do exactly the same in Node with streams.
7+
38
Ans. Typically, Stream is a mechanism for transferring data between two points. Node.js provides you streams
49
to read data from the source or to write data to the destination. In Node.js, Streams can be readable, writable, or
510
both and all streams are instances of EventEmitter class.
@@ -33,7 +38,17 @@ Node.js supports four types of streams as given below:
3338
### Transform - A type of duplex stream where the output is computed based on input. Both operations are linked via some transform.
3439

3540

41+
### Many of the built-in modules in Node implement the streaming interface:
42+
43+
<img src="Streams-Node.jpeg">
44+
45+
The list above has some examples for native Node.js objects that are also readable and writable streams. Some of these objects are both readable and writable streams, like TCP sockets, zlib and crypto streams.
46+
47+
Notice that the objects are also closely related. While an HTTP response is a readable stream on the client, it’s a writable stream on the server. This is because in the HTTP case, we basically read from one object (http.IncomingMessage) and write to the other (http.ServerResponse).
48+
3649

3750
### Further Reading
3851

39-
1> [https://www.sitepoint.com/basics-node-js-streams/](https://www.sitepoint.com/basics-node-js-streams/)
52+
1> [https://www.sitepoint.com/basics-node-js-streams/](https://www.sitepoint.com/basics-node-js-streams/)
53+
54+
2> [https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93](https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93)

‎Node-Express/busboy-why-its-needed.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
### Why [bus-boy](https://github.com/mscdex/busboy) is needed
2+
3+
4+
#### A node.js module for parsing incoming HTML form data. Its used to upload files. Busboy is a Writable stream and its an alternative for multer. A writable stream is an abstraction for a destination to which data can be written. An example of that is the ``fs.createWriteStream`` method.
5+
6+
On busboy 'file' event you get parameter named 'file' and this is a stream so you can pipe it.
7+
8+
9+
```js
10+
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) => {
11+
file.pipe(streamToSQS)
12+
})
13+
```
14+
[An example of file upload with busboy and express](https://gist.github.com/shobhitg/5b367f01b6daf46a0287)
15+
16+
```js
17+
// accept POST request on the homepage
18+
app.post('/', function (req, res) {
19+
var busboy = new Busboy({ headers: req.headers });
20+
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
21+
var saveTo = path.join('.', filename);
22+
console.log('Uploading: ' + saveTo);
23+
file.pipe(fs.createWriteStream(saveTo));
24+
});
25+
busboy.on('finish', function() {
26+
console.log('Upload complete');
27+
res.writeHead(200, { 'Connection': 'close' });
28+
res.end("That's all folks!");
29+
});
30+
return req.pipe(busboy);
31+
32+
});
33+
```
34+
35+
The function ``fs.createWriteStream()`` creates a writable stream in a very simple manner. After a call to fs.createWriteStream with the filepath, you have a writeable stream to work with.
36+
37+
## Comparison with multer
38+
39+
40+
Some developers opine that Multer is easier because it abstracts away some of the details of Busboy.
41+
42+
### Difference between busboy and connect-busboy
43+
44+
[https://stackoverflow.com/questions/39439922/difference-between-busboy-and-connect-busboy](https://stackoverflow.com/questions/39439922/difference-between-busboy-and-connect-busboy)
45+
Connect is a middleware layer for building servers in Node.js. It was originally the basis for the Express web framework.
46+
47+
What middleware here really means is essentially an array of functions that conform to an interface which get called on each request in the order they are defined.
48+
49+
connect-busboy wraps the busboy library into a connect compatible middleware. You can see in the source it really just returns a function.
50+
51+
If you're using express you might want to take a look at express-busboy which uses connect-busboy under the hood and has recent updates.
52+
53+
### What exactly are streams?*
54+
55+
Streams are collections of data — just like arrays or strings. The difference is that streams might not be available all at once, and they don’t have to fit in memory. This makes streams really powerful when working with large amounts of data, or data that’s coming from an external source one chunk at a time.
56+
57+
However, streams are not only about working with big data. They also give us the power of composability in our code. Just like we can compose powerful linux commands by piping other smaller Linux commands, we can do exactly the same in Node with streams.
58+
59+
#### Here’s the magic line about pipe() and stream that you need to remember:
60+
61+
``readableSrc.pipe(writableDest)``
62+
63+
In this simple line, we’re piping the output of a readable stream — the source of data, as the input of a writable stream — the destination.

‎Node-Express/pipe-in-node.md‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,11 @@ The pipe() function reads data from a readable stream as it becomes available, a
44

55
### What are Streams
66

7+
8+
Streams are collections of data — just like arrays or strings. The difference is that streams might not be available all at once, and they don’t have to fit in memory. This makes streams really powerful when working with large amounts of data, or data that’s coming from an external source one chunk at a time.
9+
10+
However, streams are not only about working with big data. They also give us the power of composability in our code. Just like we can compose powerful linux commands by piping other smaller Linux commands, we can do exactly the same in Node with streams.
11+
712
Streams are unix pipes that let you easily read data from a source and pipe it to a destination. Simply put, a stream is nothing but an EventEmitter and implements some specials methods. Depending on the methods implemented, a stream becomes Readable, Writable, or Duplex (both readable and writable). Readable streams let you read data from a source while writable streams let you write data to a destination.
813

914
#### An implementation, for a functionality of uploading image to mongodb, where I am using grid-stream packages to read and write directly to mongodb and then making it available to the write-stream
@@ -32,8 +37,37 @@ readableStream.pipe(writableStream);
3237
```
3338
The above snippet makes use of the pipe() function to write the content of file1 to file2. As pipe() manages the data flow for you, you should not worry about slow or fast data flow. This makes pipe() a neat tool to read and write data. You should also note that pipe() returns the destination stream. So, you can easily utilize this to chain multiple streams together. Let’s see how!
3439

40+
#### Here’s the magic line that you need to remember:
41+
42+
``readableSrc.pipe(writableDest)``
3543

44+
In this simple line, we’re piping the output of a readable stream — the source of data, as the input of a writable stream — the destination. The source has to be a readable stream and the destination has to be a writable one. Of course, they can both be duplex/transform streams as well. In fact, if we’re piping into a duplex stream, we can chain pipe calls just like we do in Linux:
45+
46+
```js
47+
readableSrc
48+
.pipe(transformStream1)
49+
.pipe(transformStream2)
50+
.pipe(finalWrtitableDest)
51+
52+
```
53+
The pipe method returns the destination stream, which enabled us to do the chaining above. For streams a (readable), b and c (duplex), and d (writable), we can:
54+
55+
```js
56+
a.pipe(b).pipe(c).pipe(d)
57+
# Which is equivalent to:
58+
a.pipe(b)
59+
b.pipe(c)
60+
c.pipe(d)
61+
```
62+
63+
### Which, in Linux, is equivalent to:
64+
65+
``$ a | b | c | d``
66+
67+
The pipe method is the easiest way to consume streams. It’s generally recommended to either use the pipe method or consume streams with events, but avoid mixing these two. Usually when you’re using the pipe method you don’t need to use events, but if you need to consume the streams in more custom ways, events would be the way to go.
3668

3769
### Good Reference
3870

39-
https://www.sitepoint.com/basics-node-js-streams/
71+
1> https://www.sitepoint.com/basics-node-js-streams/
72+
73+
2> https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

0 commit comments

Comments
 (0)