Comments on Just a little Python: Distributed Systems with ZeroMQ

The only things you need are the Zeromq libraries ...

2013-04-18T08:31:57.252-04:00

The only things you need are the Zeromq libraries and the pyzmq package.

actually anyone can help what are tools that need ...

2013-04-18T06:48:53.623-04:00

actually anyone can help what are tools that need to instal when we want build a zeromq program in python?

I'm not completely sure I understand the quest...

2013-01-09T17:05:10.436-05:00

I'm not completely sure I understand the question, but I'll try to answer anyway ;-)

First off, you should *not* expose a ZeroMQ socket over the internet, so if this was in the plans for your architecture, ZMQ is a poor choice.

Secondly, ZeroMQ has its own protocol on top of TCP, so it can't be run over HTTP. Again, if that's what the architecture calls for, ZMQ is a non-starter.

ZeroMQ is a good choice if (and only if) you are communicating between nodes over a trusted network and want "a better socket." ZeroMQ is not a replacement for a message broker like AMQP (though it could be used to implement such a broker).

I'm not sure if I answered your question, but I hope that helped anyway.... Thanks for the comment!

How could zeromq help in the following scenario: ...

2013-01-09T11:31:16.697-05:00

How could zeromq help in the following scenario: Requests (from clients) and responses (from server) are serviced with a json-based api ie. webserver <-> server application <-> api <-> json <-> http ... <-> http <-> json <-> api <-> client. The client can be any device, OS, browser. Developers could use any programming language/tool to build the client-side application. Thx.

Thanks for the comment! I'd say that pickle, j...

2012-09-27T11:52:34.550-04:00

Thanks for the comment! I'd say that pickle, json, and xml are all fine from a *functional* perspective, but they'll be fairly low-performance. My personal preference is using bson (http://bsonspec.org), but protocol buffers (https://developers.google.com/protocol-buffers/docs/pythontutorial) also seem to be a good option.

Hi, I'm a newbie in zmq that seems to be the p...

2012-09-27T06:01:08.528-04:00

Hi, I'm a newbie in zmq that seems to be the perfect solution for distributing tasks to workers through a PUSH/PULL pipeline, thanks to your article. What's the best practice for (un)marshalling complex and big objects (+100K) that transit through zmq sockets? pickle ? json ? xml ?
Thanks again for these excellent tutorials.

Great! Glad to hear that ZeroMQ's working out ...

2012-08-27T11:56:35.792-04:00

Great! Glad to hear that ZeroMQ's working out for you. Is the reporting system open-source and available anywhere?

Thanks for the comment! I'll definitely check ...

2012-08-27T11:55:41.786-04:00

Thanks for the comment! I'll definitely check out PyPln!

Leveraging the weather broadcast example given in ...

2012-08-26T07:44:45.666-04:00

Leveraging the weather broadcast example given in the 0mq learning guide, I'm playing with system status reporting (cpu, mem, drive space, etc) using a pub connect pattern to on the reporting systems to a centralized sub/bind socket. Seems to be working great!

Hello Rick, I'm using pyzmq on a project to di...

2012-08-25T20:59:37.996-04:00

Hello Rick,
I'm using pyzmq on a project to distribute jobs along a cluster -- probably in the future the code responsible for manage the cluster and the jobs will pop-up from this software and I'll create a library based on this code, so anyone can benefit of this.
The whole project's focus is to do [scalable] natural language processing and you can fetch it at: https://github.com/namd/pypln

[]s

Really the only flow control problems we had were ...

2012-08-15T19:05:04.655-04:00

Really the only flow control problems we had were in figuring out the correct settings to use so we didn't buffer unlimited data on a PUSH socket when all the PULLers were busy.

Thanks for all the info, it sounds like you really pushed the ZMQ system to its limits (which I'd expect, since you're building a high performance distributed task framework).

In a nutshell, we had some issues with the socket ...

2012-08-15T18:39:11.747-04:00

In a nutshell, we had some issues with the socket shutdown function. After a shutdown, it needs some time before we could create a new connection, which slows our unit tests with delays. I agree, it's not a big deal.

Second minor I can think of is the random port selector (binding to port *) which is really not random... It simply tries again from the beginning port and increments. We first had a topology that needed multiple connections for each node, which means looping over the same ports for each bind. The reason stated in their forum/mailing list were that they haven't thought of propagating information back from the OS socket, thus they can't let the kernel call efficiently choose a port for the library.
I agree again, not such a big deal since socket binding is not frequently called, but still.

We tried to be as by-the-book as possible for our patterns so we haven't undergone any flow control issues, I guess the bandwidth tweaks, notably on the Nagle algorithm, may have had some drawbacks for you?

SCOOP also looks pretty cool. I'm interested i...

2012-08-15T17:28:48.947-04:00

SCOOP also looks pretty cool. I'm interested in hearing what the minor negative points you encountered with ZeroMQ. (A few I ran across were that it behaves badly with bad input and flow control is a little tricky to configure.) Thanks for the info!

We are currently using ZeroMQ in our distributed t...

2012-08-15T15:37:28.174-04:00

We are currently using ZeroMQ in our distributed task framework in Python, SCOOP (http://scoop.googlecode.com).

ZeroMQ was chosen as the communication library because it simply works and isn't bloated. No need to implement the state-machines for common patterns in our sockets, ZMQ does it and fast. It doesn't replace a standard socket, though, it only add a layer of functionalities over it.

We found some minor negative points while using it, but overall, ZMQ is a tool that saved us much developing time and should definitively be looked by distribution systems.

ZeroRPC looks pretty cool; definitely something I&...

2012-08-15T11:39:18.143-04:00

ZeroRPC looks pretty cool; definitely something I'll consider for future projects. Thanks, Ken!

You should check out ZeroRPC http://zerorpc.dotclo...

2012-08-15T10:42:10.595-04:00

You should check out ZeroRPC http://zerorpc.dotcloud.com it handles a lot of the boilerplate code for you, when working with python and ZeroMQ when building RPC based services.