Skip to content

Conversation

@LewisJEllis
Copy link
Member

@LewisJEllis LewisJEllis commented Apr 4, 2018

I have moved everything over to ioredis, torn out all the callback stuff, and gotten all the tests back to passing! It took a while, but it's pretty much the main hurdle/technical risk for making the 2.0 plan work. That's enough to warrant a WIP branch. I will ping people here when it's less WIP and more ready for review.

Some notes/thoughts:

  • Can't wait to refactor a bunch of promise stuff to async/await now that tests are back to green
  • I really really like ava, and wow the tests run so fast! 3-4 seconds on my laptop.
  • ioredis
    • Their auto-reconnection stuff is great - actually simplified some tests and makes me feel good about trying to test and trust connection re-establishment more
      • keeping an eye on ioredis issue 610 to cover bases/maybe repro potential issues
    • Still lots of little todos to utilize some of their nice features, cluster capabilities, etc
      • also a cool defineCommand capability for lua scripts - could obviate a lot of the load-script-cache stuff we've implemented
    • filed issue 614 on ioredis to resolve a confusion on status/ready detection/timing
  • tearing out callbacks
    • Deleted so many instances of if (cb) helpers.asCallback(promise, cb);
    • new rule that your .process handler should just be an async func is great
  • Moved promise-callbacks down to a devDependency
    • using several of them in tests so kept whole thing as devDependency
    • implementation code is basically only using waitOn and withTimeout though, so plucked those two into helpers file

Other todos:

  • maybe add debug logs, ioredis's debug logs were v helpful at times
  • figure out where a percent of test coverage went and get it back
  • investigate viability of sharing event subscriber connections, worker connections
  • bunch of readme/example updates to expunge callbacks
  • redo benchmarks
  • probably include some other features in 2.0 for completeness, like pluggable job data serializers and stallLimit

Refs #104

@coveralls
Copy link

coveralls commented Apr 4, 2018

Coverage Status

Coverage decreased (-1.4%) to 98.603% when pulling 4923705 on 2.0dev into c92207e on master.

@skeggse
Copy link
Member

skeggse commented Apr 4, 2018

🔥

@bradvogel
Copy link

🙏 🎉

@dylanjha
Copy link
Contributor

@LewisJEllis thank you! I didn't know there was a 2.0 in the works!

I'm very happy to help with some of the development here. So far bee-queue has been working really well for us and I want to see this project continue to develop.

@eugene1g
Copy link

Would you consider a PR for transferring the current 2.0 codebase to TypeScript?

@sarendipitee
Copy link

I'm sure you've been busy on other stuff, @LewisJEllis, but if you've got a sec could you update what's done, not done, and what others could help on?

@syuilo
Copy link

syuilo commented Mar 10, 2019

any updates?

@hughsw
Copy link
Collaborator

hughsw commented Apr 3, 2020

What's the status of this PR and @LewisJEllis involvement with bee-queue?

@jasrusable
Copy link

Hey all :) I am the author of ConveyorMQ which I have built with the intention of it being a fast, robust and reliable distributed task queue for Node using Redis. My intention is to make ConveyorMQ the most performant, feature-full and defacto-standard job/task queue system for Node.

ConveyorMQ already implements all of bee-queue's features, as well as the following:

  • Simple, efficient design
  • Very fast (Benchmarks are faster than bee-queue)
  • High test coverage
  • Uses ioredis
  • Queue pause/resume
  • Written in Typescript with async/await & promise API
  • Supports large job/task data

Additionally, my road map currently includes:

  • Task prioritization
  • Queue rate limiting
  • A Dashboard/WebUI

I would be grateful for any feedback you might have to offer :)

@bradvogel
Copy link

@jasrusable awesome project! Just curious - have you considered using Redis Streams (introduced in Redis 5.0)? If we were to build bee-queue again from the ground up, we likely would have used Streams since they simplify so much (see this comment).

@compwright
Copy link
Collaborator

This PR has many conflicts and appears abandoned, closing.

@compwright compwright closed this Nov 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.