Skip to content

Added an HeuristicPolicy with Control Lyapunov Controller#74

Merged
matteobettini merged 12 commits intoproroklab:mainfrom
gy2256:main
Jan 2, 2024
Merged

Added an HeuristicPolicy with Control Lyapunov Controller#74
matteobettini merged 12 commits intoproroklab:mainfrom
gy2256:main

Conversation

@gy2256
Copy link
Contributor

@gy2256 gy2256 commented Dec 28, 2023

QP controller with CLF.

Copy link
Member

@matteobettini matteobettini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for this!

I left some initial comments.

My main question is. Is there a roadmap you see that we can take to make this type of controllers generally available to any vmas task and not specific to navigation?

gy2256 and others added 5 commits December 28, 2023 10:50
Co-authored-by: Matteo Bettini <55539777+matteobettini@users.noreply.github.com>
Co-authored-by: Matteo Bettini <55539777+matteobettini@users.noreply.github.com>
Co-authored-by: Matteo Bettini <55539777+matteobettini@users.noreply.github.com>
Co-authored-by: Matteo Bettini <55539777+matteobettini@users.noreply.github.com>
Co-authored-by: Matteo Bettini <55539777+matteobettini@users.noreply.github.com>
@gy2256
Copy link
Contributor Author

gy2256 commented Dec 28, 2023

Yes, this type of controller has two purposes once Control Barrier Functions are included: (1) It can steer agents to desired goals. (2) It can be used as a safety shielding (For example, to avoid collisions or never exceed certain relative velocities with respect to other agents). With (2), the controller can take an action from another policy and use it as a reference. The safety shielding will only be activated if the safety constraints are violated. This feature can be used for safe RL learning.



class HeuristicPolicy(BaseHeuristicPolicy):
def __init__(self, ctrl_params=[0.2, 1.0, 1.0, 100.0], *args, **kwargs):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here could we have clf_epsilon=0.2, cbf_eta=1.0 etc. ?

Copy link
Contributor Author

@gy2256 gy2256 Jan 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clf_epsilon defines the convergence rate of the Lyapunov function. Let me remove cbf parameters for now because it's not implemented yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just meant having each parameter separate instead of a list.
So that users can change just one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. One moment, let me modify it.

@matteobettini
Copy link
Member

matteobettini commented Jan 2, 2024

Yes, this type of controller has two purposes once Control Barrier Functions are included: (1) It can steer agents to desired goals. (2) It can be used as a safety shielding (For example, to avoid collisions or never exceed certain relative velocities with respect to other agents). With (2), the controller can take an action from another policy and use it as a reference. The safety shielding will only be activated if the safety constraints are violated. This feature can be used for safe RL learning.

Ok so here is what we can do. As I understand it, this heuristic provides an example of (1). We can merge this as a heuristic for navigation as it is specific to goal following.

Regarding (2), it would be nice to have a customizable component in https://github.com/proroklab/VectorizedMultiAgentSimulator/tree/main/vmas/simulator/controllers that takes some custom constraints and applies the CBF to the agent action. This compenents will then be called in scenario.process_action()

A challenge here is how to create such a component so that it is able to take general constraints.

Copy link
Member

@matteobettini matteobettini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, could you format with black 23.1.0

@matteobettini matteobettini merged commit 127a9ec into proroklab:main Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants