Skip to content

NSX: (temp fix) Skip adding firewall rules for CKS Clusters on VPC tiers #56

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
weizhouapache merged 2 commits into from
Jan 19, 2024
Merged

NSX: (temp fix) Skip adding firewall rules for CKS Clusters on VPC tiers #56

weizhouapache merged 2 commits into from
Jan 19, 2024

Conversation

@Pearl1594
Copy link
Contributor

@Pearl1594 Pearl1594 commented Jan 18, 2024

Currently CKP does not setup NetworkACLs for CKS clusters on VPC tiers, and fails to add Firewall rules - as Firewall isn't supported on VPCs. This is a partial fix, to skip setting up Firewall rules if the network doesn't support the service.

@weizhouapache
Copy link
Member

weizhouapache commented Jan 19, 2024

@Pearl1594
I tested with CKS cluster deployment and nginx deployment/service.

without this change

E0119 13:23:09.742798       1 controller.go:244] error processing service default/nginx-service (will retry): failed to ensure load balancer: error creating new firewall rule for public IP cd17cbc3-538a-439b-b7a0-4f493d8443f5, proto tcp, port 80, allowed [0.0.0.0/0]: CloudStack API error 431 (CSExceptionErrorCode: 9999): There is no new provider for IP 10.0.53.175 of service Firewall!

with this change, it crashed at line 166

I0119 14:05:45.321563       1 event.go:294] "Event occurred" object="default/nginx-service" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
E0119 14:05:52.775883       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 299 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1d8c880?, 0x33f2140})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0007756e0?})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/runtime/runtime.go:49 +0x75
panic({0x1d8c880, 0x33f2140})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/apache/cloudstack-go/v2/cloudstack.(*NetworkService).GetNetworkByID(0xc00041dce0, {0xc0005cdb30, 0x24}, {0xc000012fc0, 0x1, 0x50?})
	/go/pkg/mod/github.com/apache/cloudstack-go/v2@v2.15.0/cloudstack/NetworkService.go:3557 +0x1b6
github.com/apache/cloudstack-kubernetes-provider.(*CSCloud).EnsureLoadBalancer(0xc0004a7980, {0xc00068f800?, 0x21fca68?}, {0x20f7fd6, 0xa}, 0xc0007b6000, {0xc000012db8, 0x1, 0x1})
	/go/src/github.com/apache/cloudstack-kubernetes-provider/cloudstack_loadbalancer.go:166 +0x1543
k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0x20f1674?)
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:459 +0x117
k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0xc0007b6000, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:414 +0x6cc
k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0xc0007b6000, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:346 +0x146
k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc00044d260, {0x23f4d70, 0xc00065e240}, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:887 +0x257
k8s.io/cloud-provider/controllers/service.(*Controller).processNextWorkItem(0xc00044d260, {0x23f4d70, 0xc00065e240})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:304 +0x127
k8s.io/cloud-provider/controllers/service.(*Controller).worker(0xc0008e0600?, {0x23f4d70, 0xc00065e240})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:283 +0x39
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:190 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x603c86b68e?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x23d9960, 0xc0008e0600}, 0x1, 0xc00066a300)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2fea4e8bf0db330b?, 0x3b9aca00, 0x0, 0xe2?, 0x714083b09f70d83c?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x23f4d70, 0xc00065e240}, 0xc00068e7c0, 0xac5c70001ad37c31?, 0x255bd85d5287fd9c?, 0xe2?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:190 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x23f4d70?, 0xc00065e240?}, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:101 +0x2b
created by k8s.io/cloud-provider/controllers/service.(*Controller).Run
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:241 +0x246
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x193b676]

goroutine 299 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0007756e0?})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x1d8c880, 0x33f2140})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/apache/cloudstack-go/v2/cloudstack.(*NetworkService).GetNetworkByID(0xc00041dce0, {0xc0005cdb30, 0x24}, {0xc000012fc0, 0x1, 0x50?})
	/go/pkg/mod/github.com/apache/cloudstack-go/v2@v2.15.0/cloudstack/NetworkService.go:3557 +0x1b6
github.com/apache/cloudstack-kubernetes-provider.(*CSCloud).EnsureLoadBalancer(0xc0004a7980, {0xc00068f800?, 0x21fca68?}, {0x20f7fd6, 0xa}, 0xc0007b6000, {0xc000012db8, 0x1, 0x1})
	/go/src/github.com/apache/cloudstack-kubernetes-provider/cloudstack_loadbalancer.go:166 +0x1543
k8s.io/cloud-provider/controllers/service.(*Controller).ensureLoadBalancer(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0x20f1674?)
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:459 +0x117
k8s.io/cloud-provider/controllers/service.(*Controller).syncLoadBalancerIfNeeded(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0xc0007b6000, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:414 +0x6cc
k8s.io/cloud-provider/controllers/service.(*Controller).processServiceCreateOrUpdate(0xc00044d260, {0x23f4d70, 0xc00065e240}, 0xc0007b6000, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:346 +0x146
k8s.io/cloud-provider/controllers/service.(*Controller).syncService(0xc00044d260, {0x23f4d70, 0xc00065e240}, {0xc00005cd08, 0x15})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:887 +0x257
k8s.io/cloud-provider/controllers/service.(*Controller).processNextWorkItem(0xc00044d260, {0x23f4d70, 0xc00065e240})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:304 +0x127
k8s.io/cloud-provider/controllers/service.(*Controller).worker(0xc0008e0600?, {0x23f4d70, 0xc00065e240})
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:283 +0x39
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:190 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x603c86b68e?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x23d9960, 0xc0008e0600}, 0x1, 0xc00066a300)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x2fea4e8bf0db330b?, 0x3b9aca00, 0x0, 0xe2?, 0x714083b09f70d83c?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext({0x23f4d70, 0xc00065e240}, 0xc00068e7c0, 0xac5c70001ad37c31?, 0x255bd85d5287fd9c?, 0xe2?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:190 +0x99
k8s.io/apimachinery/pkg/util/wait.UntilWithContext({0x23f4d70?, 0xc00065e240?}, 0x0?, 0x0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.12/pkg/util/wait/wait.go:101 +0x2b
created by k8s.io/cloud-provider/controllers/service.(*Controller).Run
	/go/pkg/mod/k8s.io/cloud-provider@v0.24.12/controllers/service/controller.go:241 +0x246

@weizhouapache
Copy link
Member

weizhouapache commented Jan 19, 2024

@Pearl1594
tested ok

root@cks-vpc-control-18d21c9d231:~# kubectl get svc
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP      10.96.0.1        <none>        443/TCP        4h26m
nginx-service   LoadBalancer   10.107.131.221   10.0.53.175   80:31387/TCP   2m4s

image

@weizhouapache weizhouapache merged commit 98bd3c5 into main Jan 19, 2024
@Pearl1594
Copy link
Contributor Author

Pearl1594 commented Jan 19, 2024

Thank you so much @weizhouapache for testing!!!

@DaanHoogland DaanHoogland deleted the nsx-skip-firewall branch January 19, 2024 19:11
@weizhouapache weizhouapache mentioned this pull request Jan 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rohityadavcloud rohityadavcloud rohityadavcloud approved these changes

@weizhouapache weizhouapache weizhouapache approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Pearl1594 @weizhouapache @rohityadavcloud