* wip: apply dirty NetPols every 500ms in Linux
* only build npm linux image
* fix: check for empty cache
* feat: toggle for netpol interval. default 500 ms
* ci: remove stages "build binaries" and "run windows tests"
* wip: max batched netpols (toggle-specified)
* ci: remove manifest build/push for win npm
* wip: handle ipset deletion properly and max batch for delete too
* fix: correct remove policy
* fix: only remove policy if it was in kernel
* finalize toggles, allowing ability to turn off iptablesInBackground
* ci: conf + cyc use PR's configmaps
* fix: lints
* fix dp toggle: iptablesInBackground
* fix lock typo and config logging
* fix background thread. add comments. only add tmp ref when enabled
* copy pod selector list
* fix: removepolicy needs namespace too
* rename opInfo to event
* fix: fix references and prevent concurrent map read/write
* tmp: debug logging
* fix: missing set references by swap keys and values
* Revert "tmp: debug logging"
This reverts commit 70ed34c714ea4a6d009a1fe90a7168be4bedd5bf.
* fix: add podSelectorList to fake NetPol
* log: do not print error when failing to delete non-existent nft rule
* log: verbose iptables bootup
* log: use fmt.Errorf for clean logging
* log: never return error for iptables in background and fix some lints
* fix: activate/deactivate azure chain rules
* fix: correctly decrement netpols in kernel
* ci: run UTs again
* ci: update profiles. default to placefirst=false
* address comment: rename batch to pendingPolicy
* refactor: make dirty cache OS-specific
* test: UTs
* test: put UT cfg back to placefirst to not break things
* ci: update cyclonus workflows
* fmt: address comment & lint
* fmt: rename numInKernel to policiesInKernel
* log: switch to fmt.Errorf
* fmt: whitespace
* feat: resiliency to errors while reconciling dirty netpols
* log: temporarily print everything for ipset restore
* fix: remove nomatch from ipset -D for cidr blocks
* test: UTs for non-happy path
* test: fix hns fake
* fix: don't change windows. let it delete ipsets when removing policies
* fix windows lint
* fix: ignore chain doesn't exist errors for iptables -D
* feat: latency and failure metrics
* test: update exit code for UT
* metrics: new metrics should go in node-metrics path
* style: simplify nesting
* style: move identical windows & linux code to shared file
* ci: remove v1 conformance and cyclonus
* feat: add NetPols in background from the DP (revert background code in pMgr)
* style: remove "background" from iptables metrics
* revert changes in ipsetmanager, const.go, and dp.Remove/UpdatePolicy
* style: whitespace
* perf: use len() instead of creating slice from map
* remove verbosity for iptables bootup
* build: add return statement
* style: whitespace
* build: fix variable shadowing
* build: fix more import shadowing
* build: windows pointer issue and UT issue
* test: fix UT for iptables error code 2
* ci: enable linux scale test
* ci: revert to master pipeline.yaml
* revert changes to chain-management. do changes in PR #2012
* log: change wording
* test: UTs for netpol in background
* log: wording
* feat: apply ipsets for each netpol individually
* config: rearrange ConfigMap & update capz yaml
* fix: windows bootup phase logic for addpolicy
* feat: restrict netpol in background to linux + nftables
* test: skip nftables check for UT
* style: netpols[0] instead of loop
* log: address log comments
* style: lint for long line
---------
Co-authored-by: Vamsi Kalapala <vakr@microsoft.com>
* cherry-picking stuff from apply in background POC
* add all policies poc
* add debug prints
* fix deadlock
* fix other GetPolicy deadlock
* update whitespace in yamls
* properly merge
* properly merge 2
* add ACLs in batches
* cleanup errors
* lint and log
* persist state as we add
* refactor into function so we can do UTs on batching
* fix lint
* batch struct
* successful policies
* reduce batch limit to 30
* wip
* fix UT by applying dataplane immediately for RemovePolicy()
* configmap options for apply in background
* fix deadlocks
* better logging
* rename config variables, update default config, change shouldApply check
* update configmap values
* FIXME: remove tmp commit overriding applyDP config (using for pipeline tests)
* optimize applying ipsets for add policy
* cleanup code and finalize apply ipsets for netpols
* flip order of if statement
* UTs. address comments. fix netpol behavior by waiting to start pod controller
* all UTs except ones related to issue #1729
* remove bootup phase stuff
* fix lints and move applyinbackground to toggle
* fix lint
* don't check isWindows every time
* use diff var for applyinbackground
* fix lint
* get node IP
* add allow-host-to-endpoint ACL
* update ACL ID to be equal to other ACLs in the netpol
* add node ip to acl
* UTs and make node IP a part of pMgr cfg
* fix skip test logic from #1857
* fix pMgr UTs and prom metrics
* fix lints and add comments
* fix UT and prom metrics for linux
* UT for getting node IP
* revert skipTest change
* error out if node IP is an empty string
* update logging for node ip and only get node ip for windows
---------
Co-authored-by: Vamsi Kalapala <vakr@microsoft.com>
* set kubeconfig on capz
* update dockerfile
* test network name Calico
* add base acls
* add WindowsNetworkName toggle and revert hard coded Calico parts
* update base acls for calico and add UTs
* capitalize calico network name
* fix connectivity. try with host allow acls
* revert change to policy_windows.go
* more UTs and add base ACLs for other "new endpoint" scenario
* run all UTs
* update npm image to .42
* add log line
* allow traffic going inter-node
* Revert "allow traffic going inter-node"
This reverts commit e1014822d5.
* add long-runner pod for testing vfp tags in capz
* fix lints
* fix: [NPM] Remove error on not finding server version
* removing the isnewNetpol flag
* fix UTs
* removing dependency on windows builds
* putting pipeline win dependency back
Co-authored-by: Hunter Gregory <hunterlgregory@gmail.com>
Co-authored-by: Hunter Gregory <42728408+huntergregory@users.noreply.github.com>
* add placeFirst to pMgr cfg
* add placeFirst=false functionality to v2
* use constant instead of bool values for placeFirst
* fix bug and update cyclonus profiles
* set placefirst config based on configmap
* make code cleaner
* position azure jump first or directly after kube jump
* fix npm encoder ut
* intial touches to create sets
* adding initial touches to dpshim
* deprecating initialize DP func
* feat: [NPM] Adding DPShim layer in controller pods
* correcting an import error
* Adding some UTs
* adding a UT
* Addressing some comments
* Moving an UT to linux specifc file
* Fixing some issues with controller pod
* Adding some dns poilices and logs for debugging
* Moving aroudn outchannel to help with hydration of new clients
* removeing pass by ref
* Adding http server in daemon for pprof
* Adding a new grpc option to wait
* Fixing 100% cpu in daemon
* Fixing some logic in list management
* Applying some golints
* adding mutex
* Addressing comments and solving a bug. Cyclonus seems to be good now
* Fixing a bug
* Addressing a comment
* fixing an issue and addressing comments
* call policy reconcile in dataplane
* lock to staleChains
* allow interruption of deleting stale chains while reconciling
* fix lint
* switch reconcile period back from seconds to minutes
* address comments
* address comments
* remove RunPeriodicTasks from GenericDP interface
* fix build error
* Revert "fix build error"
This reverts commit de3d6e20c5.
* make RunPeriodicTasks an interface method again
* finished logic. need to update some UTs
* address comments for chain-management_linux.go
* make an exported Dataplane config
* fix go lint and update npm start dataplane to use the config
* deactivate and activate pMgr instead of rebooting. TODO: UTs
* wip for revising reset/init
* update dataplane to bootup instead of resetting and then initializing
* fix lint
* update windows file
* update print statement
* address comments
* Update codes to enable V2 NPM
* Deleted dead codes (if we want to keep it, please let me know)
* Update azure-npm.yaml to add toggle parameters
* Fix incorrect call for v2 NPM
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* [NPM] Windows Policy Manager changes for OS22
* Adding new NPM ACLSettings with ID
* first pass on both add and remove policies
* fixing a merge issue
* Working 1st level Setpolicy CRUD operations
* have NPMACl to HNSACL conversion logic ready
* updating policy endpoints only after adding policy to an endpoint
* updating policy endpoints only after adding policy to an endpoint
* fixing a build issue
* fixing issue in linux files
* Addressing some comments and also completing some integrations with V2 control plane
* Updating policy ID logic and update pod
* Updating policy ID logic and update pod
* Addressing some comments
* adding basic reset bits
* fixnig build issue in linux
* Fixing the _linux_test.go build failures
* fix lints
* Addressing some comments and correcting windows logic to apply set policies in order
* cleaning up logic for calculating set policies
* Applying some feedback.
* fixing a failing test and panic
* Add uts for parseiptable.go
Co-authored-by: Hunter Gregory <hgregory@microsoft.com>
* test commit
* deleted file from test commit
* added a UT for convertiptable and moved shared UT functionality to a new file. also renamed some command constants to avoid confusion with real commands
* removing print statements from when I was debugging
* Add UTs for start.go
* Add simple UT for start.go
* make it clear that cache file and iptables save file need to be used together
* remove unnecessary wantEmptyOutput field in test struct
* Refactor cobra command and adjust unit tests
* UT for gettuples cmd
* comment out test without cache file and refactor args
* Delete unnecessary comments and commeted codes
* Remove lint errors
* Use correct files and expected values in UTs
Co-authored-by: Hunter Gregory <hgregory@microsoft.com>
Co-authored-by: Hunter Gregory <hunterlgregory@gmail.com>
* made prometheus exec time metrics for ipsets and iptables in line with those for network policies (exec time recorded even for failures). Also made prometheus timer variable names clearer.
* fixed faulty prometheus handler test looking for a node metric name when testing the cluster metric handler
* add clarity in comments related to the IPSetInventory metric
* Include prometheus metrics for lists and in DestroyNPMIpsets(). Only make metric updates when there's no error
* refactor prometheus testing and include metric tests for lists and NPMDestroyIpsets()
* better check for empty response to ipset list in DestroyNpmIpsets()
* remove unused clientset from controllers
* replace function for setting ipset inventory with function for removing ipset for better readability. updating comments too
* reset ipset inventory before each unit test
* added unit test for adding to set with pod cache
* remove unused cluster state function and clientset from np manager
* fix build problems: remove clientset from calls to npm.NewNetworkPolicyManager()
* fix logic for destroy ipsets for situation when destroy is called while num ipsets is 0
* delete commented out function
* encapsulated prometheus metrics, refactored prometheus testing for iptm and netpol controller, and removed clientset from controller creation in test files (fixing build error)
* update test for DestroyNpmIpsets() to always use a new Exec
* CLI functions
* fix whitespace bug in CIDRmatch + go lint issue
* update main.go from master
* addressed CR comments
* addressed Matt's comments
* make config flag to be a root cmd flag only
* make config flag to be a root cmd flag only
Hunter Gregory
Vamsi Kalapala
Mathew Merrick
Nitish Malhotra
JungukCho
Eng Zer Jun
Quang Nguyen