I fixed this by changing the JSON marshaller of the vtctl commands: For
protobuf messages we now use the "jsonpb" package which supports
emitting fields with zero values.
We already used this approach in the vtctld API endpoints. I've moved
the code to the "vtctl" package to make it more generic.
A drawback of this approach is that it doesn't cope well with our *Info
objects e.g. ShardInfo because the type switch recognizes them as
proto.Message and they have unexported fields which let jsonpb panic. To
work-around this, I've changed the vtctl code to marshal only the
embedded protobuf message and not the full *Info object anymore.
User can control through UI for running a canary task,
then running the remaining tasks. Create unit tests
and manually test the UI in e2e test environment.
It supports creating workflow to reshard multiple source shards.
It will show a phase-based tree-structure UI.
Unit Test is included for testing the happy path.
1. Improving tablet sort in dashboard.
When fetching tablet list, we sort by type as we add tablets.
Using sub-ordering for sort by type and sort by cell, to have a
deterministic order within groups.
2. Fixing initial table display in dashboard.
Only display the 'There are no tablets in this shard.' message when we
actually got a result with no tablets.
3. Fixing shard view columns width.
4. No more selection in shards view.
5. Different menus for master vs slaves in shard view.
6. Adding a refresh button to 3 dashboard views.
7. Using a single dialog for all workflows.
That way it's not linked to a single workflow component, but instead to
the window. Which as a side effect fixes the bug where opening the
dialog only greys out half of the window.
8. Moving keyspace / shard creation.
From the floating '+' button that doesn't float to a menu.
9. Using plus icon for new workflow.
Removing plus icon, as it doesn't float.
Only showing + in workflows if master.
The main goal of this change is to make not started and finished workflows
visible in the UI. Without this it was possible to create a workflow with
-skip_start option, and then it won't be visible in the UI thus not giving any
ability to actually start it. Also once workflow finishes it was disappearing
from the UI thus not giving any ability to see how it finished and whether it
existed at all or not. Besides this UI problem also workflows were never deleted
from the topo server and were accumulating indefinitely. This change makes
everything from topo server visible in the UI thus giving ability for the user
to delete already finished workflows.
This UI improvement allows the validator workflow to become less awkward and
actually finish once all fixers are done working. I'm modifying the validator
workflow to do that here as well.
- adding a flag to not display CRUD topo menus.
So that vtctld can be less dangerous in established production
environments.
- Make set RO / RW behind the replication flag.
Also add better messages, and add the ignore regexp flag.
- Tablet type changes in shard display.
Now they are sorted with master first, then replica, then rdonly, then
everything else.
Also display master in bold, replica/rdonly in italic, rest normal.
1. The schema page is now more resilient to missing data in responses.
2. The topology browser doesn't add extra '/' in topodata URLs, so we
don't do a useless roundtrip with a redirect.
3. Adding a button in UI to switch from old to new and new to old UI.
So we can enable / disable UI features.
Using it for top-level Status and Workflows links, as well as reparent
menu items.
Also re-organized the menus for shard / tablets.
- Make the first value of enums (the one that is equal to 0) to be the unknown
one, so that it was considered a programming error.
- Remove "omitempty" tag from fields where change from some value to empty value
must be passed in an update to the web UI.
- Add "omitempty" to Children because that's an important special use case.
- Modify javascript to assume that name, state, style and message of an action
is always there.
- Modify javascript to work when 'children' value was not received.
It looks like something has changed recently and now when WorkflowManager
encodes Node into json it behaves similar to proto3: when 'state' and 'style'
in an action have enum value equal to 0, it's getting omitted from the resulting
json. That caused the javascript code to skip adding the Action object to the
Node. I'm fixing the javascript to account for that and consider missing 'state'
and 'style' as being equal to 0.
This makes a minimal wiring of schema swap process into the workflow framework,
just enough to execute it and to see in the UI when it's in progress. Deeper
integration to display all the steps being done during the schema swap will be
done later.
This also fixes a few nil pointer dereference bugs discovered during testing of
the process. With those bugs fixed the rest of the code works as is.
- Including UI display of the redirect, and a fix in zk topology.
- Better Update support in vtctld2.
Now always using a single Update structure, and json-encoding it.
Omitting a bunch of fileds if empty in json encoding.
- Adding unit test for long polling.
It contains a Keyspace and Shard validator for now, but it's easy to add
new ones.
In the process:
- Fixing minor bugs in vtctld2 web ui.
null and undefined are different in TS.
- Ignoring ErrNoNode in FindAllShardsInKeyspace.
It most likely means a shard creation was aborted in the middle, and we
don't want to error everything out when that happens.
- Adding workflow creation dialog in vtctld.
- Adding 'Stop' button to workflows in vtctld.
- Fixing exception for redirect keyspaces display.
- Adding skip_start to vtctld.
- Adding Ui side for showing non-running workflows.
- Adding a vtctld HTTP based long-polling mechanism.
- Releasing UI resources for workflow closure.
Also fixing a race condition in manager shutdown:
We were using (manager.ctx == nil) as a test to check out if the manager
was stopped. This is not reliable, as manager.ctx is updated after the
context cancelation and after taking the manager mutex. In the
meantime, a job could have also realized its context was canceled and
started its shutdown, which takes the manager mutex and checks for
manager.ctx == nil as well.
The new logic uses a new 'stopped' flag instead to differentiate the
Manager shutdown from a workflow Stop. It's better documented too.
* Added dsahboard view and created keyspace service
* Implemented most of the views necessary. Checking in to sync up.
* restructured file tree
* Working on CRUD interface
* Finished adding mock up views to dashboard
* Implemented keyspace read functionality
* Adds CRUD functionality for Keyspaces and infrastructure
* Started exposing Shard API Endpoints
* gofmt
* Added support for shard endpoints
* Addressed all comments by enisoc and mberlin
* vtctld: added full crud functionality for UI.
Pointed UI to endpoint that leverages vtctl for commands.
Added support for Validation at all levels, RebuildKeyspaceGraph,
RemoveKeyspaceCell, InitShardMaster, Ping, and RefreshState.
* Added new icons on the sidenav, these do not point to new views.
* Removed extraneous merge files and adde new line to dialog-content
* handle partial list error
* gofmt indented return in api.go
* gofmt
* moved material imports to modules in app.module
* fix line length for linter
* adding command support and prime dropdown
* Dashboard has commands working
* removed dashboard prepare function
* finished transition to commands in dialog setting, added tablet actions
* Started adding dropdowns for schema view
* updated to webpack
* adding tablet fetching
* Updateed Tablet service
* Added single schema view for selected schema.
* Changed how columns are processed and moved to prime dialog
* Clean up titles and remove debugging code
* Changed to schema to use vtctlService.runCommand.
* Added spaces for style and removed unnecessary css
* Reworked UI based on team comments
* Removed an unnecessary !important tag
Michael Berlin
Yipei Wang
Alain Jobart
thompsonja
Pavel Ivanov
Joshua Thompson
Priyanka Kulshreshtha
pkulsh
dsslater