Edited collectsfdata.options.json to reflect new local ingestion fields (LocalPath, DatabasePersistence, DatabasePersistencePath). Added examples to show how to setup your configuration file to do local ingestion.
I added changes primarily in Collector.cs (UploadLocalData() function and other helper functions) to grab files from the LocalPath field location specified in the configuration file. Then go through those files and filter out those whose 'LastModified' time does not fit the StartTimeStamp/EndTimeStamp times. Then, with the files that fit the time range, I created a copy of those files and put them in the CacheLocation path specified in the configuration file. The files in the CacheLocation are used later for formatting and local ingestion.
I added changes to the other files to support the LocalPath field and allow you to test the code I put in the Collector.cs file. Those changes are the same as what is in my prior PR but doesn't include all of them. I only added what was necessary to reach the new functionality I described above and test that the file copies were created in the CacheLocation. You won't be able to actually completely do local ingestion with my changes in this PR, only in the prior PR.
I made changes to the CollectServiceFabricData tool to support local ingestion. My first main changes involve adding 2 new configuration field options, "LocalPath" and "OverwriteTable." "LocalPath" is the field that contains a path to a folder where the original, unmodified trace files are put. If this has a value, then that indicates that the user wants to do local ingestion (depending on what we decide with time stamp filtering, this could change). "OverwriteTable" is a boolean indicating whether to append or replace the data in the table the user is ingesting traces into. A value of "true" indicates that the table will be cleared before ingesting new data. A value of "false" (default) will just add new traces to the table without replacement. For local ingestion, I am utilizing the preexisting "KustoCluster" field to take in a local web server url with a database name (example: http://localhost:8080/MyDBName). To enable local ingestion, you must provide a url like the example in the "KustoCluster" field and a path in the "LocalPath" field in your collectsfdata.options.json file.
I created/edited functions in KustoConnection.cs and KustoEndpoint.cs to support connecting to the Kusto emulator, creating a new database/table if they don't exist, and ingesting traces after they've been formatted. Rather than making use of an ingestion queue, my local ingestion process executes KQL commands to directly ingest data into a table. At the end, you should be able to use the Kusto Explorer to view and evaluate traces.
What I tested:
- local ingestion with one short trace file (5 rows) with 'OverrideTable' set to false
- local ingestion with one short trace file (5 rows) with 'OverrideTable' set to true
(image shows example of ingested data in table)
- local ingestion with 2-10 trace files (the short one from above and a large one (CreateSelfReconfiguringService.test.trace) with 35,622 rows) "OverrideTable" set to true. Clears table and uploads correct number of rows.
- local ingestion with the 2-10 trace files with "OverrideTable" set to false. Successfully adds to the table
- local ingestion with 2-10 trace files with no DB / table already existing. They are created successfully, and ingestion continues
- local ingestion with 5 trace files with no "overrideTable" field in config file - sucessfully ingests into new table
- Attempting local ingestion without providing a "cachelocation" value in my configuration file - successfully gave the user an error and in the message says to provide a value. Safely exits the program.
- Attempting local ingestion with a localhost url that doesn't match my pattern in the 'KustoCluster' field of my configuration file. Ex: "http://localhost:808000/MyDBName". The program safely exits and provides appropriate...
block net462 and kusto federated authentication without azure client id as it is not supported
remove runtime property IsARMValid when saving configuration
add additional vscode launch configurations for net462 and net48
Steve Edwards
jagilber
Victoria Chin
dbucce
Jason Gilbertson
