* Initial commit.
The intent is to review this in its current feature-complete state, so I can also commit the equivalent changes in the Python SDK.
Then, in the next PR (or before merging this PR), I will make some additional improvements as discussed, including break main into individual methods and refactor this "query-ingest-query" approach into an additional "query-only" mode.
* Remove comment
* Optimize if/else
* Minor verbiage changes
* Minor verbiage changes
* Minor verbiage changes
* Update README.md
Co-authored-by: Yochai Gilad <yogilad@microsoft.com>
* - Break main up into methods
- Consider new parameters
- Few code review changes, like consistent periods at end of sentences and filename consistency with Python
* Only alter batching policy if !useExistingTable
* - Pass filesize of 0 for autodetection
- Do not create a File object to get its path; just pass along the provided path
- Move alterBatchingPolicy to within createNewTable, and wrap it in if(false) so it won't run
- Add comment explaining that we generally wouldn't print the output of a control command
* - Improve README text
- Change .alter command to .alter-merge command
* - Change filename kusto-sample-config.json to kusto_sample_config.json to be consistent with Python
* - Fix reference to Python
- Aad a comment for AppCertificate auth
Co-authored-by: Yochai Gilad <yogilad@microsoft.com>
Breaking Changes:
Reverted mandatory DataFormat
Features:
Added github actions for running builds
Fixes:
Fixed pipeline to run and test
Fixed failing tests and other minor fixes
* Support AAD Managed Identity authentication
* Update data/src/main/java/com/microsoft/azure/kusto/data/auth/ManagedIdentityTokenProvider.java
Co-authored-by: AsafMah <asafmahlev@microsoft.com>
* Fix scope
* Fix space
* Downgrade azure-identity package to avoid showstopper bug causing ManagedIdentity auth to fail.
Note that there still remains 1 change to be made in this PR - to set the scope based on the Kusto Service Resource ID from CloudInfo. In support of this, I have a 22-file change to broadly improve how the TokenProviders consume from CloudInfo, but am hesitating to push that before understanding the initial CloudInfo design decisions.
At a high level, I would like to change the throws list when creating a client. It currently throws URISyntaxException, because it only uses CloudInfo when a token is requested. However, I don't see why we don't make use of CloudInfo when creating the client, in which case we should return a DataClientException/DataServiceException. This is a breaking change and a non-trivial paradigm shift, so it's worth discussing first.
I will begin discussing this with Asaf tomorrow.
* Resolve dependency conflicts
* Reorganize CloudInfo code
* Fix compilatino error
* Remove superfluous method (determineAadAuthorityUrl)
* Remove duplicate line
* Fix cloudinfo initialization
* Fix E2E tests (have been broken for a while)
* Switch order of clientUrl to be first
Co-authored-by: AsafMah <asafmahlev@microsoft.com>
* Implement MSAL Subject Name Issuer authentication
* Update ConnectionStringBuilder.java
* Update ConnectionStringBuilder.java
* Fix build failure after OnCloudInit() was added to abstract parent
* Ensure we send the public certificate when using SNI
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
* Kusto now queries the provided cluster for metadata. This fixes incompatibility issues with other clouds, or services like Synapse. The results are cached and don't require any changes to the user's code or behavior.
* new API to get primitive as objects to handle null values
* nulls
* better
* comments
* comments
Co-authored-by: KustoIbizaExtension Build <kustodev@microsoft.com>
* Add streaming query API
* - Add some resource closing enhancements
- Wrap the CloseParentResourcesStream in a EofSensorInputStream, as opposed to the converse
* Add TODO to reuse HttpClient
* Use existing Java exception formatting, which is an improvement over the .NET exception
* Add new API to query so that the result is a json. When the caller doesn't need the Java object, this saves time converting to a Java object in the SDK and from the caller converting from the Java object.
* - Split input variables validation
- trim() command
* Rename from executeForJsonResult to executeToJsonResult
* Rename from executeForJsonResult to executeToJsonResult
* Streaming endpoint is always v1
* - Optimize and upgrade dependency versions
- Remove dependency duplicates/conflicts
- Resolve mvn warnings
* - Use range to ensure specific minor version of jackson dependencies
- Upgrade flatten-maven-plugin
* added classes for generating pojos from KustoResultSetTable instances
This commit includes three new classes:
1. KustoType which is a generic class for representing different Kusto
column types and bind them to different Java types
2. KustoColumn which represents a Kusto column - it has a name,
nullability information, and a type (KustoType)
3. KustoResultMapper which can be set up to parse KustoResultSetTable
instances and return a list of pojos. The setup phase is done using a
builder that takes the pojo constructor, as well as zero or more
instances of KustoColumn along with setter functions to set pojo's
fields based on the column values.
This mechanism can potentially be extended to use reflection based on
the target pojo class instead of the manual KustoResultMapper setup
* Update data/src/main/java/com/microsoft/azure/kusto/data/KustoType.java
Changed documentation link to international version
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
* Use String.format() for exception messages
* Introduced ObjectPopulator class, ordinal column mapping
- Unified KustoColumn and QueryResultColumnInfo into the new
ObjectPopulator class
- Added support for manually specified column ordinals
- extractList finds column indexes once per result set to save on column
lookups
* KustoResultMapper extractSingle - don't create empty objects
If the KustoResultSetTable is null, the proper response is to return a
null object, not an unpopulated one
* ObjectPopulator - always use column index, formatting
populateFrom used to get objects from the ResultSet by the column name
unless an ordinal was set. Now it always uses the ordinal (looking it up
if necessary).
* KustoType formatting - don't align field declarations
* KustoResultMapperTest
The test uses a mocked KustoResultMapper to test the pojo mapping
functionality of the KustoResultMapper
* Update KustoType.java
* Removed addNonNullableColumn/addNonNullableColumn, added javadoc
Instead of individual addNonNullableColumn/addNullableColumn methods,
exposed the more generic addColumn methods. Additionally, added javadoc
to document the addColumn parameters
* Renamed ObjectPopulator to KustoResultColumnPopulator
* KustoResultColumnPopulator::populateFrom use passed ordinal
populateFrom used to always lookup the column ordinal by name instead of
using the passed ordinal. Now it's always using the ordinal, moving the
responsibility of correctly determining that ordinal to the caller.
* renamed variable in KustoResultColumnPopulator::populateFrom
* KustoResultMapper::extractList renamed variable
* added "single" to names of tests of KustoResultMapper::extractSingle
* Javadoc fixes in KustoType and KustoResultMapper
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
* Rearchitect the AAD authentication code to be object oriented and follow best-practices (Java and otherwise)
* Rearchitect the AAD authentication code to be object oriented and follow best-practices (Java and otherwise)
* Code review comments:
- Rename to CallbackTokenProvider
- Remove 2 unnecessarily-passed authorityIds
* Add another layer of abstraction (MsalTokenProvider)
* Add another layer of abstraction (MsalTokenProvider)
* Add Override annotation per code review comment
* Convert from ADAL to MSAL (include a few questions to be resolved during code review)
* Resolve some TODOs
* Some minor changes and some changes from code review
* Incorporate most review comments
* Add '/' to authority to resolve issue.
* Per Yochai's request, first try to access the token silently, and if that fails authenticate from scratch
* Make the method determineAadAuthorityUrl() less confusing
* Fix error
* Fix test
* remove federated url path if set
* ignore case
* better
* test
* fix test
* raw format
* comments
* comments
Co-authored-by: Ohad Bitton <ohbitton@microsoft.com>
* Streaming client wrong endpoint detection
* Fix invalid character for the file's encoding
* Add parent class to both Queued and Streaming IngestClients for the local purpose of removing significant code duplication in the wrong-endpoint-detection feature.
This change represents a better architecture that will yield further benefits.
* Rename AbstractIngestClient to IngestClientBase
Adds ManagedStreamingIngestClient to support multiple streaming retries and queue fallback.
commits:
* First commit, create file and basic members
* basic copy-paste argument validation, close streams on close.
no logic yet
* Copy ctors for all the needed types
* Filled out other methods (Like in the c# manager, all of them redirect to stream, except blob which goes directly to the queued ingestor)
* Simplest support for actual streaming
* Fixed with new name for the queued client
* Removed unsued stuff for now, added doc comment.
* Some small fixes from testing -
- Log exception
- Add a direct c-tor
- Catch all exception types, since they may be unpredictable
* Reset the stream after an exception
* Adding tests for the class
* cleanup imports
* Added annotations as an explicit dependency(like in data) to appease CI.
* Update ingest/src/main/java/com/microsoft/azure/kusto/ingest/ManagedStreamingIngestClient.java
grammar
Co-authored-by: boazsha <15858748+boazsha@users.noreply.github.com>
* Update ingest/src/main/java/com/microsoft/azure/kusto/ingest/ManagedStreamingIngestClient.java
grammar
Co-authored-by: boazsha <15858748+boazsha@users.noreply.github.com>
* Change retry count to 3
* Made the json file tests different from each other, plus they now check the stream.
The gz file was changed since the old one had CRLF line endings, while the uncompressed one had LF.
* Documentation - added a note to explain ingestFromBlob's behaviour, and some tiny grammar fixes.
* Removed default compile annotation
* Update ingest/src/main/java/com/microsoft/azure/kusto/ingest/ManagedStreamingIngestClient.java
Co-authored-by: Boaz Shalom <15858748+boazsha@users.noreply.github.com>
* Removed todo comments from tests, since we decided to keep them this way
* Pass the to the StreamSourceInfo ctor whenever applicable
* Make comments fit the style
* Added doc for ingestFromBlob function
* Added factory for managed streaming ingest client
* Fixed managedingest from files - since FileInputStream isn't resettable, we can't use managed with it, so I added a class for it and a test.
* Reset the mock properly
* Add license
* Parsing permanent exceptions from the api (marked as @permanent=true in the returning json) and stopping if it is one.
* Reset the client in all tests
* Made compressedJson test work despite the fact that the stream closes in the end
* Adding debug prints to the comparison test since I don't know how else to check tests that run on my machine but fail on the testing machine
* Try using alternative method of reading string to avoid newline issue
* Also fixed the other one to make the tests pass, but we'll still have to look into why the file changes on the test machine
* Add UncloseableStream class
* Fixed typo
* Solve the newlines problem by hardcoding the string result into the test file.
* Use UncloseableStream in utils (still needs tests)
* Update ingest/src/main/java/com/microsoft/azure/kusto/ingest/ManagedStreamingIngestClient.java
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
* Use proper reset function in tests
* Unit test UncloseableStream
* Correct format for web error
Co-authored-by: boazsha <15858748+boazsha@users.noreply.github.com>
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
* Add detection of wrong endpoint with correct endpoint URI suggestion
* Fix code review comments
* Remove extraneous comment from test
* Queued Ingestion's wrong endpoint detection is more generic to reduce code duplication if the same feature is added for Streaming Ingestion.
Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com>
Yihezkel Schoenbrun
AsafMah
ohad bitton
KustoIbizaExtension Build
maltalex
Ohad Bitton
ohbitton