Azure
/
azure-kusto-parquet-conv
зеркало из https://github.com/Azure/azure-kusto-parquet-conv.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
65 коммитов 5 Ветки 2 Теги 9,9 MiB
Граф коммитов

65 Коммитов

Автор SHA1 Сообщение Дата
Michael Spector 4a8f36a4b5 Incremented Nuget package version 2022-09-02 13:59:09 +03:00
Michael Spector 3f77a196be
Merge pull request #29 from Azure/cargo_update 
cargo update
 2022-09-02 13:54:41 +03:00
Michael Spector c7fd426123 cargo update 
Updating autocfg v1.0.1 -> v1.1.0
    Updating brotli v3.3.3 -> v3.3.4
    Updating cc v1.0.72 -> v1.0.73
    Updating chrono v0.4.19 -> v0.4.22
    Updating crc32fast v1.3.0 -> v1.3.2
    Updating either v1.6.1 -> v1.8.0
    Updating flate2 v1.0.22 -> v1.0.24
    Updating itoa v1.0.1 -> v1.0.3
    Updating libc v0.2.112 -> v0.2.132
    Updating log v0.4.14 -> v0.4.17
    Updating lz4 v1.23.2 -> v1.24.0
    Updating lz4-sys v1.9.2 -> v1.9.4
    Updating memchr v2.4.1 -> v2.5.0
    Updating miniz_oxide v0.4.4 -> v0.5.3
    Updating num-integer v0.1.44 -> v0.1.45
    Updating num-traits v0.2.14 -> v0.2.15
    Updating ryu v1.0.9 -> v1.0.11
    Updating serde v1.0.133 -> v1.0.144
    Updating serde_json v1.0.75 -> v1.0.85
 2022-09-02 10:03:24 +03:00
Igor Borodin e59b4f22a0 Promote NuGet to 0.1.19 2022-06-07 16:25:15 +03:00
sigorbor cf8a622183
Merge pull request #26 from Azure/less_deps 
Cut unused dependencies and default features
 2022-06-07 15:52:13 +03:00
Igor Borodin 25c8597850 Cut unused dependencies 2022-06-07 15:44:06 +03:00
Igor Borodin 2584132a1c Revert "Update Package.nuspec to pack all the output files" 
This reverts commit c406594bba.
 2022-05-01 15:48:41 +03:00
sigorbor c406594bba
Update Package.nuspec to pack all the output files 2022-05-01 15:45:18 +03:00
urishapira 2dee789156
Merge pull request #25 from urishapira/dev 
Return cslschema fix
 2022-04-26 13:30:47 +03:00
urishapira 4fa4174b37
Merge branch 'dev' into dev 2022-04-25 15:14:53 +03:00
Uri Shapira 389a5d85b9 updated version to 18 2022-04-25 15:12:48 +03:00
Uri Shapira ccc66a9589 reverted the revert 2022-04-25 13:09:13 +03:00
sigorbor d43a7a3f4f
Update LinuxPackage.nuspec 
trim trailing zeros (and dot) for floats when generating CSV
 2022-04-13 19:02:59 +03:00
sigorbor 335ab83b98
Update Package.nuspec to 0.1.17 
+ trim trailing zeros and dot for floats when generating CSV
 2022-04-13 15:24:26 +03:00
sigorbor e890fb2472
Merge pull request #24 from Azure/floats_truncate_trailing_zeros 
Trim trailing zeros and '.' (dot) for float numbers when generating CSV
 2022-04-13 11:23:42 +03:00
Igor Borodin 95f2673938 Trim trailing zeros and '.' (dot) for float numbers when generating CSV 2022-04-13 02:44:48 +03:00
sigorbor 856a707840
Merge pull request #23 from urishapira/dev 
reverted cslschema changes
 2022-04-06 21:20:19 +03:00
Uri Shapira 5256bc681a reverted cslschema changes 2022-04-06 20:40:20 +03:00
sigorbor 45cb5c2df6
Merge pull request #21 from urishapira/pq2json-linux-config 
added target condition
 2022-03-27 09:30:57 +03:00
urishapira 0f94bcd44b added target condition 2022-03-24 17:27:50 +02:00
sigorbor 89612edb8e
Merge pull request #18 from urishapira/linux-nuspec 
added linux nuspec
 2022-03-03 16:26:20 +02:00
urishapira b98c3394d6 added linux nuspec 2022-03-03 15:51:20 +02:00
Michael Spector b189180aa1
Merge pull request #14 from urishapira/cslschame-update 
updated cslschama types
 2022-02-20 08:45:16 +02:00
Michael Spector 5462c9b3c1
Merge branch 'dev' into cslschame-update 2022-02-20 08:44:24 +02:00
sigorbor dfa72004fd
Update Package.nuspec to 0.1.12 
To avoid conflict between Nuget feeds, promote the version
 2022-02-15 12:15:57 +02:00
Michael Spector c3541e81aa
Merge pull request #15 from urishapira/gt-parquet-row-groups-data 
Get parquet row groups data
 2022-02-15 11:06:48 +02:00
urishapira 7bdca360a3 updated package.nuspec 2022-02-15 09:54:24 +02:00
urishapira 06fa6ab965 format 2022-02-15 09:51:38 +02:00
urishapira 2586c53db0 updated schema 2022-02-15 09:41:51 +02:00
urishapira 28b50bf4d9 Added row groups data 2022-02-14 16:35:44 +02:00
urishapira 8baec66f0e updated cslschama types 2022-02-09 16:37:03 +02:00
Michael Spector 9f00a1f95b Update Cargo.lock 2022-01-18 09:15:37 +02:00
sigorbor 23a6f6ad2b
Merge pull request #13 from sigorbor/nuget0.1.10 
Promote NuGet package version + cargo fmt
 2021-07-05 16:12:52 +03:00
Igor Borodin 8e0b51486f Promote NuGet package version 
cargo fmt the code
 2021-07-05 15:54:48 +03:00
Michael Spector ed1e7ecffd
Merge pull request #12 from sigorbor/missing_columns_support 
Add empty CSV values for missing PQ columns
 2021-06-30 11:52:43 +03:00
Igor Borodin 9a6308d861 DOn't require --columns argument 2021-06-29 13:34:24 +03:00
Igor Borodin a14622fe7c Add empty CSV values for missing PQ columns 2021-06-28 16:36:48 +03:00
Michael Spector 16a0031c37 Increment version (badly uploaded package) 2021-03-11 15:31:33 +02:00
Michael Spector a787dc5c4f
Merge pull request #10 from Azure/convert_types 
Enable implicit Parquet to Kusto types conversion
 2021-03-11 14:24:52 +02:00
Michael Spector 328e336d9f Added configuration option that enables implicit Parquet to Kusto types conversion 2021-03-11 14:19:52 +02:00
Michael Spector ce0ff04e6b Support numeric keys in MAP type 2021-02-10 07:15:06 +02:00
Michael Spector d5172cbbf1
Merge pull request #9 from Azure/cslschema 
Added an option for getting CSL schema of Parquet file
 2020-12-01 13:17:41 +00:00
Michael Spector 881d4c789b Added an option for getting CSL schema of Parquet file 2020-12-01 15:13:39 +02:00
Michael Spector 38159e93ba
Merge pull request #8 from Azure/csv 
Support CSV output format
 2020-11-15 07:07:15 +00:00
Michael Spector 64b1046380 Format date logical type as string 2020-11-14 13:06:28 +02:00
Michael Spector 75527e2d25 Use ryu for rendering floats (the same way as serde_json) 2020-11-13 08:42:30 +02:00
Michael Spector f88a5485b3 Eliminate double line terminators 2020-11-12 14:15:40 +02:00
Michael Spector c965c9d313 Fixed escaping of strings with quotes by proper usage 2020-11-12 11:05:06 +02:00
Michael Spector 45e6d85e5e Add support for CSV output format. 
When `--csv` argument is used, the utility produces CSV format.
Nested top level elements (arrays, objects) are formatted as JSON.

Using the option alone saves some time already:

```
PS C:\Users\mispecto\Projects\azure-kusto-parquet-conv> Measure-Command
{ .\target\debug\pq2json.exe ..\..\Downloads\20200809201109710_53ea103e_e9fabfe6_001.parquet > 1 }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 11
Milliseconds      : 361
Ticks             : 113610039
TotalDays         : 0.000131493100694444
TotalHours        : 0.00315583441666667
TotalMinutes      : 0.189350065
TotalSeconds      : 11.3610039
TotalMilliseconds : 11361.0039

PS C:\Users\mispecto\Projects\azure-kusto-parquet-conv> Measure-Command
{ .\target\debug\pq2json.exe --csv ..\..\Downloads\20200809201109710_53ea103e_e9fabfe6_001.parquet > 1 }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 5
Milliseconds      : 624
Ticks             : 56247514
TotalDays         : 6.51012893518518E-05
TotalHours        : 0.00156243094444444
TotalMinutes      : 0.0937458566666667
TotalSeconds      : 5.6247514
TotalMilliseconds : 5624.7514
```
 2020-11-11 16:15:14 +02:00
Michael Spector d6d3fb9fc7
Merge pull request #7 from Azure/upgrade_arrow 
Fixed crash when reading a null byte array field.
 2020-05-05 13:31:56 +03:00