Граф коммитов

273 Коммитов

Автор SHA1 Сообщение Дата
microsoft-github-policy-service[bot] 8ede19d766
Auto merge mandatory file pr
This pr is auto merged as it contains a mandatory file and is opened for more than 10 days.
2022-11-28 19:13:49 +00:00
microsoft-github-policy-service[bot] a0decea44d
Microsoft mandatory file 2022-08-15 21:16:37 +00:00
Junjie Qian 7f47dadffb
Fix the merge from upstream (#9)
* implement client options, support multiple namenodes with failover

* Fix an alignment bug causing us to send very small packets

The faulty math in block_write_stream was causing us to always send
512 byte packets.

* rename host to namenodeHost

* document standby retry behaviour

* include detail in namenode error

* deprecate WrapNamenodeConnection

* replace goto behaviour with for loop

* fallback to error from last failure

* travis: bump go version to 1.x

* Regenerate proto files

(this should fix travis issues, hopefully)

* travis: touch *.pb.go before running make

For some reason, make on travis recently started deciding that the
*.pb.go files were older than the *.proto files, ever though they
originate from the same `git checkout`. The easiest way to fix this
is simply to touch them right before we run make, so that they look
newer.

* travis: attempt to fix hadoop tarball caching

* Fix comment for CopyToRemote

Fixes #89, #90

* Deprecate NewForUser and NewForConnection

* Add the ability for 'put -' to read from stdin

Fixes #44, closes #98

* Convert from govendor to dep

* Update dependencies

* Tweak put tests

Add a very basic test, and put everything in the same directory so it
gets cleaned up properly.

* Improve conf loading and surrounding tests

- Fix a bug where configuration wasn't loaded from specified paths
- Only search one path at a time, rather than deduping from all
- Correctly handle fs.default.name
- Improve test coverage and documentation

* Fix the merge from upstream

* more fixes to merge master

* more fixes
2018-03-17 13:22:31 -07:00
Colin Marc 6910b40450 Improve conf loading and surrounding tests
- Fix a bug where configuration wasn't loaded from specified paths
- Only search one path at a time, rather than deduping from all
- Correctly handle fs.default.name
- Improve test coverage and documentation
2018-03-16 12:59:08 -07:00
Colin Marc f4280a2b2c Tweak put tests
Add a very basic test, and put everything in the same directory so it
gets cleaned up properly.
2018-03-16 12:59:08 -07:00
Colin Marc c63d32e4bc Update dependencies 2018-03-16 12:59:08 -07:00
Colin Marc 437573d5c2 Convert from govendor to dep 2018-03-16 12:59:08 -07:00
Shastick 5434f3676c Add the ability for 'put -' to read from stdin
Fixes #44, closes #98
2018-03-16 12:59:08 -07:00
Colin Marc 715f1cbbb6 Deprecate NewForUser and NewForConnection 2018-03-16 12:59:08 -07:00
Colin Marc 5f7f8dda00 Fix comment for CopyToRemote
Fixes #89, #90
2018-03-16 12:59:08 -07:00
Colin Marc 63b6f7b227 travis: attempt to fix hadoop tarball caching 2018-03-16 12:59:08 -07:00
Colin Marc 51cb745606 travis: touch *.pb.go before running make
For some reason, make on travis recently started deciding that the
*.pb.go files were older than the *.proto files, ever though they
originate from the same `git checkout`. The easiest way to fix this
is simply to touch them right before we run make, so that they look
newer.
2018-03-16 12:59:08 -07:00
Colin Marc 107883f4fe Regenerate proto files
(this should fix travis issues, hopefully)
2018-03-16 12:59:08 -07:00
Colin Marc c57337ce74 travis: bump go version to 1.x 2018-03-16 12:59:08 -07:00
Richard Jones af088d6254 fallback to error from last failure 2018-03-16 12:59:08 -07:00
Richard Jones b875ddb934 deprecate WrapNamenodeConnection 2018-03-16 12:59:08 -07:00
Richard Jones ce75329b67 include detail in namenode error 2018-03-16 12:59:08 -07:00
Wang Dafeng d559d5f139 Merge master branch to our branch (#6)
* implement client options, support multiple namenodes with failover

* Fix an alignment bug causing us to send very small packets

The faulty math in block_write_stream was causing us to always send
512 byte packets.

* rename host to namenodeHost

* document standby retry behaviour

* include detail in namenode error

* deprecate WrapNamenodeConnection

* replace goto behaviour with for loop

* fallback to error from last failure
2017-10-26 11:44:59 -07:00
Junjie Qian a783f7badd Merge pull request #4 from colinmarc/master
Latest stuff from Colin Marc's hdfs
2017-03-14 21:13:55 -07:00
Junjie Qian d961456920 Add StatFs and the df [-h] command 2017-03-14 00:47:54 +01:00
Colin Marc 89165ebd1b Align to chunk boundaries when appending (fixes #61) 2017-02-25 20:03:39 +01:00
Colin Marc 0e04ab0f44 Fix weird naming in NewBlockWriter 2017-02-25 19:36:03 +01:00
Colin Marc c263b7b08a Merge pull request #67 from sakserv/cli-nn-from-config
Allow the CLI to load the NameNodes from the Hadoop configuration
2017-02-13 22:43:02 +01:00
Shane Kumpf a732047294 Allow the CLI to load the NameNodes from the Hadoop configuration 2017-02-13 11:34:53 -07:00
Colin Marc af44627e28 Allow failures on OSX 2017-01-23 15:27:42 +01:00
Colin Marc 815c508b3b typo 2017-01-23 14:58:17 +01:00
Colin Marc bfec16f19d Merge pull request #63 from advincze/conn
new client consturctor with rpc.NamenodeConnection
2017-01-23 13:21:59 +01:00
Colin Marc 1a7827b227 Update installation instructions 2017-01-23 13:19:00 +01:00
Colin Marc 5ca9568680 Add a -v / --version flag 2017-01-23 13:08:15 +01:00
Colin Marc d3d0a3f030 Only do releases once 2017-01-23 12:59:51 +01:00
Colin Marc d3d7c7752d Remove stray get-deps invocation 2017-01-23 12:55:36 +01:00
Colin Marc c271c30522 set up releases 2017-01-23 12:52:27 +01:00
Colin Marc 5b6dfde9aa Build on golang 1.7 2017-01-23 12:30:26 +01:00
Colin Marc b17fab75c8 Switch to vendored dependencies 2017-01-23 12:29:46 +01:00
adam 1d7222509c added new client consturctor 2016-12-06 08:04:55 +01:00
Colin Marc cefa4bc7e7 Make mv behavior match the posix spec a little better
- Moving a non-directory on top of a directory should never work
 - Replacing an existing file should be the default
2016-10-26 21:05:48 +02:00
Junjie Qian 5d2b6c17a6 Merge pull request #3 from colinmarc/master
Port change from Colin's head fork
2016-10-26 11:48:42 -07:00
Colin Marc 91fa6b4ca6 Merge pull request #55 from junjieqian/rename-clobber
Clobber in rename: provide overwrite option in Rename
2016-10-26 19:32:48 +02:00
Junjie Qian 8774aa7d4c Force overwrite in rename and provide option support in mv 2016-10-26 10:15:58 -07:00
Marko Radmilac 7253051eab Merge pull request #2 from colinmarc/master
Merge latest changes including Go 1.6 testing, etc.
2016-09-28 11:39:37 -07:00
Colin Marc 948b8ef728 Merge pull request #53 from colinmarc/block-incrementing
Don't increment the generation timestamp when appending
2016-09-28 20:31:20 +02:00
Colin Marc 8f59cc5f80 Don't increment the generation timestamp when appending 2016-09-23 13:04:13 +02:00
Colin Marc 1a4155475b Merge pull request #45 from veonik/append-2
Fixed incorrect Append implementation.
2016-08-17 21:52:41 +02:00
Colin Marc b482024c6a Add a note about HADOOP_USER_NAME
fixes #48 #49
2016-08-17 21:51:24 +02:00
Tyler Sommer 365db1e6a2 Fixed incorrect Append implementation.
The previous implementation considered existing blocks immutable, which is not
correct.

This commit attempts to unfinalize the last existing block during an append
operation so that it may be written to again. This is done by submitting the
correct stage `hdfs.OpWriteBlockProto_PIPELINE_SETUP_APPEND` during the
initial OpWriteBlock operation. Additionally, `blockWriteStream`s are now
always initialized with an offset, which will be > 0 in case of append
operation. Finally, the GenerationStamp field is incremented at the beginning
of the append operation, which is an incrementing integer that records the
version of a particular block.
2016-07-01 11:05:57 -06:00
Colin Marc 1fe2183ac9 Merge pull request #43 from veonik/append
Implemented Append operation to append to existing files.
2016-06-30 22:35:04 +02:00
Tyler Sommer 97274ea184 Implemented Append operation to append to existing files.
This commit adds support for APPEND mode file operations on existing files
stored on HDFS.
2016-06-30 13:47:15 -06:00
Colin Marc 7bd4aae133 Test on 1.6 2016-06-30 21:30:43 +02:00
Marko Radmilac adf6763a6d Latest stuff from Colin 2016-06-02 15:28:27 -07:00
Colin Marc 1ab3163168 Merge pull request #39 from colinmarc/proto-generation
Fix proto generation so we don't have to munge imports
2016-04-12 02:21:40 +02:00