Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
This commit is contained in:
Derrick Stolee 2021-06-18 14:41:15 -04:00
Родитель 848cc6157b
Коммит f0bb0d3aa6
5 изменённых файлов: 0 добавлений и 475 удалений

Просмотреть файл

@ -6,106 +6,3 @@ Frequently Asked Questions
[microsoft-git]: https://github.com/microsoft/git
[git-faq]: https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/faq.md
Using Scalar
------------
### I don't want a sparse clone, I want every file after I clone!
Run `scalar clone --full-clone <url>` to initialize your repo to include
every file. You can switch to a sparse-checkout later by running
`git sparse-checkout init --cone`.
### I already cloned without `--full-clone`. How do I get everything?
Run `git sparse-checkout disable`.
Scalar Design Decisions
-----------------------
There may be many design decisions within Scalar that are confusing at first
glance. Some of them may cause friction when you use Scalar with your existing
repos and existing habits.
> Scalar has the most benefit when users design repositories
> with efficient patterns.
For example: Scalar uses the sparse-checkout feature to limit the size of the
working directory within a large monorepo. It is designed to work efficiently
with monorepos that are highly componentized, allowing most developers to
need many fewer files in their daily work.
### Why does `scalar clone` create a `<repo>/src` folder?
Scalar uses a file system watcher to keep track of changes under this `src` folder.
Any activity in this folder is assumed to be important to Git operations. By
creating the `src` folder, we are making it easy for your build system to
create output folders outside the `src` directory. We commonly see systems
create folders for build outputs and package downloads. Scalar itself creates
these folders during its builds.
Your build system may create build artifacts such as `.obj` or `.lib` files
next to your source code. These are commonly "hidden" from Git using
`.gitignore` files. Having such artifacts into your source tree creates
additional work for Git because it needs to look at these files and match them
against the `.gitignore` patterns.
By following the pattern Scalar tries to establish and placing your build
intermediates and outputs parallel with the `src` folder and not inside it,
you can help optimize Git command performance for developers in the repository
by limiting the number of files Git needs to consider for many common
operations.
### Why is Scalar a C# tool instead of built directly into Git?
Currently, Scalar does a few things that Git does not do and a few
things that Git will never do.
Git does not have a concept of running background jobs, which is critical to
keeping our repositories running smoothly. Git does not have a concept of
cache servers, so we need to use the GVFS protocol during `scalar clone` to
set up the cache server URL. Scalar also creates the `.scalarCache` directory
so multiple enlistments share objects from the remote, allowing a second
`scalar clone` operation to be much faster than the first. All of these concepts
are on our backlog for contributing to Git. Once we do, we will transition
from the Scalar implementation to the Git implementation.
Scalar uses the GVFS protocol for some repositories. This protocol was created
by Azure Repos to support large repositories before the Git protocol supported
partial clone and the Git client supported missing objects. In a way, the GVFS
protocol proved that there was value in creating the partial clone feature.
This shows precedent for how our C# tools show value that leads to Git client
features. However, Git will never adopt the GVFS protocol, so we need to use
an tool other than the Git client to support teams that use it.
For now, Scalar requires a
[custom version of Git](https://github.com/microsoft/git), but we hope to relax
that restriction eventually.
We try to keep our fork of Git as close to [upstream](https://github.com/git-for-windows/git)
as we can, but some patches are necessary _for now_. Still, maintaining those
patches as we take updates from upstream has a significant cost. It is better
to create features in C# that use Git as a black box, then remove those C#
features as we contribute replacements to Git upstream.
Finally, as Git gains the features that we need in Scalar, Scalar will need to
support backwards compatibility with repositories cloned with older versions of
Scalar. Using the C# layer, we can update the Git version and upgrade old
repositories to whatever new features are added to Git.
### Why doesn't Scalar support case-sensitive directories (or case-insensitive ones on Linux)?
Although Scalar is agnostic about the contents of your repository and
largely leaves file handling within your working tree to Git, it does check
some file paths and manage some cache files, and to do so Scalar assumes
your filesystem has the filename case-matching semantics typical for your
operating system. On Windows and macOS this means being case-insensitive
when matching filenames, and on Linux it means matching case-sensitively.
Because all three operating systems support exceptions to their case-matching
conventions (e.g., [case-sensitive NTFS directories][ntfs-case] or
[case-insensitive ext4 directories][ext4-case]), a possible future
enhancement for Scalar would be to support these alternatives using
a per-repository custom Git configuration value.
[ext4-case]: https://github.com/torvalds/linux/blob/171d4ff79f965c1f164705ef0aaea102a6ad238b/Documentation/admin-guide/ext4.rst#case-insensitive-file-name-lookups
[ntfs-case]: https://devblogs.microsoft.com/commandline/per-directory-case-sensitivity-and-wsl/#per-directory-case-sensitivity

Просмотреть файл

@ -6,113 +6,3 @@ Getting Started
[microsoft-git]: https://github.com/microsoft/git
[git-getting-started]: https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/getting-started.md
Registering existing Git repos
------------------------------
To add a repository to the list of registered repos, run `scalar register [<path>]`.
If `<path>` is not provided, then the "current repository" is discovered from
the working directory by scanning the parent paths for a path containing a `.git`
folder, possibly inside a `src` folder.
To see which repositories are currently tracked by the service, run
`scalar list`.
Run `scalar unregister [<path>]` to remove the repo from this list.
Creating a new Scalar clone using the GVFS Protocol
---------------------------------------------------
The `clone` verb creates a local enlistment of a remote repository using the
[GVFS protocol](https://github.com/microsoft/VFSForGit/blob/HEAD/Protocol.md).
```
scalar clone [options] <url> [<dir>]
```
Create a local copy of the repository at `<url>`. If specified, create the `<dir>`
directory and place the repository there. Otherwise, the last section of the `<url>`
will be used for `<dir>`.
At the end, the repo is located at `<dir>/src`. By default, the sparse-checkout
feature is enabled and the only files present are those in the root of your
Git repository. Use `git sparse-checkout set` to expand the set of directories
you want to see, or `git sparse-checkout disable` to expand to all files. You
can explore the subdirectories outside your sparse-checkout specification using
`git ls-tree HEAD`.
### Sparse Repo Mode
By default, Scalar reduces your working directory to the only the files at the
root of the repository. You need to add the folders you care about to build up
to your working set.
* `scalar clone <url>`
* Please choose the **Clone with HTTPS** option in the `Clone Repository` dialog in Azure Repos, not **Clone with SSH**.
* `cd <root>\src`
* At this point, your `src` directory only contains files that appear in your root
tree. No folders are populated.
* Set the directory list for your sparse-checkout using:
1. `git sparse-checkout set <dir1> <dir2> ...`
2. `git sparse-checkout set --stdin <dir-list.txt`
* Run git commands as you normally would.
* To fully populate your working directory, run `git sparse-checkout disable`.
If instead you want to start with all files on-disk, you can clone with the
`--full-clone` option. To enable sparse-checkout after the fact, run
`git sparse-checkout init --cone`. This will initialize your sparse-checkout
patterns to only match the files at root.
If you are unfamiliar with what directories are available in the repository,
then you can run `git ls-tree -d --name-only HEAD` to discover the directories
at root, or `git ls-tree -d --name-only HEAD <path>` to discover the directories
in `<path>`.
### Options
These options allow a user to customize their initial enlistment.
* `--full-clone`: If specified, do not initialize the sparse-checkout feature.
All files will be present in your `src` directory. This behaves very similar
to a Git partial clone in that blobs are downloaded on demand. However, it
will use the GVFS protocol to download all Git objects.
* `--cache-server-url=<url>`: If specified, set the intended cache server to
the specified `<url>`. All object queries will use the GVFS protocol to this
`<url>` instead of the origin remote. If the remote supplies a list of
cache servers via the `<url>/gvfs/config` endpoint, then the `clone` command
will select a nearby cache server from that list.
* `--branch=<ref>`: Specify the branch to checkout after clone.
* `--local-cache-path=<path>`: Use this option to override the path for the
local Scalar cache. If not specified, then Scalar will select a default
path to share objects with your other enlistments. On Windows, this path
is a subdirectory of `<Volume>:\.scalarCache\`. On Mac, this path is a
subdirectory of `~/.scalarCache/`. The default cache path is recommended so
multiple enlistments of the same remote repository share objects on the
same device.
### Advanced Options
The options below are not intended for use by a typical user. These are
usually used by build machines to create a temporary enlistment that
operates on a single commit.
* `--single-branch`: Use this option to only download metadata for the branch
that will be checked out. This is helpful for build machines that target
a remote with many branches. Any `git fetch` commands after the clone will
still ask for all branches.
* `--no-prefetch`: Use this option to not prefetch commits after clone. This
is not recommended for anyone planning to use their clone for history
traversal. Use of this option will make commands like `git log` or
`git pull` extremely slow and is therefore not recommended.
Removing a Scalar Clone
-----------------------
Since the `scalar clone` command sets up a file-system watcher (when available),
that watcher could prevent deleting the enlistment. Run `scalar delete <path>`
from outside of your enlistment to unregister the enlistment from the filesystem
watcher and delete the enlistment at `<path>`.

Просмотреть файл

@ -3,145 +3,3 @@ Scalar: Enabling Git at Scale
> **Note: Scalar has moved!** It is now part of [the `microsoft/git` fork][microsoft-git]
> and thus this document has [a new version in that repository][git-index].
[microsoft-git]: https://github.com/microsoft/git
[git-index]: https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/index.md
Scalar is a tool that helps Git scale to some of the largest Git repositories.
It achieves this by enabling some advanced Git features, such as:
* *Sparse-checkout:* limits the size of your working directory.
* *File system monitor:* tracks the recently modified files and eliminates
the need for Git to scan the entire worktree.
* *Commit-graph:* accelerates commit walks and reachability calculations,
speeding up commands like `git log`.
* *Multi-pack-index:* enables fast object lookups across many pack-files.
* *Incremental repack:* Repacks the packed Git data into fewer pack-file
without disrupting concurrent commands by using the multi-pack-index.
By running `scalar register` in any Git repo, Scalar will automatically enable
these features for that repo and start running suggested maintenance in the
background.
Repos cloned with the `scalar clone` command use the
[GVFS protocol](https://github.com/microsoft/VFSForGit/blob/HEAD/Protocol.md)
to significantly reduce the amount of data required to get started
using a repository. By delaying all blob downloads until they are required,
Scalar allows you to work with very large repositories quickly. This protocol
allows a network of _cache servers_ to serve objects with lower latency and
higher throughput. The cache servers also reduce load on the central server.
Installing on macOS
------------------
Scalar is installed and updated on macOS via [Homebrew](https://brew.sh/).
```sh
brew tap microsoft/git
brew install --cask scalar
```
If you wish to use the [GVFS Protocol][gvfs-protocol], then you will
instead need the cask that uses [our custom fork of Git][microsoft-git]:
```sh
brew tap microsoft/git
brew install --cask scalar-azrepos
```
When new versions of Scalar are available, you can upgrade in a few
different ways. First, you can use `brew`:
```sh
brew update
# Use only one of the following, depending on which you have installed:
brew upgrade --cask scalar
brew upgrade --cask scalar-azrepos
```
Alternatively, you can run `scalar upgrade` and it will run the necessary
`brew` commands on your behalf.
If your repository has many files in the working directory, then you might
want to install [Watchman](https://github.com/facebook/watchman), which
Scalar will detect and configure with Git's File System Monitor feature.
```sh
brew install watchman
```
Installing on Windows
--------------------
To install Scalar on Windows,
[download the `Installers_Windows_Release.zip` from the releases page](https://github.com/microsoft/scalar/releases).
Extract the `Installers_Windows_Release` folder, open it in a command prompt, and
run `InstallScalar.bat`. This will install the following components:
* [Git for Windows](https://github.com/microsoft/git) (with custom patches)
* Scalar
* [Watchman](https://github.com/facebook/watchman), if you use the `--watchman` argument.
Installing on Linux
-------------------
Currently, we package a custom version of Git and Scalar as `.deb` packages
that can be installed after downloading from the GitHub releases pages.
At this point, you must install our custom version of Git in order to get
background maintenance as part of `scalar clone` or `scalar register`. As
that feature is accepted and merged into the core Git client, then you can
take advantage of the feature without the custom fork.
The latest releases can be downloaded and installed as follows:
> Note: If you don't have `wget`, then try `sudo apt-get install wget` first.
```sh
# Install git-vfs, a custom fork of Git
wget https://github.com/microsoft/git/releases/download/v2.29.0.vfs.0.0/git-vfs_2.29.0.vfs.0.0.deb
sudo dpkg -i git-vfs_2.29.0.vfs.0.0.deb
# Install GCM Core
wget https://github.com/microsoft/Git-Credential-Manager-Core/releases/download/v2.0.252-beta/gcmcore-linux_amd64.2.0.252.766.deb
sudo dpkg -i gcmcore-linux_amd64.2.0.252.766.deb
git-credential-manager-core configure
# Install Scalar
wget https://github.com/microsoft/scalar/releases/download/v20.10.178.6/scalar-azrepos-linux_amd64.20.10.178.0.deb
sudo dpkg -i scalar-azrepos-linux_amd64.20.10.178.0.deb
# Test installation
git version
git-credential-manager-core version
scalar version
```
We are working to provide a package via `apt-get` to make this installation
easier and better for automatically upgrading.
The current installation via `.deb` package only works on Debian-based
Linux distributions. The software has only been compiled and tested with x86_64/amd64
architectures. Motivated users can install [`microsoft/git`](https://github.com/microsoft/git)
and Scalar from source. See [the `InstallFromSource.sh` script](https://github.com/microsoft/scalar/blob/2dc48243c50763024b048c5f36d5f50835943dda/Scripts/Linux/InstallFromSource.sh#L62-L76)
for assistance installing Scalar from source.
Documentation
-------------
* [Getting Started](getting-started.md): Get started with Scalar.
Includes `scalar register`, `scalar unregister`, `scalar clone`, and
`scalar delete`.
* [Troubleshooting](troubleshooting.md):
Collect diagnostic information or update custom settings. Includes
`scalar diagnose`, `scalar config`, `scalar upgrade`, and `scalar cache-server`.
* [The Philosophy of Scalar](philosophy.md): Why does Scalar work the way
it does, and how do we make decisions about its future?
* [Frequently Asked Questions](faq.md)

Просмотреть файл

@ -6,66 +6,3 @@ The Philosophy of Scalar
[microsoft-git]: https://github.com/microsoft/git
[git-philosophy]: https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/philosophy.md
The team building Scalar has **opinions** about Git performance. Scalar
takes out the guesswork by automatically configuring your Git repositories
to take advantage of the latest and greatest features. It is difficult to
say that these are the absolute best settings for every repository, but
these settings do work for some of the largest repositories in the world.
Scalar intends to do very little more than the standard Git client. We
actively implement new features into Git instead of Scalar, then update
Scalar only to configure those new settings. In particular, we are porting
features like background maintenance to Git to make Scalar simpler and
make Git more powerful.
Scalar ships with [a custom version of Git][microsoft-git],
but performs most of its benefits without that version. The only feature
that is not intended to ever reach the standard Git client is Scalar's use
of [the GVFS Protocol][gvfs-protocol].
The GVFS Protocol is a way to reduce object transfer between client and
server, named for its use in [VFS for Git](https://github.com/microsoft/vfsforgit).
If you don't use the GVFS Protocol, then most of the value of Scalar can
be found in the core Git client. However, most of the advanced features
that really optimize Git's performance are off by default for compatibility
reasons. To really take advantage of Git's latest and greatest features,
you either need to study the [`git config` documentation](https://git-scm.com/docs/git-config)
and regularly read [the Git release notes](https://github.com/git/git/tree/master/Documentation/RelNotes).
Even if you do all that work and customize your Git settings on your machines,
you likely will want to share those settings with other team members.
Or, you can just use Scalar!
Using `scalar register` on an existing Git repository will give you these
benefits:
* Additional compression of your `.git/index` file.
* Hourly background `git fetch` operations, keeping you in-sync with your
remotes.
* Advanced data structures, such as the `commit-graph` and `multi-pack-index`
are updated automatically in the background.
* If Watchman is installed, then the FileSystem Monitor hook is configured
to use Watchman's change-tracking, providing faster commands such as
`git status` or `git add`.
Additionally, if you use `scalar clone` to create a new repository, then
you will automatically get these benefits:
* Use Git's partial clone feature to only download the files you need for
your current checkout.
* Use Git's [sparse-checkout feature][sparse-checkout] to minimize the
number of files required in your working directory.
[Read more about sparse-checkout here.][sparse-checkout-blog]
* Create the Git repository inside `<repo-name>/src` to make it easy to
place build artifacts outside of the Git repository, such as in
`<repo-name>/bin` or `<repo-name>/packages`.
We also admit that these **opinions** can always be improved! If you have
an idea of how to improve our setup, consider [creating an issue](https://github.com/microsoft/scalar/issues/new) or contributing a pull request! Some [existing](https://github.com/microsoft/scalar/issues/382)
[issues](https://github.com/microsoft/scalar/issues/388) have already
improved our configuration settings and roadmap!
[gvfs-protocol]: https://github.com/microsoft/VFSForGit/blob/HEAD/Protocol.md
[microsoft-git]: https://github.com/microsoft/git
[sparse-checkout]: https://git-scm.com/docs/git-sparse-checkout
[sparse-checkout-blog]: https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/

Просмотреть файл

@ -6,60 +6,3 @@ Troubleshooting
[microsoft-git]: https://github.com/microsoft/git
[git-troubleshooting]: https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/troubleshooting.md
Upgrade
-------
The Scalar service checks for new versions of Scalar daily and will prompt you
for upgrade using a notification. To check yourself, run `scalar upgrade` to
see if an upgrade is available. Run `scalar upgrade --confirm` to actually
perform the upgrade, if you wish.
Diagnosing Issues
-----------------
The `scalar diagnose` command collects logs and config details for the current
repository. The resulting zip file helps root-cause issues.
When run inside your repository, creates a zip file containing several important
files for that repository. This includes:
* All log files from `scalar` commands run in the enlistment, including
maintenance steps.
* Log files from the Scalar service.
* Configuration files from your `.git` folder, such as the `config` file,
`index`, `hooks`, and `refs`.
* A summary of your Git object database, including the number of loose objects
and the names and sizes of pack-files.
As the `diagnose` command completes, it provides the path of the resulting
zip file. This zip can be sent to [the support team](mailto:git-client@github.com) for investigation.
Modifying Configuration Values
------------------------------
The Scalar-specific configuration is only available for repos using the
GVFS protocol.
### Cache Server URL
When using an enlistment cloned with `scalar clone` and the GVFS protocol,
you will have a value called the cache server URL. Cache servers are a feature
of the GVFS protocol to provide low-latency access to the on-demand object
requests. This modifies the `gvfs.cache-server` setting in your local Git config
file.
Run `scalar cache-server --get` to see the current cache server.
Run `scalar cache-server --list` to see the available cache server URLs.
Run `scalar cache-server --set=<url>` to set your cache server to `<url>`.
### Scalar Config
The `scalar config` command is used for customizing the feed used for
Scalar upgrades. This is so large teams can bundle a custom installer
or other tools along with Scalar upgrades.