diff --git a/docs/AMBROSIA_client_network_protocol.md b/CONTRIBUTING/AMBROSIA_client_network_protocol.md similarity index 100% rename from docs/AMBROSIA_client_network_protocol.md rename to CONTRIBUTING/AMBROSIA_client_network_protocol.md diff --git a/CONTRIBUTING/README.md b/CONTRIBUTING/README.md new file mode 100644 index 0000000..d283895 --- /dev/null +++ b/CONTRIBUTING/README.md @@ -0,0 +1,89 @@ + + +CONTRIBUTING GUIDE +================== + +For developers interested in adding to AMBROSIA, or developing new +[language-level or RPC-framework bindings to AMBROSIA](#new-client-bindings), +this document provides a few pointers. + +Overview of repository +---------------------- + +AMBROSIA is implemented in C# and built with Visual Studio or dotnet +CLI tooling. Within the top level of this source repository, you will +find. + +(1) Core libraries and tools: + + * `./Ambrosia`: the core reliable messaging and runtime coordination engine. + + * `./ImmortalCoordinator`: the wrapper program around the core library that + must be run as a daemon alongside each AMBROSIA application process. + + * `./DevTools`: additional console tools for interacting with the + Azure metadata that supports an Ambrosia service. + + * `./AKS-scripts`: scripts to get a user started with AMBROSIA on + Kubernetes on Azure. + + * `./Scripts`: scripts used when running automated tests (CI) as well + as the runAmbrosiaService.sh script which provides an example means + of executing an app+coordinator. + +(2) Client libraries: + + * `./Clients`: these provide idiomatic bindings into different + programming languages. + +(3) Sample programs and tests: + + * `./Samples`: starting point examples for AMBROSIA users. + + * `./InternalImmortals`: internal test AMBROSIA programs, demos, and + benchmarks. + + * `./AmbrosiaTest`: testing code + + +New Client Bindings +=================== + +AMBROSIA is designed to keep its runtime components in a separate +process (ImmortalCoordinator) than the running application process. +The coordinator and the application communicate over a pair of TCP +connections. + +This separation makes the runtime component of AMBROSIA completely +language-agnostic. All that is needed is for the application +processes to speak the low-level messaging protocol with the +coordinator. + +For a new language or RPC framewrok, there are two ways to accomplish +this: (1) do the work yourself to implement the wire protocol, (2) +wrap the provided standalone native code library (which is small with +zero dependencies), to create a higher-level language binding. + + +Implement the low-level wire protocol +------------------------------------- + +Refer to +[AMBROSIA_client_network_protocol.md](AMBROSIA_client_network_protocol.md) +for details on the specification applications must meet to communicate +with ImmortalCoordinator at runtime over TCP sockets. + + +Wrap the Native Client +---------------------- + +`Clients/C` contains a small library that handles the wire protocol: +i.e, decoding headers, variable width integer encodings, and so on. +It provides a primitive messaging abstraction for writing payloads of +bytes with different method IDs, but nothing more. + +This native code client library is written in vanilla C code and is +free of runtime dependencies. It should be wrappable in any +high-level language that supports the C calling conventions with its +foreign function interface (FFI). + diff --git a/README.md b/README.md index 7092094..5bf6e75 100644 --- a/README.md +++ b/README.md @@ -29,34 +29,25 @@ execution, without requiring developers to weave together such complex systems, or use overly expensive mechanisms. Check out the overview deck linked to the left to learn more or email us. -Quick Start ------------ +Quick Start: Fetch a binary distribution +---------------------------------------- -Build the PerformanceTestInterruptible example, using this one-line -powershell command: - - .\CmdLine-FreshBuild.ps1 - -Overview of directories ------------------------ - - * LocalAmbrosiaRuntime: the core reliable messaging and coordination engine. - - * InternalImmortals: example programs and services built on Ambrosia. - - * Tools: additional console tools for interacting with the Azure - metadata that supports an Ambrosia service. - - * AmbrosiaTest: integration tests. - - * Ambrosia : Language binding for C# - -Out of place things that should move: - - * Franklin_TestApp/ - - this contained the first kubernetes prototype, obsolete [2018.09.13] +FINISHME - +Quick Start: Build from Source +------------------------------ +Build the Ambrosia Immortal coordinator and C# client code generator +with this Bash script: + ./build_dotnetcore_bindist.sh +Given a .NET Core SDK, this will work on Windows, Mac OS, or Linux. +After that, you have an AMBROSIA binary distribution built inside the +`./bin` directory within your working copy. + +Running a Sample +---------------- + +FINISHME - AmbrosiaDocs.md content will move here!! diff --git a/ValidatePreCommit.ps1 b/ValidatePreCommit.ps1 deleted file mode 100644 index 0c17ef2..0000000 --- a/ValidatePreCommit.ps1 +++ /dev/null @@ -1,178 +0,0 @@ -# A basic sanity check before committing. -# Sets up a performance test between processes on the local machine. -# Requires an internet connection for accessing Azure Table Storage. - -param ( - # The below switch enables super-aggressive cleaning with - # "git clean". Be careful you don't have un-added files! - [switch]$distclean = $false, - - # Skip the build phase if you want to go right to testing. - [switch]$nobuild = $false, - - # Skip the removing of previous service registrations from Azure table. - [switch]$noremove = $false, - - # Skip removing and adding stuff to Azure, - [switch]$noupload = $false, - - # Skip the test-running phase, meaning this is only useful for - # setting variables, registering and uploading, and printing hints. - [switch]$notest = $false, - - # Be chatty - [switch]$debug = $false - -) - -$ErrorActionPreference = "Stop" - -# Configuration -# ------------------------------------------------------------ - -# Binary locations: -# The fact that we're using TargetFrameworkVersion 4.6.1 is specified in the csproj files: -$localruntime = "Ambrosia\bin\x64\Release\net46\LocalAmbrosiaRuntime.exe" -$craworker = "ImmortalCoordinator\bin\x64\Release\net46\ImmortalCoordinator.exe" -$removeservice = "Tools\RemoveService\bin\Release\RemoveService.exe" - -# Allow the caller to provide their own AZURE_STORAGE_CONN_STRING if desired: -if (! $env:AZURE_STORAGE_CONN_STRING) { - Throw "ERROR: env var AZURE_STORAGE_CONN_STRING must be set." -} else { - Write-Host "Heads up: using this value of AZURE_STORAGE_CONN_STRING:" - Write-Host "$env:AZURE_STORAGE_CONN_STRING" -} - -Write-Host "Using Azure Storage connection string: $env:AZURE_STORAGE_CONN_STRING" - -# Set a suffix to avoid interfering with other runs: -# $UNIQ = (Get-Date -UFormat "%s_") + (Get-Random) -# $UNIQ = (Get-Date -UFormat "_%s") -# $UNIQ = (Get-Random).ToString() -$UNIQ = "" -Write-Host "Using Uniq suffix: $UNIQ" - -if ($debug) { Set-PSDebug -Strict -Trace 1 } - -# TODO: Add unique suffix: -# $jobservice = "validateJobServ" -# $jobinstance = "validateJobInst" -# $serverservice = "validateServServ" -# $serverinstance = "validateServInst" -# $binstag = "validateBins" - -$jobservice = "rrnjob$UNIQ" -$serverservice = "rrnserver$UNIQ" -$jobinstance = "$jobservice" -$serverinstance = "$serverservice" -#$jobinstance = "rnjobmachine" -#$serverinstance = "rnservermachine" -$binsserver = "rrnbinaries$UNIQ" -$binsjob = "rrnbinaries$UNIQ" - -# TODO: make this cross platform: -$logsdir = "c:\logs\validate_$UNIQ\" -# $logsdir = "./logs" - - -# Execution -# ------------------------------------------------------------ - -# (0) -Write-Host "First, blow away logs that might interfere with our test." -if (Test-Path $logsdir) { rm $logsdir -r -fo } -New-Item -ItemType Directory -Force -Path $logsdir | Out-Null - -# (1) Clean if needed. -# (2) Build everything we need for the tests. -if ($nobuild) { - Write-Host "Skipping build, straight to testing..." -} else { - if ($distclean) { - git clean -fxd - { cd deps\CRA; git clean -fxd } - .\CmdLine-FreshBuild.ps1 -noclean - } else { - .\CmdLine-FreshBuild.ps1 - } -} - -# (3) Remove services from any previous run. This had better be idempotent. -if (-not ($noremove)) { - Write-Host "Now we require internet access to remove previously registered services" - & $removeservice $jobservice $jobinstance - & $removeservice $serverservice $serverinstance - Write-Host "Validate job/server unregistered." -} - -$ptestdir = "InternalImmortals\PerformanceTestInterruptible" - - -# RRN: FIXME - looks like the CLI has changed for LAR: -# Ambrosiaruntime -# OR -# Ambrosiaruntime - -Write-Host "To reproduce the below test by hand, run the following commands:" -Write-Host " (C1) $localruntime $serverinstance 2000 2001 $serverservice $logsdir $binsserver a n y 1000 n 0 0" -Write-Host " (C2) $localruntime $jobinstance 1000 1001 $jobservice $logsdir $binsjob a n y 1000 n 0 0" -Write-Host " (C3) $craworker $jobinstance 1500" -Write-Host " (C4) $craworker $serverinstance 2500" -Write-Host " (C5) .\$ptestdir\Client\bin\x64\Release\net46\Job.exe 1001 1000 $jobservice $serverservice" -Write-Host " (C6) .\$ptestdir\Server\bin\x64\Release\net46\Server.exe 2001 2000 $jobservice $serverservice" - -# (4) -if (-not ($noupload)) { - Write-Host "Now register and upload binaries (C1,C2)" - Start-Process -NoNewWindow -Wait -FilePath "$localruntime" -ArgumentList "$jobinstance 1000 1001 $jobservice $logsdir $binsjob a n y 1000 n 0 0" - Start-Process -NoNewWindow -Wait -FilePath "$localruntime" -ArgumentList "$serverinstance 2000 2001 $serverservice $logsdir $binsserver a n y 1000 n 0 0" -} -# You can check that the above worked like so: -# az storage table list --connection-string $AZURE_STORAGE_CONN_STRING --output table | grep $UNIQ -# (This takes many seconds for me. -RRN) - -if (-not ($notest)) -{ - # (5) - Write-Host "Bring up the CRA worker processes (C3,C4)" - $c1 = Start-Process -NoNewWindow -PassThru -FilePath $craworker -ArgumentList "$jobinstance 1500"; - $c2 = Start-Process -NoNewWindow -PassThru -FilePath $craworker -ArgumentList "$serverinstance 2500" - # RN: ^ oddity, if I allow NewWindow then an extra process gets thrown - # in the middle that interferes with things. - - # TODO: could read CRA worker outputs until they say "Ready". - Start-Sleep -Seconds 20 - - # (5B) Bring up the Ambrosia service processes and test. - # $env:AMBROSIA_JOB_INSTANCE = $jobinstance - # $env:AMBROSIA_SERVER_INSTANCE = $serverinstance - $p1 = Start-Process -NoNewWindow -PassThru -WorkingDirectory "$ptestdir\Client" -FilePath "$ptestdir\Client\bin\x64\Release\net46\Job.exe" -ArgumentList "1001 1000 $jobservice $serverservice" - $p2 = Start-Process -NoNewWindow -PassThru -WorkingDirectory "$ptestdir\Server" -FilePath "$ptestdir\Server\bin\x64\Release\net46\Server.exe" -ArgumentList "2001 2000 $jobservice $serverservice" - - $c1.WaitForExit() - $c2.WaitForExit() - $p1.WaitForExit() - $p2.WaitForExit() - # Write-Host "Exit codes: " $c1.ExitCode $c2.ExitCode - # if ( $c1.ExitCode -ne 0 ) { - # Throw Format("{0} child process exited with return code: {1}", $c1.ProcessName, $c1.ExitCode) - # } - - Write-Host "Passed rudimentary testing." -} - -# (6) Repeat step (3) from above. -if (-not ($noremove)) { - Write-Host "Cleanup: Finally, remove services we just uploaded to avoid pollution." - & $removeservice $jobservice $jobinstance - & $removeservice $serverservice $serverinstance - Write-Host "Validate job/server unregistered." -} - -Write-Host "Validate script finished." - -if ($debug) { Set-PSDebug -Trace 0 } - - -# Ambrosia\bin\x64\Release\net46\LocalAmbrosiaRuntime.exe validateServInst 2000 2001 validateServServ c:\logs\ validateBins a n y 1000 n 0 0 diff --git a/docs/AmbrosiaDocs.md b/docs/AmbrosiaDocs.md index 690cc8e..592d2a3 100644 --- a/docs/AmbrosiaDocs.md +++ b/docs/AmbrosiaDocs.md @@ -37,6 +37,8 @@ RPCs. ## Requirements/Platforms supported for development and deployment +FINISHME - + ## Setup How to get set up to use Ambrosia @@ -118,6 +120,8 @@ open a PowerShell prompt and enter the following commands: .\Generate-Assemblies-NetCore.ps1 ## Developing with Ambrosia +FINISHME - + small snippet example of how to send messages through ambrosia how to handle impulses recovery, active active @@ -126,19 +130,23 @@ time travel debugging how to portability ## Deployment of services built on Ambrosia -Windows, Linux +FINISHME - Windows, Linux ## Best Practices -how to ensure you get all the guarantees that Ambrosia provides +FINISHME - how to ensure you get all the guarantees that Ambrosia provides ## Examples + +FINISHME - Brief description of each example and what it demonstrates instructions for cloning/downloading and modifying examples ## References -configurations, command line options, parameters +FINISHME - configurations, command line options, parameters ## Roadmap + +FINISHME - planned future features other language support support non-Azure storage