Cast the Hashtable returned from JsonObject.CovnertFromJson to a case-insensitive Hashtable as long as it's doable.
If the cast fails, because of keys differ only in case, then we use the original case-sensitive Hashtable.
Major changes are as follows:
- Add `Utils.TransformInBindingValueAsNeeded` and `Utils.TransformOutBindingValueAsNeeded` for additional data transformation before passing input data in as argument and after retrieving output data back.
- Update `AzFunctionInfo` to capture the static types of script parameters. The parameter types can be used as hints for the additional data transformation.
- Update `PowerShellManager` to call `Utils.TransformInBindingValueAsNeeded` before passing argument in; update `RequestProcessor` to call `Utils.TransformOutBindingValueAsNeeded` before sending out the output data.
- Update `PowerShellWorker.psm1` to remove `Convert-OutputBindingValue`. That logic is moved to `Utils.TransformOutBindingValueAsNeeded`. The reason to move the transformation to C# code is:
- A script transform function will always unroll the collection value passed to the function when returning the value, which will result in losing the actual type of the collection value. For example, if a byte array is passed to `Convert-OutputBindingValue`, after `return $Value`, the caller will get back an object array instead. `Write-Output -NoEnumerate` doesn't work either -- it returns an array of `PSObject` even though `-NoEnumerate` is specified. In order to preserve the original type of a collection value, I have to implement the transformation function either as a PowerShell class method, or in C# directly. I think the latter is more desirable.
- Keep consistent with `Utils.TransformInBindingValueAsNeeded`
- Remove `FunctionMetadata.cs`. The `FunctionMetadata` type was added to support doing data transformation on output value within the script. Now that we move that operation in C#, this type is no longer needed. It could be useful in future, but I don't intend to keep an unused data structure around. If we need it in future, we can always add it back since there is no much code about it.
The major changes are:
- Use `PSThreadOptions.ReuseThread` for the `InitialSessionState` when creating `Runspace`, so that every `PowerShellManager` only creates one thread and then reuse it afterwards. The default behavior is to create a new thread every time `PowerShell.Invoke` is called.
- Update `RequestProcessor` to process `InvocationRequest` in asynchronously via tasks.
- Implement `PowerShellManagerPool` using `BlockingCollection`
- make upper bound of the pool configurable via an environment variable `PSWorkerInProcConcurrencyUpperBound`
- make the pool able to expand in a lazy way
- checkout `PowerShellManager` via `CheckoutIdleWorker` on the main thread. Once getting an idle instance back, the main thread will queue a task to process an invocation request on a thread-pool thread and forget about it -- the main thread then can go ahead to process the next message.
- Update the `RpcLogger` and make every `PowerShellManager` have its own logger instance.
- also update the way to set the `RequestId` and `InvocationId` for logger. The original way to setup the context only works for single-thread design.
- Update `MessagingStream` to use a `BlockingCollection` to hold all messages that are about to be written out, then use a single thread-pool thread to take out items and write them to the gRPC channel.
- currently, the way we write out response/log messages is completely task-based/async, using a semaphore for synchronization. However, this approach doesn't guarantee the order of the message.
- this is because there could be multiple tasks blocked on the semaphore, and releasing the semaphore allows a blocked task to enter the semaphore, but there is no guaranteed order, such as first-in-first-out, for blocked threads to enter the semaphore.
- so, the unblocked task could be a random one, and thus change the arrival order of the message when writing the message to the gRPC channel.
- Remove the two system logging we have in our worker, because they drastically worsen the processing time per an invocation request when there are a lot in-coming invocation requests.
- the logging for "TriggerMetadata" parameter is not that useful, and should be removed
- the execution time logging is good to have, but not necessary, especially when it impact the throughput.
Formatting tables is different in pwsh 6.2-preview.3. We can no longer be sure that the last 2 items in a format-table string are new lines.
Now we walk backwards until we hit the last not white-space line.
Refactor the code base to make it easy to support concurrency within a worker in future.
- Add `PowerShellManagerPool` , but it's just the skeleton. Today the pool only has one PowerShellManager instance. We can add the real pool implementation if we decide to support concurrency within a worker process.
- Setup the `PSModule` environment variable as part of `Runspace.Open` by using `InitialSessionState.EnvironmentVariables`. This fixes the mysterious `ModulePathShouldBeSetCorrectly` test failure by making sure the env variable is set to the expected one as part of the `Runspace.Open` of every `PowerShellManager`. It failed before because:
1. `dotnet test` runs tests in parallel by default
2. when creating a new `PowerShellManager`, the `PSModulePath` environment variable will be set again within `Runspace.Open` (by `SetModulePath` called from `ModuleIntrisic` constructor). That makes value unexpected.
- Update tests to make them more reliable.
- Refactor the code to do initialization in a more organized way.
- Pre-validate the script against the function metadata when loading a 'FunctionLoadRequest'.
- Support `EntryPoint` only when `ScriptFile` points to a `.psm1`; support `.psm1` script file only when `EntryPoint` is specified.
- Add unit tests and e2e tests
In order to trace the pipeline output in a streaming way, the `DataAdding` event won't work, because even though you can get the pipeline object right after it's inserted into the pipeline, you will have to call `Out-String` to get the formatted output. You cannot call `Out-String` within the same Runspace as it's busying running the user's function script, so you will have to create a separate Runspace just for this purpose. This won't work given the concurrency support PS worker needs to have.
The approach I choose is to have a special cmdlet `Trace-PipelineObject` behaves like `Tee-Object`. For each object from the pipeline, `Trace-PipelineObject` pushes the object through a `SteppablePipeline` that consists of `Out-String -Stream | Write-Information -Tags "__PipelineObject__"`, and then writes the object back to pipeline. By using this `SteppablePipeline`, every pipeline object is properly formatted and the resulted strings are written to the information stream.
To properly convert a value for a specific output binding name implicitly, we need to know the metadata of the binding referred by that name, such as the type of the output binding.
The changes in this PR register the output binding metadata to a concurrent dictionary with the Runspace's InstanceId as the key before start running a function script, and then unregister after the function script finishes running. When a function script that calls `Push-OutputBinding` is running, we can figure out the current Runspace by `Runspace.DefaultRunspace`, and then we can query for the output binding metadata by using the InstanceId.
This approach take into the concurrency support that we will have in future. `Push-OutputBinding` is able to get the metadata about the function script that is running even with multiple Runspaces/PowerShellManagers processing requests at the same time.
* Make 'StatusCode' accept status code in the form of integer, string, and enum
* Update the property to type 'HttpStatusCode'
* Update the E2E test to cover different forms of status code
* Use status code 202 for success case to make sure status code is set
* Minor fix in test
* first e2e test
* conditionally chmod +x
* actually run the tests :)
* get job differently
* don't fail if we cant get the job
* honor configuration
* handle env:CONFIGURATION not set
* address Pragna's feedback
* add TestFunctionApp change
* elaborate test name
* add additional test cases
* typo
* add trigger metadata tests and Error test
* new lines
* address Steve's feedback
* address Steve's feedback
* nit extra line
There are 2 changes:
1. Use `JsonObject.ConvertFromJson` to serialize JSON binding input. We have to use reflection because the reference assemblies are missing in our SDK NuGet packages.
2. Check for `PSObject` in our `ToTypedData` method, so that it handles wrapped objects properly.
When the return/output object is a complex one, the powershell worker will freeze when converting it to JSON due to `-Depth 10` in the call to `ConvertTo-Json`. Talked to @eamonoreilly and decided to change to 3, which should be deep enough for most real scenarios.
- Use `ConvertTo-Json` for converting the results returned from the function to JSON. So objects that are wrapped in `PSObject` won't be a problem anymore.
- Refactor the code to make it ready for durable function implementation. The implementation of invoking an orchestration function is made a placeholder in this PR, where it throws `NotImplementedException` when an orchestration function is triggered.
See `RequestProcessor.InvokeOrchestrationFunction`.
* initial refactor of logger
* misc
* add PowerShellManagerTests
* newline
* use test data and move RpcLogger to logger folder with ILogger
* add copyright
* add appveyor
* add PSDepend and hard code configuration for now
* Scope CurrentUser
* no .
* upload Pester tests
* configuration
* CONFIGURATION
* $(configuration)
* * instead of env var and ErrorActionPref
* more spaces
* explictly say paths and remove bogus failing test
* add logic to grab dependencies
* move psdepends logic
* PSDepends is not a required dependency
* move comment
* actually include install deps script
* fix path
* commit handle Azure authentication
* move WriteAsync out of using
* authenticate to azure per-request and address feedback
* moved Auth to Azure in Worker Init for now
Dongbo Wang
SatishRanjan
Francisco Gamino
Tyler James Leonhardt
Pragna Gopa