Merged PR 227890: Spec for @napajs/lock, @napajs/memory and @napajs/config

Spec for lock, memory and config
2017-04-05 23:02:13 +00:00 · 2017-04-05 23:02:13 +00:00 · 2fe725a590
--- a/docs/modules/napa-module-index.md
+++ b/docs/modules/napa-module-index.md
@ -1,24 +1,36 @@
 # Index of Modules@napajs

-## Napa.JS fundementals
- [@napajs/logger](../../modules/@napajs/logger/README.md) : Logging API for Napa.JS. 
- [@napajs/metrics](./@napajs/metrics/README.md) : Metric API for Napa.JS.
- [@napajs/shared-depot](): Sharing native objects across V8 isolates.
- [@napajs/multi-task](): (TBD) High level API for multi-tasks.
- [@napajs/vineyard](./@napajs/vineyard/README.md): Application framework to build highly iterative applications.
+## Napa.JS foundation modules
+- [napajs](../../modules/napajs/README.md) : A multi-thread JavaScript runtime on Node.
+- [@napajs/config](../../modules/@napajs/config/README.md) : Utilities to read configuration files.
+- [@napajs/lock](../../modules/@napajs/lock/README.md) : Synchronization between multiple isolates in Napa.JS
+- [@napajs/logger](../../modules/@napajs/logger/README.md) : Access logging in Napa.JS. 
+- [@napajs/memory](../../modules/@napajs/memory/README.md): Handling memories in Napa.JS.
+- [@napajs/metrics](../../modules/@napajs/metrics/README.md) : Access metrics in Napa.JS.
+- [@napajs/workflow](../../modules/@napajs/workflow/README.md) : (TBD) Flow control in Napa.JS

-## General purpose modules
- [@napajs/edge](): Interop with C# libraries
+## Public
+### General purpose framework
+- [edge](): Interop with C# libraries
+- [storage-client](): Common API to access key/value storage.
+- [winery](./@napajs/vineyard/README.md): Application framework to build highly iterative applications.

-## General machine learning
- [@napajs/feature-evaluation](): Feature evaluation library for machine learning 
- [@napajs/tf-serve](): (TBD) Tensorflow serving library
+### General machine learning
+- [expression-compiler](): Compiler for generating execution sequence for a list of expressions.
+- [tf-serve](): (TBD) Tensorflow serving library

 ## Microsoft internal
- [@napajs/ms-autopilot](): Common Autopilot APIs.
- [@napajs/ms-bing-query](): Bing query parser and query object.
- [@napajs/ms-bing-features](): Bing ranking features.
- [@napajs/ms-bing-wordbreaker](): Bing word breaker.
- [@napajs/ms-storage-client](): General API to access KV stores.
- [@napajs/ms-object-store](): (TBD) API to access object store. 
- [@napajs/ms-sonoma](): Abstraction layer to serve Tensorflow and CNTK models.
+- [@ms/autopilot](): Common Autopilot APIs.
+- [@ms/cosmos-client](): API to access cosmos.
+- [@ms/object-store-client](): Access object stores.
+- [@ms/sonoma](): Abstraction layer to serve Tensorflow and CNTK models.
+- [@ms/storage-client](): General API to access KV stores.
+
+## Bing internal
+- [@bing/features](): Bing ranking features.
+- [@bing/knowledge-models](): Bing knowledge models
+- [@bing/query](): Bing query parser and query object.
+- [@bing/query-rewrite](): (TBD) Bing query rewrite.
+- [@bing/trie](): Read and write trie files.
+- [@bing/tree-ensemble](): Boost gradient decision tree ensemble used in Bing.
+- [@bing/word-breaker](): Bing word breaker.
--- a/modules/@napajs/config/README.md
+++ b/modules/@napajs/config/README.md
@ -0,0 +1,102 @@
+# @napajs/config
+
+A facade module to access configuration objects. There are 3 aspects of configuration reading:
+1) File locator
+
+## API
+
+```ts
+/// <summary> Function interface for file locator. </summary>
+export interface FileLocator {
+    (configName: string): string
+}
+
+/// <summary> Function interface for config parser. </summary>
+export interface Parser {
+    parse(fileName: string): any;
+}
+
+/// <summary> Class for config object schema. </summary>
+export declare class Schema {
+    /// <summary> Construct a schema object. </summary>
+    /// <param name="jsonSchemaFile"> JSON schema describe the final object read from file. This schema is not coupled with config format.</param>
+    constructor(jsonSchemaFile: string);
+
+    /// <summary> Validate an object against the schema and fill default values. </summary>
+    /// <param name="jsValue"> A Javascript value. </param>
+    validate(jsValue: any): boolean;
+}
+
+/// <summary> Set a custom parser for a file extension. 
+/// If multiple calls are made to setParser on a file extension, the last one will be used.
+/// </summary>
+/// <param name="fileExtension"> File extension name without '.' . </param>
+/// <param name="parser"> A parser function for the file extension . </param>
+export function setParser(fileExtension: string, parser: Parser): void;
+
+/// <summary> Set a file locator for resolving config file. 
+/// If multiple calls are made to setLocator, the last one will be used.
+/// </summary>
+/// <param name="locator"> A file locator function. </param>
+export function setLocator(locator: FileLocator): void;
+
+/// <summary> Read a config object from a config identifier. Validate against schema and fill default values if provided. </summary>
+/// <param name="configId"> An identifier of config that will be passed to FileLocator to resolve into a file path. </param>
+/// <param name="schema"> (optional) schema object to validate config and fill default values. </schema>
+export function read(configId:string, schema?: Schema): any;
+
+/// <summary> Namespace for default facilities. </summary>
+namespace defaults {
+    /// <summary> Default file locator, which resolves the configName as file name in working directory. </summary>
+    export var FileLocator;
+
+    /// <summary> Default Ini parser. </summary>
+    export var IniParser;
+
+    /// <summary> Default Xml parser. </summary>
+    export var XmlParser;
+
+    /// <summary> Default Json parser, which is JSON.parse. </summary>
+    export var JsonParser;
+}
+
+```
+## Usage
+
+Set up custom locator and parser.
+
+```ts
+import * as config from '@napajs/config';
+import * as path from 'path';
+import * as fs from 'fs';
+
+config.setLocator((configName: string): string => {
+    let extensions = ['json', 'xml', 'ini'];
+    for (let ext of extensions) {
+        let filePath = path.resolve(process.cwd(), `${configName}.${ext}`);
+        if (fs.existsSync(filePath)) {
+            return filePath;
+        }
+    }
+    throw new Error("File doesn't exist for config: " + configName);
+});
+
+config.setParser('json', (fileName: string): any => {
+    return JSON.parse(fileName);
+});
+
+```
+Parse config.
+```ts
+
+// Read test.json, or test.xml or test.ini if exists.
+// test.json:
+// { "threshold": 1 }
+// test-schema.json has default value for 'enabled': true.
+let schema = new Schema('test-schema.json')
+let testConfig = config.read('test', schema);
+
+// returns {"threshold": 1, "enabled": true }
+console.log(testConfig);
+
+```
--- a/modules/@napajs/lock/README.md
+++ b/modules/@napajs/lock/README.md
@ -0,0 +1,89 @@
+# @napajs/lock - Lock for thread synchronization under NapaJS
+
+
+## API
+### Simple Lock
+#### API
+```ts
+guard(lockName: string, func: {(): any}): any;
+```
+Guard execution of a function with a lock name. Return value from the function will be relayed.
+Exception thrown in the function will be propagate to external.
+
+#### Example
+```ts
+import * as lock from "@napajs/locks"
+import * as assert from "assert"
+
+{
+    let returnValue = lock.guard('lockName', () => {
+        // Do something here.
+        return [1, 2, 3];
+    });
+
+    assert.deepEqual(returnValue, [1, 2, 3]);
+}
+{
+    try {
+        lock.guard('lockName', () => {
+            // Exception is thrown.
+            throw new Error();
+        });
+    }
+    catch (e) {
+        console.log(e);
+    }
+}
+
+```
+### Read-write lock
+#### API
+```ts
+guardRead(lockName: string, func: {(): any}): any;
+```
+Guard execution of a function as a reader. That is, multiple readers can obtain the lock at the same time without blocking, if lock is not obtained by the writer. 
+
+Return value from the function will be relayed. Exception thrown in the function will be propagate to external.
+
+```ts
+guardWrite(lockName: string, func: {(): any}): any;
+```
+Guard execution of a function as a writer. That is, only one writer can obtain the lock, all other readers and writers will wait. 
+
+Return value from the function will be relayed. Exception thrown in the function will be propagate to external.
+
+#### Examples
+
+```ts
+import * as lock from "@napajs/locks"
+import * as assert from "assert"
+
+{
+    let returnValue = lock.guardRead('lockName', () => {
+        // Do read here.
+        return "hello-world";
+    });
+    assert.equal(returnValue, "hello-world");
+}
+
+// Handle exceptions thrown from input functions.
+{
+    let succeeded: boolean = false;
+    
+    // Do some work...
+    try {
+        let returnValue = lock.guardWrite('lockName', () => {
+            // Do write here.
+            if (!succeeded)
+                throw new Error();
+
+            return 123;
+        });
+        assert.equal(returnValue, 123);
+    }
+    catch (e) {
+        console.log(e);
+    }
+}
+```
+
--- a/modules/@napajs/memory/README.md
+++ b/modules/@napajs/memory/README.md
@ -0,0 +1,252 @@
+# @napajs/memory: Handling memory in NapaJS
+
+## Working with native objects
+
+Interface for NativeObject, as a convention, to enable us to provide common facilities for native objects. A readonly property `handle` is the only obligation for now.
+
+```ts
+/// <summary> Handle defines minimum information to access a native object. 
+/// Property 'type' is a string to identify the object type, which can be used for debugging and type checking in JavaScript.
+/// Property 'pointer' represents a 64-bit pointer in a 2-element number array, pointer[0] is the higher 32-bit, and pointer[1] is the lower 32-bit.
+/// </summary>
+type Handle = { type: string, pointer: [number, number]};
+
+/// <summary> Interface for all native object. </summary>
+interface NativeObject {
+    readonly handle: Handle;
+}
+
+```
+## Sharing object across isolates
+- Provide facilities for sharing objects across isolates at the global scope or a local scope (like within a request).
+- Support both plain JS objects and wrappers for native objects
+
+### API
+```ts
+/// <summary> Barrel defines a scope for sharing objects. </summary>
+declare class Barrel {
+    /// <summary> Get id for current barrel. </summary>
+    readonly id: string;
+
+    /// <summary> Set a shared object. </summary>
+    set(key: string, value: any);
+
+    /// <summary> Get a shared object by key. </summary>
+    get(key: string): any;
+
+    /// <summary> Check if a key exists. </summary>
+    exists(key: string): boolean;
+
+    /// <summary> Remove a shared object by key. </summary>
+    remove(key: string): void;
+
+    /// <summary> Release a barrel. </summary>
+    release(): void;
+
+    /// <summary> Get number of objects shared in current barrel. </summary>
+    count(): number:
+}
+
+/// <summary> Create a new barrel for sharing. </summary>
+declare function createBarrel() : Barrel;
+
+/// <summary> Find an existing barrel by ID </summary>
+/// <returns> A Barrel object, if not found, exception will be thrown. </returns>
+declare function findBarrel(id) : Barrel;
+
+/// <summary> Returns number of barrels. </summary>
+declare function barrelCount(): number;
+
+/// <summary> Barrel for sharing at global namespace. </summary>
+declare var global: Barrel;
+
+```
+### Examples:
+#### Sharing JS objects at global namespace.
+
+```ts
+let mem = require('@napajs/memory');
+
+mem.global.set('sample', {
+    foo: 1,
+    bar: "hello world"
+});
+
+/// In another isolate.
+assert(mem.global.exists('sample'));
+let object = mem.global.get('sample');
+assert.deepEqual(object, {
+    foo: 1,
+    bar: "hello world"
+})
+```
+#### Sharing JS objects at request level.
+```ts
+let mem = require('@napajs/memory');
+
+let barrel = mem.createBarrel());
+
+barrel.set('sample', {
+    foo: 1,
+    bar: "hello world"
+});
+
+/// In another isolate.
+declare var id: string;
+let barrel = mem.findBarrel(id);
+assert(barrel.exists('sample'));
+let object = barrel.get('sample');
+assert.deepEqual(object, {
+    foo: 1,
+    bar: "hello world"
+})
+
+// Release barrel at end of request.
+barrel.release();
+```
+#### Sharing native objects
+TODO: move shared-depot examples here.
+
+## Allocator support
+- Create customized allocator
+- Pass allocators across isolates
+- Consume allocator in C++
+- Consume allocator in JavaScript (optional)
+
+### API
+#### JS interface (Optional)
+```ts
+/// <summary> 
+interface Allocator : NativeObject {
+    /// <summary> Allocate memory of requested size. </summary>
+    allocate(size: number): Buffer
+
+    /// <summary> Free allocated memory. </summary>
+    free(buffer: Buffer): void;
+
+    /// <summary> Handle of this allocator. </summary>
+    readonly handle: Handle;
+
+    /// <summary> Release current allocator. </summary>
+    release(): void;
+}
+
+interface AllocatorFactory {
+    create(allocatorType: string): Allocator;
+}
+
+/// <summary> add an allocator factory. </summary>
+declare function registerAllocatorFactory(factory: AllocatorFactory);
+
+/// <summary> create a new allocator instance of a type. </summary>
+declare function createAllocator(allocatorType: string): Allocator;
+
+```
+
+#### CPP interfaces
+
+`Header`
+```cpp
+namespace napa {
+    namespace memory {
+        /// <summary> Interface for allocator factory. </summary>
+        class AllocatorFactory {
+            // ...
+            public:
+                Allocator* create(const char* allocatorType) = 0;
+        };
+        
+        /// <summary> Concrete allocator class. 
+        /// which will bind to different implementations.
+        /// </summary>
+        class Allocator {
+            public:
+                Allocator(AllocatorImpl* impl);
+
+                void* allocate(size_t);
+
+                void deallocate(void*);
+        };
+
+        /// <summary> Stl-style allocator adaptor from allocator. </summary>
+        template <typename T>
+        class StlAllocator {
+            public:
+                StlAllocator(Allocator* allocator);
+
+                T* allocate(std::size_t n, const void *hint);
+
+                void deallocate(T* pointer, std::size_t n);
+        };
+    }
+}
+
+```
+`Implementation`
+```cpp
+
+/// <summary> This is defined in CPP. </summary>
+struct AllocatorImpl {
+    typedef void* (*AllocateFunction)(size_t);
+    typedef void (*DeallocateFunction)(void*);
+    typedef void (*DestructorFunction)(void*);
+
+    void* allocatorHandle;
+
+    AllocateFunction allocate;
+    DeallocateFunction deallocate;
+    DestructorFunction destruct;
+};
+
+/// <summary> Allocator factory that loads allocator from Dll. </summary>
+class DllAllocatorFactory: public AllocatorFactory {
+    // ...
+    public:
+        DllAllocatorFactory(const char* dllFileName);
+
+        Allocator* create(const char* allocatorType);
+};
+
+```
+`Dll exported functions`
+```cpp
+
+/// <summary> Create an allocator instance. </summary>
+void* CreateAllocator(const char* allocatorType);
+
+/// <summary> Allocate memory from an allocator. </summary>
+void* Allocate(void* allocator, size_t size);
+
+/// <summary> Free memory from an allocator. </summary>
+void Deallocate(void* allocator, HANDLE memory);
+
+/// <summary> Release allocator. </summary>
+void ReleaseAllocator(void* allocator);
+```
+
+#### Built-in allocators
+- StdAllocator
+
+MS internal implementations
+- Arena
+- ThreadLocalAllocator
+
+#### Stl containers with allocator
+
+```cpp
+namespace nape {
+    template <typename T>
+    using Vector = std::vector<T, napa::memory::StlAllocator<T>>;
+
+    template <typename Key, typename T, typename Compare = std::less<Key> >
+    using Map = std::map<Key, T, Compare, napa::memory::StlAllocator<std::pair<const Key, T>>>;
+
+    template <typename Key, typename T, typename Compare = std::less<Key> >
+    using UnorderedMap = std::unordered_map<Key, T, Compare, napa::memory::StlAllocator<std::pair<const Key, T>>>;
+
+    /// More STL containers ...
+}
+
+```
+
+*End of Document*