diff --git a/CHANGES.rst b/CHANGES.rst index 3e94dc44..76aa4829 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -85,7 +85,7 @@ Version 0.4.0 ------------- - Add `data` package that holds all data-related codes - Reform the data provider structure -- Create a server for data centralized management `qlib-server`_ +- Create a server for data centralized management `qlib-server `_ - Add a `ClientProvider` to work with server - Add a pluggable cache mechanism - Add a recursive backtracking algorithm to inspect the furthest reference date for an expression @@ -166,12 +166,12 @@ Version 0.8.0 - Nested decision execution framework is supported - There are lots of changes for daily trading, it is hard to list all of them. But a few important changes could be noticed - The trading limitation is more accurate; - - In `previous version `_, longing and shorting actions share the same action. - - In `current version `_, the trading limitation is different between logging and shorting action. + - In `previous version `__, longing and shorting actions share the same action. + - In `current version `__, the trading limitation is different between logging and shorting action. - The constant is different when calculating annualized metrics. - - `Current version `_ uses more accurate constant than `previous version `_ - - `A new version `_ of data is released. Due to the unstability of Yahoo data source, the data may be different after downloading data again. - - Users could check out the backtesting results between `Current version `_ and `previous version `_ + - `Current version `_ uses more accurate constant than `previous version `__ + - `A new version `__ of data is released. Due to the unstability of Yahoo data source, the data may be different after downloading data again. + - Users could check out the backtesting results between `Current version `__ and `previous version `__ Other Versions diff --git a/docs/advanced/alpha.rst b/docs/advanced/alpha.rst index 797eb19d..88d65074 100644 --- a/docs/advanced/alpha.rst +++ b/docs/advanced/alpha.rst @@ -38,7 +38,7 @@ Example DIF = \frac{EMA(CLOSE, 12) - EMA(CLOSE, 26)}{CLOSE} - `DEA`means a 9-period EMA of the DIF. + `DEA` means a 9-period EMA of the DIF. .. math:: diff --git a/docs/advanced/task_management.rst b/docs/advanced/task_management.rst index d45c7b97..70b6bcfc 100644 --- a/docs/advanced/task_management.rst +++ b/docs/advanced/task_management.rst @@ -18,7 +18,7 @@ With this module, users can run their ``task`` automatically at different period This whole process can be used in `Online Serving <../component/online.html>`_. -An example of the entire process is shown `here `_. +An example of the entire process is shown `here `__. Task Generating =============== @@ -33,7 +33,7 @@ Here is the base class of ``TaskGen``: :members: ``Qlib`` provides a class `RollingGen `_ to generate a list of ``task`` of the dataset in different date segments. -This class allows users to verify the effect of data from different periods on the model in one experiment. More information is `here <../reference/api.html#TaskGen>`_. +This class allows users to verify the effect of data from different periods on the model in one experiment. More information is `here <../reference/api.html#TaskGen>`__. Task Storing ============ @@ -54,7 +54,7 @@ Users need to provide the MongoDB URL and database name for using ``TaskManager` .. autoclass:: qlib.workflow.task.manage.TaskManager :members: -More information of ``Task Manager`` can be found in `here <../reference/api.html#TaskManager>`_. +More information of ``Task Manager`` can be found in `here <../reference/api.html#TaskManager>`__. Task Training ============= diff --git a/docs/component/data.rst b/docs/component/data.rst index b8279432..d3b8cafe 100644 --- a/docs/component/data.rst +++ b/docs/component/data.rst @@ -24,8 +24,8 @@ The introduction of ``Data Layer`` includes the following parts. Here is a typical example of Qlib data workflow - Users download data and converting data into Qlib format(with filename suffix `.bin`). In this step, typically only some basic data are stored on disk(such as OHLCV). -- Creating some basic features based on Qlib's expression Engine(e.g. "Ref($close, 60) / $close", the return of last 60 trading days). Supported operators in the expression engine can be found `here `_. This step is typically implemented in Qlib's `Data Loader `_ which is a component of `Data Handler `_ . -- If users require more complicated data processing (e.g. data normalization), `Data Handler `_ support user-customized processors to process data(some predefined processors can be found `here `_). The processors are different from operators in expression engine. It is designed for some complicated data processing methods which is hard to supported in operators in expression engine. +- Creating some basic features based on Qlib's expression Engine(e.g. "Ref($close, 60) / $close", the return of last 60 trading days). Supported operators in the expression engine can be found `here `__. This step is typically implemented in Qlib's `Data Loader `_ which is a component of `Data Handler `_ . +- If users require more complicated data processing (e.g. data normalization), `Data Handler `_ support user-customized processors to process data(some predefined processors can be found `here `__). The processors are different from operators in expression engine. It is designed for some complicated data processing methods which is hard to supported in operators in expression engine. - At last, `Dataset `_ is responsible to prepare model-specific dataset from the processed data of Data Handler Data Preparation @@ -37,7 +37,7 @@ Qlib Format Data We've specially designed a data structure to manage financial data, please refer to the `File storage design section in Qlib paper `_ for detailed information. Such data will be stored with filename suffix `.bin` (We'll call them `.bin` file, `.bin` format, or qlib format). `.bin` file is designed for scientific computing on finance data. -``Qlib`` provides two different off-the-shelf datasets, which can be accessed through this `link `_: +``Qlib`` provides two different off-the-shelf datasets, which can be accessed through this `link `__: ======================== ================= ================ Dataset US Market China Market @@ -47,7 +47,7 @@ Alpha360 √ √ Alpha158 √ √ ======================== ================= ================ -Also, ``Qlib`` provides a high-frequency dataset. Users can run a high-frequency dataset example through this `link `_. +Also, ``Qlib`` provides a high-frequency dataset. Users can run a high-frequency dataset example through this `link `__. Qlib Format Dataset ------------------- @@ -512,7 +512,7 @@ Data and Cache File Structure We've specially designed a file structure to manage data and cache, please refer to the `File storage design section in Qlib paper `_ for detailed information. The file structure of data and cache is listed as follows. -.. code-block:: json +.. code-block:: - data/ [raw data] updated by data providers diff --git a/docs/component/online.rst b/docs/component/online.rst index c72c77a8..351098db 100644 --- a/docs/component/online.rst +++ b/docs/component/online.rst @@ -1,4 +1,4 @@ -.. _online: +.. _online_serving: ============== Online Serving diff --git a/docs/component/report.rst b/docs/component/report.rst index 6ed87eef..30fca078 100644 --- a/docs/component/report.rst +++ b/docs/component/report.rst @@ -174,6 +174,7 @@ Graphical Result The `Information Ratio` without cost. - `excess_return_with_cost` The `Information Ratio` with cost. + To know more about `Information Ratio`, please refer to `Information Ratio – IR `_. - `max_drawdown` - `excess_return_without_cost` diff --git a/docs/component/rl/framework.rst b/docs/component/rl/framework.rst index 7edb08ef..a31cb1b4 100644 --- a/docs/component/rl/framework.rst +++ b/docs/component/rl/framework.rst @@ -28,7 +28,7 @@ In QlibRL, EnvWrapper is a subclass of gym.Env, so it implements all necessary i EnvWrapper will organically organize these components. Such decomposition allows for better flexibility in development. For example, if the developers want to train multiple types of policies in the same environment, they only need to design one simulator and design different state interpreters/action interpreters/reward functions for different types of policies. -QlibRL has well-defined base classes for all these 4 components. All the developers need to do is define their own components by inheriting the base classes and then implementing all interfaces required by the base classes. The API for the above base components can be found `here <../../reference/api.html#module-qlib.rl>`_. +QlibRL has well-defined base classes for all these 4 components. All the developers need to do is define their own components by inheriting the base classes and then implementing all interfaces required by the base classes. The API for the above base components can be found `here <../../reference/api.html#module-qlib.rl>`__. Policy ------------ @@ -42,4 +42,4 @@ As you may have noticed, a training vessel itself holds all the required compone With a training vessel, the trainer could finally launch the training pipeline by simple, Scikit-learn-like interfaces (i.e., ``trainer.fit()``). -The API for Trainer and TrainingVessel and can be found `here <../../reference/api.html#module-qlib.rl.trainer>`_. \ No newline at end of file +The API for Trainer and TrainingVessel and can be found `here <../../reference/api.html#module-qlib.rl.trainer>`__. \ No newline at end of file diff --git a/docs/component/strategy.rst b/docs/component/strategy.rst index 919551fb..910ebf70 100644 --- a/docs/component/strategy.rst +++ b/docs/component/strategy.rst @@ -80,6 +80,7 @@ TopkDropoutStrategy In most cases, ``TopkDrop`` algorithm sells and buys `Drop` stocks every trading day, which yields a turnover rate of 2$\times$`Drop`/$K$. The following images illustrate a typical scenario. + .. image:: ../_static/img/topk_drop.png :alt: Topk-Drop diff --git a/docs/conf.py b/docs/conf.py index a7147a96..442c89da 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -77,7 +77,7 @@ language = "en_US" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path -exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "hidden"] # The name of the Pygments (syntax highlighting) style to use. pygments_style = "sphinx" diff --git a/docs/developer/code_standard_and_dev_guide.rst b/docs/developer/code_standard_and_dev_guide.rst index ae5927c8..79a7778a 100644 --- a/docs/developer/code_standard_and_dev_guide.rst +++ b/docs/developer/code_standard_and_dev_guide.rst @@ -15,7 +15,8 @@ Continuous Integration (CI) tools help you stick to the quality standards by run When you submit a PR request, you can check whether your code passes the CI tests in the "check" section at the bottom of the web page. 1. Qlib will check the code format with black. The PR will raise error if your code does not align to the standard of Qlib(e.g. a common error is the mixed use of space and tab). - You can fix the bug by inputing the following code in the command line. + + You can fix the bug by inputing the following code in the command line. .. code-block:: bash @@ -32,7 +33,8 @@ When you submit a PR request, you can check whether your code passes the CI test 3. Qlib will check your code style flake8. The checking command is implemented in [github action workflow](https://github.com/microsoft/qlib/blob/0e8b94a552f1c457cfa6cd2c1bb3b87ebb3fb279/.github/workflows/test.yml#L73). - You can fix the bug by inputing the following code in the command line. + + You can fix the bug by inputing the following code in the command line. .. code-block:: bash @@ -40,7 +42,8 @@ When you submit a PR request, you can check whether your code passes the CI test 4. Qlib has integrated pre-commit, which will make it easier for developers to format their code. - Just run the following two commands, and the code will be automatically formatted using black and flake8 when the git commit command is executed. + + Just run the following two commands, and the code will be automatically formatted using black and flake8 when the git commit command is executed. .. code-block:: bash diff --git a/docs/hidden/client.rst b/docs/hidden/client.rst index 7ca0d680..de6e2e68 100644 --- a/docs/hidden/client.rst +++ b/docs/hidden/client.rst @@ -81,6 +81,7 @@ If running on Windows, open **NFS** features and write correct **mount_path**, i * Open ``Programs and Features``. * Click ``Turn Windows features on or off``. * Scroll down and check the option ``Services for NFS``, then click OK + Reference address: https://graspingtech.com/mount-nfs-share-windows-10/ 2.config correct mount_path * In windows, mount path must be not exist path and root path, @@ -161,7 +162,7 @@ Limitations API *** -The client is based on `python-socketio`_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure. +The client is based on `python-socketio `_ which is a framework that supports WebSocket client for Python language. The client can only propose requests and receive results, which do not include any calculating procedure. Class ----- diff --git a/docs/index.rst b/docs/index.rst index 0d8cad81..3adf9049 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -56,6 +56,12 @@ Document Structure Task Management Point-In-Time database +.. toctree:: + :maxdepth: 3 + :caption: FOR DEVELOPERS: + + Code Standard & Development Guidance + .. toctree:: :maxdepth: 3 :caption: REFERENCE: diff --git a/docs/introduction/quick.rst b/docs/introduction/quick.rst index 364e58ca..78d9c208 100644 --- a/docs/introduction/quick.rst +++ b/docs/introduction/quick.rst @@ -21,6 +21,7 @@ Users can easily intsall ``Qlib`` according to the following steps: - Before installing ``Qlib`` from source, users need to install some dependencies: .. code-block:: + pip install numpy pip install --upgrade cython diff --git a/docs/reference/api.rst b/docs/reference/api.rst index 98f50fc2..4e6a7a85 100644 --- a/docs/reference/api.rst +++ b/docs/reference/api.rst @@ -1,4 +1,5 @@ .. _api: + ============= API Reference ============= diff --git a/docs/start/getdata.rst b/docs/start/getdata.rst index 8849eb87..cea9c2b0 100644 --- a/docs/start/getdata.rst +++ b/docs/start/getdata.rst @@ -83,15 +83,14 @@ Load features of certain instruments in a given time range: >> from qlib.data import D >> instruments = ['SH600000'] >> fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low'] - >> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head() - - $close $volume Ref($close, 1) Mean($close, 3) $high-$low - instrument datetime - SH600000 2010-01-04 86.778313 16162960.0 88.825928 88.061483 2.907631 - 2010-01-05 87.433578 28117442.0 86.778313 87.679273 3.235252 - 2010-01-06 85.713585 23632884.0 87.433578 86.641825 1.720009 - 2010-01-07 83.788803 20813402.0 85.713585 85.645322 3.030487 - 2010-01-08 84.730675 16044853.0 83.788803 84.744354 2.047623 + >> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head().to_string() + ' $close $volume Ref($close, 1) Mean($close, 3) $high-$low + ... instrument datetime + ... SH600000 2010-01-04 86.778313 16162960.0 88.825928 88.061483 2.907631 + ... 2010-01-05 87.433578 28117442.0 86.778313 87.679273 3.235252 + ... 2010-01-06 85.713585 23632884.0 87.433578 86.641825 1.720009 + ... 2010-01-07 83.788803 20813402.0 85.713585 85.645322 3.030487 + ... 2010-01-08 84.730675 16044853.0 83.788803 84.744354 2.047623' Load features of certain stock pool in a given time range: @@ -105,15 +104,14 @@ Load features of certain stock pool in a given time range: >> expressionDFilter = ExpressionDFilter(rule_expression='$close>Ref($close,1)') >> instruments = D.instruments(market='csi300', filter_pipe=[nameDFilter, expressionDFilter]) >> fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low'] - >> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head() - - $close $volume Ref($close, 1) Mean($close, 3) $high-$low - instrument datetime - SH600655 2010-01-04 2699.567383 158193.328125 2619.070312 2626.097738 124.580566 - 2010-01-08 2612.359619 77501.406250 2584.567627 2623.220133 83.373047 - 2010-01-11 2712.982422 160852.390625 2612.359619 2636.636556 146.621582 - 2010-01-12 2788.688232 164587.937500 2712.982422 2704.676758 128.413818 - 2010-01-13 2790.604004 145460.453125 2788.688232 2764.091553 128.413818 + >> D.features(instruments, fields, start_time='2010-01-01', end_time='2017-12-31', freq='day').head().to_string() + ' $close $volume Ref($close, 1) Mean($close, 3) $high-$low + ... instrument datetime + ... SH600655 2010-01-04 2699.567383 158193.328125 2619.070312 2626.097738 124.580566 + ... 2010-01-08 2612.359619 77501.406250 2584.567627 2623.220133 83.373047 + ... 2010-01-11 2712.982422 160852.390625 2612.359619 2636.636556 146.621582 + ... 2010-01-12 2788.688232 164587.937500 2712.982422 2704.676758 128.413818 + ... 2010-01-13 2790.604004 145460.453125 2788.688232 2764.091553 128.413818' For more details about features, please refer `Feature API <../component/data.html>`_. diff --git a/docs/start/integration.rst b/docs/start/integration.rst index 801bb819..a9eecc4e 100644 --- a/docs/start/integration.rst +++ b/docs/start/integration.rst @@ -21,84 +21,88 @@ The Custom models need to inherit `qlib.model.base.Model <../reference/api.html# - ``Qlib`` passes the initialized parameters to the \_\_init\_\_ method. - The hyperparameters of model in the configuration must be consistent with those defined in the `__init__` method. - Code Example: In the following example, the hyperparameters of model in the configuration file should contain parameters such as `loss:mse`. - .. code-block:: Python - def __init__(self, loss='mse', **kwargs): - if loss not in {'mse', 'binary'}: - raise NotImplementedError - self._scorer = mean_squared_error if loss == 'mse' else roc_auc_score - self._params.update(objective=loss, **kwargs) - self._model = None + .. code-block:: Python + + def __init__(self, loss='mse', **kwargs): + if loss not in {'mse', 'binary'}: + raise NotImplementedError + self._scorer = mean_squared_error if loss == 'mse' else roc_auc_score + self._params.update(objective=loss, **kwargs) + self._model = None - Override the `fit` method - ``Qlib`` calls the fit method to train the model. - The parameters must include training feature `dataset`, which is designed in the interface. - The parameters could include some `optional` parameters with default values, such as `num_boost_round = 1000` for `GBDT`. - Code Example: In the following example, `num_boost_round = 1000` is an optional parameter. - .. code-block:: Python - def fit(self, dataset: DatasetH, num_boost_round = 1000, **kwargs): + .. code-block:: Python - # prepare dataset for lgb training and evaluation - df_train, df_valid = dataset.prepare( - ["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L - ) - x_train, y_train = df_train["feature"], df_train["label"] - x_valid, y_valid = df_valid["feature"], df_valid["label"] + def fit(self, dataset: DatasetH, num_boost_round = 1000, **kwargs): - # Lightgbm need 1D array as its label - if y_train.values.ndim == 2 and y_train.values.shape[1] == 1: - y_train, y_valid = np.squeeze(y_train.values), np.squeeze(y_valid.values) - else: - raise ValueError("LightGBM doesn't support multi-label training") + # prepare dataset for lgb training and evaluation + df_train, df_valid = dataset.prepare( + ["train", "valid"], col_set=["feature", "label"], data_key=DataHandlerLP.DK_L + ) + x_train, y_train = df_train["feature"], df_train["label"] + x_valid, y_valid = df_valid["feature"], df_valid["label"] - dtrain = lgb.Dataset(x_train.values, label=y_train) - dvalid = lgb.Dataset(x_valid.values, label=y_valid) + # Lightgbm need 1D array as its label + if y_train.values.ndim == 2 and y_train.values.shape[1] == 1: + y_train, y_valid = np.squeeze(y_train.values), np.squeeze(y_valid.values) + else: + raise ValueError("LightGBM doesn't support multi-label training") - # fit the model - self.model = lgb.train( - self.params, - dtrain, - num_boost_round=num_boost_round, - valid_sets=[dtrain, dvalid], - valid_names=["train", "valid"], - early_stopping_rounds=early_stopping_rounds, - verbose_eval=verbose_eval, - evals_result=evals_result, - **kwargs - ) + dtrain = lgb.Dataset(x_train.values, label=y_train) + dvalid = lgb.Dataset(x_valid.values, label=y_valid) + + # fit the model + self.model = lgb.train( + self.params, + dtrain, + num_boost_round=num_boost_round, + valid_sets=[dtrain, dvalid], + valid_names=["train", "valid"], + early_stopping_rounds=early_stopping_rounds, + verbose_eval=verbose_eval, + evals_result=evals_result, + **kwargs + ) - Override the `predict` method - The parameters must include the parameter `dataset`, which will be userd to get the test dataset. - Return the `prediction score`. - Please refer to `Model API <../reference/api.html#module-qlib.model.base>`_ for the parameter types of the fit method. - Code Example: In the following example, users need to use `LightGBM` to predict the label(such as `preds`) of test data `x_test` and return it. - .. code-block:: Python - def predict(self, dataset: DatasetH, **kwargs)-> pandas.Series: - if self.model is None: - raise ValueError("model is not fitted yet!") - x_test = dataset.prepare("test", col_set="feature", data_key=DataHandlerLP.DK_I) - return pd.Series(self.model.predict(x_test.values), index=x_test.index) + .. code-block:: Python + + def predict(self, dataset: DatasetH, **kwargs)-> pandas.Series: + if self.model is None: + raise ValueError("model is not fitted yet!") + x_test = dataset.prepare("test", col_set="feature", data_key=DataHandlerLP.DK_I) + return pd.Series(self.model.predict(x_test.values), index=x_test.index) - Override the `finetune` method (Optional) - This method is optional to the users. When users want to use this method on their own models, they should inherit the ``ModelFT`` base class, which includes the interface of `finetune`. - The parameters must include the parameter `dataset`. - Code Example: In the following example, users will use `LightGBM` as the model and finetune it. - .. code-block:: Python - def finetune(self, dataset: DatasetH, num_boost_round=10, verbose_eval=20): - # Based on existing model and finetune by train more rounds - dtrain, _ = self._prepare_data(dataset) - self.model = lgb.train( - self.params, - dtrain, - num_boost_round=num_boost_round, - init_model=self.model, - valid_sets=[dtrain], - valid_names=["train"], - verbose_eval=verbose_eval, - ) + .. code-block:: Python + + def finetune(self, dataset: DatasetH, num_boost_round=10, verbose_eval=20): + # Based on existing model and finetune by train more rounds + dtrain, _ = self._prepare_data(dataset) + self.model = lgb.train( + self.params, + dtrain, + num_boost_round=num_boost_round, + init_model=self.model, + valid_sets=[dtrain], + valid_names=["train"], + verbose_eval=verbose_eval, + ) Configuration File ================== @@ -107,21 +111,21 @@ The configuration file is described in detail in the `Workflow <../component/wor - Example: The following example describes the `model` field of configuration file about the custom lightgbm model mentioned above, where `module_path` is the module path, `class` is the class name, and `args` is the hyperparameter passed into the __init__ method. All parameters in the field is passed to `self._params` by `\*\*kwargs` in `__init__` except `loss = mse`. -.. code-block:: YAML + .. code-block:: YAML - model: - class: LGBModel - module_path: qlib.contrib.model.gbdt - args: - loss: mse - colsample_bytree: 0.8879 - learning_rate: 0.0421 - subsample: 0.8789 - lambda_l1: 205.6999 - lambda_l2: 580.9768 - max_depth: 8 - num_leaves: 210 - num_threads: 20 + model: + class: LGBModel + module_path: qlib.contrib.model.gbdt + args: + loss: mse + colsample_bytree: 0.8879 + learning_rate: 0.0421 + subsample: 0.8789 + lambda_l1: 205.6999 + lambda_l2: 580.9768 + max_depth: 8 + num_leaves: 210 + num_threads: 20 Users could find configuration file of the baselines of the ``Model`` in ``examples/benchmarks``. All the configurations of different models are listed under the corresponding model folder.