microsoft
/
data-accelerator
зеркало из https://github.com/microsoft/data-accelerator.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
Страница: Schedule batch job
Страницы
Arm Parameters Azure deployment Cloud Deployment On Linux Cloud Simulator Cloud deployment Configuring the Arm template Create new metric Creating your first pipeline in 5 minutes! Data Accelerator with Databricks Data Accelerator Data Accumulator Diagnose issues using Telemetry FAQ Find Applications For Your Environment Home Inviting others and RBAC Live query Local Cloud Debugging Local Tutorial Add an Alert Local Tutorial Adding SQL to your flow and outputs to Metrics dashboard Local Tutorial Advanced Aggregate alerts Local Tutorial Creating your first Flow in local mode Local Tutorial Custom schema Local Tutorial Debugging using Spark logs Local Tutorial Extending with UDF UDAF custom code Local Tutorial Outputs to disk Local Tutorial Reference data Local Tutorial Scaling the docker host Local Tutorial Tag Aggregate to metrics Local Tutorial Tag Rules output to local file Local create metric Local mode with Docker Local running sample Output data to Azure SQL Database Run Data Accelerator Flows on Databricks Scale Schedule batch job Set up aggregate alert Set up new outputs Set up simple alert Spark logs Tagged data flowing to CosmosDB Tagging aggregate rules Tagging simple rules Tutorials Upgrade Existing Data Accelerator Environment to v1.1 Upgrade Existing Data Accelerator Environment to v1.2 Use Input in different tenant Windowing functions functions readme reference data sql query
14 Schedule batch job
Kwangje Cho редактировал(а) эту страницу 2019-08-13 15:39:42 -07:00
Содержание

In this tutorial you will learn how to:

  • Create a new Flow for batching
  • Configure input source of data (Supported today: blob)
  • Inspect schema of incoming data
  • Schedule jobs
  • Save

Steps to follow

  • Open Data Accelerator portal and go to the Flows tab

  • Click on "New" on the Flows tab, this will create a new Flow:

    New Flow

  • Switch to the Input tab and select Batching for Mode and Azure Blob for Type. Data Accelerator currently supports only Blob for Batching

  • Provide the following information of your input Azure blob

    • The Connection string
    • The Blob Path. This should be a wasbs URL format. If your path contains date and time, you can use DateTime pattern with curly brackets '{' and '}'
      e.g. wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<folder_name>/{yyyy-MM-dd}.
      Scheduling

  • Click "GetSchema" button. This reads the blobs from the given path and infer the schema.

  • Switch to the schedule tab and click Add button. There are two types of scheduling.

    • Recurring This creates a job based on the given recurrence. e.g. If Recurrence is 1 day, it will create one job per day.
      Scheduling

    • One Time This creates all jobs for the given the date range at once (between the start time and the end time). And once all jobs are scheduled, the schedule will be disabled and read-only.
      Scheduling

    • Recurrence: It specifies an interval between jobs. The available values are day, hour and minute.

    • Delay: It specifies an offset for a job. The offset is negative so an actual start time will be calculated by subtracting offset from the original start time. The available values are day, hour and minute.

    • Window: It specifies how much a job will look back. The available values are day, hour and minute.

  • Click "Save" button to save the Flow. Once it's saved, the jobs will be created and be running by the scheduler. Currently the interval of the scheduler is 1 hour.

  • For details on other tabs, please check this https://github.com/Microsoft/data-accelerator/wiki/Creating-your-first-pipeline-in-5-minutes!

Data Accelerator

Install

Docs