← Back to the journal

DataFocus · Journal

Scheduled Table Import Tasks in OT Flash: Keeping Doris Tables Fresh

How DataFocus OT Flash keeps imported tables current with scheduled import tasks instead of an ETL engine: connect a read-only source, pick tables, set a schedule, and watch health in System Manager.

To schedule recurring table imports from a source database into DataFocus OT Flash, first add the database as an external data source with a dedicated read-only account. Then run the three-step import wizard (select the data source, select tables, configure how the tables load) and set an optional schedule on the last step. Scheduled imports are managed in the Table Task tab, and the imported tables are stored in Flash's built-in data source, "DataFocus system Doris". This tutorial walks through that path and the checks that tell you whether the refreshes are healthy.

Why does Flash refresh data with import tasks instead of ETL pipelines?

Its manual lists table-level tools for shaping data: Join Worksheet, Merge Worksheet (which stacks tables with the same structure), SQL views with an editor, datasets that combine related tables, scheduled table import tasks in a Table Task tab, and "norm" formulas. These are explicitly not the ETL engine that DataFocus OT Heavy provides. In the Flash agent manual, the dfc_etl tool only reports capabilities; it can return available=false with the code ETL_NOT_INCLUDED_IN_OT_FLASH, so Flash does not run Heavy-style ETL jobs.

The practical consequence is a simple division of labor:

  • The source system stays the system of record.
  • A scheduled import copies the tables you chose into Flash.
  • Reshaping happens inside Flash with worksheets and views, after the data has landed.

For a team that wants fresh tables for search-style analysis and does not need a pipeline orchestrator, this is a smaller moving-parts count: one connection, one wizard, one schedule per table selection. The trade-off is covered at the end.

Where does imported data live?

OT Flash ships with a built-in data source called "DataFocus system Doris". Doris is also the engine behind table queries in Flash: the agent manual states that queries go through Doris and that original Trino, Hive and Hadoop services are not part of Flash's agent entry points.

The Apache Doris project describes Doris in its official documentation as an MPP-based real-time data warehouse, with frontend (FE) nodes that handle requests, query planning and metadata, and backend (BE) nodes that handle storage and query execution. That split shows up in Flash too. The System Manager monitoring page lists df-doris-fe and df-doris-be among the service containers, and its Doris panel reports query status, active FE and BE counts, storage and the internal table count.

One wording note: the r57 manual calls the data source "DataFocus system Doris" and, for external databases, lists "SelectDB / Doris" as a source type. When you search the interface, look for those names.

How do I set up the source connection?

Flash supports a long list of external database types, grouped as Database, API source and HTTP source. The Database group includes PostgreSQL, MySQL (including MariaDB), Oracle, IBM DB2, GaussDB 100, SQL Server, Sybase, Greenplum, ClickHouse, Kudu, ElasticSearch, TeraData, MongoDB, Prometheus, Hive, Delta Lake, Iceberg, TiDB, SelectDB / Doris, OceanBase, SAP HANA, ClickZetta, Impala, DM8 (Dameng), ArgoDB and KingBase.

Connecting a database asks for these fields:

FieldRequiredNote
Source NameyesThe label you will see in the import wizard
ServeryesHost of the source database
PortyesDatabase port
DatabaseyesDatabase to read from
SchemaoptionalWhere the source type uses schemas
JDBC ParamoptionalExtra connection parameters
Username / passwordyesThe account Flash uses to read

Use Connect to test before saving. The manual recommends a dedicated read-only database account for OT Flash, limited to the databases and tables you want to analyze. Treat that as a design rule, not a formality: a recurring import has to read the source without anyone present, so size the account's privileges for exactly that job.

An illustrative sketch for a PostgreSQL source (illustrative, not run; adapt names and privileges to your environment):

CREATE ROLE flash_reader LOGIN PASSWORD '<set-a-strong-password>';
GRANT CONNECT ON DATABASE sales TO flash_reader;
GRANT USAGE ON SCHEMA public TO flash_reader;
GRANT SELECT ON public.orders, public.customers TO flash_reader;

Only the tables you list are readable, so a later mistake in the import wizard cannot reach beyond them.

How do I create a scheduled import in the Table Task tab?

The import is a three-step wizard.

  1. Select data source. Choose the connection created above.
  2. Select tables. Tick the source tables to bring in. Start with the tables your questions actually need rather than the whole schema.
  3. Config load tables. Choose the import type and, if you want the data to stay current, the optional schedule.

The schedule is what turns a one-time copy into a refresh. Once saved, the task appears in the Table Task tab alongside your other table tasks, which is where you return to review it. The manual also covers other ways to bring in a table (local CSV, Excel or JSON files, an API, or an HTTP source), but only a database import has a source that changes continuously, so it is the case that benefits from a schedule.

A few habits make scheduled imports easier to operate:

  • One purpose per task. Group tables that are always queried together so they refresh together.
  • Name by source and cadence. A name such as sales_orders_daily tells whoever reads the Table Task tab what to expect.
  • Check the first load. Open the imported table and confirm the columns and row shape are what you expect before you rely on the schedule.
  • Match the cadence to the question. A weekly management review does not need hourly refreshes of the underlying tables.

How do I reshape data after import?

Because Flash has no ETL engine, shaping happens in the table-level tools, on data that is already inside Doris:

  • Join Worksheet combines tables on shared keys.
  • Merge Worksheet stacks tables with the same structure, for example monthly extracts that share one layout.
  • SQL views use the built-in editor to define a reusable query over imported tables.
  • Datasets combine related tables so that search and Q&A can use them together.
  • Norm formulas add calculated values.

A view is the natural place for logic that would otherwise sit in a pipeline. Here is an illustrative example (illustrative, not run; table and column names are hypothetical):

SELECT o.order_id,
       o.order_date,
       c.region,
       o.amount
FROM   orders o
JOIN   customers c ON c.customer_id = o.customer_id
WHERE  o.status <> 'cancelled';

Since the view reads the imported tables, every refresh of orders or customers flows through it with no extra step. Keep views over the imported copies rather than copying the same logic into several places.

How do I watch import and scheduling health?

System Manager is where you check that the platform doing the work is healthy. Its monitoring page shows the service containers (for example df-webserver, df-search, df-biserver, df-doris-fe, df-doris-be, df-pgsql, df-redis and df-nginx, one running instance each in normal operation), the infrastructure services (Doris, PostgreSQL, RabbitMQ and Redis) and the business services, which include Task Manager alongside BI Server, Search Server and Web Server.

Alarms turn that view into something you do not have to watch. You can set thresholds on nodes (CPU, memory, disks), pods, Doris, PostgreSQL, RabbitMQ, Redis, service memory, task scheduling and business data, using the operators >, >=, <, <= and =. Alarms are delivered in the app, by email, or both. The email cooldown defaults to 60 minutes, the notification language is Chinese or English, and every alarm is added to the Alarm Log.

For a scheduled import setup, a sensible starting set is:

  1. A Doris storage threshold, so a growing set of imported tables does not fill the disks unnoticed.
  2. A task scheduling alarm, so scheduling problems are flagged rather than discovered when a dashboard looks old.
  3. A disk threshold on the nodes hosting the stack.

Review the Alarm Log after the first week and tune thresholds to match what is normal for your data volume.

When is this not the right fit?

Scheduled imports copy data. That has consequences worth stating plainly:

  • Freshness is bounded by your schedule. A table refreshed every day is up to a day behind the source. If you need answers from the live source at every query, a copy on a schedule is a different model.
  • Storage is duplicated. Imported tables occupy space in the built-in Doris in addition to the source database. Plan resources around data volume, concurrency and model requirements.
  • Heavy transformations belong elsewhere. If you need ETL-engine capabilities, look at DataFocus OT Heavy, which provides the ETL engine; Flash's worksheets and SQL views cover table-level shaping.

If your needs sit within "copy these tables on a schedule, reshape them with joins and views, ask questions of the result", the Table Task tab covers the workflow with one connection and one wizard.

Next step

Download OT Flash from the editions and download page, or browse the OT Flash user manual, whose chapter on tables covers import, worksheets, views and table tasks. System requirements and supported platforms are listed in the release notes in the OT Flash GitHub repository.