diff --git a/content/en-us/docs/docs/admin/audit-log.md b/content/en-us/docs/docs/admin/audit-log.md index e82ac176d0f6a004d3c190a1c87a89e4f06a8b9a..9f89d515e8401f3cdb16a5b929b0ba2bbe9ba72a 100644 --- a/content/en-us/docs/docs/admin/audit-log.md +++ b/content/en-us/docs/docs/admin/audit-log.md @@ -1,6 +1,6 @@ # Audit Log -openLooKeng audit logging functionality is a custom event listener that is invoked for query creation and query completion (success or failure) +openLooKeng audit logging functionality is a custom event listener, which monitors the start and stop of openLooKeng cluster and the dynamic addition and deletion of nodes in the cluster; Listen to WebUi user login and exit events; Listen for query events and call when the query is created and completed (success or failure). An audit log contains the following information: 1. time when an event occurs @@ -24,15 +24,17 @@ To enable audit logging feature, the following configs must be present in `etc/e hetu.event.listener.type=AUDIT hetu.event.listener.listen.query.creation=true hetu.event.listener.listen.query.completion=true +hetu.auditlog.logoutput=/var/log/ +hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH ``` -Other audit logging properties include: +The following is a detailed description of audit logging properties: -`hetu.event.listener.audit.file`: Optional property to define absolute file path for the audit file. Ensure the process running the openLooKeng server has write access to this directory. +`hetu.event.listener.type`: property to define logging type for audit files. Allowed values are AUDIT and LOGGER. -`hetu.event.listener.audit.filecount`: Optional property to define the number of files to use +`hetu.auditlog.logoutput`: property to define absolute file directory for audit files. Ensure the process running the openLooKeng server has write access to this directory. -`hetu.event.listener.audit.limit`: Optional property to define the maximum number of bytes to write to any one file +`hetu.auditlog.logconversionpattern`: property to define the conversion pattern of audit files. Allowed values are yyyy-MM-dd.HH and yyyy-MM-dd. Example configuration file: @@ -41,7 +43,6 @@ event-listener.name=hetu-listener hetu.event.listener.type=AUDIT hetu.event.listener.listen.query.creation=true hetu.event.listener.listen.query.completion=true -hetu.event.listener.audit.file=/var/log/hetu/hetu-audit.log -hetu.event.listener.audit.filecount=1 -hetu.event.listener.audit.limit=100000 +hetu.auditlog.logoutput=/var/log/ +hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH ``` diff --git a/content/en-us/docs/docs/admin/dynamic-catalog.md b/content/en-us/docs/docs/admin/dynamic-catalog.md index 263e366e7f726d4fcdde3f5abb2aace2811ac543..dc5e7a70217cff9e536fbf0346f3487daf9b9d2e 100644 --- a/content/en-us/docs/docs/admin/dynamic-catalog.md +++ b/content/en-us/docs/docs/admin/dynamic-catalog.md @@ -137,7 +137,7 @@ An UPDATE operation is a combination of DELETE and ADD operations. First the adm Other coordinators and workers periodically check the catalog properties file in the shared filesystem and perform changes accordingly on the local file system. -Catalog properties including ``connector-name`` and ``properties`` can be modified. However, the **catalog name** CAN NOT be changed. +Catalog properties including ``properties`` can be modified. However, the **catalog name** CAN NOT be changed. Users are not advised to update `connector-name` using a PUT request, and you can first delete the catalog and then perform a POST request. Take Hive as an example, by curl you can update the catalog with the following command, the following updated `hive.allow-drop-table` parameters: diff --git a/content/en-us/docs/docs/admin/extension-execution-planner.md b/content/en-us/docs/docs/admin/extension-execution-planner.md new file mode 100644 index 0000000000000000000000000000000000000000..f1dd46f96c8c1e3a68d2485a632742f8f79e865b --- /dev/null +++ b/content/en-us/docs/docs/admin/extension-execution-planner.md @@ -0,0 +1,31 @@ +# Extension Physical Execution Planner +This section describes how to add an extension physical execution planner in openLooKeng. With the extension physical execution planner, openLooKeng can utilize other operator acceleration libraries to speed up the execution of SQL statements. + +## Configuration +First, you should get the below jar package and c++ libraries from links: + +- `boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.jar`: [Link](https://github.com/kunpengcompute/boostkit-bigdata/releases/download/v1.3.0/boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.zip) + +- `libboostkit-omniop-xx-1.0.0-aarch.so`: [Link](https://www.hikunpeng.com/en/developer/boostkit/big-data?acclerated=3) + +Then, to enable extension physical execution feature, the following configs must be added in +`config.properties`： + +``` properties +extension_execution_planner_enabled=true +extension_execution_planner_jar_path=file:///xxPath/boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.jar +extension_execution_planner_class_path=nova.hetu.olk.OmniLocalExecutionPlanner +``` + +The above attributes are described below: + +- `extension_execution_planner_enabled`: Enable extension physical execution feature. +- `extension_execution_planner_jar_path`: Set the file path of the extension physical execution jar package. +- `extension_execution_planner_class_path`: Set the package path of extension physical execution generated class in jar。 + + +## Usage +The below command can control the enablement of extension physical execution feature in WebUI or Cli while running openLooKeng: +``` +set session extension_execution_planner_enabled=true/false +``` \ No newline at end of file diff --git a/content/en-us/docs/docs/admin/multi-split-for-jdbc-data-source.md b/content/en-us/docs/docs/admin/multi-split-for-jdbc-data-source.md index 0eca4dfbb0a5624206d2eecc634e77676548e33e..3748942e6ff8dbbc1aaad01c34d85cd5fe8d2114 100644 --- a/content/en-us/docs/docs/admin/multi-split-for-jdbc-data-source.md +++ b/content/en-us/docs/docs/admin/multi-split-for-jdbc-data-source.md @@ -6,7 +6,7 @@ This function applies to JDBC data sources. Data tables to be read are divided i ## Properties -Multi-split management is based on connectors. For a data table with this function enabled, add the following attributes to the configuration file of the connector to which the data table belong. For example, the configuration file corresponding to the **mysql** connector is **etc/mysql.properties**. +Multi-split management is based on connectors. For a data table with this function enabled, add the following attributes to the configuration file of the connector to which the data table belong. For example, the configuration file corresponding to the **mysql** connector is **etc/catalog/mysql.properties**. Property list: diff --git a/content/en-us/docs/docs/admin/properties.md b/content/en-us/docs/docs/admin/properties.md index effcb1b3919b691f520d687966f1dfb97c7bcfb8..04ea155608be53b3e959f9d75c78e3b9d8c51e0c 100644 --- a/content/en-us/docs/docs/admin/properties.md +++ b/content/en-us/docs/docs/admin/properties.md @@ -118,6 +118,20 @@ This section describes the most important config properties that may be used to > > This is the amount of memory set aside as headroom/buffer in the JVM heap for allocations that are not tracked by openLooKeng. +### `query.suspend-query-enabled` + +> - **Type:** `boolean` +> - **Default value:** `false` +> +> Enables running query temporary suspension when system is in low resource situation. + +### `query.max-suspended-queries` + +> - **Type:** `integer` +> - **Default value:** `10` +> +> Maximum number of queries to attempt suspension before starting of killing the queries. This property comes in effect only if `query.suspend-query-enabled` is configured `true` + ## Spilling Properties ### `experimental.spill-enabled` @@ -364,39 +378,136 @@ Exchanges transfer data between openLooKeng nodes for different stages of a quer > > Maximum size of a response returned from an exchange request. The response will be placed in the exchange client buffer which is shared across all concurrent requests for the exchange. > -> -> > Increasing the value may improve network throughput if there is high latency. Decreasing the value may improve query performance for large clusters as it reduces skew due to the exchange client buffer holding responses for more tasks (rather than hold more data from fewer tasks). -### `exchange.max-error-duration` +### `sink.max-buffer-size` -> - **Type:** `duration` -> - **Minimum value:** `1m` -> - **Default value:** `7m` +> - **Type:** `data size` +> - **Default value:** `32MB` > -> The maximum amount of time coordinator waits for inter-task related errors to be resolved before it's considered a failure. +> Output buffer size for task data that is waiting to be pulled by upstream tasks. If the task output is hash partitioned, then the buffer will be shared across all of the partitioned consumers. Increasing this value may improve network throughput for data transferred between stages if the network has high latency or if there are many nodes in the cluster. -### `exchange.is-timeout-failure-detection-enabled` +### `query-resource-tracking` > - **Type:** `boolean` -> - **Default value:** `true` +> - **Default value:** `false` > -> The failure detection mechanism in use. Default is timeout based failure detection. Otherwise, i.e. when this property is set to false, maximum retry based failure detection mechanism is enabled. +> Disable query level resource tracking and mitigation actions. + +### `query-no-resource-retry-count` + +> - **Type:** `integer` +> - **Default value:** `5` > +> Query shall be retried at the given number of times when resources available is not enough. This is enabled only when `query-resource-tracking` is set to be `true`. +> In case the value specified is less than 5, it will automatically be reverted to default value 5. -### `exchange.max-retry-count` +## Failure Recovery handling Properties + +### Failure Retry Policies + +### `failure.recovery.retry.profile` + +> - **Type:** `String` +> - **Default value:** `default` +> +> This property defines the failure detection profile used to determine if failure has happened for a http client. The value `` set for this property has to correspond to `.properties` file in `etc/failure-retry-policy/`. In case no such profile is available, and this property is not set, "default" profile is used. +> For example, `failure.recovery.retry.profile=test` requires `test.properties` file to be present in `etc/failure-retry-policy`. +> The file `test.properties` must contain `failure.recovery.retry.type` specified. + + +### `failure.recovery.retry.type` + +> - **Type:** `String` +> - **Default value:** `timeout` +> +> The failure detection mechanism in use. Default is timeout based failure detection. +> +#### `timeout` based failure detection. +> Using this mechanism, HTTP client failures are retried for a specific duration before considering it as a permanent failure. +> Additional properties `max.error.duration` can be defined for this type of failure detection. +> +#### `max-retry` based failure detection. +> Using this mechanism, HTTP client failures are retried for a specific number of times before considering it as a permanent failure. +> Additional properties `max.retry.count` and `max.error.duration` can be defined for this type of failure detection. +> Using this type of failure detection is configured to be used, `max.retry.count` times retry is performed before consulting the failure detector module. When the remote node is failed as per the failure detector module, HTTP client considers it a permanent failure. Otherwise, i.e. When remote worker node is alive but not sending response, retry happens for `max.error.duration` before considering it as permanent failure. + +### `max.error.duration` +> - **Type:** `duration` +> - **Default value:** `300s` +> +> The maximum amount of time coordinator waits for inter-task related errors to be resolved before it's considered a permanent failure. + + +### `max.retry.count` > - **Type:** `integer` > - **Default value:** `100` > -> The maximum number of retry for failed task performed by the coordinator before consulting the failure detector module about the remote node status. If the remote node status is failed as per the failure detector module, it is considered as a permanent failure. This parameter is the minimum count which is required to decide, not necessarily the exact count. Based on the cluster size, load on the cluster the exact count may vary slightly. This property is used only when exchange.is-timeout-failure-detection-enabled is set to false. This value needs to be at least 100 to take effect. +> The maximum number of retry for failed task performed by the coordinator before consulting the failure detector module about the remote node status. +> This parameter is the minimum count before consulting the failure detection module. Hence, the actual number of failures may vary slightly based on the cluster size, and load on the cluster. +> This property is used only for `max-retry` based failure detection profiles. +> The minimum value for this parameter is 100. -### `sink.max-buffer-size` +### Gossip Protocol Configurations for Failure Detection -> - **Type:** `data size` -> - **Default value:** `32MB` +### `failure-detection-protocol` + +>- **Type:** String +>- **Default value:** `heartbeat` > -> Output buffer size for task data that is waiting to be pulled by upstream tasks. If the task output is hash partitioned, then the buffer will be shared across all of the partitioned consumers. Increasing this value may improve network throughput for data transferred between stages if the network has high latency or if there are many nodes in the cluster. +> This property defines the type of failure detector in use. Default configuration is `heartbeat` failure detector. +> Gossip protocol can be enabled by specifying this parameter in `config.properties` file, with the value `gossip`. +> All nodes (i.e. coordinator as well as workers) in a cluster should have this property specified in their respective `etc/config.properties` file. + +### `failure-detector.heartbeat-interval` + +>- **Type:** Duration +>- **Default value:** `500ms` (500 miliseconds) +> +> This is the interval of gossip between two nodes in the cluster. +> In gossip protocol, two workers are expected to gossip with higher frequency than the coordinator and a worker. +> In `config.properties` for the coordinator, this property can be set with a reasonably higher value, such as `5s` (5 seconds). +> In workers, this property can be left to use the default value. +> +### `failure-detector.worker-gossip-probe-interval` +> +> - **Type:** Duration +>- **Default value:** `5s` (5 seconds) +> +> Gossip protocol uses monitoring tasks (same as the heartbeat failure detector) to keep tab on the other nodes. +> This property specifies the interval of refreshing the monitoring tasks to trigger worker to worker gossip. +> This property, if needed to be configured with any other value than the default, should be specified only for the worker nodes. +> This parameter should have higher value than `failure-detector.heartbeat-interval`. +> +### `failure-detector.coordinator-gossip-probe-interval` +> +> - **Type:** Duration +>- **Default value:** `5s` (5 seconds) +> +> Gossip protocol uses monitoring tasks (same as the heartbeat failure detector) to keep tab on the other nodes. +> This property specifies the interval of refreshing the monitoring tasks to trigger coordinator to worker gossip. +> This property, if needed to be configured with any other value than the default, should be specified only for the coordinator. +> This parameter should have higher value than `failure-detector.heartbeat-interval` and `failure-detector.worker-gossip-probe-interval`. +> +### `failure-detector.coordinator-gossip-collate-interval` +> +> - **Type:** Duration +>- **Default value:** `2s` (2 seconds) +> +> This property specifies the interval in which the coordinator collates all the gossips it obtained from all the workers. +> This property has to be specified only for the coordinator. +> This parameter should have higher value than `failure-detector.heartbeat-interval`. +> +### `failure-detector.gossip-group-size` +> +> - **Type:** Integer +>- **Default value:** `Integer.MAX_VALUE` +> +> A worker should gossip with how many other workers in the cluster, is defined by this parameter. +> Any value higher than the cluster-size (i.e. the number of workers) implies all-to-all gossip. +> To keep the network overhead low, this value should be reasonably low for a big cluster (e.g. 10 for a cluster size of 100). +> On each refresh of the worker-monitoring tasks at the coordinator, the coordinator defines the list of worker URIs of size `failure-detector.gossip-group-size` to trigger worker-to-worker gossip. ## Task Properties @@ -630,6 +741,84 @@ Exchanges transfer data between openLooKeng nodes for different stages of a quer > > **Note:** This is supported only for Hive connector. +### `optimizer.transform-self-join-to-window` + +> - **Type:** `boolean` +> - **Default value:** `true` +> +> Enable optimization to remove self join with window function using lead/lag methods. + +### `optimizer.transform-self-join-aggregates-to-window` + +> - **Type:** `boolean` +> - **Default value:** `true` +> +> Enable optimization to remove self joins with aggregations with window function using lead/lag methods. + +### `optimizer.join-partitioned-build-min-row-count` + +> - **Type:** `long` +> - **Default value:** `1,000,000` +> +> Use gathering exchange for exchanges with less than the specified rows produced, else replace with partitioning exchange. + +### `optimizer.use-exact-partitioning` + +> - **Type:** `boolean` +> - **Default value:** `false` +> +> When enabled this forces data repartitioning unless the partitioning of upstream stage matches exactly what downstream stage expects. + +### `enable-cte-result-cache` + +> - **Type:** `boolean` +> - **Default value:** `false` +> +> Enable CTE materialization cache feature. This can also be specified on a per-query basis using the enable_cte_result_cache session property. + +### `cte-result-cache-threshold-size` + +> - **Type:** `Data Size` +> - **Default value:** `128MB` +> +> Maximum allowed size to be stored as part of cte materialization cache per CTE per query. + +### `adaptive-partial-aggregation.enabled` + +> - **Type:** `boolean` +> - **Default value:** `true` +> +> Enable adaptive partial aggregation feature. + +### `adaptive-partial-aggregation.min-rows` + +> - **Type:** `long` +> - **Default value:** `100,000` +> +> Minimum number of processed rows before partial aggregation might be adaptively turned off. + +### `adaptive-partial-aggregation.unique-rows-ratio-threshold` + +> - **Type:** `double` +> - **Default value:** `0.8` +> +> Ratio between aggregation output and input rows above which partial aggregation might be adaptively turned off. + +### `optimizer.join-multi-clause-independence-factor` + +> - **Type:** `double` +> - **Default value:** `0.25` +> +> Scales the strength of independence assumption for selectivity estimates of multi-clause joins. + +### `optimizer.filter-conjunction-independence-factor` + +> - **Type:** `double` +> - **Default value:** `0.75` +> +> Scales the strength of independence assumption for selectivity estimates of the conjunction of multiple filters. + + ## Regular Expression Function Properties The following properties allow tuning the [regexp](../functions/regexp.html). @@ -756,6 +945,34 @@ of constructing another execution plan, thus reducing the amount of query pre-pr > > Time in milliseconds to expire cached execution plans after the last access +### `hetu.execution.data-cache.enabled` +> +> - **Type:** `boolean` +> - **Default value:** `false` +> +> Enable caching materialization of _Common Table Execution_(**CTE**) thereby, by-passing frequently executed sub-plans. + +### `hetu.execution.data-cache.max-size` +> +> - **Type:** `long` +> - **Default value:** `2147483648` +> +> Maximum size of the total data stored in CTE materialization storage cache. + +### `hetu.execution.data-cache.schema-name` +> +> - **Type:** `string` +> - **Default value:** `cache` +> +> Schema which shall include the materialized cache tables. + +### `hetu.execution.data-cache.connector-name` +> +> - **Type:** `string` +> - **Default value:** `hive` +> +> Catalog name which shall include the materialized cache tables. + ## SplitCacheMap Properties SplitCacheMap must be enabled to support caching row data. When enabled, the coordinator stores table, partition and split scheduling metadata that @@ -856,14 +1073,32 @@ helps with cache affinity scheduling. > Note: > For snapshot recovery `query.remote-task.max-error-duration` should be greater than `exchange.max-error-duration`. -## Distributed Snapshot +### `query.execution-policy` +> - **Type:** `string` +> - **Default value:** `all-at-once` +> +> Specifies the execution policy enforced by the scheduler. One of following set of execution policies can be configured: - +> 1. _**all-at-once**_: This policy makes available all stages for scheduler to process and start. +> 2. _**phased**_: This policy follows the stage dependency based on the producer source, and schedule all independent stages together. +> 3. _**prioritize-utilization**_: This policy follows the stage dependency in addition to producer source, it also looks at dynamic filters producers for dependent paths. +> + + +## Query Recovery + +### `recovery_enabled` + +> - **Type:** `boolean` +> - **Default value:** `false` +> +> This session property is used to enable or disable the recovery framework, which enables to restart/resume the query in case of failure. ### `snapshot_enabled` > - **Type:** `boolean` > - **Default value:** `false` > -> This session property is used to enable or disable the distributed snapshot functionality. +> This session property is enabled to capture snapshots during query execution, when recovery framework is enabled. Without recovery framework enabled this flag has no significance ### `hetu.experimental.snapshot.profile` @@ -875,23 +1110,23 @@ helps with cache affinity scheduling. > > This is an experimental property. In the future it may be allowed to store snapshots in non-file-system locations, e.g. in a connector. -### `hetu.snapshot.maxRetries` +### `hetu.recovery.maxRetries` > - **Type:** `int` > - **Default value:** `10` > > This property defines the maximum number of error recovery attempts for a query. When the limit is reached, the query fails. > -> This can also be specified on a per-query basis using the `snapshot_max_retries` session property. +> This can also be specified on a per-query basis using the `recovery_max_retries` session property. -### `hetu.snapshot.retryTimeout` +### `hetu.recovery.retryTimeout` > - **Type:** `duration` > - **Default value:** `10m` (10 minutes) > > This property defines the maximum amount of time for the system to wait until all tasks are successfully restored. If any task is not ready within this timeout, then the recovery attempt is considered a failure, and the query will try to resume from an earlier snapshot if available. > -> This can also be specified on a per-query basis using the `snapshot_retry_timeout` session property. +> This can also be specified on a per-query basis using the `recovery_retry_timeout` session property. ### `hetu.snapshot.useKryoSerialization` @@ -900,6 +1135,12 @@ helps with cache affinity scheduling. > > Enables Kryo based serialization for snapshot, instead of default java serializer. +### `experimental.eliminate-duplicate-spill-files` + +> - **Type:** `boolean` +> - **Default value:** `false` +> +> Enables elimination of duplicate spill files storage as part of snapshot capture. ## HTTP Client Configurations @@ -922,3 +1163,239 @@ helps with cache affinity scheduling. > After the configured time elapsed and no response received, then client connection consider that to be failure in submission of request. > > (Note: this parameter should be configured with higher time when in high load environment) + +## Connector Properties configuration + +### `case-insensitive-name-matching` +> +> - **Type:** `boolean` +> - **Default value:** `false` +> +> Case-insensitive matching between database and collection names. The default is case sensitive. + +## Fault Tolerant Execution + +### `exchange-manager.name` + +> - **Type:** `string` +> - **Default value:** `filesystem` +> +> The name of task snapshot manager. + +### `exchange-filesystem-type` +> - **Type:** `string` +> - **Default value:** `exchange` +> +> The file system client name used by the task snapshot. If `exchange-filesystem-type=exchange`, the client named `exchange` will be used to store the Task snapshot during query execution. The configuration file path for file system clients is `etc/filesystem/exchange.properties` (the properties file name needs to be consistent with exchange-filesystem-type), see [Exchange File System Client](<./properties.html#exchange-file-system-client>). + +### `exchange.base-directories` +> +> - **Type:** `URI` +> +> The initial file directory which is utilized to store the snapshots of tasks. If there is more than one root directory, commas are used to separate them. + +### `exchange.compression-enabled` +> - **Type:** `boolean` +> - **Default value:** `false` +> +> The property enables the compression feature of task snapshot. + +### `exchange.max-page-storage-size` +> +> - **Type:** `data size` +> - **Default value:** `16MB` +> +> Max storage size of a page written to a sink, including the page itself and its size represented by an int + +### `exchange.sink-buffer-pool-min-size` +> +> - **Type:** `int` +> - **Default value:** `10` +> +> The minimum buffer pool size for an exchange sink. The larger the buffer pool size, the larger the write parallelism and memory usage. + +### `exchange.sink-buffers-per-partition` +> +> - **Type:** `int` +> - **Minimum value:** `2` +> - **Default value:** `2` +> +> The number of buffers per partition in the buffer pool. The larger the buffer pool size, the larger the write parallelism and memory usage. + +### `exchange.sink-max-file-size` +> +> - **Type:** `data size` +> - **Default value:** `1GB` +> +> Max size of files written by exchange sinks. + +### `exchange.source-concurrent-readers` +> +> - **Type:** `int` +> - **Default value:** `4` +> +> Number of concurrent readers to read from spooling storage. The larger the number of concurrent readers, the larger the read parallelism and memory usage. + +### `exchange.max-output-partition-count` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `50` +> +> Max number of distinct partitions to be created by the exchange. + +### `exchange.file-listing-parallelism` +> +> - **Type:** `int` +> - **Default value:** `50` +> +> Max parallelism of file listing calls when enumerating spooling files + +### `retry-policy` +> +> - **Type:** `string` +> - **Allowed values:** `NONE`, `TASK` +> - **Default value:** `NONE` +> +> Configures what is retried in the event of failure, TASK to retry tasks individually if they fail. + +### `task-retry-attempts-overall` +> +> - **Type:** `int` +> - **Default value:** `null(no limit)` +> +> Maximum number retries across all tasks within a given query before declaring the query as failed. + +### `task-retry-attempts-per-task` +> +> - **Type:** `int` +> - **Minimum value:** `0` +> - **Default value:** `4` +> +> Maximum number of times openLooKeng may attempt to retry a single task before declaring the query as failed. + +### `retry-initial-delay` +> +> - **Type:** `duration` +> - **Default value:** `10s` +> +> Minimum time that a failed task must wait before it is retried. May be overridden with the retry_initial_delay session property. + +### `retry-max-delay` +> +> - **Type:** `duration` +> - **Default value:** `1m` +> +> Maximum time that a failed task must wait before it is retried. Wait time is increased on each subsequent failure. May be overridden with the retry_max_delay session property. + +### `retry-delay-scale-factor` +> +> - **Type:** `double` +> - **Default value:** `2.0` +> +> Factor by which retry delay is increased on each task failure. May be overridden with the retry_delay_scale_factor session property. + +### `max-tasks-waiting-for-node-per-stage` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `5` +> +> Allow for up to configured number of tasks to wait for node allocation per stage, before pausing scheduling for other tasks from this stage. + +### `fault-tolerant-execution-target-task-input-size` +> +> - **Type:** `data size` +> - **Default value:** `4GB` +> +> Target size in bytes of all task inputs for a single fault-tolerant task. Applies to tasks that read input from spooled data written by other tasks. +> May be overridden for the current session with the fault_tolerant_execution_target_task_input_size session property. + +### `fault-tolerant-execution-target-task-split-count` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `64` +> +> Target number of standard splits processed by a single task that reads data from source tables. Value is interpreted with split weight taken into account. If the weight of splits produced by a catalog denotes that they are lighter or heavier than “standard” split, then the number of splits processed by single task is adjusted accordingly. +> May be overridden for the current session with the fault_tolerant_execution_target_task_split_count session property. + +### `fault-tolerant-execution-preserve-input-partitions-in-write-stage` +> +> - **Type:** `boolean` +> - **Default value:** `true` +> +> Ensure single task reads single hash partitioned input partition for stages which write table data + +### `fault-tolerant-execution-min-task-split-count` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `16` +> +> Minimum number of splits processed by a single task. This value is not split weight-adjusted and serves as protection against situations where catalogs report an incorrect split weight. +> May be overridden for the current session with the fault_tolerant_execution_min_task_split_count session property. + +### `fault-tolerant-execution-max-task-split-count` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `256` +> +> Maximum number of splits processed by a single task. This value is not split weight-adjusted and serves as protection against situations where catalogs report an incorrect split weight. +> May be overridden for the current session with the fault_tolerant_execution_max_task_split_count session property. + +### `fault-tolerant-execution-partition-count` +> +> - **Type:** `int` +> - **Minimum value:** `1` +> - **Default value:** `50` +> +> Number of partitions to use for distributed joins and aggregations. May be overridden for the current session with the fault_tolerant_execution_partition_count session property. + +### `fault-tolerant-execution-task-descriptor-storage-max-memory` +> +> - **Type:** `data size` +> - **Default value:** `(JVM heap size * 0.15)` +> +> Maximum amount of memory to be used to store task descriptors for fault tolerant queries on coordinator. Extra memory is needed to be able to reschedule tasks in case of a failure. + +### `node-scheduler.allowed-no-matching-node-period` +> +> - **Type:** `duration` +> - **Default value:** `2m` +> +> Wait time for scheduler before failing a query for which hard task requirements cannot be satisfied. + +### `node-scheduler.allocator-type` +> +> - **Type:** `string` +> - **Allowed values:** `BIN_PACKING`, `FIXED_COUNT` +> - **Default value:** `BIN_PACKING` +> +> Configures node allocator type. + +## Exchange File System Client + +### `fs.client.type` +> - **Type:** `string` +> - **Allowed values:** `local`, `hdfs` +> - **Default value:** `local` +> +> The type of file system client is configured by the property. if `fs.client.type=local`, recover framework uses local file system client, else if `fs.client.type=hdfs` recover framework uses hdfs file system client. + +### `hdfs.config.resources` +> - **Type:** `string` +> - **Default value:** `etc/filesystem/core-site.xml,etc/filesystem/hdfs-site.xml` +> +> The URI of resource configuration file that the hdfs file system depends on. +> +> (Note that this is a configuration property required by the hdfs file system, and when there is more than one resource configuration file, it is separated by ','.) + +### `hdfs.authenticcation.type` +> - **Type:** `string` +> - **Default value:** `NONE` +> +> The property specifies the authentication type for the hdfs file system. +> +> (Note that this is a configuration property required by the hdfs file system.) diff --git a/content/en-us/docs/docs/admin/reliable-query/_index.md b/content/en-us/docs/docs/admin/reliable-query/_index.md new file mode 100644 index 0000000000000000000000000000000000000000..01fe9a243b8aba54a214f87980c27bbdbdaefb70 --- /dev/null +++ b/content/en-us/docs/docs/admin/reliable-query/_index.md @@ -0,0 +1,13 @@ + +# Overview + +When a node in a cluster fails as result of network, hardware, or software issues, all queries with tasks running on the failing node will be lost. This can significantly impact cluster productivity and waste precious resources, especially for long running queries. + +One way to overcome this is to automatically rerun those impacted queries. This reduces the need for human intervention and increases fault tolerance, but as a result, the total execution time can be much longer. + +To achieve better performance while maintaining execution reliability, there are two mechanisms in openLooKeng provided: + +- [Operator Snapshot](./reliable-query/operator-snapshot.html) +- [Task Snapshot](./reliable-query/task-snapshot.html) + + diff --git a/content/en-us/docs/docs/admin/reliable-execution.md b/content/en-us/docs/docs/admin/reliable-query/operator-snapshot.md similarity index 66% rename from content/en-us/docs/docs/admin/reliable-execution.md rename to content/en-us/docs/docs/admin/reliable-query/operator-snapshot.md index 8ef03fd475c7d72139f2119e3d111a2bac1cd5b6..83c95dad473a4a180a7aee45ad1aab5da2fe7c26 100644 --- a/content/en-us/docs/docs/admin/reliable-execution.md +++ b/content/en-us/docs/docs/admin/reliable-query/operator-snapshot.md @@ -1,19 +1,15 @@ -# Reliable Query Execution +# Operator Snapshot ## Overview -When a node in a cluster fails as result of network, hardware, or software issues, all queries with tasks running on the failing node will be lost. This can significantly impact cluster productivity and waste precious resources, especially for long running queries. - -One way to overcome this is to automatically rerun those impacted queries. This reduces the need for human intervention and increases fault tolerance, but as a result, the total execution time can be much longer. - -To achieve better performance while maintaining execution reliability, the *distributed snapshot* mechanism in openLooKeng takes periodic snapshots of the complete state of the query execution. When an error occurs, the query can resume execution from the last successful snapshot. The implementation is based on the standard [Chandy-Lamport algorithm](https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport_algorithm). +The Operator Snapshot in openLooKeng takes periodic snapshots of the complete state of the query execution. When an error occurs, the query can resume execution from the last successful snapshot. The implementation is based on the standard [Chandy-Lamport algorithm](https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport_algorithm). As of release 1.2.0, openLooKeng supports recovery of tasks and worker node failures. -## Enable Distributed Snapshot +## Enable Recovery framework -Distributed snapshot is most useful for long running queries. It is disabled by default, and must be enabled and disabled via a session property [`snapshot_enabled`](properties.html#snapshot_enabled). It is recommended that the feature is only enabled for complex queries that require high reliability. +Recovery framework is most useful for long running queries. It is disabled by default, and must be enabled or disabled via a session property [`recovery_enabled`](../properties.html#recovery_enabled). It is recommended that the feature is only enabled for complex queries that require high reliability. ## Requirements @@ -37,7 +33,7 @@ When a query that does not meet the above requirements is submitted with distrib ## Detection -Error recovery is triggered when communication between the coordinator and a remote task fails for an extended period of time, as controlled by the [`query.remote-task.max-error-duration`](properties.html#queryremote-taskmax-error-duration) configuration. +Error recovery is triggered when communication between the coordinator and a remote task fails for an extended period of time, as controlled by the [`Failure Recovery handling Properties`](<../properties.html#failure-recovery-handling-properties>) configuration. ## Storage Considerations @@ -55,18 +51,20 @@ Each query execution may produce multiple snapshots. Contents of these snapshots The ability to recover from an error and resume from a snapshot does not come for free. Capturing a snapshot, depending on complexity, takes time. Thus it is a trade-off between performance and reliability. -It is suggested to only turn on distributed snapshot when necessary, i.e. for queries that run for a long time. For these types of workloads, the overhead of taking snapshots becomes negligible. +It is suggested to turn on snapshot capture when necessary, i.e. for queries that run for a long time. For these types of workloads, the overhead of taking snapshots becomes negligible. ## Snapshot statistics Snapshot capture and restore statistics are displayed in CLI along with query result when CLI is launched in debug mode -Snapshot capture statistics covers size of snapshots captured, CPU Time taken for capturing the snapshots and Wall Time taken for capturing the snapshots during the query. These statistics are displayed for all snapshots and for last snapshot separately. +Snapshot capture statistics includes number of snapshots captured, size of snapshots captured, CPU Time taken for capturing the snapshots and Wall Time taken for capturing the snapshots during the query. These statistics are displayed for all snapshots and for last snapshot separately. Snapshot restore statistics covers number of times restored from snapshots during query, Size of the snapshots loaded for restoring, CPU Time taken for restoring from snapshots and Wall Time taken for restoring from snapshots. Restore statistics are displayed only when there is restore(recovery) happened during the query. -![](../images/snapshot_statistics.png) +Additionally, while query is in progress number of capturing snapshots and id of the restoring snapshot will be displayed. Refer below picture for more details + +![](../../images/snapshot_statistics.png) ## Configurations -Configurations related to distributed snapshot feature can be found in [Properties Reference](properties.html#distributed-snapshot). +Configurations related to recovery framework feature can be found in [Properties Reference](<../properties.html#query-recovery>). \ No newline at end of file diff --git a/content/en-us/docs/docs/admin/reliable-query/task-snapshot.md b/content/en-us/docs/docs/admin/reliable-query/task-snapshot.md new file mode 100644 index 0000000000000000000000000000000000000000..1f7c96a53c51747c4a2fc1bc585e9f28c9dd5b33 --- /dev/null +++ b/content/en-us/docs/docs/admin/reliable-query/task-snapshot.md @@ -0,0 +1,78 @@ + +# Task Snapshot + +## Overview + +Task Snapshot is another mechanism for reliable query, it is a trade-off of Retry Query and Reliable Query Execution by Operator Snapshot. +Compared with Retry Query, query can be resumed through task snapshot instead of re-execution after a failure occurs, thereby optimizing the recovery time. +Compared with Reliable Query Execution by Operator Snapshot, the granularity space usage of task snapshot is smaller than that of operator snapshot. + +By default, a query needs to be retried manually after the query fails. Task Snapshot in openLooKeng is used to ensure reliable query execution by resuming automatically. +After the Task Snapshot is enabled, the exchange data between tasks are materialized through snapshots during the worker in query process. +When the coordinator in query process detects that worker is faulty, or the query failures, the coordinator schedules the idle worker to read the last saved task snapshot by the faulty worker, and then automatically resume query execution. +This can ensure the success of long-running queries, therefore, users don't need to watch over the long-running queries all the time in case there might be a failure, which enables openLooKeng for batch scenarios. + +![](../../images/task_snapshot_overview.png) + +## Enable Recovery framework + +Recovery framework is most useful for complex queries that need long-running time. It is disabled by default, and must be enabled or disabled via a configuration property `retry-policy` in the configuration file `etc/config.properties`. +- **Enable**: `retry-policy=TASK` +- **Disable**: `retry-policy=NONE` + +## Requirements + +To ensure that query execution can be automatically resumed through a previously saved task snapshot and query execution is reliable, the cluster needs to meet the following requirements: +- the recovery framework is enabled +- the cluster contains at least 2 worker nodes +- the surviving worker nodes in the cluster are still active + +## Limitations + +- **Supported Statements**: only `SELECT`, `INSERT` and `CREATE TABLE AS SELECT(CTAS)` types of statements are supported + - This does *not* include statements like `INSERT INTO CUBE` +- **Source tables**: can only read from tables in `Hive` catalog +- **Target table**: can only write into tables in `Hive` catalog, with `ORC` format +- **Interaction with other features**: Task Snapshot feature and Operator Snapshot feature are not supported at the same time + +## Detection + +The recovery query execution is triggered when the communication between the coordinator and the worker in query process failure or times out. The retry configuration of the relevant fault recovery properties is as follows: +- property: `task-retry-attempts-per-task` + - default value: 4 + - description: Maximum number of times may attempt to retry a single task before declaring the query as failed. +- property: `task-initial-delay` + - default value: 10s + - description: Minimum time that a failed task must wait before it is retried. +- property: `task-max-delay` + - default value: 1m + - description: Maximum time that a failed task must wait before it is retried. + +## Storage Considerations + +The exchange data and status between tasks in worker nodes are materialized to the file system as task snapshot when the query executes, when the file system supports two types: local and hdfs. +When a failure occurs, the coordinator in query process schedules other idle worker nodes to read task snapshot from the specified file directory in the file system, such as to implement reliable query execution. +Therefore, all worker nodes are required to be able to access the file system that stores task snapshots. The exchange manager is adopted to manage file system related properties in the configuration file etc/exchange-manager.properties. +Related properties are as follows: +- property: `exchange-manager.name` + - description: The name of exchange manager is configured by the property. +- property: `exchange-filesystem-type` + - description: The type name of file system client used by the task snapshot. If `exchange-filesystem-type=exchange`, the client named `exchange` will be used to store the Task snapshot during query execution. The configuration file path for file system clients is `etc/filesystem/exchange.properties` (the properties file name needs to be consistent with exchange-filesystem-type), see [Exchange File System Client Properties Reference](<../properties.html#exchange-file-system-client>). +- property: `exchange.base-directories` + - description: The property specifies the URI shared by the file system and accessible to all worker nodes. All task snapshots are stored under the current URI. +- property: `exchange.compression-enabled` + - description: The property enables whether to compress task snapshots when they are stored. + +Note that task snapshots are used to save the exchange data and status between tasks in worker nodes during query execution. ensure that the cluster has enough free storage space to store these task snapshots before query execution, and the site is automatically cleaned to release storage space after query finished. + + +## Performance Overhead + +The ability to recover from an error and resume from a snapshot does not come for free. Capturing a snapshot, depending on complexity, takes time. Thus it is a trade-off between performance and reliability. + +It is suggested to turn on snapshot capture when necessary, i.e. for queries that run for a long time. For these types of workloads, the overhead of taking snapshots becomes negligible. + + +## Configurations + +Configurations related to recovery framework feature can be found in [Properties Reference](<../properties.html#fault-tolerant-execution>). \ No newline at end of file diff --git a/content/en-us/docs/docs/admin/tuning.md b/content/en-us/docs/docs/admin/tuning.md index db40a51db76e07d954d3aec4d4cc86aea2d7f709..e05a2ded06da2daea58f2742589446958293ae4e 100644 --- a/content/en-us/docs/docs/admin/tuning.md +++ b/content/en-us/docs/docs/admin/tuning.md @@ -27,3 +27,35 @@ The following can be helpful for diagnosing GC issues: -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 ``` + +## Performance Tuning Notes +Below are few notes on configurations which help with optimization and tuning query execution performance. + +### Common Table Expression(CTE) Optimization +Common Table expressions are common sub-plans under the query plan which are used more than ones within the query. +openLooKeng engine assesses the usage of such CTE plan nodes for feasibility of reuse based optimization. This optimization is applied in 2 scenarios: + +> #### Reuse Execution Pipeline +> Given the configuration [optimizer.cte-reuse-enabled](./properties.html#optimizercte-reuse-enabled) is enabled: Same CTE nodes in a given query are arranged in the stage pipeline such that only 1 executes and other reuse the output from the first node wiz. deemed as producer and reset as consumers. +> Its important to note that CTE pipeline cannot be used if CTE is used in Self join with the same CTE. +> Pipeline mode execution reduces the duplicate scanning and processing of the operations, thereby reducing the round trip time for the query. + +> #### Materialize and Reused +> Given the configuration [enable-cte-result-cache](./properties.html#enable-cte-result-cache) is enabled: CTE results are materialized to user's choice of storage(refer [data-cache configuration](./properties.html#hetuexecutiondata-cacheschema-name)). This approach caches the output of the CTE node which is read and reused for subsequent queries after materialization succeeds. + +### Plan optimizations +* **Use exact partitioning**: When enabled this forces data repartitioning unless the partitioning of upstream stage matches exactly what downstream stage expects (refer: [exact partitioning](./properties.html#optimizeruse-exact-partitioning)). + +* **Adaptive aggregations**: This feature engages adaptive partial aggregation; it is governed by following 3 configurations: +> 1) [Enable Adaptive Partial Aggregation](./properties.html#adaptive-partial-aggregationenabled): enable the feature. +> 2) [Minimum rows threshold](./properties.html#adaptive-partial-aggregationmin-rows): Minimum number of processed rows before partial aggregation might be adaptively turned off. +> 3) [Unique rows ratio threshold](./properties.html#adaptive-partial-aggregationunique-rows-ratio-threshold): Ratio between aggregation output and input rows above which partial aggregation might be adaptively turned off. + +* **Join Independence assumptions for Selectivity estimates**: +> 1) [Multi-Clause join Independence factor](./properties.html#optimizerjoin-multi-clause-independence-factor): Scales the strength of independence assumption for selectivity estimates of multi-clause joins. +> 2) [Filter Conjunction Independence factor](./properties.html#optimizerfilter-conjunction-independence-factor): Scales the strength of independence assumption for selectivity estimates of the conjunction of multiple filters. + +* **Execution policy for better filtering**: Specifies the execution policy enforced by the scheduler. One of following set of execution policies can be configured (refer [execution policy specification](./properties.html#queryexecution-policy)): +> 1. _**all-at-once**_: This policy makes available all stages for scheduler to process and start. +> 2. _**phased**_: This policy follows the stage dependency based on the producer source, and schedule all independent stages together. +> 3. _**prioritize-utilization**_: This policy follows the stage dependency in addition to producer source, it also looks at dynamic filters producers for dependent paths. diff --git a/content/en-us/docs/docs/admin/web-interface.md b/content/en-us/docs/docs/admin/web-interface.md index 4d6e383bd6a79a0c28b1d992ad93577c34eed44f..6105100408f6728815bb18da21e7fb2201699a99 100644 --- a/content/en-us/docs/docs/admin/web-interface.md +++ b/content/en-us/docs/docs/admin/web-interface.md @@ -61,4 +61,26 @@ and statistics about the query is available by clicking the *JSON* link. These v > - **Type:** `duration` > - **Default value:** `1 DAYS` > -> UI session timeout is set to 1 day as default. This could be overridden via `hetu.queryeditor-ui.session-timeout` of `etc/config.properties` \ No newline at end of file +> UI session timeout is set to 1 day as default. This could be overridden via `hetu.queryeditor-ui.session-timeout` of `etc/config.properties` + +### `hetu.queryhistory.max-count` + +> - **Type:** `int` +> - **Default value:** `1000` +> +> The maximum number of query history stored by openLooKeng. This could be overridden via "hetu.queryhistory.max-count" of "etc/config.properties". + +### `hetu.collectionsql.max-count` + +> - **Type:** `int` +> - **Default value:** `100` +> +> The Maximum number of SQL collected by each user. This could be overridden via "hetu.collectionsql.max-count" of "etc/config.properties". + +## Remarks + +The max length of the favorite SQL is 600 by default. You can modify it through the following steps: + +1. Login MySQL database according to the JDBC configuration of `hetu-metastore.properties` + +2. Select table hetu_favorite, execute script `alter table hetu_favorite modify query varchar(2000) not null;` to modify the max length of the favorite SQL. \ No newline at end of file diff --git a/content/en-us/docs/docs/connector/Hudi.md b/content/en-us/docs/docs/connector/Hudi.md index e9b0319eaafa3e21eda90190898d6c3f6ed42256..5b93d3038d779f6f35df7aff2c51f75272ac23fa 100644 --- a/content/en-us/docs/docs/connector/Hudi.md +++ b/content/en-us/docs/docs/connector/Hudi.md @@ -1,5 +1,8 @@ # Hudi Connector +### Release Notes +Currently Hudi only supports version 0.7.0. + ### Hudi Introduction Apache Hudi is a fast growing data lake storage system that helps organizations build and manage petabyte-scale data lakes. Hudi enables storing vast amounts of data on top of existing DFS compatible storage while also enabling stream processing in addition to typical batch-processing. This is made possible by providing two new primitives. Specifically, diff --git a/content/en-us/docs/docs/connector/dameng.md b/content/en-us/docs/docs/connector/dameng.md new file mode 100644 index 0000000000000000000000000000000000000000..ef9c1df1b1f681515ee75d76a55c8ddad11cbf4b --- /dev/null +++ b/content/en-us/docs/docs/connector/dameng.md @@ -0,0 +1,162 @@ +# DM(dameng) Connector + +## Overview + +The DM connector allows querying and creating tables in an external DM database. This can be used to join data between different systems like DM and Hive, or between two different DM instances. + +## Configuration + +Before using the DM connector, you should prepare: + +- JDBC connection details for connecting to the DM database + +To configure the DM connector, create a catalog properties file in`etc/catalog` named, for example, `dameng.properties`. Create the file with the following contents, replacing the connection properties as appropriate for your setup: + +``` properties +connector.name=dameng +connection-url=jdbc:dm://host:port/SYSDBA +connection-user=username +connection-password=password +``` + +- Adding the DM driver + +The DM JDBC driver is not provided in a common repository. If you are an DM database user, you can visit the official DM website, and download and deploy the DM JDBC driver to the repository on the condition that the license for the DM JDBC driver is complied with. The DM JDBC driver (**DmJdbcDriver***X*.**jar** where *X* is a number and varies according to the DM version) may be installed as a part of the DM client. After obtaining the DM JDBC driver, you can deploy the **DmJdbcDriver***X*.**jar** file to the openLooKeng plugin folder on the coordinator and worker. For example, if the JDBC driver file is **DmJdbcDriver***X*.**jar** and the openLooKeng plugin package folder is **/opt/hetu-server/plugin**, run the following command: **cp DmJdbcDriverX.jar /opt/hetu-server/plugin/dm**。 Restart the coordinator and worker. Then, the DM connector can work properly. + +- Whether to enable the query pushdown function + +The pushdown function of the DM connector is enabled by default, and you do not need to perform any operation. You can also set the parameter as follows: + +``` properties +jdbc.pushdown-enabled=true +#true indicates that pushdown is enabled, and false indicates that pushdown is disabled. +``` + +- Mode for the push-down feature + +If you want to enable the connector all push down feature for DM connector, you do not need to do any things for DM connector's push down feature, which is FULL_PUSHDOWN on by default. But you can also set as below: + +``` properties +jdbc.pushdown-module=FULL_PUSHDOWN +#FULL_PUSHDOWN: All push down. BASE_PUSHDOWN: Partial push down, which indicates that filter, aggregation, limit, topN and project can be pushed down. +``` + +- More configurations + +DM and Oracle are of the same origin, some configurations of Oracle connector are reused when implementing DM connector. Please refer to [ORACLE](./oracle.html). + +## Querying DM Using openLooKeng + +For the DM connector named **dameng**, each DM database user can run the **SHOW SCHEMAS** command to obtain the available schemas: + + SHOW SCHEMAS FROM dameng; + +If you have obtained the available schemas, run the **SHOW TABLES** command to view the tables owned by the DM database named **data**: + + SHOW TABLES FROM dameng.data; + +To view a list of columns in a table named **hello** in data schema, run either of the following commands: + + DESCRIBE dameng.data.hello; + SHOW COLUMNS FROM dameng.data.hello; + +You can access the **hello** table in the **data** schema: + + SELECT * FROM dameng.data.hello; + +The connector's permissions in these schemas are your permissions configured in the connection property file. If you cannot access the tables, a specific connector cannot access them. + +## DM Update/Delete Support + +### Create DM Table + +Example： + +```sql +CREATE TABLE dameng_table ( + id int, + name varchar(255)); +``` + +### INSERT on DM tables + +Example： + +```sql +INSERT INTO dameng_table + VALUES + (1, 'foo'), + (2, 'bar'); +``` + +### UPDATE on DM tables + +Example： + +```sql +UPDATE dameng_table + SET name='john' + WHERE id=2; +``` + +Above example updates the column `name`'s value to `john` of row with column `id` having value `2`. + +SELECT result before UPDATE: + +```sql +lk:default> SELECT * FROM dameng_table; +id | name +----+------ + 2 | bar + 1 | foo +(2 rows) +``` + +SELECT result after UPDATE + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 2 | john + 1 | foo +(2 rows) +``` + +### DELETE on DM tables + +Example： + +```sql +DELETE FROM dameng_table + WHERE id=2; +``` + +Above example delete the row with column `id` having value `2`. + +SELECT result before DELETE: + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 2 | john + 1 | foo +(2 rows) +``` + +SELECT result after DELETE: + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 1 | foo +(1 row) +``` + +## Restrictions on the DM Connector + +- The openLooKeng can connect to DM Database 8. + +- The DM Connector does not support query pushdown for DM Update yet. \ No newline at end of file diff --git a/content/en-us/docs/docs/connector/elasticsearch.md b/content/en-us/docs/docs/connector/elasticsearch.md index 424d66699ffb48b6ca97da598db354e1cb845da7..1652ce0e892fa4a1985a0aa0c115eb9821a1fc10 100644 --- a/content/en-us/docs/docs/connector/elasticsearch.md +++ b/content/en-us/docs/docs/connector/elasticsearch.md @@ -45,6 +45,40 @@ Defines the schema that will contain all tables defined without a qualifying sch This property is optional; the default is `default`. +### `elasticsearch.pushdown.enabled` +Whether to enable the query pushdown function + +The pushdown function of the Elasticsearch connector is disabled by default. You can set the parameter as follows: + +``` properties +elasticsearch.pushdown.enabled=true +#true indicates that pushdown is enabled, and false indicates that pushdown is disabled. +``` +NOTE: Currently pushdown is only supported as part of where clause partially such as EQUAL, AND, OR, Arithmetic operators, BETWEEN, IN +and in the case of aggregations such as GROUP BY , SUM, COUNT functions. + +Support matrix for pushdown for different datatypes is as follows: + +| Elasticsearch | openLooKeng | Aggregation Pushdown Supported | Predicate Pushdown Supported | +|:---------------|:------------|:-------------------------------|:-----------------------------| +| `binary` | VARBINARY | No | No | +| `boolean` | BOOLEAN | Yes | Yes | +| `double` | DOUBLE | Yes | Yes | +| `float` | REAL | Yes | Yes | +| `byte` | TINYINT | Yes | Yes | +| `short` | SMALLINT | Yes | Yes | +| `integer` | INTEGER | Yes | Yes | +| `long` | BIGINT | Yes | Yes | +| `keyword` | VARCHAR | Yes | Yes | +| `text` | VARCHAR | No | Yes | +| `date` | TIMESTAMP | Yes | Yes | +| `ip` | IPADDRESS | Yes | Yes | +| `(all others)` | | No | No | + +**Note** + +Due to functional limitations of Elasticsearch on aggregations for TEXT field types, the type of pushdown will always fail with an error message returned. + ### `elasticsearch.scroll-size` This property defines the maximum number of hits that can be returned with each Elasticsearch scroll request. @@ -192,4 +226,4 @@ The array fields of this structure can be defined by using the following command ## Limitations 1. openLooKeng does not support to query table in Elasticsearch which has duplicated columns, such as column "name" and "NAME"; -2. openLooKeng does not support to query the table in Elasticsearch that the name has special characters, such as '-', '.', etc. \ No newline at end of file +2. openLooKeng does not support to query the table in Elasticsearch that the name has special characters, such as '-', '.', etc. diff --git a/content/en-us/docs/docs/connector/iceberg.md b/content/en-us/docs/docs/connector/iceberg.md new file mode 100644 index 0000000000000000000000000000000000000000..9526f7924a8c5af88db8826fa63acce8dbb1c450 --- /dev/null +++ b/content/en-us/docs/docs/connector/iceberg.md @@ -0,0 +1,371 @@ +# Iceberg connector +============== + +## Overview + +openLooKeng Iceberg is an open table format for huge analytic datasets. The Iceberg connector allows querying data stored in files written in Iceberg format. +Iceberg data files can be stored in either Parquet, ORC format, as determined by the format property in the table definition. The table format defaults to ORC. + +## Requirements + +To use Iceberg, you need: + +- Network access from the openLooKeng coordinator and workers to the distributed object storage. +- Access to a Hive metastore service (HMS) . +- Network access from the openLooKeng coordinator to the HMS. Hive metastore access with the Thrift protocol defaults to using port 9083. + +## Hive metastore catalog + +The Hive metastore catalog is the default implementation. When using it, the Iceberg connector supports the same metastore configuration properties as the Hive connector. At a minimum, hive.metastore.uri must be configured. + +```properties +connector.name=iceberg +hive.metastore.uri=thrift://localhost:9083 +``` + +### General configuration + +These configuration properties are independent of which catalog implementation is used. + +Iceberg general configuration properties +With the following content creation`etc/catalog/iceberg.properties`，please repalce`localhost:9083` with the right ip and port： + +| Attribute name | Attribute value |necessary | Description | +|:------------------------------------|:----------------------- |:-------|:-------------------------------------| +| connector.name | iceberg |true | connector.name | +| hive.metastore.uri | thrift://localhost:9083 |true | Hive connector.uri | +| iceberg.file-format | ORC |false | Define the data storage file format for Iceberg tables. Possible values are PARQUET、ORC | +|iceberg.compression-codec|ZSTD|false|The compression codec to be used when writing files. Possible values are (NONE SNAPPY LZ4 ZSTD GZIP) +|iceberg.use-file-size-from-metadata|true|false|Read file sizes from metadata instead of file system. This property should only be set as a workaround for this issue. The problem was fixed in Iceberg version 0.11.0. +|iceberg.max-partitions-per-writer|100|false|Maximum number of partitions handled per writer. +|iceberg.unique-table-location|true|false|Use randomized, unique table locations. +|iceberg.dynamic-filtering.wait-timeout|0s|false|Maximum duration to wait for completion of dynamic filters during split generation. +|iceberg.table-statistics-enabled|true|false|Enables Table statistics. The equivalent catalog session property is for session specific use. Set to to disable statistics. Disabling statistics means that Cost based optimizations can not make smart decisions about the query plan.statistics_enabledfalse +|iceberg.minimum-assigned-split-weight|0.05|false|A decimal value in the range (0, 1] used as a minimum for weights assigned to each split. A low value may improve performance on tables with small files. A higher value may improve performance for queries with highly skewed aggregations or joins. +## SQL support + +This connector provides read access and write access to data and metadata in Iceberg. In addition to the globally available and read operation statements, the connector supports the following features: + +### Table Related Statements +- create +```sql +CREATE TABLE ordersg ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) +WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']); +``` +### Column related statements +- INSERT +```sql +alter table ordersg add COLUMN zip varchar; +``` +- RENAME +```sql +ALTER TABLE ordersg RENAME COLUMN zip TO zip_r; +``` +- DELETE +```sql +ALTER TABLE ordersg DROP COLUMN zip_r; +``` +### Data related Statements +- INSERT +```sql +insert into ordersg values(1,date'1988-11-09',666888,'tim','US'); +``` +- DELETE +```sql +Delete from ordersg where order_id = 1; +``` +- UPDATE +```sql +update ordersg set customer = 'Alice' where order_id = 2; +``` +- SELECT +```sql +select * from ordersg; +``` + +## Partitioned tables + +Iceberg supports partitioning by specifying transforms over the table columns. A partition is created for each unique tuple value produced by the transforms. Identity transforms are simply the column name. Other transforms are: + +| Transform | Description | +|:------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| +| year(ts) | A partition is created for each year. The partition value is the integer difference in years between ts and January 1 1970. | +| month(ts) | A partition is created for each month of each year. The partition value is the integer difference in months between ts and January 1 1970. | +| day(ts) | A partition is created for each day of each year. The partition value is the integer difference in days between ts and January 1 1970. | +| hour(ts) | A partition is created hour of each day. The partition value is a timestamp with the minutes and seconds set to zero. | +| bucket(x,nbuckets) | The data is hashed into the specified number of buckets. The partition value is an integer hash of x, with a value between 0 and nbuckets - 1 inclusive. | +| truncate(s,nchars) | The partition value is the first nchars characters of s. | + +In this example, the table is partitioned by the month of order_date, a hash of account_number (with 10 buckets), and country: +```sql +CREATE TABLE ordersg ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) + WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']); +``` +Manually Modifying Partitions +```sql +ALTER TABLE ordersg SET PROPERTIES partitioning = ARRAY['month(order_date)']; +``` +For partitioned tables, the Iceberg connector supports the deletion of entire partitions if the WHERE clause specifies filters only on the identity-transformed partitioning columns, that can match entire partitions. Given the table definition above, this SQL will delete all partitions for which country is US: + +```sql +DELETE FROM iceberg.testdb.ordersg WHERE country = 'US'; +``` + +### Rolling back to a previous snapshot + +Iceberg supports a “snapshot” model of data, where table snapshots are identified by an snapshot IDs. + +The connector provides a system snapshots table for each Iceberg table. Snapshots are identified by BIGINT snapshot IDs. You can find the latest snapshot ID for table customer_orders by running the following command: +```sql +SELECT snapshot_id FROM "ordersg $snapshots" ORDER BY committed_at DESC LIMIT 1; +``` + +|snapshot_id| +|:----------| +|921254093881523606| +|535467754709887442| +|343895437069940394| +|34i302849038590348| +|(4 rows)| + +A SQL procedure system.rollback_to_snapshot allows the caller to roll back the state of the table to +a previous snapshot id: +```sql +CALL iceberg.system.rollback_to_snapshot('testdb', 'ordersg', 8954597067493422955); +``` + +## Metadata tables + +The connector exposes several metadata tables for each Iceberg table. These metadata tables contain information about the internal structure of the Iceberg table. You can query each metadata table by appending the metadata table name to the table name: + +```sql +SELECT * FROM "ordersg$data"; +``` + +### $data table# + +The $data table is an alias for the Iceberg table itself. + +The statement: +```sql +SELECT * FROM "ordersg$data"; +``` +is equivalent to: +```sql +SELECT * FROM ordersg; +``` + +### $properties table +The $properties table provides access to general information about Iceberg table configuration and any additional metadata key/value pairs that the table is tagged with. + +You can retrieve the properties of the current snapshot of the Iceberg table by using the following query: +```sql +SELECT * FROM "ordersg$properties"; +``` + +key | value | +-----------------------+----------+ +write.format.default | PARQUET | + + +### $history table# + +The $history table provides a log of the metadata changes performed on the Iceberg table. + +You can retrieve the changelog of the Iceberg table by using the following query: +```sql +SELECT * FROM "ordersg$history"; +``` + +made_current_at | snapshot_id | parent_id | is_current_ancestor +----------------------------------+----------------------+----------------------+-------------------- +2022-08-19 05:42:37.854 UTC | 7464177163099901858 | 7924870943332311497 | true +2022-08-19 05:44:35.212 UTC | 2228056193383591891 | 7464177163099901858 | true + +The output of the query has the following columns: + +| Name | Type |Description | +|:--------------------|:----------------------------|:----------------------| +| made_current_at | timestamp(3)with time zone |The time when the snapshot became active | +| snapshot_id | bigint |The identifier of the snapshot | +| parent_id | bigint |The identifier of the parent snapshot| +| is_current_ancestor | boolean |Whether or not this snapshot is an ancestor of the current snapshot| + +### $snapshots table + +The $snapshots table provides a detailed view of snapshots of the Iceberg table. A snapshot consists of one or more file manifests, and the complete table contents is represented by the union of all the data files in those manifests. + +You can retrieve the information about the snapshots of the Iceberg table by using the following query: + +```sql +SELECT * FROM "ordersg$snapshots"; +``` + +| committed_at | snapshot_id | parent_id | operation | manifest_list | summary | +| 2022-08-08 08:20:04.126 UTC | 7026585913702073835 | | append | hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/snap-7026585913702073835-1-d0b5ba3d-6363-4f32-974e-79bb68d19423.avro | {changed-partition-count=0, total-equality-deletes=0, total-position-deletes=0, total-delete-files=0, total-files-size=0, total-records=0, total-data-files=0} | +| 2022-08-08 08:21:58.343 UTC | 629134202395791160 | 7026585913702073835 | append | hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/snap-629134202395791160-1-b6e9c1c3-0532-4bf8-a814-a159494e272d.avro | {changed-partition-count=1, added-data-files=1, total-equality-deletes=0, added-records=1, total-position-deletes=0, added-files-size=289, total-delete-files=0, total-files-size=289, total-records=1, total-data-files=1} | + +The output of the query has the following columns: + +| Name | Type | Description | +|:--------------|:-----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| committed_at | timestamp(3) with time zone | The time when the snapshot became active | +| snapshot_id | bigint | The identifier for the snapshot | +| parent_id |bigint | The identifier for the parent snapshot | +| operation | varchar | The type of operation performed on the Iceberg table. The supported operation types in Iceberg are: -append when new data is appended -replace when files are removed and replaced without changing the data in the table -overwrite when new data is added to overwrite existing data -delete when data is deleted from the table and no new data is added| +| manifest_list | varchar | The list of avro manifest files containing the detailed information about the snapshot changes. | +| summary | map(varchar, varchar) | A summary of the changes made from the previous snapshot to the current snapshot | + +### $manifests table# + +The $manifests table provides a detailed overview of the manifests corresponding to the snapshots performed in the log of the Iceberg table. +You can retrieve the information about the manifests of the Iceberg table by using the following query: +```sql +SELECT * FROM "ordersg$manifests"; +``` + +Path | length | partition_spec_id | added_snapshot_id | added_data_files_count | existing_data_files_count | deleted_data_files_count | partitions +---------------------------------------+---------------------+---------------------+---------------------+-------------------+--------------------+--------------------+--------------------+--------------------+------------------- +hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/b6e9c1c3-0532-4bf8-a814-a159494e272d-m0.avro | 6534 | 0 | 629134202395791160 | 1 | 0 | 0 | [ ] + +The output of the query has the following columns: + +|Name| Type | Description | +|:----|:---------------------|:-------------------------------------------------------------------| +|path| varchar | The manifest file location | +|length | bigint | length | +|partition_spec_id| integer | partition_spec_id | +|added_snapshot_id | bigint | added_snapshot_id | +|added_data_files_count| integer | The number of data files with status ADDED in the manifest file | +|existing_data_files_count | integer | The number of data files with status EXISTING in the manifest file | +|deleted_data_files_count | integer | The number of data files with status DELETED in the manifest file | +|partitions| array(row(contains_null boolean, contains_nan boolean, lower_bound varchar, upper_bound varchar)) | Partition range metadata | + + +### $partitions table + +The $partitions table provides a detailed overview of the partitions of the Iceberg table. +You can retrieve the information about the partitions of the Iceberg table by using the following query: +```sql +SELECT * FROM "ordersg$partitions"; +``` + +| record_count | file_count | total_size | data | --------------+-------------+---------------------+---------------------+------------------------------------| 1 | 1 | 289 | {id={min=null, max=null, null_count=0, nan_count=null}, name={min=null, max=null, null_count=0, nan_count=null}} | + +The output of the query has the following columns: +Partitions columns + +|Name|Type |Description | +|:-----|:-----|:-------| +|record_count |bigint |The number of records in the partition| +|file_count| bigint|The number of files mapped in the partition | +|total_size |bigint|The size of all the files in the partition | +|data |row(... row (min ..., max ... , null_count bigint, nan_count bigint)) |Partition range metadat| + +### $files table# + +The $files table provides a detailed overview of the data files in current snapshot of the Iceberg table. + +To retrieve the information about the data files of the Iceberg table use the following query: +```sql +SELECT * FROM "ordersg$files"; +``` + +| content | file_path | file_format | record_count | file_size_in_bytes | column_sizes | value_counts | null_value_counts | nan_value_counts | lower_bounds | upper_bounds | key_metadata | split_offsets | equality_ids | +| 0 | hdfs://192.168.31.120:9000/user/hive/warehouse/orders19/data/20220819_034313_39152_vdmku-1709db2a-dc6f-4ef9-bb77-23f4c150801f.orc | ORC | 1 | 354 | | {1=1, 2=1, 3=1}","{1=0, 2=0, 3=0} | | | | | | | +| 0 | hdfs://192.168.31.120:9000/user/hive/warehouse/orders19/data/20220819_054009_11365_xq568-1803130c-6b7b-4da6-b460-dfb44f176ef4.orc | ORC | 1 | 413 | | {1=1, 2=1, 3=1, 4=1} | {1=0, 2=0, 3=0, 4=1} | | | | | | | + +The output of the query has the following columns: +Files columns + +|Name| Type | Description | +|:----|:--------|:------------------------------------------------------------------------------------------------------| +|content| integer| Type of content stored in the file. The supported content types in Iceberg are: -DATA(0) - POSITION_DELETES(1) - EQUALITY_DELETES(2)| +|file_path| varchar| The data file location | +|file_format| varchar | The format of the data file | +|record_count| bigint | The number of entries contained in the data file | +|file_size_in_bytes |bigint | The data file size | +|column_sizes| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding size in the file | +|value_counts| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding count of entries in the file | +|null_value_counts| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding count of NULL values in the file | +|nan_value_counts| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding count of non numerical values in the file | +|lower_bounds| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding lower bound in the file | +|upper_bounds| map(integer, bigint) | Mapping between the Iceberg column ID and its corresponding upper bound in the file | +|key_metadata |varbinary | Metadata about the encryption key used to encrypt this file, if applicable | +|split_offsets| array(bigint) | List of recommended split locations | +|equality_ids |array(integer) | The set of field IDs used for equality comparison in equality delete files | + + + +### ALTER TABLE EXECUTE + +The connector supports the following commands for use with ALTER TABLE EXECUTE(For details, see Merging Files). + +### File merging + +The optimize command is used for rewriting the active content of the specified table so that it is merged into fewer but larger files. In case that the table is partitioned, the data compaction acts separately on each partition selected for optimization. This operation improves read performance. + +All files with a size below the optional file_size_threshold parameter (default value for the threshold is 100MB) are merged: +```sql +ALTER TABLE ordersg EXECUTE optimize; +``` +The following statement merges the files in a table that are under 10 megabytes in size: +```sql +ALTER TABLE ordersg EXECUTE optimize(file_size_threshold => '10MB'); +``` + +### ALTER TABLE SET PROPERTIES + +The connector supports modifying the properties on existing tables using ALTER TABLE SET PROPERTIES. + +The following table properties can be updated after a table is created: +- format +- partitioning + +```sql +ALTER TABLE ordersg SET PROPERTIES format ='PARQUET'; +``` + +Or to set the column country as a partition column on a table: +```sql +ALTER TABLE ordersg SET PROPERTIES partitioning = ARRAY[, 'country']; +``` + +You can use SHOW CREATE TABLE Ordersg to display the current values of the TABLE properties. + + +### openLooKeng to Iceberg type mapping +| openLooKeng type | Iceberg type +|:-----------------------|:------------------------ +|BOOLEAN | BOOLEAN +|INTEGER | INT +|BIGINT | LONG +|REAL | FLOAT +|DOUBLE | DOUBLE +|DECIMAL(p,s) | DECIMAL(p,s) +|DATE | DATE +|TIME | TIME +|TIMESTAMP | TIMESTAMP +|TIMESTAMP WITH TIME ZONE | TIMESTAMPTZ +|VARCHAR | STRING +|VARBINARY | BINARY +|ROW(...) |STRUCT(...) +|ARRAY(e) |LIST(e) +|MAP(k,v) |MAP(k,v) + + + + + + + diff --git a/content/en-us/docs/docs/connector/mariadb.md b/content/en-us/docs/docs/connector/mariadb.md new file mode 100644 index 0000000000000000000000000000000000000000..50f5f9c10aeaaccc727aff1374b17ceba96de6ce --- /dev/null +++ b/content/en-us/docs/docs/connector/mariadb.md @@ -0,0 +1,97 @@ +The MariaDB connector allows querying and creating tables in external MariaDB databases. This can be used to join data between different systems such as MySQL and MariaDB or between two different MariaDB instances. + +## Configuration + +To configure the MariaDB connector, create a catalog properties file in `etc/catalog` in the openLooKeng installation directory, for example `maria.properties`, and mount the MariaDB connector as the maria directory. Create the file using the following and replace the connection properties according to the settings. + +```properties +connector.name=maria +connection-url=jdbc:mariadb://Deployed MairaDB ip:MariaDB service port (default 3306) +connection-user=Your data source user +connection-password=Your data source password +``` + +- Whether to enable the query push-down feature + +If you want to enable the connector push-down feature of MariaDB connector, you don't need to do anything, the MariaDB connector push-down feature is turned on by default. However, it can also be set as follows. + +```properties +#true means push down is turned on, false means off. +jdbc.pushdown-enabled=true +``` + +- Push down mode selection + +The default push-down mode of MariaDB connector is partial push-down, if you want to enable the full push-down function of MariaDB connector, you can set it as follows. + +```properties +#FULL_PUSHDOWN, means all push down; BASE_PUSHDOWN, means partial push down, where partial push down means filter/aggregation/limit/topN/project which can be pushed down. +jdbc.pushdown-module=FULL_PUSHDOWN +``` + +### External function registration + +The MariaDB connector supports registration of external functions. + +Configure the external function registration namespace `catalog.schema` that supports push-down. For example, in `etc/catalog/maria.properties` configure. + +```properties +jdbc.pushdown.remotenamespace=mariafun.default +``` + +### External function push-down + +Push down external functions to MariaDB data sources for execution. + +Configure the external function registration namespace `catalog.schema` that supports push-down. For example, in `etc/catalog/maria.properties` configure. + +```properties +jdbc.pushdown.remotenamespace=mariafun.default +``` + +You can declare that you support functions in multiple function namespaces, just use '|' split in the `jdbc.pushdown.remotenamespace` configuration item. For example + +```properties +# indicates that the current Connector instance supports both mysqlfun1.default, mysqlfun2.default, and mysqlfun3.default function namespaces final function push down to the currently connected data source for execution. +jdbc.pushdown.remotenamespace=mariafun1.default|mariafun2.default|mariafun3.default +``` + +### Multiple MariaDB servers + +As many catalogs as needed can be created, so if there are additional MariaDB servers, just add another properties file with a different name to `etc/catalog` (make sure it ends with `.properties`). For example, if the properties file is named `sales.properties`, openLooKeng will create a catalog named `sales` using the configured connector. + +## Querying MariaDB + +The MariaDB Connector provides a schema for each MariaDB database. Available MariaDB databases can be viewed by executing `SHOW SCHEMAS` + +```sql +-- The catalog name for this case is called maria +SHOW SCHEMAS FROM maria; +``` + +If you have a MariaDB database named `test`, you can view the tables in the database by executing `SHOW TABLES` + +```sql +SHOW TABLES FROM maria.test; +``` + +You can view a list of columns in the `user` table in the `test` database using one of the following methods + +```sql +DESCRIBE maria.test.user; +SHOW COLUMNS FROM maria.test.user; +``` + +Finally, you can access the `user` table in the `test` database + +```sql +SELECT * FROM maria.test.user; +``` + +If you use a different name for the directory properties file, use that directory name instead of `maria` in the above example. + +## MariaDB Connector Restrictions + +The following SQL statements are not supported at this time + +[DELETE](../sql/delete.html)、[GRANT](../sql/grant.html)、[REVOKE](../sql/revoke.html)、[SHOW GRANTS](../sql/show-grants.html)、[SHOW ROLES](../sql/show-roles.html)、[SHOW ROLE GRANTS](../sql/show-role-grants.html) \ No newline at end of file diff --git a/content/en-us/docs/docs/connector/oracle.md b/content/en-us/docs/docs/connector/oracle.md index c608ef1e215faa71888ec682fd498f1c26db3229..51e1f9eb33d41fe49d9e96c8ae320e6bd5d59f5e 100644 --- a/content/en-us/docs/docs/connector/oracle.md +++ b/content/en-us/docs/docs/connector/oracle.md @@ -177,7 +177,7 @@ Data type mapping > | Oracle Database Type| openLooKeng Type| Description| > |:----------|:----------|:----------| -> | DECIMAL(p, s)| DECIMAL(p, s)| +> | NUMBER(p, s)| DECIMAL(p, s)| > | NUMBER(p)| DECIMAL(p, 0)| > | FLOAT(p)| DOUBLE| > | BINARY\_FLOAT| REAL| diff --git a/content/en-us/docs/docs/connector/redis.md b/content/en-us/docs/docs/connector/redis.md new file mode 100644 index 0000000000000000000000000000000000000000..a6b0d37f796979089d4bb63f2a027abbc6d46743 --- /dev/null +++ b/content/en-us/docs/docs/connector/redis.md @@ -0,0 +1,173 @@ +Redis Connector +==================== +Overview +-------- +this connector allows the use of Redis key/value pair is presented as a single row in openLooKeng. + +**Note** + +*In Redis,key/value pair can only be mapped to string or hash value types.keys can be stored in a zset,then keys can split into multiple slice* + +*Support Redis 2.8.0 or higher* + +Configuration +------------- +To configure the Redis connector, create a catalog properties file `etc/catalog/redis.properties` with the following contents, replacing the properties as appropriate: +``` properties +connector.name=redis +redis.table-names=schema1.table1,schema1.table2 +redis.nodes=host1:port +``` +### Multiple Redis Servers +You can have as many catalogs as you need. If you have additional +Redis servers, simply add another properties file to ``etc/catalog`` +with a different name, making sure it ends in ``.properties``. +For example, if you name the property file `sales.properties`, openLooKeng will create a catalog named `sales` using the configured connector. + +Configuration properties +------------------------ +The following configuration properties are available: + +| Property Name | Description | +|:-----------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------| +| `redis.table-names` | List of all tables provided by the catalog | +| `redis.default-schema` | Default schema name for tables (default `default`) | +| `redis.nodes` | List of nodes in the Redis server | +| `redis.connect-timeout` | Timeout for connecting to the Redis server (ms) (default 2000) | +| `redis.scan-count` | The number of keys obtained from each scan for string and hash value types (default 100) | +| `redis.key-prefix-schema-table` | Redis keys have schema-name:table-name prefix (default false) | +| `redis.key-delimiter` | Delimiter separating schema_name and table_name if redis.key-prefix-schema-table is used (default `:`) | +| `redis.table-description-dir` | Directory containing table description files (default `etc/redis/`) | +| `redis.hide-internal-columns` | Whether internal columns are shown in table metadata or not. (default true) | +| `redis.database-index` | Redis database index (default 0) | +| `redis.password` | Redis server password (default null) | +| `redis.table-description-interval` | the interval of flush description files (ms) (default no flush,table description will be memoized without expiration) | + + +Internal columns +---------------- + +| Column name | Type | Description | +|:-------------------| :------ |:-----------------------------------------------------------------------------------------------------------------------------------------| +| `_key` | VARCHAR | Redis key. | +| `_value` | VARCHAR | Redis value corresponding to the key | +| `_key_length` | BIGINT | Number of bytes in the key. | +| `_key_corrupt` | BOOLEAN | True if the decoder could not decode the key for this row. When true, data columns mapped from the key should be treated as invalid. | +| `_value_corrupt` | BOOLEAN | True if the decoder could not decode the value for this row. When true, data columns mapped from the value should be treated as invalid. | + + +Table Definition Files +---------------------- +For openLooKeng, every key/value pair must be mapped into columns to allow queries against the data. It is like kafka conntector,so you can refer to kafka-tutorial + +A table definition file consists of a JSON definition for a table. The name of the file can be arbitrary but must end in `.json`. + +for example,there is a nation.json +``` json +{ + "tableName": "nation", + "schemaName": "tpch", + "key": { + "dataFormat": "raw", + "fields": [ + { + "name": "redis_key", + "type": "VARCHAR(64)", + "hidden": "true" + } + ] + }, + "value": { + "dataFormat": "json", + "fields": [ + { + "name": "nationkey", + "mapping": "nationkey", + "type": "BIGINT" + }, + { + "name": "name", + "mapping": "name", + "type": "VARCHAR(25)" + }, + { + "name": "regionkey", + "mapping": "regionkey", + "type": "BIGINT" + }, + { + "name": "comment", + "mapping": "comment", + "type": "VARCHAR(152)" + } + ] + } +} +``` +In redis,such data exists +```shell +127.0.0.1:6379> keys tpch:nation:* + 1) "tpch:nation:2" + 2) "tpch:nation:4" + 3) "tpch:nation:16" + 4) "tpch:nation:18" + 5) "tpch:nation:10" + 6) "tpch:nation:17" + 7) "tpch:nation:1" +``` +```shell +127.0.0.1:6379> get tpch:nation:1 +"{\"nationkey\":1,\"name\":\"ARGENTINA\",\"regionkey\":1,\"comment\":\"al foxes promise slyly according to the regular accounts. bold requests alon\"}" +``` +Now we can use redis connector get data from redis,(redis_key don't show,because we set "hidden": "true" ) +```shell +lk> select * from redis.tpch.nation; + nationkey | name | regionkey | comment +-----------+----------------+-----------+-------------------------------------------------------------------------------------------------------------------- + 3 | CANADA | 1 | eas hang ironic, silent packages. slyly regular packages are furiously over the tithes. fluffily bold + 9 | INDONESIA | 2 | slyly express asymptotes. regular deposits haggle slyly. carefully ironic hockey players sleep blithely. carefull + 19 | ROMANIA | 3 | ular asymptotes are about the furious multipliers. express dependencies nag above the ironically ironic account + 2 | BRAZIL | 1 | y alongside of the pending deposits. carefully special packages are about the ironic forges. slyly special +``` +**Note** + +*if redis.key-prefix-schema-table is false (default is false),all keys in redis will be mapped to table's key,no matching occurs* + +Please refer to the `kafka-tutorial` for the description of the ``dataFormat`` as well as various available decoders. + +In addition to the above Kafka types, the Redis connector supports ``hash`` type for the ``value`` field which represent data stored in the Redis hash. +Redis connector use `hgetall key` to get data. +``` json + { + "tableName": ..., + "schemaName": ..., + "value": { + "dataFormat": "hash", + "fields": [ + ... + ] + } + } +``` + +the Redis connector supports ``zset`` type for the ``key`` field which represent key stored in the Redis zset. +if and only if ``zset`` is used as key datafomart,the split is truly supported , because we can use `zrange zsetkey split.start split.end` to get keys of a split. +``` json + { + "tableName": ..., + "schemaName": ..., + "key": { + "dataFormat": "zset", + "name": "zsetkey", //zadd zsetkey score member + "fields": [ + ... + ] + } + } +``` + +Redis Connector Limitations +--------------------------- +only support read operation,don't support write operation. + + diff --git a/content/en-us/docs/docs/connector/singledata.md b/content/en-us/docs/docs/connector/singledata.md new file mode 100644 index 0000000000000000000000000000000000000000..f243aa92a7d9dd1238f6e93bf5b061d106d9f9c6 --- /dev/null +++ b/content/en-us/docs/docs/connector/singledata.md @@ -0,0 +1,98 @@ +# SingleData Connector + +The singleData connector supplements the OLAP capability for the openGauss, enhances the capability of data analysis. + +The singleData mode can be ShardingSphere or tidRange + +## ShardingSphere Mode + +### Overview + +Apache ShardingSphere is a distributed database ecosystem. It can convert any database into a distributed database. SingleData Connector can use ShardingSphere to shard openGauss data to improve analysis efficiency. + +For more information, please visit: [ShardingSphere official website](https://shardingsphere.apache.org/) + +### Configuration + +To configure the singleData Connector for ShardingSphere mode, create a catalog properties file `etc/catalog` named, for example, `shardingsphere.properties`. Create the file with the following contents, replacing the connection properties as appropriate for your setup: + +- Basic Configuration + +``` properties +connector.name=singledata +singledata.mode=SHARDING_SPHERE +shardingsphere.database-name=sharding_db +shardingsphere.type=zookeeper +shardingsphere.namespace=governance_ds +shardingsphere.server-list=localhost:2181 +``` + +- Configuration Properties + +| Property Name | Description | required | +|------------------------------|------------------------------------------------------------------------------------------------------------------|----------| +| shardingsphere.database-name | Name of the connected shardingSphere database | Yes | +| shardingsphere.type | Persistent repository type of the registration center.Zookeeper and etcd are supported. Zookeeper is recommended | Yes | +| shardingsphere.namespace | NameSpace of the registry center | Yes | +| shardingsphere.server-lists | Connection address list of the registry center | Yes | + +- Optional configuration when the persistent repository type is zookeeper + +| Property Name | Description | Default | +|---------------------------------------------------------|---------------------------------------------------------|---------| +| shardingsphere.zookeeper.retry-interval-milliseconds | Retry interval after connection failure in milliseconds | 500 | +| shardingsphere.zookeeper.max-retries | Maximum retry connections | 3 | +| shardingsphere.zookeeper.time-to-live-seconds | Temporary node lifetime seconds | 60 | +| shardingsphere.zookeeper.operation-timeout-milliseconds | Operation timeout milliseconds | 500 | +| shardingsphere.zookeeper.digest | Token | "" | + +- Optional configuration when the persistent repository type is etcd + +| Property Name | Description | Default | +|------------------------------------------|---------------------------------|---------| +| shardingsphere.etcd.time-to-live-seconds | Temporary node lifetime seconds | 30 | +| shardingsphere.etcd.connection-timeout | Connection timeout seconds | 3 | + +****Note:**** + +- SingleData connector only supports ShardingSphere 5.2.0 now + +## TidRange Mode + +### Overview + +OpenGauss data is stored in HEAP PAGE by row. Each row of data has a corresponding ctid (line number). The tidrangescan plugin of openGauss can use the range of ctid for query. The tidRange mode uses this plugin to implement parrallel data analysis. + +To obtain the tidrangescan plugin, please visit: [Plugin](https://gitee.com/opengauss/Plugin) + +### Connection Configuration + +To configure the singleData Connector for tidRange mode, create a catalog properties file `etc/catalog` named, for example, `tidrange.properties`. Create the file with the following contents, replacing the connection properties as appropriate for your setup: +```properties +connection.name=singledata +connection.mode=TID_RANGE +connection-url=jdbc:opengauss://master-host:port/database;jdbc:opengauss://slave-host:port/database +connection-user=user +connection-password=password +``` + +- connection-url can be configured multiple JDBC connection addresses. The addresses are separated by `;`. During query, each segment randomly selects a connection address for connection. +- For other connection configurations, see the openGauss connector + +### Split Configuration + +| Property Name | Description | Required | Default | +|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------| +| tidrange.max-split-count | Max count of split, that is, the maximum number of JDBC connections. The value must be less than or equal to the value of max_connections in openGauss | No | 100 | +| tidrange.page-size | Page size of the openGauss. Ensure that the value is same as the value of block_size of the openGauss. Otherwise, the query result may be incorrect | No | 8kB | +| tidrange.default-split-size | Default size of each split. When the data size is small, the singleData connector fragments the data bases on this configuration. The value ranges from 1MB to 1GB | No | 32MB | + +****NOTE:**** + +- TidRange mode needs to be used with tidrangescan plugin of the openGauss, without tidrangescan plugin, the singledata connector can be used for query, but the performance deteriorates greatly. +- TidRange will be not enabled when an index exists in the table + +## Limitations +- SingleData Connector only support select statement now, statement such as INSERT/UPDATE/DELETE that modify data or data structures are not supported. +- The maximum decimal precision of openLookeng is 38. If the precision of Decimal or Numeric is greater than 38 in openGauss, it will be not supported. +- The openGauss version required 3.0.0 or later diff --git a/content/en-us/docs/docs/develop/_index.md b/content/en-us/docs/docs/develop/_index.md index 79ed92bc1ad7d0e2c1265b0b64a9ad4d33d32bf9..67fd7cb7ec52e168acd1ded8dbe49f4f8815f813 100644 --- a/content/en-us/docs/docs/develop/_index.md +++ b/content/en-us/docs/docs/develop/_index.md @@ -1,5 +1,5 @@ # Developer Guide -openLooKeng is based on Trino(formerly known as PrestoSQL), and has been forked from the Trino open source project. openLooKeng has additional optimizations, and enhanced features to allow in-situ analytics on any data, anywhere, including geographically remote data sources. This guide is intended for openLooKeng contributors and plugin developers. +openLooKeng is based on Trino 316(formerly known as PrestoSQL), and has been forked from the Trino open source project. openLooKeng has additional optimizations, and enhanced features to allow in-situ analytics on any data, anywhere, including geographically remote data sources. This guide is intended for openLooKeng contributors and plugin developers. diff --git a/content/en-us/docs/docs/faq/faq.md b/content/en-us/docs/docs/faq/faq.md index db83806ff842db0a4d2bd1b9d8005e8b0c349dd6..d7a9f47c55574a14ed09cbf3ad7d4caf20c9630e 100644 --- a/content/en-us/docs/docs/faq/faq.md +++ b/content/en-us/docs/docs/faq/faq.md @@ -40,6 +40,10 @@ > > openLooKeng Bilibili channel: https://space.bilibili.com/627629884 +8. Which version of Trino is openLooKeng developed on? + + > Based on Trino 316 version development. + ## Functions 1. What connectors does the openLooKeng support? @@ -154,7 +158,7 @@ 1. Are SQL statements of the openLooKeng case-sensitive? If SQL statements of the data source are case-sensitive, how does the openLooKeng process them? - > SQL statements of the openLooKeng are case-insensitive. Uppercase letters in SQL statements are converted to lowercase letters. If SQL statements of the data source are case-sensitive, queries may fail. For example, SQL statements of the Elasticsearch is case-sensitive. If the column name in Elasticsearch is in uppercase, an error occurs when the column is queried in the openLooKeng. + > SQL statements of the openLooKeng are case-insensitive. Uppercase letters in SQL statements are converted to lowercase letters. If SQL statements of the data source are case-sensitive, queries may fail. For example, SQL statements of the Elasticsearch is case-sensitive. If the column name in Elasticsearch is in uppercase, an error occurs when the column is queried in the openLooKeng.Insensitive When JDBC data sources match database and schema, "case-insensitive name-matching" is specified in the corresponding data source profile. For details, see https://docs.openlookeng.io/en/docs/docs/admin/properties.html. 2. What do I do If the error message "cannot change stats state for a transactional table" is displayed when an update operation fails after the openLooKeng connects to the Hive? diff --git a/content/en-us/docs/docs/images/snapshot_statistics.png b/content/en-us/docs/docs/images/snapshot_statistics.png index f41da3e3e48dde82fe1191f5551f5f07e937f2d4..d6918f25f26a767d9ca92ec02569c4b5b09e67fe 100644 Binary files a/content/en-us/docs/docs/images/snapshot_statistics.png and b/content/en-us/docs/docs/images/snapshot_statistics.png differ diff --git a/content/en-us/docs/docs/images/task_snapshot_overview.png b/content/en-us/docs/docs/images/task_snapshot_overview.png new file mode 100644 index 0000000000000000000000000000000000000000..84286f9e0e31384665111932188481301c731122 Binary files /dev/null and b/content/en-us/docs/docs/images/task_snapshot_overview.png differ diff --git a/content/en-us/docs/docs/indexer/overview.md b/content/en-us/docs/docs/indexer/overview.md index b90dbd95d8050b57ea4c4c9ab09c2ff2fd802ccb..9eef7edb7ad2aeddb6904fa79229f99614053081 100644 --- a/content/en-us/docs/docs/indexer/overview.md +++ b/content/en-us/docs/docs/indexer/overview.md @@ -91,7 +91,7 @@ Here is a reference `etc/hetu-metastore.properties`: As mentioned in the Hetu Metastore setup, the global cache must be enabled, this requires the Hetu Statestore to be configured. For more information about how to configure it, please check [Hetu Statestore](../admin/state-store.html). -In `etc/config.propertes`: +In `etc/config.properties`: ```properties hetu.embedded-state-store.enabled=true diff --git a/content/en-us/docs/docs/installation/deployment.md b/content/en-us/docs/docs/installation/deployment.md index 9ff12775796fb3800455fdf30d870fb28a1084b0..09d13d5e8ead293ea11fc68b42d74c5b6ba4e1f3 100644 --- a/content/en-us/docs/docs/installation/deployment.md +++ b/content/en-us/docs/docs/installation/deployment.md @@ -158,7 +158,15 @@ See [Connectors](../connector.html) for more information about configuring conne ## Running openLooKeng -The installation directory contains the launcher script in `bin/launcher`. openLooKeng can be started as a daemon by running the following: +The installation directory contains the launcher script in `bin/launcher`. There is a preparation before running: + +Edit `bin/launcher.py` in the installation directory, and respectively replace `options.launcher_log_file` and `options.server_log_file` with `node_properties.get('node.launcher-log-file')` and `node_properties.get('node.server-log-file')` on lines 440 and 441. +The modified code should be as follows: +``` +o.launcher_log = realpath(node_properties.get('node.launcher-log-file' ) or pathjoin(o.data_dir, 'var/log/launcher.log')) +o.server_log = realpath(node_properties.get('node.server-log-file' ) or pathjoin(o.data_dir, 'var/log/server.log')) +``` +When the cluster starts, the log files will be preferentially saved in the path specified by `etc/node.properties`. openLooKeng can be started as a daemon by running the following: ``` shell bin/launcher start diff --git a/content/en-us/docs/docs/releasenotes/releasenotes-1.0.1.md b/content/en-us/docs/docs/releasenotes/releasenotes-1.0.1.md index 5d9c845b469dea9d17798e8295cb9a0c2bc19e37..b7741562c64b18102f02525e3a6b1e73262df6fb 100644 --- a/content/en-us/docs/docs/releasenotes/releasenotes-1.0.1.md +++ b/content/en-us/docs/docs/releasenotes/releasenotes-1.0.1.md @@ -7,7 +7,7 @@ This release is mainly for fixing version number issue and index-cli issues. Bes | Area | Feature | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | Index | Merge the index-cli into hetu-cli, and introduce the basic access control to index command | 283, 298 | -| Security | Handle CVEs | 289,287,295,288,285,292，297,249,302 | +| Security | Handle CVEs | 289,287,295,288,285,292,297,249,302 | ## Obtaining the Document diff --git a/content/en-us/docs/docs/releasenotes/releasenotes-1.7.0.md b/content/en-us/docs/docs/releasenotes/releasenotes-1.7.0.md new file mode 100644 index 0000000000000000000000000000000000000000..2f4c6ca379f1b2342a25a81800407f44f6c52923 --- /dev/null +++ b/content/en-us/docs/docs/releasenotes/releasenotes-1.7.0.md @@ -0,0 +1,22 @@ +# Release 1.7.0 + +## Key Features + +| Area | Feature | +| --------------------- | ------------------------------------------------------------ | +| Operator processing extension | Users can customize the physical execution plan of worker nodes. Users can use their own operator pipelines to replace the native implementation to accelerate operator processing. | +| Task Recovery | 1. Enables query recovery by restarting the query, without capturing snapshots during query execution
2. Support snapshot related information display in CLI debug mode: Number of snapshots being captured currently/ Number of snapshots captured already/ Number of times query recovered (either by resuming or restarting)/ Size of the snapshots captured and Time taken to capture snapshots/ Size of snapshots restored and time taken to restore
3. Resource Manager: Suspend a query on low memory situation which is consuming most memory and resume. User can suspend/resume a query（limit sql in CTAS and INSERT INTO SELECT）
4. Add customizable failure retry policies: Timeout based retry policy and Max-retry based retry policy
5. Add Gossip based failure detection mechanism: Coordinator continuously monitors the workers whether they are alive by receiving periodic heartbeats. Gossip mechanism enables all workers to monitor each other, and let the coordinator know about their local knowledge on each other’s availability. Gossiping can prevent premature query failing in case of transient network failures in the cluster. | +| openLooKeng Web UI Enhancement | 1. openLooKeng audit log feature enhancements
2. openLooKeng SQL history query feature enhancements
3. SQL Editor supports automatic association
4. Supports the collection of common sql statements
5. When a data source loads an exception, the front end can be displayed through exception tags and does not affect the normal operation of the openLooKeng service
6. The UI interface supports displaying specific information about the data source(Show catalog). | +| openLooKeng Cluster Management | 1. Provides graphical indicator monitoring and customization to obtain key system information
2. Provides configuration management of service attributes to meet actual service performance requirements
3. Provides operations on clusters, services, and instances to meet one-click start and stop requirements
4. Provides the openLooKeng online deployment function and supports online upgrade
5. Provides the node scaling service
6. Integrate the existing web UI of openLooKeng
7. The openLooKeng Manager tool supports user login and logout. | + +## Known Issues + +| Category | Description | Gitee issue | +| ------------- | ------------------------------------------------------------ | --------------------------------------------------------- | +| Task Recovery | When a snapshot is enabled and a CTAS with transaction is executed, kill one worker, an error is reported in the SQL statement. | [I5EAM3](https://e.gitee.com/open_lookeng/issues/list?issue=I5EAM3) | + + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.7.0/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.7.0/hetu-docs/en) + diff --git a/content/en-us/docs/docs/releasenotes/releasenotes-1.8.0.md b/content/en-us/docs/docs/releasenotes/releasenotes-1.8.0.md new file mode 100644 index 0000000000000000000000000000000000000000..65d6eff0319bf165cdce11d1fb462be9db3835dc --- /dev/null +++ b/content/en-us/docs/docs/releasenotes/releasenotes-1.8.0.md @@ -0,0 +1,14 @@ +# Release 1.8.0 + +## Key Features + +| Area | Feature | +|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Task Level Recovery | 1. Enables query recovery by retrying only failed task(s) using exchange materialization
2. Supported using HDFS for spooling
3. Supported Direct Serialization and compression too while materializing the exchange data. | +| Query Resource Management | Query level resource monitoring that triggers mitigation actions like suspend/resume query or task, spill revocable memory, throttle scheduling, and kill for problematic queries in a resource group. | +| Connector Extension | The singleData connector supplements the OLAP capability for the openGauss, enhances the capability of data analysis. Supports adaptor to the ShardingSphere and tidRange modes. | + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.8.0/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.8.0/hetu-docs/en) + diff --git a/content/en-us/docs/docs/releasenotes/releasenotes-1.9.0.md b/content/en-us/docs/docs/releasenotes/releasenotes-1.9.0.md new file mode 100644 index 0000000000000000000000000000000000000000..6feb02d3305549738bc84f8df7896dc4d9430f74 --- /dev/null +++ b/content/en-us/docs/docs/releasenotes/releasenotes-1.9.0.md @@ -0,0 +1,13 @@ +# Release 1.9.0 + +## Key Features + +| Area | Feature | +|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Task Level Recovery | Support task retry and recovery in the Create Table As Select scenario | +| Connector Extension | 1. Add support Iceberg connector to add/delete/modify/access data, hidden partitioning, time travel, transaction and small file compaction
2. Enhance Elastic Search connector to support filter and aggregation pushdown | +| Performance Enhancement | 1. CTE optimizations including reuse execution pipeline and cte results materialization and reuse
2. Plan optimizations including use exact partitioning, adaptive aggregations, join independence assumptions for selectivity estimates and execution policy for better filtering | + +## Obtaining the Document + +For details, see [https://gitee.com/openlookeng/hetu-core/tree/1.9.0/hetu-docs/en](https://gitee.com/openlookeng/hetu-core/tree/1.9.0/hetu-docs/en) diff --git a/content/en-us/docs/docs/security/built-in-system-access-control.md b/content/en-us/docs/docs/security/built-in-system-access-control.md index 39d702325a7f8388356d3ce050d72fcaa3bc6b95..f8664d7dea60154d39378f47f44128d9c4b5d90a 100644 --- a/content/en-us/docs/docs/security/built-in-system-access-control.md +++ b/content/en-us/docs/docs/security/built-in-system-access-control.md @@ -1,4 +1,3 @@ - Built-in System Access Control ============================== @@ -57,7 +56,10 @@ composed of the following fields: - `user` (optional): regex to match against user name. Defaults to `.*`. - `catalog` (optional): regex to match against catalog name. Defaults to `.*`. -- `allow` (required): boolean indicating whether a user has access to the catalog +- ``allow`` (required): string indicating whether a user has access to the catalog. + This value can be ``all``, ``read-only`` or ``none``, and defaults to ``none``. + Setting this value to ``read-only`` has the same behavior as the ``read-only`` + system access control plugin. **Note** @@ -65,7 +67,9 @@ composed of the following fields: *By default, all users have access to the `system` catalog. You can override this behavior by adding a rule.* -For example, if you want to allow only the user `admin` to access the `mysql` and the `system` catalog, allow all users to access the `hive` catalog, and deny all other access, you can use the following rules: +For example, if you want to allow only the user ``admin`` to access the``mysql`` and the ``system`` catalog, +allow all users to access the ``hive`` catalog, allow the user ``alice`` read-only access to the ``postgresql`` +catalog, and deny all other access, you can use the following rules: ``` json { @@ -73,15 +77,20 @@ For example, if you want to allow only the user `admin` to access the `mysql` an { "user": "admin", "catalog": "(mysql|system)", - "allow": true + "allow": all }, { "catalog": "hive", - "allow": true + "allow": all + }, + { + "user": "alice", + "catalog": "postgresql", + "allow": "read-only" }, { "catalog": "system", - "allow": false + "allow": none } ] } @@ -195,6 +204,7 @@ If you want to allow users to use the extactly same name as their Kerberos prin ``` ### Node State Rules + These rules govern the node state info particular users can access. The user is granted access to update a node state based on the first matching rule read from top to bottom. If no rule matches, access is denied. Each rule is composed of the following fields: diff --git a/content/en-us/docs/docs/security/ranger-access-control.md b/content/en-us/docs/docs/security/ranger-access-control.md index 2291c68af55c451653a516b1aded793814b90b22..ff01e350eb6bda6ed981a821a5fa883a14932654 100644 --- a/content/en-us/docs/docs/security/ranger-access-control.md +++ b/content/en-us/docs/docs/security/ranger-access-control.md @@ -7,7 +7,7 @@ Overview Apache Ranger delivers a comprehensive approach to security for a Hadoop cluster. It provides a centralized platform to define, administer and manage security policies consistently across Hadoop components. Check [Apache Ranger Wiki](https://cwiki.apache.org/confluence/display/RANGER/Index) for detail introduction and user guide. -[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) is a ranger plugin for openLooKeng to enable, monitor and manage comprehensive data security. +[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) is developed based on Ranger 2.1.0, which is a ranger plugin for openLooKeng to enable, monitor and manage comprehensive data security. Build Process ------------------------- diff --git a/content/en-us/docs/docs/security/server.md b/content/en-us/docs/docs/security/server.md index 66f29d501d603ba2d0d9a630a4bdf81beb1bc13f..445f940eed2b86aad00e437f8a032ef2f6af0b1b 100644 --- a/content/en-us/docs/docs/security/server.md +++ b/content/en-us/docs/docs/security/server.md @@ -131,7 +131,7 @@ http-server.https.excluded-cipher= The Java documentation lists the [supported cipher suites](http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuites). -### access-controls.properties +### access-control.properties At a minimum, an `access-control.properties` file must contain an `access-control.name` property. All other configuration is specific for the implementation being configured. See System Access Control for details. diff --git a/content/en-us/docs/menu/index.md b/content/en-us/docs/menu/index.md index d40055905cde837c927abed71158c63b613cd933..55e49bb152a73c941bcf9f29700a7cd0548b7fc4 100644 --- a/content/en-us/docs/menu/index.md +++ b/content/en-us/docs/menu/index.md @@ -44,8 +44,9 @@ headless: true - [State store]({{< relref "./docs/admin/state-store.md" >}}) - [Meta Store]({{< relref "./docs/admin/meta-store.md" >}}) - [Audit Log]({{< relref "./docs/admin/audit-log.md" >}}) - - [Reliable Execution]({{< relref "./docs/admin/reliable-execution.md" >}}) + - [Reliable Execution]({{< relref "./docs/admin/reliable-query/_index.md" >}}) - [JDBC Data Source Multi-Split Management]({{< relref "./docs/admin/multi-split-for-jdbc-data-source.md" >}}) + - [Extension Physical Execution Planner]({{< relref "./docs/admin/extension-execution-planner.md" >}}) - [Query Optimizer]("#") - [Table Statistics]({{< relref "./docs/optimizer/statistics.md" >}}) @@ -72,6 +73,7 @@ headless: true - [Carbondata]({{< relref "./docs/connector/carbondata.md" >}}) - [ClickHouse]({{< relref "./docs/connector/clickhouse.md" >}}) - [DataCenter]({{< relref "./docs/connector/datacenter.md" >}}) + - [DaMeng]({{< relref "./docs/connector/dameng.md" >}}) - [Elasticsearch]({{< relref "./docs/connector/elasticsearch.md" >}}) - [Greenplum]({{< relref "./docs/connector/greenplum.md" >}}) - [Hana]({{< relref "./docs/connector/hana.md" >}}) @@ -83,13 +85,16 @@ headless: true - [JMX]({{< relref "./docs/connector/jmx.md" >}}) - [Kafka]({{< relref "./docs/connector/kafka.md" >}}) - [Kafka Connector Tutorial]({{< relref "./docs/connector/kafka-tutorial.md" >}}) + - [Redis]({{< relref "./docs/connector/redis.md" >}}) - [Local File]({{< relref "./docs/connector/localfile.md" >}}) + - [MariaDB]({{< relref "./docs/connector/mariadb.md" >}}) - [Memory]({{< relref "./docs/connector/memory.md" >}}) - [MongoDB]({{< relref "./docs/connector/mongodb.md" >}}) - [MySQL]({{< relref "./docs/connector/mysql.md" >}}) - [openGauss]({{< relref "./docs/connector/opengauss.md" >}}) - [Oracle]({{< relref "./docs/connector/oracle.md" >}}) - [PostgreSQL]({{< relref "./docs/connector/postgresql.md" >}}) + - [SingleData]({{< relref "./docs/connector/singledata.md" >}}) - [SQL Server]({{< relref "./docs/connector/sqlserver.md" >}}) - [System]({{< relref "./docs/connector/system.md" >}}) - [Thrift]({{< relref "./docs/connector/thrift.md" >}}) @@ -217,6 +222,9 @@ headless: true - [Task Resource]({{< relref "./docs/rest/task.md" >}}) - [Release Notes]("#") + - [1.9.0 (30 Dec 2022)]({{< relref "./docs/releasenotes/releasenotes-1.9.0.md" >}}) + - [1.8.0 (30 Nov 2022)]({{< relref "./docs/releasenotes/releasenotes-1.8.0.md" >}}) + - [1.7.0 (30 Jun 2022)]({{< relref "./docs/releasenotes/releasenotes-1.7.0.md" >}}) - [1.6.1 (27 Apr 2022)]({{< relref "./docs/releasenotes/releasenotes-1.6.1.md" >}}) - [1.6.0 (30 Mar 2022)]({{< relref "./docs/releasenotes/releasenotes-1.6.0.md" >}}) - [1.5.0 (30 Dec 2021)]({{< relref "./docs/releasenotes/releasenotes-1.5.0.md" >}}) diff --git a/content/zh-cn/docs/docs/admin/audit-log.md b/content/zh-cn/docs/docs/admin/audit-log.md index 24fa70016e268ab393bee2bdf61eaa1e040389f2..2eac453b82b4fc34c80443b59189d1bb7d7670f0 100644 --- a/content/zh-cn/docs/docs/admin/audit-log.md +++ b/content/zh-cn/docs/docs/admin/audit-log.md @@ -1,6 +1,6 @@ # 审计日志 -openLooKeng审计日志记录功能是一个自定义事件监听器，在查询创建和完成（成功或失败）时调用。审计日志包含以下信息： +openLooKeng审计日志记录功能是一个自定义事件监听器，监听openLooKeng集群启停与集群中节点的动态添加与删除事件；监听WebUi用户登录与退出事件；监听查询事件,在查询创建和完成（成功或失败）时调用。审计日志包含以下信息： 1. 事件发生时间 2. 用户ID @@ -23,15 +23,17 @@ openLooKeng审计日志记录功能是一个自定义事件监听器，在查询 hetu.event.listener.type=AUDIT hetu.event.listener.listen.query.creation=true hetu.event.listener.listen.query.completion=true +hetu.auditlog.logoutput=/var/log/ +hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH ``` 其他审计日志记录属性包括： -`hetu.event.listener.audit.file`：可选属性，用于定义审计文件的绝对文件路径。确保运行openLooKeng服务器的进程对该目录有写权限。 +`hetu.event.listener.type`：用于定义审计日志的记录类型，允许的值为AUDIT和LOGGER。 -`hetu.event.listener.audit.filecount`：可选属性，用于定义要使用的文件数。 +`hetu.auditlog.logoutput`：用于定义审计文件的绝对目录路径。确保运行openLooKeng服务器的进程对该目录有写权限。 -`hetu.event.listener.audit.limit`：可选属性，用于定义写入任一文件的最大字节数。 +`hetu.auditlog.logconversionpattern`：用于定义审计日志的轮转模式。允许的值为yyyy-MM-dd.HH和yyyy-MM-dd。 配置文件示例： @@ -43,4 +45,6 @@ hetu.event.listener.listen.query.completion=true hetu.event.listener.audit.file=/var/log/hetu/hetu-audit.log hetu.event.listener.audit.filecount=1 hetu.event.listener.audit.limit=100000 +hetu.auditlog.logoutput=/var/log/ +hetu.auditlog.logconversionpattern=yyyy-MM-dd.HH ``` \ No newline at end of file diff --git a/content/zh-cn/docs/docs/admin/dynamic-catalog.md b/content/zh-cn/docs/docs/admin/dynamic-catalog.md index c18f0b1af1cb11f09d3930a6d35e272f00827d9b..c18b5429924a94da9e7a54bb45c29db0313225e1 100644 --- a/content/zh-cn/docs/docs/admin/dynamic-catalog.md +++ b/content/zh-cn/docs/docs/admin/dynamic-catalog.md @@ -144,7 +144,7 @@ UPDATE操作是DELETE和ADD操作的组合。首先管理员向协调节点发 其他协调节点和工作节点定期检查共享文件系统中的目录属性文件，并在本地文件系统上执行相应的更改。 -目录属性，包括 `connector-name`、`properties`等，支持修改。但是**目录名称**不能更改。 +支持修改`properties`目录属性，但是**目录名称**不能更改。不建议通过PUT请求直接修改`connector-name`，可以通过先DELETE后POST的方式进行修改。 以Hive为例，通过curl可以用以下命令更新目录，以下更新了`hive.allow-drop-table`参数： diff --git a/content/zh-cn/docs/docs/admin/extension-execution-planner.md b/content/zh-cn/docs/docs/admin/extension-execution-planner.md new file mode 100644 index 0000000000000000000000000000000000000000..b3250ee35bcea83775d6c3644be4036d25ce10aa --- /dev/null +++ b/content/zh-cn/docs/docs/admin/extension-execution-planner.md @@ -0,0 +1,29 @@ +# 扩展物理执行计划 +本节介绍openLooKeng如何添加扩展物理执行计划。通过物理执行计划的扩展，openLooKeng可以使用其他算子加速库来加速SQL语句的执行。 + +## 配置 +首先，需要从以下链接获取相关jar包和c++库： +- `boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.jar`: [Link](https://github.com/kunpengcompute/boostkit-bigdata/releases/download/v1.3.0/boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.zip) + +- `libboostkit-omniop-xx-1.0.0-aarch.so`: [Link](https://www.hikunpeng.com/zh/developer/boostkit/big-data?acclerated=3) + +然后，在配置文件`config.properties`增加如下配置： + +``` properties +extension_execution_planner_enabled=true +extension_execution_planner_jar_path=file:///xxPath/boostkit-omniop-openlookeng-1.6.1-1.0.0-aarch64.jar +extension_execution_planner_class_path=nova.hetu.olk.OmniLocalExecutionPlanner +``` + +上述属性说明如下： + +- `extension_execution_planner_enabled`：是否开启扩展物理执行计划特性。 +- `extension_execution_planner_jar_path`：指定扩展jar包的文件路径。 +- `extension_execution_planner_class_path`：指定扩展jar包中执行计划生成类的包路径。 + + +## 使用 +当运行openLooKeng时，可在WebUI或Cli中通过如下命令控制扩展物理执行计划的开启: +``` +set session extension_execution_planner_enabled=true/false +``` \ No newline at end of file diff --git a/content/zh-cn/docs/docs/admin/multi-split-for-jdbc-data-source.md b/content/zh-cn/docs/docs/admin/multi-split-for-jdbc-data-source.md index f87413b8a89fd7550fbaaa32cef04c2b3cb00fad..2ee1d559ebca64e048d1f71b4e77286bb350a5fb 100644 --- a/content/zh-cn/docs/docs/admin/multi-split-for-jdbc-data-source.md +++ b/content/zh-cn/docs/docs/admin/multi-split-for-jdbc-data-source.md @@ -5,7 +5,7 @@ ## 属性配置 -多分片管理的配置以连接器为单位，开启该功能的数据表需要在所属的连接器的配置文件（例如名为mysql的连接器对应的配置文件为etc/mysql.properties）添加以下属性： +多分片管理的配置以连接器为单位，开启该功能的数据表需要在所属的连接器的配置文件（例如名为mysql的连接器对应的配置文件为etc/catalog/mysql.properties）添加以下属性： 属性列表： 配置文件配置项说明: diff --git a/content/zh-cn/docs/docs/admin/properties.md b/content/zh-cn/docs/docs/admin/properties.md index 35cf280ef4a6a26b8d001e3298e8ebf2276309a3..bf0024fc3bf71c57fdf18d3a2f2b6cc8d57906bc 100644 --- a/content/zh-cn/docs/docs/admin/properties.md +++ b/content/zh-cn/docs/docs/admin/properties.md @@ -115,6 +115,20 @@ > > 此属性是在JVM堆中为openLooKeng不跟踪的分配留作裕量/缓冲区的内存量。 +### `query.suspend-query-enabled` + +> - **类型：** `boolean` +> - **默认值：** `false` +> +> 系统资源不足时，临时挂起运行中的查询。 + +### `query.max-suspended-queries` + +> - **类型：** `integer` +> - **默认值：** `10` +> +> 终止查询之前，查询挂起尝试的最大次数。仅当`query.suspend-query-enabled`设置为`true`时，此属性才生效。 + ## 溢出属性 ### `experimental.spill-enabled` @@ -340,35 +354,139 @@ > > 如果网络延迟较高，增大该值可以提高网络吞吐量。减小该值可以提高大型集群的查询性能，因为它减少了由于交换客户端缓冲区保存了较多任务（而不是保存较少任务中的较多数据）的响应而导致的倾斜。 -### `exchange.max-error-duration` +### `sink.max-buffer-size` -> - **类型：** `duration` -> - **最小值：** `1m` -> - **默认值：** `7m` +> - **类型：** `data size` +> - **默认值：** `32MB` > -> 交换错误最大缓冲时间，超过该时限则查询失败。 +>等待上游任务拉取的任务数据的输出缓冲区大小。如果任务输出是哈希分区的，则缓冲区将在所有分区的消费者之间共享。如果网络延迟高或集群中有许多节点，则增加此值可以提高阶段之间传输数据的网络吞吐量。 -### `exchange.is-timeout-failure-detection-enabled` +### `query-resource-tracking` > - **类型：** `boolean` -> - **默认值：** `true` +> - **默认值：** `false` > -> 正在使用的故障检测机制。默认值是基于超时的故障检测。但是，当该属性设置为`false`时，启用基于最大重试次数的故障检测机制。 - -### `exchange.max-retry-count` +>禁用查询级别资源跟踪和减缓操作。 +> +### `query-no-resource-retry-count` > - **类型：** `integer` -> - **默认值：** `10` +> - **默认值：** `5` +> +> 当现有资源不够充足时，将会根据指定的重试次数重新执行查询语句。只有当`query-resource-tracking`被设置为`true`时，此特性才会被开启。 +> 如果该值被设定为小于5，该值会被自动调整为默认值5。 +> + ## 故障恢复处理属性 + + ### 失败重试策略 + + ### `failure.recovery.retry.profile` + + > - **类型：** `string` + > - **默认值：** `default` + > + > 此属性定义用于确定HTTP客户端上是否发生故障的故障检测配置文件。此属性的值``必须对应`etc/failure-retry-policy/`路径中的`.properties`文件。如果没有此类配置文件可用，并且未设置此属性，则使用“default”配置文件。 + > 例如，`failure.recovery.retry.profile=test`要求`test.properties`文件存在于`etc/failure-retry-policy`路径中。 + > `test.properties`文件必须包含指定的`failure.recovery.retry.type`。 + + + ### `failure.recovery.retry.type` + + > - **类型：** `string` + > - **默认值：** `timeout` + > + > 此属性用来设置正在使用的故障检测机制。默认值是基于`timeout`的故障检测。 + + #### 基于`timeout`的故障检测 + + > 如果使用此机制，HTTP客户端故障将在指定时间段内重试，重试失败则被视为永久故障。 + > + > 可以为此类故障检测定义`max.error.duration`属性。 + + #### 基于`max-retry`的故障检测 + + > 如果使用此机制，HTTP客户端故障将在被视为永久故障之前重试指定次数。 + > 可以为此类故障检测定义`max.retry.count`和`max.error.duration`属性。 + > 在这种类型的故障检测中，在查询故障检测模块之前，会执行`max.retry.count`次重试。当故障检测器模块检测到远程节点发生故障时，HTTP客户端将此故障视为永久故障。否则，例如，当远程工作节点处于活动状态但没有响应时，在`max.error.duration`指定的时间段内重试，重试失败则被视为永久故障。 + + ### `max.error.duration` + + > - **类型：** `duration` + > - **默认值：** `300s` + > + > 被视为永久故障前，协调器等待解决任务间相关错误的最长时间。 + + ### `max.retry.count` + + > - **类型：** `integer` + > - **默认值：** `100` + > + > 协调器在向故障检测器模块查询远程节点状态之前，对失败任务执行的最大重试次数。 + > 此属性指定查询失败检测模块之前的最小重试次数。因此，实际故障数量可能会因为集群大小和集群负载而略有不同。 + > 此属性仅用于基于`max-retry`的故障检测配置文件。 + > 最小值为100。 + + ### 故障检测Gossip协议配置 + + ### `failure-detection-protocol` + + >- **类型：** `string` + >- **默认值：** `heartbeat` + > + >此属性定义正在使用的故障检测器的类型。默认配置为`heartbeat`故障检测器。 + >在`config.properties`文件中，将此属性配置为`gossip`，可以启用Gossip协议。 + >集群中的所有节点（即协调器和工作节点）都应在其各自的`etc/config.properties`文件中指定此属性。 + + ### `failure-detector.heartbeat-interval` + + >- **类型：** `duration` + >- **默认值：** `500ms` + > + >集群中两个节点之间的消息散播间隔。 + >在Gossip协议中，两个工作节点间的消息散播频率高于协调器和一个工作节点间。 + >在协调器的`config.properties`文件中，可以为此属性配置一个较大的值，例如`5s`（5秒）。 + >在工作节点中，可以使用默认值。 + + ### `failure-detector.worker-gossip-probe-interval` + + >- **类型：** `duration` + >- **默认值：** `5s` + > + >Gossip协议使用监控任务（与`heartbeat`故障检测器相同）来监控其他节点。 + >此属性指定监控任务刷新间隔，以触发工作节点消息散播。 + >仅可以为工作节点指定默认值以外的任何其他值。 + >该属性的值必须大于`failure-detector.heartbeat-interval`的值。 + + ### `failure-detector.coordinator-gossip-probe-interval` + + >- **类型：** `duration` + >- **默认值：** `5s` + > + >Gossip协议使用监控任务（与heartbeat故障检测器相同）来监控其他节点。 + >此属性指定监控任务刷新间隔，以触发协调器参与工作节点消息散播。 + >仅可以为协调器指定默认值以外的任何其他值。 + >该属性的值必须大于`failure-detector.heartbeat-interval`和`failure-detector.worker-gossip-probe-interval`的值。 + + ### `failure-detector.coordinator-gossip-collate-interval` + + >- **类型：** `duration` + >- **默认值：** `2s` + > + >此属性指定协调器整理从所有工作节点获得的所有散播消息的间隔。 + >此属性只支持为协调器配置。 + >该属性的值必须大于`failure-detector.heartbeat-interval`的值。 + + ### `failure-detector.gossip-group-size` + + >- **类型：** `integer` + >- **默认值：** `Integer.MAX_VALUE` + > + >此属性定义单个工作节点在集群中散播消息的工作节点数量。 + >任何大于集群大小（即工作节点数量）的值都意味着all-to-all消息散播。 + >要保持较低的网络开销，针对大型集群，请将此属性设置为一个较小的值（例如，100工作节点的集群设置为10）。 + >每次刷新协调器上的工作节点监视任务时，协调器都会定义工作节点URI列表，其大小由`failure-detector.gossip-group-size`指定，以触发worker-to-worker消息散播。 + > > -> Coordinator在将失败任务视为永久失败之前对其执行的最大重试次数。仅当`exchange.is-timeout-failure-detection-enabled`设置为`false`时，才使用此属性。 - -### `sink.max-buffer-size` - -> - **类型：** `data size` -> - **默认值：** `32MB` -> -> 上游任务等待拉取任务数据的输出缓冲区大小。如果任务输出是经过哈希分区的，那么缓冲区将在所有分区的使用者之间共享。如果网络延迟较高或集群中有多个节点，增加此值可以提高在阶段之间传输的数据的网络吞吐量。 - ## 任务属性 ### `task.concurrency` @@ -582,6 +700,83 @@ > > **注意：** 仅适用于Hive连接器。 +### `optimizer.transform-self-join-to-window` + +> - **类型：** `boolean` +> - **默认值：** `true` +> +> 使用带有lead/lag方法的窗口函数消除自连接（self join）。 + +### `optimizer.transform-self-join-aggregates-to-window` + +> - **类型：** `boolean` +> - **默认值：** `true` +> +> 使用带有lead/lag方法的窗口聚合函数消除自连接（self join）。 + +### `optimizer.join-partitioned-build-min-row-count` + +> - **类型：** `long` +> - **默认值：** `1,000,000` +> +> 当需要交换的数据行数少于该值时使用gathering exchange，否则使用partitioning exchange。 + +### `optimizer.use-exact-partitioning` + +> - **类型：** `boolean` +> - **默认值：** `false` +> +> 启用后，如果上游阶段的分区不能精准地匹配到下游阶段的分区，则强制对数据重新分区。 + +### `enable-cte-result-cache` + +> - **类型：** `boolean` +> - **默认值：** `false` +> +> 将CTE结果物化到缓存中。也可以使用enable_cte_result_cache会话属性在每个查询基础上指定。 + +### `cte-result-cache-threshold-size` + +> - **类型：** `Data Size` +> - **默认值：** `128MB` +> +> 每个查询每个CTE可以物化到缓存的最大大小。 + +### `adaptive-partial-aggregation.enabled` + +> - **类型：** `boolean` +> - **默认值：** `true` +> +> 启用自适应部分聚合特性。 + +### `adaptive-partial-aggregation.min-rows` + +> - **类型：** `long` +> - **默认值：** `100,000` +> +> 部分聚合可能自适应关闭的最小处理数据行数。 + +### `adaptive-partial-aggregation.unique-rows-ratio-threshold` + +> - **类型：** `double` +> - **默认值：** `0.8` +> +> 部分聚合可能自适应关闭的聚合输出、输入数据行数的比值。 + +### `optimizer.join-multi-clause-independence-factor` + +> - **类型：** `double` +> - **默认值：** `0.25` +> +> 多子句连接的选择率估计的数据独立性假设因子。 + +### `optimizer.filter-conjunction-independence-factor` + +> - **类型：** `double` +> - **默认值：** `0.75` +> +> 多过滤条件的选择率估计的数据独立性假设因子。 + ## 正则表达式函数属性 下列属性允许调优[正则表达式函数](../functions/regexp.html)。 @@ -612,7 +807,7 @@ ## 启发式索引属性 -启发式索引是外部索引模块，可用于过滤连接器级别的行。 位图，Bloom和MinMaxIndex是openLooKeng提供的索引列表。 到目前为止，位图索引支持使用ORC存储格式的表支持蜂巢连接器。 +启发式索引是外部索引模块，可用于过滤连接器级别的行。 位图，Bloom和MinMaxIndex是openLooKeng提供的索引列表。 到目前为止，位图索引支持Hive连接器的ORC存储格式的表。 ### `hetu.heuristicindex.filter.enabled` @@ -707,6 +902,34 @@ > > 上次访问后使缓存的执行计划失效的时间（以毫秒为单位） +### `hetu.execution.data-cache.enabled` +> +> - **类型：** `boolean` +> - **默认值：** `false` +> +> 启用缓存CTE结果，从而避免频繁执行子执行计划。 + +### `hetu.execution.data-cache.max-size` +> +> - **类型：** `long` +> - **默认值：** `2147483648` +> +> CTE结果可以物化的最大数据量大小。 + +### `hetu.execution.data-cache.schema-name` +> +> - **类型：** `string` +> - **默认值：** `cache` +> +> 用来物化缓存CTE结果的模式名字。 + +### `hetu.execution.data-cache.connector-name` +> +> - **类型：** `string` +> - **默认值：** `hive` +> +> 用来物化缓存CTE结果的目录名字。 + ## SplitCacheMap属性 必须启用SplitCacheMap以支持缓存行数据。 启用后，协调器将存储表，分区和分片调度元数据 帮助进行缓存亲和力调度。 @@ -790,23 +1013,40 @@ ### `query.remote-task.max-error-duration` -> - 类型：`duration` -> - **默认值**：`5m` +> - **类型：** `duration` +> - **默认值: ** `5m` > > 远程任务错误最大缓冲时间，超过该时限则查询失败。 -## 分布式快照 +### `query.execution-policy` +> - **类型：** `string` +> - **默认值：** `all-at-once` +> +> 指定调度器实施的执行策略。可以配置以下一组执行策略： +> 1. _**all-at-once**_：该策略下调度器启动和处理所有的阶段。 +> 2. _**phased**_：该策略下调度器调度遵循阶段间生产者源这样的依赖关系，可以将所有独立的阶段一起调度。 +> 3. _**prioritize-utilization**_：该策略下调度器调度除了遵循生产者源这样的阶段依赖关系以外，它还查看动态过滤生产者的依赖路径。 +> + +## 查询恢复 + +### `recovery_enabled` + +> - **类型：** `boolean` +> - **默认值：** `false` +> +> 此会话属性用于启用或禁用恢复框架，该框架在发生故障时启用或禁用查询重启/恢复。 ### `snapshot_enabled` -> - 类型：`boolean` -> - **默认值**：`false` +> - **类型：** `boolean` +> - **默认值：** `false` > -> 此会话属性用于启用或禁用分布式快照功能。 +> 启用恢复框架时，启用此会话属性可以在查询执行期间捕获快照。如果未启用恢复框架，则此属性不生效。 ### `hetu.experimental.snapshot.profile` -> - 类型：`string` +> - **类型：** `string` > > 此属性定义用于存储快照的[文件系统](../develop/filesystem.html)配置文件。对应的配置文件必须存在于`etc/filesystem`中。例如，如果将该属性设置为`hetu.experimental.snapshot.profile=snapshot-hdfs1`，则必须在`etc/filesystem`中创建描述此文件系统的配置文件`snapshot-hdfs1.properties`，其中包含的必要信息包括身份验证类型、配置和密钥表（如适用）。具体细节请参考[文件系统](../develop/filesystem.html)相关章节。 > @@ -814,23 +1054,23 @@ > > 作为实验性属性，或可以将快照存储在非文件系统位置，如连接器。 -### `hetu.snapshot.maxRetries` +### `hetu.recovery.maxRetries` -> - 类型：`int` -> - **默认值**：`10` +> - **类型：** `integer` +> - **默认值：** `10` > -> 此属性定义查询的错误恢复尝试的最大次数。达到限制时，查询失败。 +> 此属性定义查询错误恢复尝试的最大次数。当达到限制时，查询失败。 > -> 也可以使用`snapshot_max_retries`会话属性在每个查询基础上指定。 +> 也可以使用`recovery_max_retries`会话属性为每个查询指定此属性。 -### `hetu.snapshot.retryTimeout` +### `hetu.recovery.retryTimeout` > - **类型：** `duration` > - **默认值：** `10m`（10分钟） > -> 此属性定义系统等待所有任务成功恢复的最大时长。如果在此超时时限内任何任务未就绪，则认为恢复失败，查询将尝试从较早快照恢复（如果可用）。 +> 此属性定义系统等待所有任务成功恢复的最长时间。如果在此时间内有任何任务未就绪，则恢复尝试将被视为失败，查询将尝试从较早的快照恢复（如果可用）。 > -> 也可以使用`snapshot_retry_timeout`会话属性在每个查询基础上指定。 +> 也可以使用`recovery_retry_timeout`会话属性为每个查询指定此属性。 ### `hetu.snapshot.useKryoSerialization` @@ -839,6 +1079,13 @@ > > 为快照启用基于Kryo的序列化，而不是默认的Java序列化。 +### `experimental.eliminate-duplicate-spill-files` + + > - **类型：** `boolean` + > - **默认值：** `false` + > + > 快照捕获时，允许消除重复溢出文件写盘。 + ## HTTP客户端属性配置 ### `http.client.idle-timeout` @@ -859,4 +1106,244 @@ > 此参数定义了http客户端接收响应的时间阈值。 > 当超过所配置时间，客户端没有接收到任何响应，则视为客户端的请求提交失败。 > -> （注意： 建议在高负载环境下，该参数配置大一点。） \ No newline at end of file +> （注意： 建议在高负载环境下，该参数配置大一点。） + +## 连接器属性配置 + +### `case-insensitive-name-matching` + +> - **类型:** `boolean` +> - **默认值:** `false` +> +> 不区分大小写匹配数据库和集合名称，默认区分大小写。 + +## 查询故障容错配置 + +### `exchange-manager.name` + +> - **类型：**`string` +> - **默认值：**`filesystem` +> +> Task快照管理器的名称。 + +### `exchange-filesystem-type` + +> - **类型：**`string` +> - **默认值：**`exchange` +> +> 存储Task快照的文件系统客户端名称。如果exchange-filesystem-type=exchange，名称为exchange客户端将用于存储查询执行产生的Task快照。文件系统客户端的配置文件路径etc/filesystem/exchange.properties，参见[交换文件系统客户端配置](<./properties.html#交换文件系统客户端配置>)。 + +### `exchange.base-directories` + +> - **类型：**`URI` +> +> 存储Task快照的起始文件路径。存在多个根目录则以逗号分隔区分。 + +### `exchange.compression-enabled` + +> - **类型：**`boolean` +> - **默认值：**`false` +> +> 该属性启停Task快照的压缩功能。 + +### `exchange.max-page-storage-size` +> +> - **类型：** `data size` +> - **默认值：** `16MB` +> +> 可写入Task快照数据空间的最大页面大小，包含页面数据内容和记录页面大小的int类型元数据共同占用的空间。 + +### `exchange.sink-buffer-pool-min-size` +> +> - **类型：** `int` +> - **默认值：** `10` +> +> Task快照数据空间的最小缓冲池大小。缓冲池大小越大，写入并行度和内存使用量就越大。 + +### `exchange.sink-buffers-per-partition` +> +> - **类型：** `int` +> - **最小值：** `2` +> - **默认值：** `2` +> +> 缓冲池中每个分区的写入缓冲区个数。缓冲区个数越大，写入并行度和内存使用量就越大。 + +### `exchange.sink-max-file-size` +> +> - **类型：** `data size` +> - **默认值：** `1GB` +> +> Task快照数据空间可写入的最大文件大小。 + +### `exchange.source-concurrent-readers` +> +> - **类型：** `int` +> - **默认值：** `4` +> +> Task快照数据空间中并发阅读器的数量。并发阅读器的数量越多，读取并行度和内存使用量就越大。 + +### `exchange.max-output-partition-count` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `50` +> +> Task快照数据空间中可创建分区的最大数量。 + +### `exchange.file-listing-parallelism` +> +> - **类型：** `int` +> - **默认值：** `50` +> +> 枚举task快照文件时，调用文件列表的最大并行度。 + +### `retry-policy` +> +> - **类型：** `string` +> - **允许值：** `NONE`, `TASK` +> - **默认值：** `NONE` +> +> 在查询发生故障时的重试策略，`TASK策略`表示在任务失败时，只单独重试失败的task。 + +### `task-retry-attempts-overall` +> +> - **类型：** `int` +> - **默认值：** `null（(无限制）` +> +> 查询返回失败前，查询中所有task允许的最大重试次数。 + +### `task-retry-attempts-per-task` +> +> - **类型：** `int` +> - **最小值：** `0` +> - **默认值：** `4` +> +> 查询返回失败前，查询中单个task允许的最大重试次数。 + +### `retry-initial-delay` +> +> - **类型：** `duration` +> - **默认值：** `10s` +> +> Task在重试之前必须等待的最短时间，也可以使用retry_initial_delay会话属性进行配置。 + +### `retry-max-delay` +> +> - **类型：** `duration` +> - **默认值：** `1m` +> +> Task在重试之前必须等待的最长时间，每次故障后的等待时间都会相应增加，也可以使用retry_max_delay会话属性进行配置。 + +### `retry-delay-scale-factor` +> +> - **类型：** `double` +> - **默认值：** `2.0` +> +> 每次任务失败后，等待重试增加的系数。也可以使用retry_delay_scale_factor会话属性进行配置。 + +### `max-tasks-waiting-for-node-per-stage` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `5` +> +> 每个阶段中允许等待分配节点的task最大个数，超过该个数的task将被暂停调度。 + +### `fault-tolerant-execution-target-task-input-size` +> +> - **类型：** `data size` +> - **默认值：** `4GB` +> +> 重试的task从快照文件中最大读取的数据大小（以字节为单位）。也可以使用fault_tolerant_execution_target_task_input_size会话属性进行配置。 + +### `fault-tolerant-execution-target-task-split-count` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `64` +> +> 单个task从源表中读取“标准”数据分片的个数。该配置会考虑数据分片权重，如果源表拆分的数据分片权重比“标准”更轻或更重，则由单个task处理的数据分片个数将相应地进行调整。 +> 也可以使用fault_tolerant_execution_target_task_split_count会话属性进行配置。 + +### `fault-tolerant-execution-preserve-input-partitions-in-write-stage` +> +> - **类型：** `boolean` +> - **默认值：** `true` +> +> 向数据表写入数据的阶段，是否要求一个task只读取一个分区的输入。 + +### `fault-tolerant-execution-min-task-split-count` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `16` +> +> 单个task处理的最小数据分片个数。此值不会按数据分片权重进行重新调整，在源表返回不正确的分片权重的情况下，依然对配置的数据提供保护不调整。 +> 也可以使用fault_tolerant_execution_min_task_split_count会话属性进行配置。 + +### `fault-tolerant-execution-max-task-split-count` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `256` +> +> 单个task处理的最大数据分片个数。此值不会按数据分片权重进行重新调整，在源表返回不正确的分片权重的情况下，依然对配置的数据提供保护不调整。 +> 也可以使用fault_tolerant_execution_max_task_split_count会话属性进行配置。 + +### `fault-tolerant-execution-partition-count` +> +> - **类型：** `int` +> - **最小值：** `1` +> - **默认值：** `50` +> +> 用于分布式join和聚合的分区数。也可以使用fault_tolerant_execution_partition_count会话属性进行配置。 + +### `fault-tolerant-execution-task-descriptor-storage-max-memory` +> +> - **类型：** `data size` +> - **默认值：** `(JVM heap size * 0.15)` +> +> 在协调节点上，存储容错查询的task描述符的最大内存量。需要额外的内存才能在发生故障时重新调度task。 + +### `node-scheduler.allowed-no-matching-node-period` +> +> - **类型：** `duration` +> - **默认值：** `2m` +> +> 在分配不到符合task运行要求的节点前，调度允许的等待时间。 + +### `node-scheduler.allocator-type` +> +> - **类型：** `string` +> - **允许值：** `BIN_PACKING`, `FIXED_COUNT` +> - **默认值：** `BIN_PACKING` +> +> 配置节点分配器类型。 +> +## 交换文件系统客户端配置 + +### `fs.client.type` + +> - **类型：**`string` +> - **允许值：** `local`，`hdfs` +> - **默认值：**`local` +> +> 配置文件系统客户端的类型。如果`fs.client.type=local`，恢复框架使用local文件系统客户端。反之如果`fs.client.type=hdfs`，恢复框架使用hdfs文件系统客户端。 + +### `hdfs.config.resources` + +> - **类型：**`string` +> - **默认值：**`etc/filesystem/core-site.xml,etc/filesystem/hdfs-site.xml` +> +> hdfs文件系统所依赖的资源配置文件的路径。 +> +>（注意这是hdfs文件系统需要的配置属性，当资源配置文件不止一个时，用','隔开！） + +### `hdfs.authentication.type` + +> - **类型：**`string` +> - **默认值：**`NONE` +> +> 指定hdfs文件系统的身份验证类型。 +> +>（注意这是hdfs文件系统所需的配置属性！） diff --git a/content/zh-cn/docs/docs/admin/reliable-query/_index.md b/content/zh-cn/docs/docs/admin/reliable-query/_index.md new file mode 100644 index 0000000000000000000000000000000000000000..60ab95e8108f89e7776d27f97f8d72e8c96fc054 --- /dev/null +++ b/content/zh-cn/docs/docs/admin/reliable-query/_index.md @@ -0,0 +1,13 @@ + +# 概述 + +当集群中的节点因网络、硬件或软件问题发生故障时，在故障节点上运行任务的所有查询都将丢失。这可能会严重影响集群生产力并造成资源浪费，尤其对于长时间运行的查询。 + +解决这一问题的一种方法是自动重新运行受影响的查询。这减少了人工干预的需要，并提高了容错性，但同时会延长总执行时间。 + +为了保持执行可靠性同时实现更好的性能，openLooKeng提供了以下两种机制: + +- [算子级别快照](./reliable-query/operator-snapshot.html) +- [Task级别快照](./reliable-query/task-snapshot.html) + + diff --git a/content/zh-cn/docs/docs/admin/reliable-execution.md b/content/zh-cn/docs/docs/admin/reliable-query/operator-snapshot.md similarity index 61% rename from content/zh-cn/docs/docs/admin/reliable-execution.md rename to content/zh-cn/docs/docs/admin/reliable-query/operator-snapshot.md index 0a9a3bf6a96879defe151ebc8f678f295beb0150..0014fb0911653d31f173275fdb44105110bb87db 100644 --- a/content/zh-cn/docs/docs/admin/reliable-execution.md +++ b/content/zh-cn/docs/docs/admin/reliable-query/operator-snapshot.md @@ -1,18 +1,14 @@ -# 可靠查询执行 +# 算子级别快照 ## 概述 -当集群中的节点因网络、硬件或软件问题发生故障时，在故障节点上运行任务的所有查询都将丢失。这可能会严重影响集群生产力并造成资源浪费，尤其对于长时间运行的查询。 - -解决这一问题的一种方法是自动重新运行受影响的查询。这减少了人工干预的需要，并提高了容错性，但同时会延长总执行时间。 - -为了保持执行可靠性同时实现更好的性能，openLooKeng中的分布式快照机制定期保存查询执行的完整状态的快照。发生错误时，查询可以从上一个成功的快照恢复执行。该实现基于标准[Chandy-Lamport算法](https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport_algorithm)。 +openLooKeng中的算子级别快照，会定期保存查询执行的完整状态的快照。发生错误时，查询可以从上一个成功的快照恢复执行。该实现基于标准[Chandy-Lamport算法](https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport_algorithm)。 自版本1.2.0起，openLooKeng支持恢复任务和工作节点故障。 -## 启用分布式快照 - -分布式快照适用于长时间运行的查询任务。该功能默认为禁用状态，可以通过会话属性[`snapshot_enabled`](properties.html#snapshot_enabled)启用或禁用。建议仅在对可靠性要求高的复杂查询场景下启用该功能。 +## 启用恢复框架 + +恢复框架对于长时间运行的查询最有用。默认禁用，可以使用会话属性[`recovery_enabled`](../properties.html#recovery_enabled)启用或禁用恢复框架。建议仅对可靠性要求高的复杂查询启用该功能。 ## 要求 @@ -35,7 +31,7 @@ ## 检测 -协调节点与远程任务之间的通信长时间失败时，将触发错误恢复，由[`query.remote-task.max-error-duration`](properties.html#queryremote-taskmax-error-duration)配置控制。 +当协调器与远程任务之间的通信长时间失败时，将触发错误恢复，由[`故障恢复处理属性`](../properties.html#故障恢复处理属性)配置控制。 ## 存储注意事项 @@ -53,18 +49,20 @@ 从错误和快照中恢复需要成本。捕获快照需要时间，时间长短取决于复杂性。因此，需要在性能和可靠性之间进行权衡。 -建议仅在必要时启用分布式快照，如运行时间较长的查询任务。对于这些类型的工作负载，捕获快照的开销可以忽略不计。 +建议在必要时打开快照捕获，例如对于长时间运行的查询。对于这些类型的工作负载，拍摄快照的开销可以忽略不计。 ## 快照统计信息 在调试模式下启动CLI时，快照捕获信息和恢复信息将与查询结果一起显示在CLI中。 -快照捕获信息包括查询期间捕获的快照大小、捕获快照的CPU时间和捕获快照的挂钟时间。分别显示所有快照和最后一个快照的捕获信息。 +快照捕获统计信息包括捕获的快照数量、捕获的快照大小、捕获快照所需的CPU时间和在查询期间捕获快照所需的挂钟时间。所有快照和最后一个快照的统计信息会分别显示。 快照恢复信息包括查询期间从快照恢复的次数、加载用于恢复的快照大小、从快照恢复所需的CPU时间和从快照恢复所需的挂钟时间。仅当查询期间发生恢复时，才会显示恢复信息。 -![](../images/snapshot_statistics.png) +此外，在查询正在进行时，将显示捕获的快照数量和恢复的快照的ID。更多详细信息，见下图。 + +![](../../images/snapshot_statistics_cn.png) ## 配置 -与分布式快照功能相关的配置可参见[属性参考](properties.html#分布式快照)。 \ No newline at end of file +与恢复框架功能相关的配置，请参见[属性参考](../properties.html#查询恢复)。 \ No newline at end of file diff --git a/content/zh-cn/docs/docs/admin/reliable-query/task-snapshot.md b/content/zh-cn/docs/docs/admin/reliable-query/task-snapshot.md new file mode 100644 index 0000000000000000000000000000000000000000..0538c98ffd23d4a27ce892e6c8a30aff6b28046b --- /dev/null +++ b/content/zh-cn/docs/docs/admin/reliable-query/task-snapshot.md @@ -0,0 +1,65 @@ +# Task级别快照 + +## 概要 + +Task快照是另一种可靠查询机制，它是query重试查询和Operator快照重试的折中机制。与query重试查询相比，在发生故障后，通过Task快照进行Task粒度重试恢复，而不是整个query重新执行，效率更高。与Operator快照相比，Task粒度的快照空间占用更小，性能也更好。 + +默认情况下，查询失败后需要手动重试查询。openLooKeng中通过Task级别快照进行自动恢复来确保可靠的查询执行。Task级别快照启用后，查询过程中task之间的交换数据通过快照进行物化，当查询过程中Coordinator检测到worker发生故障或者task失败时，Coordinator调度空闲Worker读取故障worker最近保存的Task快照，然后自动恢复查询执行，这样可以保证查询的可靠性。对于耗时长的查询，将给用户更加确切的执行结果，更适合批处理场景。 + +![](../../images/task_snapshot_overview.png) + +## 启用Task级别快照 + +Task级别快照自动恢复机制，对于需要长时间运行的复杂查询最有效果。默认情况下处于禁用状态，可通过配置文件`etc/config.properties`中的配置属性`retry-policy`进行启用。 +- **启动**：`retry-policy=TASK` +- **停用**：`retry-policy=NONE` + +## 运行要求 + +为了确保查询执行通过Task快照能够自动恢复，保证查询执行的可靠性，集群需要满足以下要求： +- 启用Task级别快照特性 +- 集群至少包含2个工作节点 +- 除故障节点外，其他工作节点处于健康状态 + +## 限制 + +- **支持的语句**：仅支持`SELECT`，`INSERT`和`CREATE TABLE AS SELECT（CTAS）`类型的语句，不包括类似`INSERT INTO CUBE`的语句。 +- **数据源**：只能从`Hive`目录中的数据表读取。 +- **目标表**：只能写入`Hive`目录中的表，格式为`ORC`。 +- **特性互斥说明**：Task快照功能和Operator快照功能不支持同时启用。 + +## 重试策略 + +当Coordinator和Worker之间的通信失败或超时，以及task执行失败时，将触发重试查询。相关故障重试的配置如下： +- 属性：`task-retry-attempts-per-task` + - 默认值：4 + - 描述：重试返回失败前，单个Task允许的最大重试次数。 +- 属性：`task-initial-delay` + - 默认值：10s + - 描述：Task在重试之前必须等待的最短时间。 +- 属性：`task-max-delay` + - 默认值：1m + - 描述：Task在重试之前必须等待的最长时间。 + +## Task快照的存储方式 + +查询过程中Task间的交换数据和状态，以Task快照的形式物化到文件系统中，其中文件系统支持两种类型：`local`和`hdfs`。当发生故障时，查询过程中Coordinator调度其他空闲的Worker节点从文件系统中的指定目录读取Task快照，进行数据再次加载和重试。因此，所有Worker节点都必须能够共享访问存储Task快照的文件系统，这样才可以进行正常的恢复。交换管理器通过配置文件`etc/exchange-manager.properties`进行共享文件系统的配置。相关属性如下： +- 属性：`exchange-manager.name` + - 描述：配置交换管理器的名称。 +- 属性：`exchange-filesystem-type` + - 描述：存储Task快照的文件系统客户端名称。如果`exchange-filesystem-type=exchange`，名称为exchange客户端将用于存储查询执行产生的Task快照。文件系统客户端的配置文件路径`etc/filesystem/exchange.properties`（properties文件名需要和exchange-filesystem-type保持一致），参见[交换文件系统客户端配置](<../properties.html#交换文件系统客户端配置>)。 +- 属性：`exchange.base-directories` + - 描述：文件系统的根目录，存储Task快照的起始文件路径。所有Worker节点都必须具备共享访问的权限。 +- 属性：`exchange.compression-enabled` + - 描述：该属性启停Task快照的压缩功能。 + +请注意，Task快照用于保存Worker节点查询执行过程中，各Task之间的交换数据和状态。查询执行前确保集群有足够的空闲空间存储这些Task快照，查询完成后会自动清理来释放存储空间。 + +## 性能开销 + +从错误和Task快照中恢复需要成本。捕获快照需要时间，时间长短取决于复杂性。因此，需要在性能和可靠性之间进行权衡。 + +建议在必要时打开Task快照机制，例如对于长时间运行的查询。对于这些类型的工作负载，拍摄快照的开销可以忽略不计。 + +## 配置 +Task快照可靠查询机制的详细配置，请参见[属性参考](<../properties.html#查询故障容错配置>)。 diff --git a/content/zh-cn/docs/docs/admin/spill.md b/content/zh-cn/docs/docs/admin/spill.md index 28448409715f6a5541a74fa207e501126cbfeb97..435b3f3114a387490df4b7422d6846c070e5731b 100644 --- a/content/zh-cn/docs/docs/admin/spill.md +++ b/content/zh-cn/docs/docs/admin/spill.md @@ -57,6 +57,8 @@ openLooKeng将溢出路径视为独立的磁盘（参见[JBOD](https://en.wikipe 通过这种机制，联接操作符使用的峰值内存可以降低到最大构建表分区的大小。假设没有数据倾斜，这个值将是整个构建表大小的`1 / task.concurrency`倍。 +注意：spill-to-disk不支持交叉连接。 + ### 聚合 聚合函数对一组值执行操作并返回一个值。如果要聚合的组数量很大，可能需要大量内存。当启用溢出到磁盘时，如果没有足够的内存，则中间累积的聚合结果将写入磁盘。结果被重新加载回来，并以较低的内存占用量合并。 diff --git a/content/zh-cn/docs/docs/admin/tuning.md b/content/zh-cn/docs/docs/admin/tuning.md index d1f27ce066f0e53b9ec8d8086d66f132482af10f..a2896f35f5091025a3e626f701ba37ffd167da77 100644 --- a/content/zh-cn/docs/docs/admin/tuning.md +++ b/content/zh-cn/docs/docs/admin/tuning.md @@ -25,4 +25,36 @@ -XX:+PrintAdaptiveSizePolicy -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -``` \ No newline at end of file +``` + +## 性能调优说明 +下面是一些有助于优化查询执行性能的配置说明。 + +### 公共表表达式（CTE） 优化 +公共表表达式是查询执行计划中被多次用到的公共子执行计划。 +openLooKeng分析引擎评估CTE计划节点的使用情况，以确定CTE重用优化的可行性。这个优化在两个场景下被应用： + +> #### 重用执行管道 +> 如果配置 [optimizer.cte-reuse-enabled](./properties.html#optimizercte-reuse-enabled)：一个给定查询里的相同CTE计划节点被安排到stage管道中，并且只有一个CTE节点（生产者）被执行，其他的CTE节点（消费者）复用第一个CTE节点的输出结果。 +> 当CTE用于自连接（self join）时，那么CTE管道不可用。 +> 管道模式执行减少了重复的数据扫描和数据处理，从而减少了查询的执行时间。 + +> #### 物化和重用 +> 如果配置 [enable-cte-result-cache](./properties.html#enable-cte-result-cache)：CTE结果将物化到用户指定的存储位置(由[数据缓存配置](./properties.html#hetuexecutiondata-cacheschema-name)指定)。这里缓存了CTE节点的输出结果，物化成功后，可被后续的查询读取和重用。 + +### 执行计划优化 +* **使用精确的分区**：启用后，除非上游阶段的分区精确地匹配到下游阶段的分区，否则强制对数据重新分区（参考: [精准分区](./properties.html#optimizeruse-exact-partitioning)）。 + +* **自适应聚合**: 该特性可以自适应地进行部分聚合； 该特性受到以下三个配置控制： +> 1) [启用自适应部分聚合](./properties.html#adaptive-partial-aggregationenabled)：启用特性。 +> 2) [最小行数阈值](./properties.html#adaptive-partial-aggregationmin-rows)：可能自适应关闭部分聚合的最小处理数据行数。 +> 3) [唯一行数比值](./properties.html#adaptive-partial-aggregationunique-rows-ratio-threshold)：可能自适应关闭部分聚合的聚合输出、输入数据行数的比值。 + +* **join数据独立性假设下的选择率估计**: +> 1) [多子句join下数据独立性因子](./properties.html#optimizerjoin-multi-clause-independence-factor)：多子句连接的选择率估计的数据独立性假设因子。 +> 2) [多过滤条件下数据独立性因子](./properties.html#optimizerfilter-conjunction-independence-factor)：多过滤条件的选择率估计的数据独立性假设因子。 + +* **执行策略**：指定调度器实施的执行策略，配置可参考[指定执行策略](./properties.html#queryexecution-policy)： +> 1. _**all-at-once**_：该策略下调度器启动和处理所有的阶段。 +> 2. _**phased**_：该策略下调度器调度遵循阶段间生产者源这样的依赖关系，可以将所有独立的阶段一起调度。 +> 3. _**prioritize-utilization**_：该策略下调度器调度除了遵循生产者源这样的阶段依赖关系以外，它还查看动态过滤生产者的依赖路径。 diff --git a/content/zh-cn/docs/docs/admin/web-interface.md b/content/zh-cn/docs/docs/admin/web-interface.md index c2eed936abe9824711c1e6b8381df5e21303507e..f595c2959c0a163a11654ecd9adc04fa2cfce791 100644 --- a/content/zh-cn/docs/docs/admin/web-interface.md +++ b/content/zh-cn/docs/docs/admin/web-interface.md @@ -1,7 +1,7 @@ # Web界面 -openLooKeng提供了一个用于监视和管理查询的Web界面。Web界面可以在openLooKeng协调器上通过HTTP协议访问，使用协调器`config_properties`中指定的HTTP端口号。 +openLooKeng提供了一个用于监视和管理查询的Web界面。Web界面可以在openLooKeng协调节点上通过HTTP协议访问，使用协调节点`config.properties`中指定的HTTP端口号。 主页有一个查询列表，其中包含诸如唯一查询ID、查询文本、查询状态、完成百分比、用户名和该查询的来源等信息。当前运行的查询位于页面的顶部，紧随其后的是最近完成或失败的查询。 @@ -58,4 +58,27 @@ openLooKeng提供了一个用于监视和管理查询的Web界面。Web界面可 > - **类型：** `duration` > - **默认值：** `1 DAYS` > -> UI会话超时默认设置为1天。可以通过配置`etc/config.properties`文件中的`hetu.queryeditor-ui.session-timeout`属性修改。 \ No newline at end of file +> UI会话超时默认设置为1天。可以通过配置`etc/config.properties`文件中的`hetu.queryeditor-ui.session-timeout`属性修改。 + +### `hetu.queryhistory.max-count` + +> - **类型:** `int` +> - **默认值:** `1000` +> +> openLooKeng储存的历史查询记录最大数量。可以通过配置`etc/config.properties`文件的`hetu.queryhistory.max-count`属性修改。 + +### `hetu.collectionsql.max-count` + +> - **类型:** `int` +> - **默认值:** `100` +> +> 每位用户收藏sql语句条数上限.可以通过配置`etc/config.properties`文件的`hetu.collectionsql.max-count`属性修改。 + + +## 备注 + +收藏sql语句的最大长度默认为600，可通过如下步骤对其进行修改： + +1. 根据hetu-metastore.properties文件中jdbc配置信息登录mysql数据库。 + +2. 选中hetu_favorite表，使用命令`alter table hetu_favorite modify query varchar(2000) not null;`修改收藏语句最大长度为2000。 \ No newline at end of file diff --git a/content/zh-cn/docs/docs/connector/Hudi.md b/content/zh-cn/docs/docs/connector/Hudi.md index 87d8b246f77363e8f679f3f7fca80ffd47444dab..76c71b2b0be438f95f1115612109ce40e20b7c77 100644 --- a/content/zh-cn/docs/docs/connector/Hudi.md +++ b/content/zh-cn/docs/docs/connector/Hudi.md @@ -1,5 +1,8 @@ # Hudi连接器 +### 版本说明 +目前Hudi只支持0.7.0版本。 + ### Hudi介绍 Apache Hudi是一个快速迭代的数据湖存储系统，可以帮助企业构建和管理PB级数据湖。它提供在DFS上存储超大规模数据集，同时使得流式处理如果批处理一样，该实现主要是通过如下两个原语实现。 diff --git a/content/zh-cn/docs/docs/connector/clickhouse.md b/content/zh-cn/docs/docs/connector/clickhouse.md index 0064c7ce8dd95072f9b8f63bf8bf91d1a354d43e..2db280c34b3d8cc4bcb9b5bf1ea49871921724ee 100644 --- a/content/zh-cn/docs/docs/connector/clickhouse.md +++ b/content/zh-cn/docs/docs/connector/clickhouse.md @@ -71,9 +71,9 @@ case-insensitive-name-matching=true ### ClickHouse到openLooKeng类型映射 -下表显示了ClickHouse数据类型到openLooKeng的的映射关系。 +下表显示了ClickHouse数据类型到openLooKeng的映射关系。 -数据类型投影表： +数据类型映射表： | ClickHouse类型 | openLooKeng类型 | 说明 | | :-------------------- | :-------------- | :--- | @@ -216,7 +216,7 @@ DAY_OF_YEAR($1) INSERT语句需要使用CAST强制转换，例如table_name_test表中的数据类型为smallint： ``` -insert into table_name_test values (cast(1 as small int)); +insert into table_name_test values (cast(1 as smallint)); ``` 由于查询语句的执行顺序不同，ClickHouse支持 as 表达式的别名在where中使用，但在openLooKeng中不允许。 diff --git a/content/zh-cn/docs/docs/connector/dameng.md b/content/zh-cn/docs/docs/connector/dameng.md new file mode 100644 index 0000000000000000000000000000000000000000..9e48c8835d04ad85f4600fde4ed717690a142baf --- /dev/null +++ b/content/zh-cn/docs/docs/connector/dameng.md @@ -0,0 +1,162 @@ +# DM(达梦数据库)连接器 + +## 概述 + +DM连接器允许在外部DM数据库中查询和创建表。这可用于在DM和Hive等不同系统之间或在两个不同的DM实例之间联接数据。 + +## 配置 + +首先，在开始使用DM连接器之前，应该先完成以下步骤： + +- 用于连接DM的JDBC连接详情 + +要配置DM连接器，在`etc/catalog`中创建一个目录属性文件，例如`dameng.properties`。使用以下内容创建文件，并根据设置替换连接属性： + +```properties +connector.name=dameng +connection-url=jdbc:dm://主机:端口/SYSDBA +connection-user=用户名 +connection-password=密码 +``` + +- 添加DM的JDBC驱动 + +DM JDBC驱动不在普通存储库中提供，如果您是DM数据库的用户，您可以选择前往DM官方网站，在确保遵守DM JDBC驱动所适用的license的条件下，下载和安装DM JDBC驱动到存储库中。DM JDBC驱动（DmJdbcDriver***X***.jar ***X***为数字，根据DM的版本不同而不同）可能会作为DM客户端安装的一部分进行安装。获取了DM JDBC驱动后，您可以将**jdbc jar**文件部署到协调节点和工作节点上的openLooKeng插件文件夹中。例如，jdbc驱动文件为**DmJdbcDriverX.jar**，openLooKeng插件包文件夹为 **/opt/hetu-server/plugin**，则拷贝命令如下： **cp DmJdbcDriverX.jar /opt/hetu-server/plugin/dm**。重启协调节点和工作节点进程，DM连接器即可正常工作。 + +- 是否开启查询下推功能 + +如果要启用DM连接器的连接器下推功能，不需要做任何操作，DM连接器的下推功能默认是打开的。但也可以按如下设置： + +``` properties +jdbc.pushdown-enabled=true +#true表示打开下推，false表示关闭。 +``` + +- 下推模式选择。 + +如果要启用DM连接器的全部下推功能，不需要做任何操作，DM连接器的下推模式默认是全部下推的。但也可以按如下设置： + +``` properties +jdbc.pushdown-module=FULL_PUSHDOWN +#FULL_PUSHDOWN，表示全部下推；BASE_PUSHDOWN，表示部分下推，其中部分下推是指filter/aggregation/limit/topN/project这些可以下推。 +``` + +- 更多配置 + +由于达梦数据库和Oracle同源，在实现DM连接器时，复用了Oracle连接器的部分配置，具体配置可参考[ORACLE](./oracle.html)。 + +## 通过openLooKeng查询DM + +对于名为**dameng**的DM连接器，每个DM数据库的用户都可以通过DM连接器获取其可用的模式，命令为**SHOW SCHEMAS**： + + SHOW SCHEMAS FROM dameng; + +如果已经拥有了可用模式，可以通过**SHOW TABLES**命令查看名为**data**的DM数据库拥有的表： + + SHOW TABLES FROM dameng.data; + +若要查看数据模式中名为**hello**的表中的列的列表，请使用以下命令中的一种： + + DESCRIBE dameng.data.hello; + SHOW COLUMNS FROM dameng.data.hello; + +你可以访问数据模式中的**hello**表： + + SELECT * FROM dameng.data.hello; + +连接器在这些模式中的权限是在连接属性文件中配置的用户的权限。如果用户无法访问这些表，则特定的连接器将无法访问这些表。 + +## DM Update/Delete 支持 + +### 使用DM连接器创建表 + +示例： + +```sql +CREATE TABLE dameng_table ( + id int, + name varchar(255)); +``` + +### 对表执行INSERT + +示例： + +```sql +INSERT INTO dameng_table + VALUES + (1, 'foo'), + (2, 'bar'); +``` + +### 对表执行UPDATE + +示例： + +```sql +UPDATE dameng_table + SET name='john' + WHERE id=2; +``` + +上述示例将值为`2`的列`id`的行的列`name`的值更新为`john`。 + +UPDATE前的SELECT结果： + +```sql +lk:default> SELECT * FROM dameng_table; +id | name +----+------ + 2 | bar + 1 | foo +(2 rows) +``` + +UPDATE后的SELECT结果 + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 2 | john + 1 | foo +(2 rows) +``` + +### 对表执行DELETE + +示例： + +```sql +DELETE FROM dameng_table + WHERE id=2; +``` + +以上示例删除了值为`2`的列`id`的行。 + +DELETE前的SELECT结果： + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 2 | john + 1 | foo +(2 rows) +``` + +DELETE后的SELECT结果： + +```sql +lk:default> SELECT * FROM dameng_table; + id | name +----+------ + 1 | foo +(1 row) +``` + +## DM连接器的限制 + +- openLooKeng支持连接DM8。 + +- DM连接器暂不支持Update下推功能。 \ No newline at end of file diff --git a/content/zh-cn/docs/docs/connector/elasticsearch.md b/content/zh-cn/docs/docs/connector/elasticsearch.md index a9be9709ef4cc43a53dfdf0707b95d97f1e37dc9..d25b547499a06855422af3553957fc85dd870219 100644 --- a/content/zh-cn/docs/docs/connector/elasticsearch.md +++ b/content/zh-cn/docs/docs/connector/elasticsearch.md @@ -39,6 +39,40 @@ elasticsearch.default-schema-name=default 此属性是可选的；默认值为`default`。 +### `elasticsearch.pushdown.enabled` +配置是否启用算子下推功能 + +默认情况下Elasticsearch连接器的算子下推功能是关闭的。你可以通过如下设置来启用它： + +``` properties +elasticsearch.pushdown.enabled=true +#true indicates that pushdown is enabled, and false indicates that pushdown is disabled. +``` +注意：目前算子下推仅支持部分SQL语句的查询关键字，包括EQUAL、AND、OR、算数运算符、BETWEEN、IN，以及部分聚合语法，包括GROUP BY、SUM、COUNT方法。 + + +数据结构算子下推支持矩阵: + +| Elasticsearch | openLooKeng | 聚合 | 查询 | +|:-----------------|:------------|:----|:----| +| `binary` | VARBINARY | 否 | 否 | +| `boolean` | BOOLEAN | 是 | 是 | +| `double` | DOUBLE | 是 | 是 | +| `float` | REAL | 是 | 是 | +| `byte` | TINYINT | 是 | 是 | +| `short` | SMALLINT | 是 | 是 | +| `integer` | INTEGER | 是 | 是 | +| `long` | BIGINT | 是 | 是 | +| `keyword` | VARCHAR | 是 | 是 | +| `text` | VARCHAR | 否 | 是 | +| `date` | TIMESTAMP | 是 | 是 | +| `ip` | IPADDRESS | 是 | 是 | +| `(其他类型)` | | 否 | 否 | + +**说明** + +由于Elasticsearch不支持针对text类型字段的聚合，因此该类型聚合下推无法成功。 + ### `elasticsearch.scroll-size` 此属性定义每个Elasticsearch滚动请求中可以返回的最大命中数。 diff --git a/content/zh-cn/docs/docs/connector/hana.md b/content/zh-cn/docs/docs/connector/hana.md index 29fddeb0d1ed9d1b73d132cb087f828cb8915ba9..89cecf6024c40990a58b1fba11ec4b8d727b5550 100644 --- a/content/zh-cn/docs/docs/connector/hana.md +++ b/content/zh-cn/docs/docs/connector/hana.md @@ -226,7 +226,7 @@ FROM SCORES GROUP BY NAME ``` -在Prersto中，可以使用`case`作为替代实现： +在openLooKeng中，可以使用`case`作为替代实现： Hana与openLooKeng SQL语法的其他差异，请参考以下官方文档列表： diff --git a/content/zh-cn/docs/docs/connector/iceberg.md b/content/zh-cn/docs/docs/connector/iceberg.md new file mode 100644 index 0000000000000000000000000000000000000000..8c3d0abb1971eb271d62d8701a5eddf3f1fd988d --- /dev/null +++ b/content/zh-cn/docs/docs/connector/iceberg.md @@ -0,0 +1,377 @@ +# iceberg连接器 +============== + +## 概述 + +openLooKeng Iceberg 是一种用于大型分析数据集的开放表式。Iceberg 连接器允许查询存储在以 Iceberg 格式编写的文件中的数据。 +Iceberg 数据文件可以存储为 Parquet、ORC 格式，由format表定义中的属性决定。该表format默认为ORC。 + +## 要求 + +要使用 Iceberg，您需要： + +- 从openLooKeng 协调节点和工作节点到分布式对象存储的网络访问。 +- 访问 Hive 元存储服务 (HMS) 。 +- 从 openLooKeng 协调器到 HMS 的网络访问。使用 Thrift 协议的 Hive Metastore 访问默认使用端口 9083。 + +## Hive元存储目录 + +Hive 元存储目录是默认实现。使用它时，Iceberg 连接器支持与 Hive 连接器相同的元存储配置属性。至少，hive.metastore.uri必须配置。 + +```properties +connector.name=iceberg +hive.metastore.uri=thrift://localhost:9083 +``` + +## 一般配置 + +这些配置属性与使用的目录实现无关。 + +Iceberg 通用配置属性 +用以下内容创建`etc/catalog/iceberg.properties`，请把`localhost:9083`替换为Hive元存储Thrift服务的正确主机和端口： + +| 属性名 | 属性值 |是否必填 | 描述 | +|:-----------------------|:------------------------|:-------------|:-------------------------------------| +|connector.name | iceberg |是 | 连接名 | +|hive.metastore.uri | thrift://localhost:9083|是 | hive连接地址 | +|iceberg.file-format | ORC |否 | 为Iceberg表定义数据存储文件格式。可能的值是PARQUET、ORC | +|iceberg.compression-codec| ZSTD|否|写入文件时要使用的压缩编解码器 可能的值是( NONE SNAPPY LZ4 ZSTD GZIP) +|iceberg.use-file-size-from-metadata|true|否|从元数据而不是文件系统中读取文件大小。此属性应仅设置为此问题的解决方法。此问题已在Iceberg版本0.11.0 中修复。 +|iceberg.max-partitions-per-writer|100|否|每个写入器处理的最大分区数。 +|iceberg.unique-table-location|true|否|使用随机的唯一表位置。 +|iceberg.dynamic-filtering.wait-timeout|0s|否|在拆分生成期间等待动态筛选器完成的最大持续时间。 +|iceberg.table-statistics-enabled|true|否|启用表统计信息。等效目录会话属性用于会话特定的用途。设置为 以禁用统计信息。禁用统计信息意味着基于开销的优化无法做出有关查询计划的明智决策。statistics_enabledfalse +|iceberg.minimum-assigned-split-weight|0.05|否|范围 （0， 1] 中的十进制值，用作分配给每个拆分的权重的最小值。较低的值可能会提高包含小文件的表的性能。较高的值可能会提高具有高度偏斜聚合或联接的查询的性能。 + +## SQL 支持 + +此连接器提供对 Iceberg 中的数据和元数据的读取访问和写入访问。除了全局可用和读取操作语句外，连接器还支持以下功能： + +### 表相关语句 +- 创建 +```sql +CREATE TABLE ordersg ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) +WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']); +``` +### 列相关语句 +- 新增 +```sql +alter table ordersg add COLUMN zip varchar; +``` +- 重命名 +```sql +ALTER TABLE ordersg RENAME COLUMN zip TO zip_r; +``` +- 删除 +```sql +ALTER TABLE ordersg DROP COLUMN zip_r; +``` +### 数据相关语句 +- 插入 +```sql +insert into ordersg values(1,date'1988-11-09',666888,'tim','US'); +``` +- 删除 +```sql +Delete from ordersg where order_id = 1; +``` +- 更新 +```sql +update ordersg set customer = 'Alice' where order_id = 2; +``` +- 查询 +```sql +select * from ordersg; +``` + +## 分区表 + +Iceberg 通过在表列上指定转换来支持分区。为转换生成的每个唯一元组值创建一个分区。身份转换只是列名。其他变换是： + +| 转换 | 描述 | +|:------------------------|:-----------------------------------------------| +| year(ts) | 每年都会创建一个分区。分区值是1970年1月1日和1970 年1月1日之间的整数差。 | +| month(ts) | 每年的每个月都会创建一个分区。分区值是1970年1月1日和1970 年1月1日之间的整数差。 | +| day(ts) | 每年的每一天创建一个分区。分区值是1970年1月1日和1970年 1月1日之间的整数差。 | +| hour(ts) | 每天每小时创建一个分区。分区值是分钟和秒设置为零的时间戳。 | +| bucket(x,nbuckets) | 数据被散列到指定数量的桶中。分区值是 的整数散列x，值介于 0 和 0 之间。nbuckets-1 | +| truncate(s,nchars) | 分区值是 的第一个nchars字符s。 | + +在此示例中，创建表并按照order_date表的月份、order_date的散列（具有 10 个存储桶），进行分区 +```sql +CREATE TABLE ordersg ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) + WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']); +``` +手动修改分区 +```sql +ALTER TABLE ordersg SET PROPERTIES partitioning = ARRAY['month(order_date)']; +``` + +按分区删除，对于分区表，如果WHERE子句仅在标识转换的分区列上指定过滤器，Iceberg 连接器支持删除整个分区，这可以匹配整个分区。鉴于上面的表定义，此 SQL 将删除所有分区country：US + +```sql +DELETE FROM iceberg.testdb.ordersg WHERE country = 'US'; +``` + +## 版本查询回溯 + +Iceberg 支持数据的“快照”模型，可以通过查询语句查询任意历史版本的数据，其中表快照由快照 ID 标识。 + +连接器为每个 Iceberg 表提供了一个系统快照表。快照由 BIGINT 快照 ID 标识。customer_orders您可以通过运行以下命令找到表的最新快照 ID ： +```sql +SELECT snapshot_id FROM "ordersg $snapshots" ORDER BY committed_at DESC LIMIT 10; +``` +|snapshot_id| +|:----------| +|921254093881523606| +|535467754709887442| +|343895437069940394| +|34i302849038590348| +|(4 rows)| + +SQL 过程system.rollback_to_snapshot允许调用者将表的状态回滚到之前的快照 ID： +```sql +CALL iceberg.system.rollback_to_snapshot('testdb', 'ordersg', 8954597067493422955); +``` + +## 元数据表 + +连接器为每个 Iceberg 表公开了几个元数据表。这些元数据表包含有关 Iceberg 表内部结构的信息。您可以通过将元数据表名称附加到表名称来查询每个元数据表： + +```sql +SELECT * FROM "ordersg$data"; +``` + +### $data表# + +该$data表是 Iceberg 表本身的别名。 + +该命名： +```sql +SELECT * FROM "ordersg$data"; +``` +相当于： +```sql +SELECT * FROM ordersg; +``` + +### $properties 表 + +该$properties表提供了对有关 Iceberg 表配置的一般信息以及该表标记的任何其他元数据键/值对的访问。 + +您可以使用以下查询检索Iceberg 表的当前快照的属性： +```sql +SELECT * FROM "ordersg$properties"; +``` + +key | value | +-----------------------+----------+ +write.format.default | PARQUET | + + +### $history表 +该$history表提供了对 Iceberg 表执行的元数据更改的日志。 + +您可以使用以下查询检索 Iceberg 表的更改日志： + +```sql +SELECT * FROM "ordersg$history"; +``` + +made_current_at | snapshot_id | parent_id | is_current_ancestor +----------------------------------+----------------------+----------------------+-------------------- +2022-08-19 05:42:37.854 UTC | 7464177163099901858 | 7924870943332311497 | true +2022-08-19 05:44:35.212 UTC | 2228056193383591891 | 7464177163099901858 | true + +查询的输出具有以下列： + +历史专栏 + +|姓名 | 类型 | 描述| +|:----------------------------|:----------------------------|:----------------------| +|made_current_at| timestamp(3)with time zone | 快照激活的时间| +|snapshot_id| bigint | 快照的标识符| +|parent_id| bigint |父快照的标识符| +|is_current_ancestor| boolean |此快照是否是当前快照的祖先| + +### $snapshots 表 + +该$snapshots表提供了 Iceberg 表快照的详细视图。快照由一个或多个文件清单组成，完整的表内容由这些清单中所有数据文件的并集表示。 + +您可以使用以下查询检索有关 Iceberg 表的快照的信息 ： + +```sql +SELECT * FROM "ordersg$snapshots"; +``` + +| committed_at | snapshot_id | parent_id | operation | manifest_list | summary | +| 2022-08-08 08:20:04.126 UTC | 7026585913702073835 | | append | hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/snap-7026585913702073835-1-d0b5ba3d-6363-4f32-974e-79bb68d19423.avro | {changed-partition-count=0, total-equality-deletes=0, total-position-deletes=0, total-delete-files=0, total-files-size=0, total-records=0, total-data-files=0} | +| 2022-08-08 08:21:58.343 UTC | 629134202395791160 | 7026585913702073835 | append | hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/snap-629134202395791160-1-b6e9c1c3-0532-4bf8-a814-a159494e272d.avro | {changed-partition-count=1, added-data-files=1, total-equality-deletes=0, added-records=1, total-position-deletes=0, added-files-size=289, total-delete-files=0, total-files-size=289, total-records=1, total-data-files=1} | + +查询的输出具有以下列： +快照列 + +|姓名 | 类型 | 描述 | +|:----------|:-----------------------------|:--------------------------------------------------------------------------------------------------------------| +|committed_at | timestamp(3) with time zone | 快照激活的时间 | +|snapshot_id| bigint | 快照的标识符 | +|parent_id |bigint | 父快照的标识符 | +|operation | varchar | 在 Iceberg 表上执行的操作类型。Iceberg 中支持的操作类型有：- append添加新数据时 -replace当文件被删除和替换而不更改表中的数据时 -overwrite添加新数据以覆盖现有数据时 -delete当从表中删除数据并且没有添加新数据时 | +|manifest_list| varchar | 包含有关快照更改的详细信息的 avro 清单文件列表。 | +|summary | map(varchar, varchar) | 从上一个快照到当前快照所做更改的摘要 | + +### $manifests表# + +该$manifests表提供了与 Iceberg 表日志中执行的快照相对应的清单的详细概述。 +您可以使用以下查询检索有关 Iceberg 表的清单的信息 ： + +```sql +SELECT * FROM "ordersg$manifests"; +``` + +Path | length | partition_spec_id | added_snapshot_id | added_data_files_count | existing_data_files_count | deleted_data_files_count | partitions +---------------------------------------+---------------------+---------------------+---------------------+-------------------+--------------------+--------------------+--------------------+--------------------+------------------- +hdfs://hadoop1:9000/home/gitama/hadoop/hive/user/hive/warehouse/test_100.db/orders08/metadata/b6e9c1c3-0532-4bf8-a814-a159494e272d-m0.avro | 6534 | 0 | 629134202395791160 | 1 | 0 | 0 | [ ] + +查询的输出具有以下列： +清单列 + +|名称| 类型 |描述| +|:----|:---------------------|:--------| +|path| varchar |清单文件位置| +|length | bigint |清单文件长度| +|partition_spec_id| integer | 用于写入清单文件的分区规范的标识符| +|added_snapshot_id | bigint | 添加此清单条目的快照的标识符| +|added_data_files_count| integer |ADDED清单文件中具有状态的数据文件的数量| +|existing_data_files_count | integer |EXISTING清单文件中具有状态的数据文件的数量| +|deleted_data_files_count | integer |DELETED清单文件中具有状态的数据文件的数量| +|partitions| array(row(contains_null boolean, contains_nan boolean, lower_bound varchar, upper_bound varchar)) | 分区范围元数据| + + +### $partitions表 +该$partitions表提供了 Iceberg 表的分区的详细概述。 +您可以使用以下查询检索有关 Iceberg 表分区的信息 ： + +```sql +SELECT * FROM "ordersg$partitions"; +``` + +| record_count | file_count | total_size | data | --------------+-------------+---------------------+---------------------+------------------------------------| 1 | 1 | 289 | {id={min=null, max=null, null_count=0, nan_count=null}, name={min=null, max=null, null_count=0, nan_count=null}} | + +查询的输出具有以下列： + +分区列 + +|姓名| 类型 | 描述| +|:-----|:-----------------------------------------------------------------------|:-------| +|record_count | bigint |分区中的记录数| +|file_count| bigint | 分区中映射的文件数| +|total_size| bigint | 分区中所有文件的大小| +|data | row(... row (min ..., max ... , null_count bigint, nan_count bigint)) |分区范围元数据| + +### $files表 + +该$files表提供了 Iceberg 表当前快照中数据文件的详细概述。 +要检索有关 Iceberg 表的数据文件的信息，请使用以下查询： + +```sql +SELECT * FROM "ordersg$files"; +``` + +| content | file_path | file_format | record_count | file_size_in_bytes | column_sizes | value_counts | null_value_counts | nan_value_counts | lower_bounds | upper_bounds | key_metadata | split_offsets | equality_ids | +| 0 | hdfs://192.168.31.120:9000/user/hive/warehouse/orders19/data/20220819_034313_39152_vdmku-1709db2a-dc6f-4ef9-bb77-23f4c150801f.orc | ORC | 1 | 354 | | {1=1, 2=1, 3=1}","{1=0, 2=0, 3=0} | | | | | | | +| 0 | hdfs://192.168.31.120:9000/user/hive/warehouse/orders19/data/20220819_054009_11365_xq568-1803130c-6b7b-4da6-b460-dfb44f176ef4.orc | ORC | 1 | 413 | | {1=1, 2=1, 3=1, 4=1} | {1=0, 2=0, 3=0, 4=1} | | | | | | | + +查询的输出具有以下列： + +文件列 + +|姓名 |类型| 描述| +|:---------|:----------|:-------------| +|content |integer| 存储在文件中的内容类型。Iceberg 中支持的内容类型有： -DATA(0) - POSITION_DELETES(1) -EQUALITY_DELETES(2)| +|file_path| varchar| 数据文件位置| +|file_format| varchar |数据文件的格式| +|record_count| bigint| 数据文件中包含的条目数| +|file_size_in_bytes| bigint |数据文件大小| +|column_sizes| map(integer, bigint)| Iceberg 列 ID 与其在文件中对应的大小之间的映射| +|value_counts| map(integer, bigint) |Iceberg 列 ID 与其对应的文件中条目数之间的映射| +|null_value_counts| map(integer,bigint) |Iceberg 列 ID 与其NULL在文件中对应的值计数之间的映射| +|nan_value_counts| map(integer,bigint)| Iceberg 列 ID 与其对应的文件中非数值计数之间的映射| +|lower_bounds| map(integer,bigint)| Iceberg 列 ID 与其在文件中对应的下限之间的映射| +|upper_bounds| map(integer,bigint)| Iceberg 列 ID 与其在文件中对应的上界之间的映射| +|key_metadata| varbinary |有关用于加密此文件的加密密钥的元数据（如果适用）| +|split_offsets| array(bigint)| 推荐拆分位置列表| +|equality_ids| array(integer) |相等删除文件中用于相等比较的字段 ID 集| + + +## 更改表执行 + +连接器支持以下用于ALTER TABLE EXECUTE的命令(具体请见文件合并)。 + +## 文件合并 + +该optimize命令用于重写指定表的活动内容，以便将其合并为更少但更大的文件。在表被分区的情况下，数据压缩分别作用于每个选择进行优化的分区，此操作提高了读取性能。 + +合并所有大小低于可选file_size_threshold参数（阈值的默认值为100MB）的文件： +```sql +ALTER TABLE ordersg EXECUTE optimize; +``` +以下语句合并表中小于 10 兆字节的文件： +```sql +ALTER TABLE ordersg EXECUTE optimize(file_size_threshold => '10MB'); +``` +## 更改表集属性 + +连接器支持使用ALTER TABLE SET PROPERTIES修改现有表的属性。 + +创建表后可以更新以下表属性： +- format +- partitioning + +```sql +ALTER TABLE ordersg SET PROPERTIES format ='PARQUET'; +``` + +或者可以将该列设置country为表上的分区列： +```sql +ALTER TABLE ordersg SET PROPERTIES partitioning = ARRAY[, 'country']; +``` + +可以使用SHOW CREATE TABLE ordersg 显示表属性的当前值。 + +### openLooKeng 与 Iceberg type 转换 +| openLooKeng 类型 | Iceberg 类型 +|:-----------------------|:------------------------ +|BOOLEAN | BOOLEAN +|INTEGER | INT +|BIGINT | LONG +|REAL | FLOAT +|DOUBLE | DOUBLE +|DECIMAL(p,s) | DECIMAL(p,s) +|DATE | DATE +|TIME | TIME +|TIMESTAMP | TIMESTAMP +|TIMESTAMP WITH TIME ZONE | TIMESTAMPTZ +|VARCHAR | STRING +|VARBINARY | BINARY +|ROW(...) |STRUCT(...) +|ARRAY(e) |LIST(e) +|MAP(k,v) |MAP(k,v) + + + + + + + + diff --git a/content/zh-cn/docs/docs/connector/kafka.md b/content/zh-cn/docs/docs/connector/kafka.md index ab1f61156cce82436cef283829cf1c08e178700d..35e87302e55bdfa911d51c99153d6cf00d0c2f7b 100644 --- a/content/zh-cn/docs/docs/connector/kafka.md +++ b/content/zh-cn/docs/docs/connector/kafka.md @@ -29,16 +29,23 @@ kafka.nodes=host1:port,host2:port 配置属性包括： -| 属性名称| 说明| -|:----------|:----------| -| `kafka.table-names`| 目录提供的所有表列表| -| `kafka.default-schema`| 表的默认模式名| -| `kafka.nodes`| Kafka集群节点列表| -| `kafka.connect-timeout`| 连接Kafka集群超时| -| `kafka.buffer-size`| Kafka读缓冲区大小| -| `kafka.table-description-dir`| 包含主题描述文件的目录| -| `kafka.hide-internal-columns`| 控制内部列是否是表模式的一部分| - +| 属性名称| 说明 | +|:----------|:-----------------------------------| +| `kafka.table-names`| 目录提供的所有表列表 | +| `kafka.default-schema`| 表的默认模式名 | +| `kafka.nodes`| Kafka集群节点列表 | +| `kafka.connect-timeout`| 连接Kafka集群超时 | +| `kafka.buffer-size`| Kafka读缓冲区大小 | +| `kafka.table-description-dir`| 包含主题描述文件的目录 | +| `kafka.hide-internal-columns`| 控制内部列是否是表模式的一部分 | +| `kerberos.on`| 是否开启Kerberos认证 | +| `user.password.auth.on`| 是否开启kafka用户密码认证 | +| `sasl.jaas.config`| 认证相关信息 | +| `java.security.krb5.conf`| krb5.conf文件路径 | +| `group.id`| kafka的groupID | +| `security.protocol`| Kafka的安全认证协议 | +| `sasl.mechanism`| sasl机制 | +| `sasl.kerberos.service.name`| kafka服务运行时的kerberos principal name | ### `kafka.table-names` 此目录提供的所有表的逗号分隔列表。表名可以是非限定的（简单名称），并将被放入默认模式（见下文）中，或者用模式名称（`.`）限定。 @@ -87,6 +94,56 @@ openLooKeng必须仍然能够连接到群集的所有节点，即使这里只指 此属性是可选的；默认值为`true`。 +### `kerberos.on` + +是否开启kerberos认证，适用于开启了kerberos认证的集群，如果在运行presto-kafka中的测试包，请置为false，因为测试程序使用内嵌Kafka，不支持认证，且该项与`user.password.auth.on`仅能选择一个，若两者均为true，则`user.password.auth.on`将覆盖`kerberos.on`且会异常。 + +此属性是可选的；默认值为`false`。 + +### `user.password.auth.on` + +是否开启用户密码认证，适用于开启了用户密码认证的集群，如果在运行presto-kafka中的测试包，请置为false，因为测试程序使用内嵌Kafka，不支持认证，且该项与`kerberos.on`仅能选择一个，若两者均为true，则`user.password.auth.on`将覆盖`kerberos.on`且会异常。 +若使用用户密码认证，需要在jvm配置中增加-Djava.ext.dirs=$JAVA_HOME/jre/lib/ext:/Users/mac/apps/lib，其中/Users/mac/apps/lib下需要放置kafka-client.jar，否则会报出 loginModule not found的相关异常 +此属性是可选的；默认值为`false`。 + +### `sasl.jaas.config` + +Kafka的认证相关信息。 +对于kerberos认证，可能的值为：sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=true serviceName=kafka keyTab=\"/Users/mac/Desktop/user01.keytab\" principal=\"user01@EXAMPLE.COM\"; +对于用户密码认证，可能的值为：sasl.jaas.config= org.apache.kafka.common.security.plain.PlainLoginModule required username=\"producer\" password=\"producerpwd\"; + +此属性是可选的；默认值为``。 + +### `java.security.krb5.conf` + +存放krb5.conf文件的路径，要注意全局配置中也需要配置此选项，例如部署后在jvm.config中配置,而在开发中需要在启动PrestoServer时使用"-D"参数配置。 + +此属性是可选的；默认值为``。 + +### `group.id` + +kafka的groupId。 + +此属性是可选的；默认值为``。 + +### `security.protocol` + +Kafka的安全协议。 + +此属性是可选的；默认值为`SASL_PLAINTEXT`。 + +### `sasl.mechanism` + +sasl机制，被用于客户端连接安全的机制。 + +此属性是可选的；默认值为`GSSAPI`。 + +### `sasl.kerberos.service.name` + +kafka运行时的kerberos principal name。 + +此属性是可选的；默认值为`kafka`。 + ## 内部列 对于每个已定义的表，连接器维护以下列： diff --git a/content/zh-cn/docs/docs/connector/mariadb.md b/content/zh-cn/docs/docs/connector/mariadb.md new file mode 100644 index 0000000000000000000000000000000000000000..6d000604aa764e0e10f0fa5adf0b4905f9bacec2 --- /dev/null +++ b/content/zh-cn/docs/docs/connector/mariadb.md @@ -0,0 +1,99 @@ +# MariaDB连接器 + +MariaDB连接器允许在外部MariaDB数据库中查询和创建表。这可用于在MySQL和MariaDB等不同系统之间或在两个不同的MariaDB实例之间联接数据。 + +## 配置 + +要配置MariaDB连接器，在openLooKeng的安装目录中`etc/catalog`中创建一个目录属性文件，例如`maria.properties`，将MariaDB连接器挂载为maria目录。使用以下内容创建文件，并根据设置替换连接属性： + +```properties +connector.name=maria +connection-url=jdbc:mariadb://部署了MairaDB的ip:MariaDB服务端口(默认3306) +connection-user=您的数据源用户 +connection-password=您的数据源密码 +``` + +- 是否开启查询下推功能 + +如果要启用MariaDB连接器的连接器下推功能，不需要做任何操作，MariaDB连接器的下推功能默认是打开的。但也可以按如下设置： + +```properties +#true表示打开下推，false表示关闭。 +jdbc.pushdown-enabled=true +``` + +- 下推模式选择 + +MariaDB连接器的下推模式默认是部分下推的，如果要启用MariaDB连接器的全部下推功能，可以按如下设置： + +```properties +#FULL_PUSHDOWN，表示全部下推；BASE_PUSHDOWN，表示部分下推，其中部分下推是指filter/aggregation/limit/topN/project这些可以下推。 +jdbc.pushdown-module=FULL_PUSHDOWN +``` + +### 外部函数注册 + +MariaDB连接器支持注册外部函数。 + +配置支持下推的外部函数注册命名空间`catalog.schema`。 例如在`etc/catalog/maria.properties`中配置： + +```Properties +jdbc.pushdown.remotenamespace=mariafun.default +``` + +### 外部函数下推 + +将外部函数下推到MariaDB数据源执行。 + +配置支持下推的外部函数注册命名空间`catalog.schema`。 例如在`etc/catalog/maria.properties`中配置： + +```Properties +jdbc.pushdown.remotenamespace=mariafun.default +``` + +可以声明自己支持多个函数命名空间中的函数，在`jdbc.pushdown.remotenamespace`配置项中使用'|'分割既可。例如： + +```Properties +jdbc.pushdown.remotenamespace=mariafun1.default|mariafun2.default|mariafun3.default +#表示当前Connector实例同时支持mysqlfun1.default、mysqlfun2.default、mysqlfun3.default三个函数命名空间最终的函数下推到当前连接的数据源中执行。 +``` + +### 多个MariaDB服务器 + +可以根据需要创建任意多的目录，因此，如果有额外的MariaDB服务器，只需添加另一个不同名称的属性文件到`etc/catalog`中（确保它以`.properties`结尾）。例如，如果将属性文件命名为`sales.properties`，openLooKeng将使用配置的连接器创建一个名为`sales`的目录。 + +## 查询MySQL + +MariaDB连接器为每个MariaDB*数据库*提供一个模式。可通过执行`SHOW SCHEMAS`来查看可用MariaDB数据库： + +```sql +-- 我们的catalog名称为maria +SHOW SCHEMAS FROM maria; +``` + +如果有一个名为`test`的MySQL数据库，可以通过执行`SHOW TABLES`查看数据库中的表： + +```sql +SHOW TABLES FROM maria.test; +``` + +可以使用以下方法之一查看`test`数据库中`user`表中的列的列表： + +```sql +DESCRIBE maria.test.user; +SHOW COLUMNS FROM maria.test.user; +``` + +最后，可以访问`test`数据库中`user`的表： + +```sql +SELECT * FROM maria.test.user; +``` + +如果对目录属性文件使用不同的名称，请使用该目录名称，而不要使用上述示例中的`maria`。 + +## MariaDB连接器限制 + +暂不支持以下SQL语句： + +[DELETE](../sql/delete.html)、[GRANT](../sql/grant.html)、[REVOKE](../sql/revoke.html)、[SHOW GRANTS](../sql/show-grants.html)、[SHOW ROLES](../sql/show-roles.html)、[SHOW ROLE GRANTS](../sql/show-role-grants.html) diff --git a/content/zh-cn/docs/docs/connector/mpp.md b/content/zh-cn/docs/docs/connector/mpp.md new file mode 100644 index 0000000000000000000000000000000000000000..c7633f7667f2e88337c9b886a5d9a5bdf62a916c --- /dev/null +++ b/content/zh-cn/docs/docs/connector/mpp.md @@ -0,0 +1,70 @@ + +# mpp连接器 + +本方案旨在通过GDS高速协议来提高openLookeng引擎读取GaussDB的性能。 + +目前对于GaussDB数据库的数据查询方式是通过JDBC Connector来实现的，和大部分关系型数据库连接类似。由于传统的MySQL等数据库存储的数据量小，因此通过JDBC方式来获取数据无可厚非。但是由于GaussDB是一种MPP类的分布式数据库，存储数据量大，且主要用于OLAP场景的分析，导致通过原生JDBC方式拉取数据的方式变得低效，因为它是一种单进单出的模式。 + +后来，社区也针对这种情况进行了基于JDBC的优化，例如增加了下推，在引擎端增加了多split并发，但是依然无法避免数据源端的单并发，性能瓶颈依然无法得到完全的解决，而且多并发还会带来多连接的情况，对数据库集群造成一定的压力。 + +本方案将会通过解决数据源端和引擎端并发问题来提高引擎查询的效率。 + +# mpp连接器设计思路 + +本方案将mpp类数据库的查询转换成对hive外表的查询，即将mpp类数据库的表数据快速导出到外部的分布式文件系统或分布式缓存，然后通过挂hive外表的方式通过hive connector来进行查询。本方案的主要权衡点在于导出数据的效率和mpp数据库通过单CN节点对外输出数据的效率的比较。随着查询数据量的递增，本方案的效率提高也会越来越高。 +更详细的内容见社区分享：https://mp.weixin.qq.com/s/Q-t592UerICHNXI63rhtPg + +## 配置 + +要配置mpp连接器，在`etc/catalog`中创建一个目录属性文件，例如`mpp.properties`，使用以下内容创建文件，并根据设置替换连接属性： + +``` properties +本方案本质上是将查询gaussdb数据库转换成查询hive，因此以下配置均基于此原理。 + +connector.name=mpp +# 配置用来做最后查询的hive仓库 +hive.metastore.uri=thrift://localhost:9083 + +etl-reuse=false #是否复用本次导数结果 + +#GDS baseinfo +#gds进程，基于postgres的fdw机制实现的一个快速导数进程，gaussdb官方插件 +gds-list=gsfs://localhost:port1|base_path #gds的ip和端口，以及该进程启动时候的basepath，多个gds进程可以通过逗号分隔 +aux-url=alluxio://localhost:19998 #alluxio的ip和端口 +base-aux=/gdsdata/ #alluxio中用来为gds导出数据服务的路径，可自定义 + +#hive info +# 用来进行创建外表等操作的hive仓库连接配置 +hive-user=username +hive-passwd=password +hive-db=xxx_db +hive-url=jdbc:hive2://localhost:10000/ + +# hive template +## 进行hive外表创建的相关SQL模板，一般无需更改 +hsql-drop=drop table if exists ${table_name} +hsql-create=CREATE EXTERNAL TABLE ${table_name} ( ${schema_info} ) COMMENT 'gds external table' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' LOCATION '${pipe_to_aux_base_path}' + +# gsDB connection info +# 要查询的GaussDB数据库相关连接信息 +gs-driver=org.postgresql.Driver +gs-url=jdbc:postgresql://localhost:25308/schema +gs-user=user +gs-passwd=password + +# gaussdb template +# 利用gds触发导数的相SQL模板，一般无需更改 +gsql-create=create foreign table ${gaussdb_name}.ext_${table_name} ( ${schema_info} ) SERVER gsmpp_server OPTIONS ( LOCATION '${gds_foreign_location}', FORMAT 'text', DELIMITER E',', NULL '', encoding 'UTF-8', noescaping 'true', EOL E'\\n', out_filename_prefix '${table_name}') WRITE ONLY; +gsql-insert=insert into ${gaussdb_name}.ext_${table_name} select ${schema_info} from ${gaussdb_name}.${table_name}; +gsql-drop=drop foreign table if exists ${gaussdb_name}.ext_${table_name}; + +``` + +## 使用 + +mpp连接器会将查询转换为对所配置的gaussdb的查询，因此如果你要查询opengauss.testdb.usertbl，如果想通过mpp +connector进行快速查询，则可以写成： + + select * from mpp.testdb.usertbl; + +如果您对mpp connector有更多的需求和见解，欢迎提issue和pr。 \ No newline at end of file diff --git a/content/zh-cn/docs/docs/connector/oracle.md b/content/zh-cn/docs/docs/connector/oracle.md index a89e04a7778c335b96507d95df8b72a5d769c249..4e219b0725626033bf9d31496b8366ce6b655224 100644 --- a/content/zh-cn/docs/docs/connector/oracle.md +++ b/content/zh-cn/docs/docs/connector/oracle.md @@ -178,7 +178,7 @@ openLooKeng支持选择以下Oracle数据库类型。下表显示了Oracle数据 > | Oracle数据库类型 | openLooKeng类型| 说明| > |:----------|:----------|:----------| -> | DECIMAL(p, s)| DECIMAL(p, s)| | +> | NUMBER(p, s)| DECIMAL(p, s)| | > | NUMBER(p) | DECIMAL(p, 0) | | > | FLOAT(p) | DOUBLE | | > | BINARY_FLOAT | REAL | | diff --git a/content/zh-cn/docs/docs/connector/redis.md b/content/zh-cn/docs/docs/connector/redis.md new file mode 100644 index 0000000000000000000000000000000000000000..1637c9200f3a642c77f2fa987fa2921d6fb4b14e --- /dev/null +++ b/content/zh-cn/docs/docs/connector/redis.md @@ -0,0 +1,171 @@ +Redis 连接器 +==================== +概述 +-------- +此连接器允许在openlookeng中，将redis中一个kv键值对映射成表的一行数据 + +**说明** + +*kv键值对只能在Reids中映射为string或hash类型。keys可以存储为一个zset,然后keys可以被分割成多个片* + +*支持 Redis 2.8.0 或更高版本* + + +配置 +------------- +要配置Redis连接器，创建具有以下内容的目录属性文件etc/catalog/redis.properties，并适当替换以下属性： +``` properties +connector.name=redis +redis.table-names=schema1.table1,schema1.table2 +redis.nodes=host1:port +``` +### 多个 Redis Servers +可以根据需要创建任意多的目录，因此，如果有额外的redis server，只需添加另一个不同的名称的属性文件到etc/catalog中（确保它以.properties结尾）。例如，如果将属性文件命名为sales.properties，openLooKeng将使用配置的连接器创建一个名为sales的目录。 + +配置属性 +------------------------ +配置属性包括： + +| 属性名称 | 说明 | +|:-----------------------------------------------------------|:----------------------------------------------------------------------------| +| `redis.table-names` | catalog 提供的所有表的列表 | +| `redis.default-schema` | 表的默认schema名 (默认`default`) | +| `redis.nodes` | Redis server的节点列表 | +| `redis.connect-timeout` | 连接Redis server的超时时间 (ms) (默认 2000) | +| `redis.scan-count` | 每轮scan获得的key的数量 (默认 100) | +| `redis.key-prefix-schema-table` | Redis keys 是否有 schema-name:table-name的前缀 (默认 false) | +| `redis.key-delimiter` | 如果`redis.key-prefix-schema-table`被启用，那么schema-name和table-name的分隔符为 (默认 `:`) | +| `redis.table-description-dir` | 存放表定义json文件的相对地址 (默认 `etc/redis/`) | +| `redis.hide-internal-columns` | 内部列是否在元数据中隐藏(默认 true) | +| `redis.database-index` | Redis database 的索引 (默认 0) | +| `redis.password` | Redis server 密码 (默认 null) | +| `redis.table-description-interval` | flush表定义文件的时间间隔（默认不会flush,意味着Plugin被加载后,表定义一直留存在内存中,不再主动读取json文件） | + +内部列 +---------------- + +| 列名 | 类型 | 说明 | +|:-------------------| :------ |:----------------------------------------------------| +| `_key` | VARCHAR | Redis key. | +| `_value` | VARCHAR | 与key相对应的值 | +| `_key_length` | BIGINT | key 的字节大小 | +| `_key_corrupt` | BOOLEAN | 如果解码器无法解码此行的key，则为true。当为 true 时，从键映射的数据列应被视为无效。 | +| `_value_corrupt` | BOOLEAN | 如果解码器无法解码此行的value，则为true。当为 true 时，从该值映射的数据列应被视为无效。 | + + + +表定义文件 +---------------------- +对于openLooKeng，每个kv键值对 必须被映射到列中，以便允许对数据查询。这很像kafka connector,所以你可以参考 kafka连接器教程 + + +表定义文件由一个表的JSON定义组成。文件名可以任意，但必须以.json结尾。 + +以nation.json为例 +``` json +{ + "tableName": "nation", + "schemaName": "tpch", + "key": { + "dataFormat": "raw", + "fields": [ + { + "name": "redis_key", + "type": "VARCHAR(64)", + "hidden": "true" + } + ] + }, + "value": { + "dataFormat": "json", + "fields": [ + { + "name": "nationkey", + "mapping": "nationkey", + "type": "BIGINT" + }, + { + "name": "name", + "mapping": "name", + "type": "VARCHAR(25)" + }, + { + "name": "regionkey", + "mapping": "regionkey", + "type": "BIGINT" + }, + { + "name": "comment", + "mapping": "comment", + "type": "VARCHAR(152)" + } + ] + } +} +``` +在redis,相应的有这样的数据 +```shell +127.0.0.1:6379> keys tpch:nation:* + 1) "tpch:nation:2" + 2) "tpch:nation:4" + 3) "tpch:nation:16" + 4) "tpch:nation:18" + 5) "tpch:nation:10" + 6) "tpch:nation:17" + 7) "tpch:nation:1" +``` +```shell +127.0.0.1:6379> get tpch:nation:1 +"{\"nationkey\":1,\"name\":\"ARGENTINA\",\"regionkey\":1,\"comment\":\"al foxes promise slyly according to the regular accounts. bold requests alon\"}" +``` +我们可以使用redis connector从redis中获取数据，（redis_key没有显示，这是因为我们设置了"hidden": "true" ） +```shell +lk> select * from redis.tpch.nation; + nationkey | name | regionkey | comment +-----------+----------------+-----------+-------------------------------------------------------------------------------------------------------------------- + 3 | CANADA | 1 | eas hang ironic, silent packages. slyly regular packages are furiously over the tithes. fluffily bold + 9 | INDONESIA | 2 | slyly express asymptotes. regular deposits haggle slyly. carefully ironic hockey players sleep blithely. carefull + 19 | ROMANIA | 3 | ular asymptotes are about the furious multipliers. express dependencies nag above the ironically ironic account + 2 | BRAZIL | 1 | y alongside of the pending deposits. carefully special packages are about the ironic forges. slyly special +``` +**说明** + +*如果属性 `redis.key-prefix-schema-table` 是false (默认是false),那么当前所有key的都会被视作的nation表的key,不会发生匹配过滤* + +有关各种可用解码器的描述，请参考kafka连接器教程 + +除了kafka支持的类型，Redis connector对value字段支持hash类型 +``` json + { + "tableName": ..., + "schemaName": ..., + "value": { + "dataFormat": "hash", + "fields": [ + ... + ] + } + } +``` + +Redis connector 支持``zset`` 作为``key``在Redis中存储类型。 +当且仅当``zset`` 作为key的存储格式时，split切片功能才能被真正支持，因为我们可以使用`zrange zsetkey split.start split.end`来得到一个切片的keys +``` json + { + "tableName": ..., + "schemaName": ..., + "key": { + "dataFormat": "zset", + "name": "zsetkey", //zadd zsetkey score member + "fields": [ + ... + ] + } + } +``` + +Redis 连接器的局限性 +--------------------------- +只支持读操作，不支持写操作. + + diff --git a/content/zh-cn/docs/docs/connector/singledata.md b/content/zh-cn/docs/docs/connector/singledata.md new file mode 100644 index 0000000000000000000000000000000000000000..4c158afcc6fec19000b283c3d92eca7b1edbe1a6 --- /dev/null +++ b/content/zh-cn/docs/docs/connector/singledata.md @@ -0,0 +1,96 @@ +# singleData连接器 + +singleData连接器为openGauss补充OLAP能力，增强数据分析竞争力。 + +singleData分为ShardingSphere和tidRange两种模式。 + +## ShardingSphere模式 + +### 概述 + +Apache ShardingSphere是一款分布式数据库生态系统，它可以将任意数据库转换为分布式数据库，singleData连接器可以使用ShardingSphere对openGauss的数据进行分片，以提升数据分析效率。 + +要了解更多shardingsphere信息请见：[ShardingSphere官方网站](https://shardingsphere.apache.org/) + +### 配置 + +要配置ShardingSphere模式的singleData连接器，在`etc/catalog`中创建一个目录属性文件，例如`shardingsphere.properties`，使用以下内容创建文件，并替换相应的属性。 + +- 基本配置 +``` properties +connector.name=singledata +singledata.mode=SHARDING_SPHERE +shardingsphere.database-name=sharding_db +shardingsphere.type=zookeeper +shardingsphere.namespace=governance_ds +shardingsphere.server-list=localhost:2181 +``` + +- 属性说明 + +| 属性名称 | 属性说明 | 是否必要 | +|------------------------------|------------------------------------------|------| +| shardingsphere.database-name | 连接的shardingsphere database的名称 | 是 | +| shardingsphere.type | 注册中心持久化仓库类型，支持zookeeper和etcd，推荐zookeeper | 是 | +| shardingsphere.namespace | 注册中心命名空间 | 是 | +| shardingsphere.server-lists | 注册中心连接地址 | 是 | + +- 仓库类型为zookeeper的可选配置 + +| 属性名称 | 说明 | 默认值 | +|---------------------------------------------------------|--------------|-----| +| shardingsphere.zookeeper.retry-interval-milliseconds | 连接失败后重试间隔毫秒数 | 500 | +| shardingsphere.zookeeper.max-retries | 连接最大重试数 | 3 | +| shardingsphere.zookeeper.time-to-live-seconds | 临时节点存活秒数 | 60 | +| shardingsphere.zookeeper.operation-timeout-milliseconds | 操作超时毫秒数 | 500 | +| shardingsphere.zookeeper.digest | 权限令牌 | "" | + +- 仓库类型为etcd的可选配置 + +| 属性名称 | 说明 | 默认值 | +|------------------------------------------|----------|-----| +| shardingsphere.etcd.time-to-live-seconds | 临时节点存活秒数 | 30 | +| shardingsphere.etcd.connection-timeout | 连接超时秒数 | 3 | + +**说明** + +- 目前singleData连接器只支持5.2.0版本的ShardingSphere,其他版本暂时不支持 + +## TidRange模式 + +### 概述 + +openGauss的数据是按行存储在HEAP PAGE中，每一行数据都会有对应的ctid（即行号）。openGauss的tidrangescan插件可以使用ctid的范围来进行查询，singleData连接器的tidRange模式利用这个插件来实现数据的并行分析。 + +tidrangescan插件获取地址：[Plugin](https://gitee.com/opengauss/Plugin) + +### 连接配置 + +要配置tidRange模式的singleData连接器，在`etc/catalog`中创建一个目录属性文件，例如`tidrange.properties`。使用以下内容创建文件，并根据设置替换连接属性 +```properties +connection.name=singledata +connection.mode=TID_RANGE +connection-url=jdbc:opengauss://master-host:port/database;jdbc:opengauss://slave-host:port/database +connection-user=user +connection-password=password +``` +- connection-url可以配置多个主备节点的jdbc连接地址，地址间以`;`作为分隔符，在进行查询时，每个分片会随机选择一个连接地址进行连接 +- 其他连接配置请参考openGauss连接器 + +### 分片配置 + +| 属性名称 | 属性说明 | 是否必须 | 默认值 | +|-----------------------------|---------------------------------------------------------------|------|------| +| tidrange.max-split-count | 最大分片数量，即最大jdbc连接数，这个数值应该不大于openGauss的max_connections配置 | 否 | 100 | +| tidrange.page-size | openGauss的page大小，请确保这个配置和openGauss的block_size一致，否则可能会导致查询结果错误 | 否 | 8kB | +| tidrange.default-split-size | 默认的每个分片的大小。当数据量较小时，singleData连接器会按此配置进行分片，配置的范围为1MB-1GB | 否 | 32MB | + +**说明** +- 本特性需配合openGauss的tidrangescan插件使用，没有tidrangescan插件singledata连接器也可以正常完成查询功能，但会导致性能大幅下降 +- 当查询的表中存在索引时，将不会启用tidrange功能 + +## 限制说明 + +- singleData连接器目前只提供对openGauss的查询功能，暂时不支持INSERT/UPDATE/DELETE等会修改数据或者数据结构的语句，查询功能请参考openGauss的连接器 +- openLookeng的Decimal精度最高支持为38，当openGauss的decimal，numeric类型的精度超过38时则无法支持 +- openGauss版本支持3.0.0及以上 \ No newline at end of file diff --git a/content/zh-cn/docs/docs/develop/_index.md b/content/zh-cn/docs/docs/develop/_index.md index 25240ded0a2bf059d268fe683002d49b88394786..6c4bce924131a7d533961fedd00a580faf8784bb 100644 --- a/content/zh-cn/docs/docs/develop/_index.md +++ b/content/zh-cn/docs/docs/develop/_index.md @@ -1,4 +1,4 @@ # 开发者指南 -openLooKeng基于Trino(之前称为PrestoSQL)，是Trino开源项目的一个分支。与Trino开源项目相比，openLooKeng提供了额外的优化和增强功能，可以在任何位置对任何数据（包括远程数据源）进行现场分析。本指南适用于openLooKeng参与者和插件开发人员。 +openLooKeng基于Trino 316版本(之前称为PrestoSQL)，是Trino开源项目的一个分支。与Trino开源项目相比，openLooKeng提供了额外的优化和增强功能，可以在任何位置对任何数据（包括远程数据源）进行现场分析。本指南适用于openLooKeng参与者和插件开发人员。 diff --git a/content/zh-cn/docs/docs/develop/functions.md b/content/zh-cn/docs/docs/develop/functions.md index b50e28346cb7e42a082c0ae1d7563c338cd88938..f18b3e550dd76f5c5479ded9cff6c9b7cee69061 100644 --- a/content/zh-cn/docs/docs/develop/functions.md +++ b/content/zh-cn/docs/docs/develop/functions.md @@ -28,7 +28,7 @@ public class ExampleFunctionsPlugin ``getFunctions()`` 方法包含我们将在本教程的下面部分实现的所有函数类。 -有关代码库的完整示例，请参见 ``presto-m`` 模块中的机器学习函数或 ``presto-teradata-functions`` 模块中与 Teradata 兼容的函数，这两个模块都位于在 openLooKeng 源代码的根目录中。 +有关代码库的完整示例，请参见 ``presto-ml`` 模块中的机器学习函数或 ``presto-teradata-functions`` 模块中与 Teradata 兼容的函数，这两个模块都位于在 openLooKeng 源代码的根目录中。 @@ -52,7 +52,7 @@ public class ExampleNullFunction ``` 函数 ``is_null`` 接受单个 ``VARCHAR`` 参数并返回一个 ``BOOLEAN`` 值，指示该参数是否为 ``NULL``。 -请注意，该函数的参数类型是 ``slice``。 +请注意，该函数的参数类型是 ``Slice``。 ``VARCHAR`` 使用 ``Slice``，后者本质上是一个对 ``byte[]`` 进行包装的包装器，而不是用于其本地容器类型的 ``String``。 diff --git a/content/zh-cn/docs/docs/faq/faq.md b/content/zh-cn/docs/docs/faq/faq.md index 2bf5da8656847a42a3f0044293062918e83a2075..6d6b7597eca44d77843512f645985096287dad9c 100644 --- a/content/zh-cn/docs/docs/faq/faq.md +++ b/content/zh-cn/docs/docs/faq/faq.md @@ -35,6 +35,10 @@ > openLooKeng微信群：请添加openLooKeng小助手微信号：openLooKengoss，小助手会拉您入群 > openLooKeng B站： https://space.bilibili.com/627629884 +8. openLooKeng基于Trino哪个版本进行开发？ + + > 基于Trino 316版本进行开发。 + ## 功能 1. openLooKeng目前支持哪些连接器? @@ -157,7 +161,7 @@ 1. openLooKeng的SQL是否是大小写敏感的？如果数据源是大小写敏感的，openLooKeng如何处理？ - > openLooKeng对大小写不敏感，对于sql中的大写，openLooKeng都会转换成小写处理。如果数据源本身是大小写敏感的，那么就可能查询失败。比如ES是大小写敏感的，所以如果ES中的列名使用大写，那么openLooKeng查询该列就会出错。 + > openLooKeng对大小写不敏感，对于sql中的大写，openLooKeng都会转换成小写处理。如果数据源本身是大小写敏感的，那么就可能查询失败。比如ES是大小写敏感的，所以如果ES中的列名使用大写，那么openLooKeng查询该列就会出错。其中，jdbc数据源匹配数据库和集合名称时，通过在对应数据源配置文件中增加配置项“case-insensitive-name-matching”控制是否对大小写敏感，详情参考: https://docs.openlookeng.io/zh/docs/docs/admin/properties.html。 2. 对接hive使用update失败，报错：cannot change stats state for a transactional table，是什么原因？ diff --git a/content/zh-cn/docs/docs/images/snapshot_statistics.png b/content/zh-cn/docs/docs/images/snapshot_statistics.png deleted file mode 100644 index f41da3e3e48dde82fe1191f5551f5f07e937f2d4..0000000000000000000000000000000000000000 Binary files a/content/zh-cn/docs/docs/images/snapshot_statistics.png and /dev/null differ diff --git a/content/zh-cn/docs/docs/images/snapshot_statistics_cn.png b/content/zh-cn/docs/docs/images/snapshot_statistics_cn.png new file mode 100644 index 0000000000000000000000000000000000000000..c90e7172a95cef40bf73e154afc6b774fb38b04b Binary files /dev/null and b/content/zh-cn/docs/docs/images/snapshot_statistics_cn.png differ diff --git a/content/zh-cn/docs/docs/images/task_snapshot_overview.png b/content/zh-cn/docs/docs/images/task_snapshot_overview.png new file mode 100644 index 0000000000000000000000000000000000000000..84286f9e0e31384665111932188481301c731122 Binary files /dev/null and b/content/zh-cn/docs/docs/images/task_snapshot_overview.png differ diff --git a/content/zh-cn/docs/docs/indexer/bloom.md b/content/zh-cn/docs/docs/indexer/bloom.md index 029214f0670d69f2549e0a4e02bfb9192bb5109e..963e0772ca9e215ad7069d70f00d141b3540fab1 100644 --- a/content/zh-cn/docs/docs/indexer/bloom.md +++ b/content/zh-cn/docs/docs/indexer/bloom.md @@ -14,7 +14,7 @@ BloomIndex仅支持相等表达式，例如`name='monkey'`。 **注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。** 1. BloomIndex用于调度时的分片过滤，被coordinator节点使用。 -2. BloomIndex也用于worker节点上，用于在读取ORC文件是过滤stripes。 +2. BloomIndex也用于worker节点上，用于在读取ORC文件时过滤stripes。 ## 选择适用的列 @@ -41,7 +41,7 @@ BloomIndex仅支持相等表达式，例如`name='monkey'`。 > > 改变布隆过滤器的FPP (false positive probability)。 > 更小的FPP会提高索引的过滤能力，但是会增加索引的体积。在大多数情况下默认值就足以够用。 -> 如果创建的索引太大，可以考虑增加这个值（例如，至0.05)。 +> 如果创建的索引太大，可以考虑增加这个值（例如：0.05)。 ### `bloom.mmapEnabled` @@ -86,9 +86,9 @@ select name from hive.hindex.users where id=123 当OLK引擎需要读取数据时，它会计划Splits（分片）。 每个分片负责读取一部分数据。 例如，当读取具有ORC数据格式的Hive表时， -每个分割将负责读取指定偏移量之间的ORC文件的一部分。 +每个Splits（分片）将负责读取指定偏移量之间的ORC文件的一部分。 -例如，`/hive/database.db/animals/000.orc`，从偏移量`0`开始，从偏移量`2000`开始。 +例如，`/hive/database.db/animals/000.orc`，起始偏移量0，结束偏移量2000。 为简单起见，我们可以假定每个分片对应于一个Stripe。 diff --git a/content/zh-cn/docs/docs/indexer/btree.md b/content/zh-cn/docs/docs/indexer/btree.md index 0c1d4ed8027b1373b7d6a7c1d194a01439fc6deb..7af3e08b3dec4c9726fe8eb4fd623df8789481d9 100644 --- a/content/zh-cn/docs/docs/indexer/btree.md +++ b/content/zh-cn/docs/docs/indexer/btree.md @@ -29,7 +29,7 @@ BTreeIndex用于调度时的分片(Split)过滤，被coordinator节点使用。 列中的数据已被过滤，`phone`列的基数较高。 在BTreeIndex和BloomIndex索引之间选择时，需要考虑： -- BloomIndex只支持`=`，而BTreeIndex提供范围咨询 +- BloomIndex只支持`=`，而BTreeIndex提供范围查询 - BloomIndex是不确定的，而BTreeIndex是确定的。因此BTreeIndex通常有更好的过滤性能 - BTreeIndex比BloomIndex索引更大 diff --git a/content/zh-cn/docs/docs/indexer/minmax.md b/content/zh-cn/docs/docs/indexer/minmax.md index b64a9b0fb4ada4d3fc33b76ed2d4ee6851eb0b1b..c0ab5ab58f5f3ec8160ea398a65ed51afb0fb65e 100644 --- a/content/zh-cn/docs/docs/indexer/minmax.md +++ b/content/zh-cn/docs/docs/indexer/minmax.md @@ -14,7 +14,7 @@ MinMaxIndex用于调度时的分片过滤，被coordinator节点使用。 在对数据进行排序的列上具有过滤predicate的query可以从MinMaxIndex中得到好的效果。 -例如，如果一下数据是根据`age`来排序的，那么一个像`SELECT name from users WHERE age> 25` +例如，如果以下数据是根据`age`来排序的，那么一个像`SELECT name from users WHERE age> 25` 之类的query则可以因有效地在`age`上利用MinMaxIndex，而从中得到好的效果。 ## 支持的运算符 diff --git a/content/zh-cn/docs/docs/indexer/overview.md b/content/zh-cn/docs/docs/indexer/overview.md index f9e9e3666161ed003e57ea8faee40d91d70aee52..a59cd0a5f2829fadd1be0966fa118607f636bc34 100644 --- a/content/zh-cn/docs/docs/indexer/overview.md +++ b/content/zh-cn/docs/docs/indexer/overview.md @@ -17,7 +17,7 @@ **注意：当前，启发式索引仅支持ORC存储格式的Hive数据源。** 1. BloomIndex，MinMaxIndex和BtreeIndex可以在Coordinator上使用，以在调度期间过滤Splits -2. 在读取ORC文件时，可以在worker上使用MinMaxIndex或者BloomIndex过滤stripe +2. 在读取ORC文件时，可以在worker上使用MinMaxIndex或者BloomIndex过滤Stripes 3. 在读取ORC文件时，可以在worker上使用BitmapIndex过滤数据行 ### 1.查询过程中过滤预定分片 @@ -36,7 +36,7 @@ *支持的索引：BloomIndex, MinMaxIndex* -与分片过滤类似，当使用Hive Connector读取ORC文件时，Stripe可以被提前过滤来减少读取的数据量，从而提升查询性能。 +与分片过滤类似，当使用Hive Connector读取ORC文件时，Stripes可以被提前过滤来减少读取的数据量，从而提升查询性能。 ### 3.读取ORC文件时筛选行 @@ -89,7 +89,7 @@ Hetu Metastore 是一个共享元数据管理实用程序，多个 openLooKeng 如Hetu Metastore 设置中提到的，必须启用全局缓存，这需要配置Hetu Statestore。 更多配置方法请查看[Hetu Statestore](../admin/state-store.html). -在`etc/config.propertes`中配置: +在`etc/config.properties`中配置: ```properties hetu.embedded-state-store.enabled=true diff --git a/content/zh-cn/docs/docs/installation/deployment.md b/content/zh-cn/docs/docs/installation/deployment.md index cb6e039741ef154b46a995df570cbce679eeec8a..9c572f1507b31f9d33029db456dd03d4ae6c6671 100644 --- a/content/zh-cn/docs/docs/installation/deployment.md +++ b/content/zh-cn/docs/docs/installation/deployment.md @@ -50,10 +50,6 @@ plugin.dir=/opt/openlookeng/hetu-server-1.1.0/plugin ### JVM配置 -JVM配置文件etc/jvm.config包含用于启动Java虚拟机的命令行选项列表。文件的格式是一个选项列表，每行一个选项。这些选项不由shell解释，因此包含空格或其他特殊字符的选项不应被引用。 - -### JVM配置 - JVM配置文件`etc/jvm.config`包含用于启动Java虚拟机的命令行选项列表。文件的格式是一个选项列表，每行一个选项。这些选项不由shell解释，因此包含空格或其他特殊字符的选项不应被引用。 以下为创建`etc/jvm.config`提供了一个良好的起点： @@ -161,7 +157,14 @@ connector.name=jmx ## 运行openLooKeng -安装目录中在`bin/launcher`中包含启动器脚本。openLooKeng可以通过运行如下命令以守护进程的方式启动： +安装目录中在`bin/launcher`中包含启动器脚本。在运行前需进行一项准备工作： + +编辑安装目录下的`bin/launcher.py`文件，将其440行与441行处的`options.launcher_log_file`与`options.server_log_file`分别修改为`node_properties.get('node.launcher-log-file')`与`node_properties.get('node.server-log-file')`。此处修改后的代码应为： +``` +o.launcher_log = realpath(node_properties.get('node.launcher-log-file' ) or pathjoin(o.data_dir, 'var/log/launcher.log')) +o.server_log = realpath(node_properties.get('node.server-log-file' ) or pathjoin(o.data_dir, 'var/log/server.log')) +``` +这使得集群启动时日志文件会优先保存在`etc/node.properties`配置的路径中。 openLooKeng可以通过运行如下命令以守护进程的方式启动： ``` shell bin/launcher start diff --git a/content/zh-cn/docs/docs/installation/odbc.md b/content/zh-cn/docs/docs/installation/odbc.md index a11a9c9406fcb3a7dc15462d1ad00ff3a1534507..cd9e7604782e1784cc5312421800c6a69749b224 100644 --- a/content/zh-cn/docs/docs/installation/odbc.md +++ b/content/zh-cn/docs/docs/installation/odbc.md @@ -22,9 +22,9 @@ ODBC驱动为应用提供了连接到数据库的能力，本产品为openLooKen **本产品支持以下版本:** -- Winodows 10 64bit +- Windows 10 64bit -- Winodows Server 2016 64bit +- Windows Server 2016 64bit > 其他版本的Windows系统未经过严格测试，用户可以自己尝试，本产品不做任何质量保证 diff --git a/content/zh-cn/docs/docs/language/types.md b/content/zh-cn/docs/docs/language/types.md index 75bfd31246786467307ca104d88216e620dbbd30..40321325693e6505ea755d59ff6743965f36ff98 100644 --- a/content/zh-cn/docs/docs/language/types.md +++ b/content/zh-cn/docs/docs/language/types.md @@ -2,7 +2,7 @@ 数据类型 ========== -openLooKeng有一组内置的数据类型，如下所述。可能通过插件提供更多的类型。 +openLooKeng有一组内置的数据类型，如下所述。可以通过插件提供更多的类型。 **注意** diff --git a/content/zh-cn/docs/docs/optimizer/table-pushdown.md b/content/zh-cn/docs/docs/optimizer/table-pushdown.md index 060029229198509de5cd23219e9ce86c92657a6e..fa4fa4063fc69f69f8bff6e50f1bd28ca4340686 100644 --- a/content/zh-cn/docs/docs/optimizer/table-pushdown.md +++ b/content/zh-cn/docs/docs/optimizer/table-pushdown.md @@ -77,8 +77,6 @@ AND lineitem.quantity < s_avg; ``` -The feature is disabled by default. The user can enable it by executing the following command to set the session parameter- - 该功能默认是关闭状态。用户可以通过以下命令设置session参数来启用该功能- ```sql diff --git a/content/zh-cn/docs/docs/preagg/statements.md b/content/zh-cn/docs/docs/preagg/statements.md index 3813abbdde253dd79f90d508ffc46d7822e465bd..4bca713fa0c883810edab89ef2d4092819088e8c 100644 --- a/content/zh-cn/docs/docs/preagg/statements.md +++ b/content/zh-cn/docs/docs/preagg/statements.md @@ -135,7 +135,7 @@ SHOW CUBES [ FOR table_name ]; ``` ### 描述 -`SHOW CUBES`列出所有立方体。添加可选的`table_name`仅列出该表的Cubes。 +`SHOW CUBES`列出所有Cubes。添加可选的`table_name`仅列出该表的Cubes。 ### 例子 diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-0.1.0.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-0.1.0.md index 1d3f8859584d3e33e2cf4e906a5863a4cd751d11..7db34f522e178899940fb1a767040b4729bdc024 100644 --- a/content/zh-cn/docs/docs/releasenotes/releasenotes-0.1.0.md +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-0.1.0.md @@ -14,7 +14,7 @@ | Compaction for ORC |支持hive ORC事务表的Compaction，通过增加每个读取器的数据获取来减少读取的文件数量，从而有助于提高查询性能，并提高并发度| | CarbonData Connector with IUD support |支持CarbonData表的插入、更新、删除操作| | Insert Overwrite |支持插入覆盖语法。这是裁剪和加载到现有表的简单方法| -| ODBC connector| ODBC PowerBI、Tableau、永洪桌面等第三方BI工具连接河图的驱动和网关| +| ODBC connector| ODBC PowerBI、Tableau、永洪桌面等第三方BI工具连接openLooKeng的驱动和网关| |Dynamic Hive UDF|将自定义的Hive UDF动态加载到openLooKeng | | HBase Connector| HBase连接器支持查询HBase数据源的数据| | Enhance Create table statement |允许在openLooKeng中执行CREATE TABLE AS命令时指定所管理的hive表的外部位置;允许在openLooKeng中创建Hive事务表| diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.0.1.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.0.1.md index 2394646747131a9e566863af1b03d6da716b3643..389e50b391f8444057e85b1ee8ed01b5a2da5ff2 100644 --- a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.0.1.md +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.0.1.md @@ -7,7 +7,7 @@ | 类别 | 特性 | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | 索引 | 将Index cli合并到hetu-cli中，并为index的命令引入基本的权限控制 | 283, 298 | -| 安全 | 处理CVEs问题 | 289,287,295,288,285,292，297,249,302 | +| 安全 | 处理CVEs问题 | 289,287,295,288,285,292,297,249,302 | ## 获取文档 diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.4.1.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.4.1.md index 9d283ff6a2a6453479d753d07bc4738130114d5f..9ae35132f525341e23a6411d0a3db46e71646ed5 100644 --- a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.4.1.md +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.4.1.md @@ -2,7 +2,7 @@ ## 关键特性 -本次发布主要新增了OmniData Connector的介紹和ARM架构下支持JDK8。 +本次发布主要新增了OmniData Connector的介绍和ARM架构下支持JDK8。 | 类别 | 特性 | PR #s | | ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.6.0.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.6.0.md index 31d96078ce3b4b72e513479bf45cd8cd697cfbec..53f0ee16f22ab962c0d53c683bccec881f2088ec 100644 --- a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.6.0.md +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.6.0.md @@ -14,7 +14,7 @@ | 分类 | 描述 | Gitee问题 | | ------------- | ------------------------------------------------------------ | --------------------------------------------------------- | | Task Recovery | 启用快照时，执行带事务的CTAS语句时，SQL语句执行报错 | [I502KF](https://e.gitee.com/open_lookeng/issues/list?issue=I502KF) | -| | 启用快照并将exchange.is-timeout-failure-detection-enable关闭时，概率性出现错误 | [I4Y3TQ](https://e.gitee.com/open_lookeng/issues/list?issue=I4Y3TQ) | +| | 启用快照并将exchange.is-timeout-failure-detection-enabled关闭时，概率性出现错误 | [I4Y3TQ](https://e.gitee.com/open_lookeng/issues/list?issue=I4Y3TQ) | | Star Tree | 在内存连接器中，启用star tree功能后，查询时偶尔出现数据不一致 | [I4QQUB](https://e.gitee.com/open_lookeng/issues/list?issue=I4QQUB) | | | 当同时对10个不同的cube执行reload cube命令时，部分cube无法重新加载 | [I4VSVJ](https://e.gitee.com/open_lookeng/issues/list?issue=I4VSVJ) | diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.7.0.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.7.0.md new file mode 100644 index 0000000000000000000000000000000000000000..b0305fb716f9320e257862dbc2df639cd8e7ead7 --- /dev/null +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.7.0.md @@ -0,0 +1,19 @@ +# Release 1.7.0 + +## 关键特性 + +| 分类 | 描述 | +| --------------------- | ------------------------------------------------------------ | +| 算子处理扩展 | 增加通过用户自定义worker结点物理执行计划的生成，用户可以实现自己的算子pipeline代替原生实现，加速算子处理 | +| Task Recovery | 1. 支持不开启快照特性时，实现失败任务重试
2. 支持在CLI端调试模式下显示快照相关统计信息：包括正在捕获的快照数量、已经捕获的快照数量、恢复次数（包括查询重启）、捕获的所有快照大小和捕获快照所需时间、用于恢复的快照大小和恢复时间
3. 资源管理：在任务消耗大量内存导致内存不足时自动暂停/恢复任务，用户也可以选择主动暂停和恢复任务（限于CTAS和INSERT INTO SELECT场景）
4. 增加可配置的失败重试策略：① 超时重试策略 ② 最大次数重试策略
5. 新增Gossip失败检测机制：控制节点通过定期接收心跳来监控工作节点是否存活，Gossip机制使得所有工作节点能够相互监测，确保所有节点所知的信息都是一致的，可以防止在集群中出现短暂网络故障时导致过早的查询失败
| +| openLooKeng WEB UI增强 | 1. openLooKeng审计日志功能增强
2. openLooKeng SQL历史记录持久化
3. SQL Editor支持自动联想
4. 支持常用SQL语句收藏
5. 数据源加载异常时，前端可通过异常标记显示
6. UI支持显示数据源信息| +| openLooKeng集群管理平台 | 1. 提供图形化的指标监控及定制，及时获取系统的关键信息
2. 提供服务属性的配置管理，满足实际业务的性能需求
3. 提供集群、服务、实例的操作功能，满足一键启停等操作需求
4. 提供openLooKeng在线部署功能，支持在线升级
5. 提供节点伸缩服务
6. 集成openLooKeng现有的Web UI
7. openLooKeng Manager工具支持用户登录/退出功能 | +## 已知问题 + +| 分类 | 描述 | Gitee问题 | +| ------------- | ------------------------------------------------------------ | --------------------------------------------------------- | +| Task Recovery | 启用快照时，执行带事务的CTAS语句时，中断一个worker服务，SQL语句执行报错 | [I5EAM3](https://e.gitee.com/open_lookeng/issues/list?issue=I5EAM3) | + +## 获取文档 + +请参考： [https://gitee.com/openlookeng/hetu-core/tree/1.7.0/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.7.0/hetu-docs/zh) \ No newline at end of file diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.8.0.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.8.0.md new file mode 100644 index 0000000000000000000000000000000000000000..dc2c690777f2571f949d4f306e07eace8bf95981 --- /dev/null +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.8.0.md @@ -0,0 +1,12 @@ +# Release 1.8.0 + +## 关键特性 + +| 分类 | 描述 | +|--------|---------------------------------------------------------------------------------------------------------------------------| +| 任务级恢复 | 1. 通过使用物化的exchange信息仅重试失败的任务来实现查询恢复，支持Select场景的任务重试和恢复
2. 用于任务恢复的快照文件支持存放在HDFS上
3. 在实现物化exchange数据时，支持直接序列化和压缩。 | +| 查询资源管理 | 实现查询级别的资源监控，当任务超出资源组配置时，可支持任务的自动暂停/恢复、溢出可撤销内存、限制调度和终止查询。 | +| 连接器扩展 | singleData连接器为openGauss补充OLAP能力，增强数据分析竞争力。支持对接ShardingSphere和tidRange两种模式。 | +## 获取文档 + +请参考： [https://gitee.com/openlookeng/hetu-core/tree/1.8.0/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.8.0/hetu-docs/zh) \ No newline at end of file diff --git a/content/zh-cn/docs/docs/releasenotes/releasenotes-1.9.0.md b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.9.0.md new file mode 100644 index 0000000000000000000000000000000000000000..a767d8dd18a46579041f119f4a8665b3d0d4fd6a --- /dev/null +++ b/content/zh-cn/docs/docs/releasenotes/releasenotes-1.9.0.md @@ -0,0 +1,13 @@ +# Release 1.9.0 + +## 关键特性 + +| 分类 | 描述 | +|--------|---------------------------------------------------------------------------------------------------------------------------| +| 任务级恢复 | 支持Create Table As Select场景的任务重试和恢复 | +| 连接器扩展 | 1. 新增Iceberg连接器，支持数据增删改查，分区演化，历史版本查询，事务，小文件合并
2. 增强Elastic Search连接器，支持算子下推，当前支持下推AND、OR、算术、比较运算符、BETWEEN、IN、GROUP BY、COUNT和SUM | | +| 性能增强 | 1. CTE优化，包括执行管道的重用，CTE结果的物化和重用
2. 执行计划优化，包括使用准确的分区，自适应聚合，join场景的选择率估计和使用能更好过滤的执行策略 | +## 获取文档 + +请参考： [https://gitee.com/openlookeng/hetu-core/tree/1.9.0/hetu-docs/zh](https://gitee.com/openlookeng/hetu-core/tree/1.9.0/hetu-docs/zh) + diff --git a/content/zh-cn/docs/docs/security/built-in-system-access-control.md b/content/zh-cn/docs/docs/security/built-in-system-access-control.md index c23f02836a18f921e506bb81826b7ecae479a852..5832acf17bcc46d81c2c24f89f189a52bd98d7d3 100644 --- a/content/zh-cn/docs/docs/security/built-in-system-access-control.md +++ b/content/zh-cn/docs/docs/security/built-in-system-access-control.md @@ -28,7 +28,7 @@ access-control.name=read-only 基于文件的系统访问控制 -------------------------------- -此插件允许您在文件中设置访问控制规则。要使用该插件，需要添加一个`etc/access-control.properties`文件，其中必需包含两个属性：`Access-control.name`属性，必须等于`file`；`security.config-file`属性，必须等于配置文件所在的位置。例如，配置文件`rules.json`位于`etc`目录，则添加`etc/access-control.properties`文件，内容如下： +此插件允许您在文件中设置访问控制规则。要使用该插件，需要添加一个`etc/access-control.properties`文件，其中必需包含两个属性：`access-control.name`属性，必须等于`file`；`security.config-file`属性，必须等于配置文件所在的位置。例如，配置文件`rules.json`位于`etc`目录，则添加`etc/access-control.properties`文件，内容如下： ``` properties access-control.name=file @@ -58,7 +58,7 @@ security.refresh-period=1s - `user`（可选）：用于匹配用户名的正则表达式。默认为`.*`。 - `catalog`（可选）:用于匹配目录名的正则表达式。默认为`.*`。 -- `allow`（必选）: 布尔类型参数，表示用户是否有访问目录的权限 +- `allow`（必选）: 字符串参数，表示用户是否有访问目录的权限。这个值可以是all、read-only或none，默认为none。将此值设置为read-only，其行为与只读的系统访问控制插件相同。 **注意** @@ -66,7 +66,7 @@ security.refresh-period=1s *默认情况下，所有用户都可以访问`system`目录。您可以通过添加规则来改变此行为。* -例如，如果希望仅允许`admin`用户访问`mysql`和`system`目录，允许所有用户访问`hive`目录，拒绝其他用户访问，则可以定义以下规则： +例如，如果希望仅允许`admin`用户访问`mysql`和`system`目录，允许所有用户访问`hive`目录，允许`alice`用户只读地访问`postgresql`，拒绝其他用户访问，则可以定义以下规则： ``` json { @@ -74,15 +74,20 @@ security.refresh-period=1s { "user": "admin", "catalog": "(mysql|system)", - "allow": true + "allow": all }, { "catalog": "hive", - "allow": true + "allow": all + }, + { + "user": "alice", + "catalog": "postgresql", + "allow": "read-only" }, { "catalog": "system", - "allow": false + "allow": none } ] } diff --git a/content/zh-cn/docs/docs/security/hazelcast.md b/content/zh-cn/docs/docs/security/hazelcast.md index 7d4da7d9f4e8c97b8e40c79080fb768dd4376538..509afc58724172b0488640c69c792a8d22e6caec 100644 --- a/content/zh-cn/docs/docs/security/hazelcast.md +++ b/content/zh-cn/docs/docs/security/hazelcast.md @@ -47,7 +47,7 @@ state-store启用后，在state-store的配置文件state-store.properties中增 注意： -所有节点的配置文件相同，包括机机用户的principal，ketTab。 +所有节点的配置文件相同，包括机机用户的principal，keyTab。 ## Hazelcast SSL/TLS @@ -55,7 +55,7 @@ state-store启用后，在state-store的配置文件state-store.properties中增 误的节点将无法与其他节点进行通信。 为了Hazelcast通信启用SSL/TLS功能，需要执行以下步骤： -1. 生成Java密钥库文件。可以使用每台主机的fully-qualified主机名为每个节点创建唯一的证书，创建时包含所有主机的所有公钥的密钥库，并为客户端 +1. 生成Java密钥库文件。可以使用每台主机的全限定主机名为每个节点创建唯一的证书，创建时包含所有主机的所有公钥的密钥库，并为客户端 指定密钥库。在大多数情况下，使用通配符进行证书的创建更加方便，如下所示： > ``` diff --git a/content/zh-cn/docs/docs/security/ranger-access-control.md b/content/zh-cn/docs/docs/security/ranger-access-control.md index 1b5a53971a7347665bc032073345bf14df4f3e08..95f109d6ee0a35d4d6bc974c0dbeeb3675923847 100644 --- a/content/zh-cn/docs/docs/security/ranger-access-control.md +++ b/content/zh-cn/docs/docs/security/ranger-access-control.md @@ -7,7 +7,7 @@ Ranger 访问控制 Apache Ranger 为 Hadoop 集群提供了一种全面的安全框架，以跨组件的、一致性的方式进行定义、授权、管理安全策略。Ranger 详细介绍和用户指导可以参考[Apache Ranger Wiki](https://cwiki.apache.org/confluence/display/RANGER/Index )。 -[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) 是为 openLooKeng 开发的 Ranger 插件，用于全面的数据安全监控和权限管理。 +[openlookeng-ranger-plugin](https://gitee.com/openlookeng/openlookeng-ranger-plugin) 基于Ranger 2.1.0版本进行开发，是为 openLooKeng 开发的 Ranger 插件，用于全面的数据安全监控和权限管理。 编译过程 ------------------------- diff --git a/content/zh-cn/docs/docs/security/server.md b/content/zh-cn/docs/docs/security/server.md index 5d3c0ffb1a3e92c9351cdbc6df6c366b259b7b7a..21820072628550b4ed9c47d5c1ead3a63c3bdead 100644 --- a/content/zh-cn/docs/docs/security/server.md +++ b/content/zh-cn/docs/docs/security/server.md @@ -131,7 +131,7 @@ http-server.https.excluded-cipher= Java资料中列出了[支持的加密套件](http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuites )。 -### access-controls.properties +### access-control.properties `access-control.properties`文件必须至少包含`access-control.name`属性。 其他配置则因配置的实现而异。有关详细信息，请参阅系统访问控制。 @@ -141,7 +141,7 @@ Java资料中列出了[支持的加密套件](http://docs.oracle.com/javase/8/do ## 异常处理 -实现Kerberos身份验证具有挑战性的。您可以独立地验证openLooKeng之外的一些配置，以便在尝试解决问题时缩小您的关注范围。 +实现Kerberos身份验证具有挑战性。您可以独立地验证openLooKeng之外的一些配置，以便在尝试解决问题时缩小您的关注范围。 ### Kerberos验证 diff --git a/content/zh-cn/docs/menu/index.md b/content/zh-cn/docs/menu/index.md index 4e5d19389a81989916cdd515a93b09e95d8aa28e..4c86f061ea0ad42a19a27b98603b0fe8a246f9f5 100644 --- a/content/zh-cn/docs/menu/index.md +++ b/content/zh-cn/docs/menu/index.md @@ -44,8 +44,10 @@ headless: true - [状态存储]({{< relref "./docs/admin/state-store.md" >}}) - [元数据存储]({{< relref "./docs/admin/meta-store.md" >}}) - [审计日志]({{< relref "./docs/admin/audit-log.md" >}}) - - [可靠查询执行]({{< relref "./docs/admin/reliable-execution.md" >}}) + - [可靠查询执行]({{< relref "./docs/admin/reliable-query/_index.md" >}}) - [JDBC数据源多分片管理]({{< relref "./docs/admin/multi-split-for-jdbc-data-source.md" >}}) + - [扩展物理执行计划]({{< relref "./docs/admin/extension-execution-planner.md" >}}) + - [查询优化器]("#") - [表统计]({{< relref "./docs/optimizer/statistics.md" >}}) - [EXPLAIN成本]({{< relref "./docs/optimizer/cost-in-explain.md" >}}) @@ -71,6 +73,7 @@ headless: true - [CarbonData]({{< relref "./docs/connector/carbondata.md" >}}) - [ClickHouse]({{< relref "./docs/connector/clickhouse.md" >}}) - [数据中心]({{< relref "./docs/connector/datacenter.md" >}}) + - [达梦数据库]({{< relref "./docs/connector/dameng.md" >}}) - [Elasticsearch]({{< relref "./docs/connector/elasticsearch.md" >}}) - [Greenplum]({{< relref "./docs/connector/greenplum.md" >}}) - [Hana]({{< relref "./docs/connector/hana.md" >}}) @@ -82,13 +85,16 @@ headless: true - [JMX]({{< relref "./docs/connector/jmx.md" >}}) - [Kafka]({{< relref "./docs/connector/kafka.md" >}}) - [Kafka连接器教程]({{< relref "./docs/connector/kafka-tutorial.md" >}}) + - [Redis]({{< relref "./docs/connector/redis.md" >}}) - [本地文件]({{< relref "./docs/connector/localfile.md" >}}) + - [MariaDB]({{< relref "./docs/connector/mariadb.md" >}}) - [内存]({{< relref "./docs/connector/memory.md" >}}) - [MongoDB]({{< relref "./docs/connector/mongodb.md" >}}) - [MySQL]({{< relref "./docs/connector/mysql.md" >}}) - [openGauss]({{< relref "./docs/connector/opengauss.md" >}}) - [Oracle]({{< relref "./docs/connector/oracle.md" >}}) - [PostgreSQL]({{< relref "./docs/connector/postgresql.md" >}}) + - [SingleData]({{< relref "./docs/connector/singledata.md" >}}) - [SQL Server]({{< relref "./docs/connector/sqlserver.md" >}}) - [系统]({{< relref "./docs/connector/system.md" >}}) - [Thrift]({{< relref "./docs/connector/thrift.md" >}}) @@ -216,6 +222,9 @@ headless: true - [任务资源]({{< relref "./docs/rest/task.md" >}}) - [发行说明]("#") + - [1.9.0 (2022年12月30日)]({{< relref "./docs/releasenotes/releasenotes-1.9.0.md" >}}) + - [1.8.0 (2022年9月30日)]({{< relref "./docs/releasenotes/releasenotes-1.8.0.md" >}}) + - [1.7.0 (2022年6月30日)]({{< relref "./docs/releasenotes/releasenotes-1.7.0.md" >}}) - [1.6.1 (2022年4月27日)]({{< relref "./docs/releasenotes/releasenotes-1.6.1.md" >}}) - [1.6.0 (2022年3月30日)]({{< relref "./docs/releasenotes/releasenotes-1.6.0.md" >}}) - [1.5.0 (2021年12月30日)]({{< relref "./docs/releasenotes/releasenotes-1.5.0.md" >}})