# elasticsearch-demo

**Repository Path**: chenwenbin_admin/elasticsearch-demo

## Basic Information

- **Project Name**: elasticsearch-demo
- **Description**: elasticsearch示例代码
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 2
- **Created**: 2021-07-30
- **Last Updated**: 2021-07-30

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Elasticsearch聚合编码实践 
—— 基于Spring Data Elasticsearch
### 本文目录

一. [使用原因](#jump1)<br>
二. [基本概念](#jump2)<br>
三. [安装调用](#jump3)<br>
四. [基础知识](#jump4)<br>
五. [项目集成](#jump5)<br>
六. [案例讲解](#jump6)<br>

### <a name="jump1"> 一、使用原因 </a>

##### 官网介绍  
```html
    Elasticsearch, 是Elastic Stack 的核心。
    Elasticsearch 是一个分布式、RESTful 风格的搜索和数据分析引擎，能够解决不断涌现出的各种用例。 
    作为 Elastic Stack 的核心，它集中存储您的数据，帮助您发现意料之中以及意料之外的情况。  
```
##### 百度百科
```html
    Elasticsearch是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。
    Elasticsearch是用Java语言开发的，并作为Apache许可条款下的开放源码发布，是一种流行的企业级搜索引擎。
    Elasticsearch用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。
    官方客户端在Java、.NET（C#）、PHP、Python、Apache Groovy、Ruby和许多其他语言中都是可用的。
    根据DB-Engines的排名显示，Elasticsearch是最受欢迎的企业搜索引擎，其次是Apache Solr，也是基于Lucene。
```

##### 使用最广泛的场景：
1、日志存储与检索<br>
2、搜索引擎<br>


### <a name="jump2"> 二. 基本概念 </a>

##### 基本概念
1. cluster 集群
2. node 节点
3. index 索引
4. type 类型
5. mapping 映射
6. document 文档

##### 对于删除type的官方说明原文段落：
```text
Why are mapping types being removed?

Initially, we spoke about an “index” being similar to a “database” in an SQL database, and a “type” being equivalent to a “table”.

This was a bad analogy that led to incorrect assumptions. In an SQL database, tables are independent of each other. The columns in one table have no bearing on columns with the same name in another table. This is not the case for fields in a mapping type.

In an Elasticsearch index, fields that have the same name in different mapping types are backed by the same Lucene field internally. In other words, using the example above, the user_name field in the user type is stored in exactly the same field as the user_name field in the tweet type, and both user_name fields must have the same mapping (definition) in both types.

This can lead to frustration when, for example, you want deleted to be a date field in one type and a boolean field in another type in the same index.

On top of that, storing different entities that have few or no fields in common in the same index leads to sparse data and interferes with Lucene’s ability to compress documents efficiently.

For these reasons, we have decided to remove the concept of mapping types from Elasticsearch
```
备注：[官方对删除type的说明链接](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/removal-of-types.html#_why_are_mapping_types_being_removed)<br>

##### 官方文档：
[官方文档](https://www.elastic.co/guide/index.html)
![avatar](./img/13.jpg)
[官方中文文档地址链接](https://www.elastic.co/guide/cn/index.html)
![avatar](./img/6.jpg)

### <a name="jump3"> 三. 安装调用 </a>
1、安装方式

![avatar](./img/4.png)

演示：
* 基于docker安装
```text
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.8.1;
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.8.1;

```
* 极简验证

![avatar](./img/5.jpg)

参照： [elasticsearch官方安装说明文档](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/install-elasticsearch.html)


2、 调用方式<br>
2.1、 http直接调用（示例：Postman）
![avatar](./img/1.jpg)

2.2、 kibana开发调试工具 Dev Tools 
![avatar](./img/2.jpg)

2.3、 代码调用<br>
2.3.1.、基于官方提供的客户端
[Elasticsearch Client](https://www.elastic.co/guide/en/elasticsearch/client/index.html)
![avatar](./img/11.jpg)
##### Java REST Client
 - Java High Level REST Client
 - Java Low Level REST Client
![avatar](./img/8.jpg)
![avatar](./img/7.jpg)
##### Transport Client
![avatar](./img/12.jpg)
##### 基于SQL JDBC

```xml
<dependency>
  <groupId>org.elasticsearch.plugin</groupId>
  <artifactId>x-pack-sql-jdbc</artifactId>
  <version>7.8.1</version>
</dependency>
```

```xml
<repositories>
  <repository>
    <id>elastic.co</id>
    <url>https://artifacts.elastic.co/maven</url>
  </repository>
</repositories>
```
参照：[elasticsearch sql jdbc官方文档](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/sql-jdbc.html)<br>

##### 基于Spring Data Elasticsearch<br>
见第五节。

### <a name="jump4"> 四. 基础知识 </a>
##### 1、DSL聚合语法演示
```text
GET /temp_lincolnshire/_search
{
  "size": 0, 
  "aggs": {
    "month": {
      "terms": {
        "field": "month",
        "size": 100
      }
    }
  }
}
```
![avatar](./img/3.jpg)
##### 2、SQL语法支持
![avatar](./img/14.jpg)

### 五. 项目集成
##### 1、 添加依赖
```$xml
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
		</dependency>
```

补充：实际项目中的依赖关系
* Spring Cloud依赖于Spring Boot
![avatar](./img/9.jpg)
* Spring Data Elasticsearch同样依赖Spring Boot，同时对于Elasticsearch版本的依赖性也很强
![avatar](./img/10.jpg)
* ES本身升级较快，同时官方客户端对版本的兼容性也较差

##### 2、数据源引入配置
###### 2.1 无权限控制
```java
@Configuration
public class RestClientConfig extends AbstractElasticsearchConfiguration {

    @Value("${es.hosts:localhost:9200}")
    private String hostAndPort;

    @Override
    public RestHighLevelClient elasticsearchClient() {
        return RestClients.create(ClientConfiguration.create(hostAndPort)).rest();
    }
}
```
###### 2.2 有权限控制
```java
@Configuration
public class RestClientByPasswordConfig extends AbstractElasticsearchConfiguration {

    @Value("${es.connect-timeout:-1}")
    private int connectTimeout;

    @Value("${es.socket-timeout:-1}")
    private int socketTimeout;

    @Value("${es.connection-request-timeout:-1}")
    private int connectionRequestTimeout;

    @Value("${es.username:es_user}")
    private String userName;

    @Value("${es.password:es_password}")
    private String password;

    @Value("${es.hosts:127.0.0.1}")
    private String hosts;

    @Value("${es.port:9200}")
    private int port;

    @Value("${es.max-retry-timeout-millis:300000}")
    private int maxRetryTimeoutMillis;

    @Override
    public RestHighLevelClient elasticsearchClient() {
        String[] hosts = this.hosts.split(",");
        HttpHost[] httpHosts = new HttpHost[hosts.length];
        for(int i=0;i<hosts.length;i++) {
            httpHosts[i] = new HttpHost(hosts[i], port, "http");
        }

        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, password));

        RestClientBuilder builder = RestClient.builder(httpHosts).setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                requestConfigBuilder.setConnectTimeout(connectTimeout);
                requestConfigBuilder.setSocketTimeout(socketTimeout);
                requestConfigBuilder.setConnectionRequestTimeout(connectionRequestTimeout);
                return requestConfigBuilder;
            }
        }).setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
            @Override
            public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                httpClientBuilder.disableAuthCaching();
                return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
            }
        });

        RestHighLevelClient client = new RestHighLevelClient(builder);
        return client;
    }
}

```

##### 3、核心交互对象
```java
@Service
public class BaseService extends LogClass {

    @Autowired
    protected ElasticsearchOperations elasticsearchOperations;

}

```

##### 4、Document类创建
```java
@Data
@EqualsAndHashCode(callSuper = false)
@Accessors(chain = true)
@Document(indexName = "temp_lincolnshire")
public class TempLincolnshireDocument implements Serializable {

    private static final long serialVersionUID = 1L;

    @Id
    @Field(type = FieldType.Integer)
    private Integer id;

    @Field(type = FieldType.Keyword)
    private String name;

    @Field(type = FieldType.Keyword)
    private String year;

    @Field(type = FieldType.Keyword)
    private String month;

    @Field(type = FieldType.Keyword)
    private String day;

    @Field(type = FieldType.Keyword)
    private String hours;

    @Field(type = FieldType.Keyword)
    private BigDecimal value;

}
```
##### 5、Elasticsearch Repositories示例

* TempLincolnshireRepositories
```java
@org.springframework.stereotype.Repository
public interface TempLincolnshireRepositories extends Repository<TempLincolnshireDocument, String> {

    TempLincolnshireDocument findById(String id);

    List<TempLincolnshireDocument> findByNameAndYearAndMonth(String name,String year, String month);

    Page<TempLincolnshireDocument> findByName(String name, Pageable pageable);

}
```

* TempLincolnshireEsRepositories
```java
@Repository
public interface TempLincolnshireEsRepositories  extends ElasticsearchRepository<TempLincolnshireDocument, String> {
}
```
返回类型支持
* List<T>
* Stream<T>
* SearchHits<T>
* List<SearchHit<T>>
* Stream<SearchHit<T>>
* SearchPage<T>

参考：[Elasticsearch Repositories说明文档](https://docs.spring.io/spring-data/elasticsearch/docs/4.2.3/reference/html/#elasticsearch.repositories)

### <a name="jump6"> 六、案例讲解</a>

* TempLincolnshireDemoService
```java
public interface TempLincolnshireDemoService {

    /**
     *@描述 criteria查询示例1: 完全匹配单个name和单个year，并支持分页
     *@参数 [name, year, pageNo, pageSize]
     *@返回值 java.util.List<cn.sunzq.springdata.elasticsearch.demo.document.TempLincolnshireDocument>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<TempLincolnshireDocument> criteriaQuery1(String name, String year, Integer pageNo, Integer pageSize);

    /**
     *@描述 criteria查询示例2: 多个name满足其一，且多个year满足其一，并支持分页
     *@参数 [name, year, pageNo, pageSize]
     *@返回值 java.util.List<cn.sunzq.springdata.elasticsearch.demo.document.TempLincolnshireDocument>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<TempLincolnshireDocument> criteriaQuery2(List<String> name, List<String> year, Integer pageNo, Integer pageSize);

    /**
     *@描述 criteria查询示例2: 在name和year中同时搜索关键字，满足其一，并支持分页
     *@参数 [keyword, pageNo, pageSize]
     *@返回值 java.util.List<cn.sunzq.springdata.elasticsearch.demo.document.TempLincolnshireDocument>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<TempLincolnshireDocument> criteriaQuery3(String keyword, Integer pageNo, Integer pageSize);

    /**
     *@描述 stringQuery示例查询1： 完全匹配单个name和单个year，并支持分页，与criteriaQuery1条件相同
     *@参数 [name, year, pageNo, pageSize]
     *@返回值 java.util.List<cn.sunzq.springdata.elasticsearch.demo.document.TempLincolnshireDocument>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<TempLincolnshireDocument> stringQuery1(String name, String year, Integer pageNo, Integer pageSize);

    /**
     *@描述 restClient示例1：基于Rest High Level Client 实现与criteriaQuery1条件相同的查询
     *@参数 [name, year, pageNo, pageSize]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<Map<String,Object>> restClient1(String name, String year, Integer pageNo, Integer pageSize);

    /**
     *@描述 聚合示例1： 完全匹配单个name和year的数据中，按hours分组得到每个分组下的数据条数
     *@参数 [name, year]
     *@返回值 java.util.Map<java.lang.String,java.lang.Object>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    Map<String,Object> aggDemo1(String name, String year);

    /**
     *@描述 聚合示例2： 完全匹配单个name和year的数据中，分别按hours和month分组，并得到hours和month两个分组每个分组下的数据条数
     *@参数 [name, year]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<Map<String,Object>> aggDemo2(String name, String year);

    /**
     *@描述 聚合示例3： 完全匹配单个name和year的数据中，分别按hours和month分组，并得到hours和month两个分组每个分组下的平均数
     *@参数 [name, year]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<Map<String, Object>> aggDemo3(String name, String year);

    /**
     *@描述 聚合示例4： 完全匹配单个name和year的数据中，分别按hours和month分组，并得到hours和month两个分组每个分组下，value按参数传递的数据范围进行再分组，得到二层分组下的数据条数
     *@参数 [name, year, valueGroup]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<Map<String,Object>> aggDemo4(String name, String year, List<Double> valueGroup);
    
    /**
     *@描述 聚合案例5： 完全匹配name，时间范围在参数传递的范围内，并得到按年、月、日、时四个分组依次向下四层结构，得到每个小时的value平均值
     *@参数 [name, dateRange]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/28
     *@修改人和其它信息
     */
    List<Map<String, Object>> aggDemo5(String  dateRangeStart, String dateRangeEnd);
    
    /**
     *@描述 聚合案例6： 时间范围在参数传递的范围内，得到按名称或者月份分组，得到分组下升序或降序时前几位value的平均值
     *@参数 [dateRangeStart, dateRangeEnd, topNo, isAsc, aggField]
     *@返回值 java.util.List<java.util.Map<java.lang.String,java.lang.Object>>
     *@创建人 孙中泉
     *@创建时间 2021/7/29
     *@修改人和其它信息
     */
    List<Map<String,Object>> aggDemo6(String dateRangeStart, String dateRangeEnd, int topNo, boolean isAsc, String aggField);

}

```

[Elasticsearch Operations官方说明文档](https://docs.spring.io/spring-data/elasticsearch/docs/4.2.3/reference/html/#elasticsearch.reactive.operations)


#### 相关资料
1. 本文链接：[Elasticsearch聚合编码实践—— 基于Spring Data Elasticsearch](https://gitee.com/szqcric/elasticsearch-demo)
2. Elasticsearch官网：[https://www.elastic.co](https://www.elastic.co/cn/downloads/)
3. Spring Data Elasticsearch官方文档：[https://docs.spring.io/spring-data/elasticsearch/docs](https://docs.spring.io/spring-data/elasticsearch/docs/4.3.x/reference/html/)
4. Java High Level REST Client: [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.x](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.x/java-rest-high.html)