forgejo/modules/indexer/issues/elasticsearch/elasticsearch_test.go

50 lines
1.2 KiB
Go
Raw Normal View History

Refactor and enhance issue indexer to support both searching, filtering and paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-07-31 08:28:53 +02:00
// Copyright 2023 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
package elasticsearch
import (
"fmt"
"net/http"
"os"
"testing"
"time"
"code.gitea.io/gitea/modules/indexer/issues/internal/tests"
)
func TestElasticsearchIndexer(t *testing.T) {
[CI] disable meilisearch/elasticsearch test, no server yet in CI (cherry picked from commit a1381d9146fba42cb97d72d38525fa3e721bfb03) (cherry picked from commit 74714e02461fb47fcc0901211668e4529fac68d0) (cherry picked from commit 7749dbfe6684498a47e3037088e7bef3542b6ce5) (cherry picked from commit 437924971136eaed795f77edd3d3dfffa5f68103) (cherry picked from commit a69f55bebf82a0b68bc0f66bc029eaea836cddb7) (cherry picked from commit 24dd5fbfdbc27c887dbc24661c1005fb2e14e3c6) (cherry picked from commit dda856d6b83936fd1c96c84544b086cbd8f63115) (cherry picked from commit bc14f4fa97fffe82d1c666e961e313f88433cb9e) (cherry picked from commit 78fef4f1379d8854901151d4bc62135c73db868e) (cherry picked from commit 69e013cc515e2a50006d8d02f575ff6490d272ff) (cherry picked from commit f173c6a2734b2dccf1424d27cd8e10fc296e44a4) (cherry picked from commit 92f9d02547017770deafd1f715c32ae4479b8ded) (cherry picked from commit c99d51e665370ceb71b96b3fb65184090c7e4442) (cherry picked from commit aa0650fd2b42738a5e564c229c3eb63b8ca77f9b) (cherry picked from commit 0a8ef91302368751df22a1967857283222bc097f) (cherry picked from commit 7b54fe01c2ded0bbbcae6b89d9e330ca4f6ab744) (cherry picked from commit 0e154f366f14d106d14f500f605380c29b5a3f21) (cherry picked from commit 02d88ee16d23b9ebb04bf1af843fc5d2074783ce) (cherry picked from commit 411924e0172a7b10de7513f2e7f60ab5341b13e4) (cherry picked from commit f4e9ca6db59f2c5c638a0560d4ea99833d61520b) (cherry picked from commit cd80126a23573dd5aea1e9674ee0bfa34c63ec5a) (cherry picked from commit da626702f9743fc6e1dd77d21aff5fc3afe75912)
2023-08-01 08:44:46 +02:00
t.Skip("elasticsearch not found in Forgejo test yet")
Refactor and enhance issue indexer to support both searching, filtering and paging (#26012) Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-07-31 08:28:53 +02:00
// The elasticsearch instance started by pull-db-tests.yml > test-unit > services > elasticsearch
url := "http://elastic:changeme@elasticsearch:9200"
if os.Getenv("CI") == "" {
// Make it possible to run tests against a local elasticsearch instance
url = os.Getenv("TEST_ELASTICSEARCH_URL")
if url == "" {
t.Skip("TEST_ELASTICSEARCH_URL not set and not running in CI")
return
}
}
ok := false
for i := 0; i < 60; i++ {
resp, err := http.Get(url)
if err == nil && resp.StatusCode == http.StatusOK {
ok = true
break
}
t.Logf("Waiting for elasticsearch to be up: %v", err)
time.Sleep(time.Second)
}
if !ok {
t.Fatalf("Failed to wait for elasticsearch to be up")
return
}
indexer := NewIndexer(url, fmt.Sprintf("test_elasticsearch_indexer_%d", time.Now().Unix()))
defer indexer.Close()
tests.TestIndexer(t, indexer)
}