1e76a824bc
Fix #24662. Replace #24822 and #25708 (although it has been merged) ## Background In the past, Gitea supported issue searching with a keyword and conditions in a less efficient way. It worked by searching for issues with the keyword and obtaining limited IDs (as it is heavy to get all) on the indexer (bleve/elasticsearch/meilisearch), and then querying with conditions on the database to find a subset of the found IDs. This is why the results could be incomplete. To solve this issue, we need to store all fields that could be used as conditions in the indexer and support both keyword and additional conditions when searching with the indexer. ## Major changes - Redefine `IndexerData` to include all fields that could be used as filter conditions. - Refactor `Search(ctx context.Context, kw string, repoIDs []int64, limit, start int, state string)` to `Search(ctx context.Context, options *SearchOptions)`, so it supports more conditions now. - Change the data type stored in `issueIndexerQueue`. Use `IndexerMetadata` instead of `IndexerData` in case the data has been updated while it is in the queue. This also reduces the storage size of the queue. - Enhance searching with Bleve/Elasticsearch/Meilisearch, make them fully support `SearchOptions`. Also, update the data versions. - Keep most logic of database indexer, but remove `issues.SearchIssueIDsByKeyword` in `models` to avoid confusion where is the entry point to search issues. - Start a Meilisearch instance to test it in unit tests. - Add unit tests with almost full coverage to test Bleve/Elasticsearch/Meilisearch indexer. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com> |
||
---|---|---|
.. | ||
migration-test | ||
schemas | ||
actions_trigger_test.go | ||
admin_config_test.go | ||
admin_user_test.go | ||
api_actions_artifact_test.go | ||
api_activitypub_person_test.go | ||
api_admin_org_test.go | ||
api_admin_test.go | ||
api_branch_test.go | ||
api_comment_attachment_test.go | ||
api_comment_test.go | ||
api_feed_user_test.go | ||
api_fork_test.go | ||
api_gitignore_templates_test.go | ||
api_gpg_keys_test.go | ||
api_helper_for_declarative_test.go | ||
api_httpsig_test.go | ||
api_issue_attachment_test.go | ||
api_issue_config_test.go | ||
api_issue_label_test.go | ||
api_issue_milestone_test.go | ||
api_issue_pin_test.go | ||
api_issue_reaction_test.go | ||
api_issue_stopwatch_test.go | ||
api_issue_subscription_test.go | ||
api_issue_test.go | ||
api_issue_tracked_time_test.go | ||
api_keys_test.go | ||
api_label_templates_test.go | ||
api_license_templates_test.go | ||
api_nodeinfo_test.go | ||
api_notification_test.go | ||
api_oauth2_apps_test.go | ||
api_org_avatar_test.go | ||
api_org_test.go | ||
api_packages_alpine_test.go | ||
api_packages_cargo_test.go | ||
api_packages_chef_test.go | ||
api_packages_composer_test.go | ||
api_packages_conan_test.go | ||
api_packages_conda_test.go | ||
api_packages_container_test.go | ||
api_packages_cran_test.go | ||
api_packages_debian_test.go | ||
api_packages_generic_test.go | ||
api_packages_goproxy_test.go | ||
api_packages_helm_test.go | ||
api_packages_maven_test.go | ||
api_packages_npm_test.go | ||
api_packages_nuget_test.go | ||
api_packages_pub_test.go | ||
api_packages_pypi_test.go | ||
api_packages_rpm_test.go | ||
api_packages_rubygems_test.go | ||
api_packages_swift_test.go | ||
api_packages_test.go | ||
api_packages_vagrant_test.go | ||
api_private_serv_test.go | ||
api_pull_commits_test.go | ||
api_pull_review_test.go | ||
api_pull_test.go | ||
api_releases_test.go | ||
api_repo_archive_test.go | ||
api_repo_avatar_test.go | ||
api_repo_branch_test.go | ||
api_repo_collaborator_test.go | ||
api_repo_edit_test.go | ||
api_repo_file_create_test.go | ||
api_repo_file_delete_test.go | ||
api_repo_file_get_test.go | ||
api_repo_file_helpers.go | ||
api_repo_file_update_test.go | ||
api_repo_files_change_test.go | ||
api_repo_get_contents_list_test.go | ||
api_repo_get_contents_test.go | ||
api_repo_git_blobs_test.go | ||
api_repo_git_commits_test.go | ||
api_repo_git_hook_test.go | ||
api_repo_git_notes_test.go | ||
api_repo_git_ref_test.go | ||
api_repo_git_tags_test.go | ||
api_repo_git_trees_test.go | ||
api_repo_hook_test.go | ||
api_repo_languages_test.go | ||
api_repo_lfs_locks_test.go | ||
api_repo_lfs_migrate_test.go | ||
api_repo_lfs_test.go | ||
api_repo_raw_test.go | ||
api_repo_tags_test.go | ||
api_repo_teams_test.go | ||
api_repo_test.go | ||
api_repo_topic_test.go | ||
api_settings_test.go | ||
api_team_test.go | ||
api_team_user_test.go | ||
api_token_test.go | ||
api_user_avatar_test.go | ||
api_user_email_test.go | ||
api_user_follow_test.go | ||
api_user_heatmap_test.go | ||
api_user_info_test.go | ||
api_user_org_perm_test.go | ||
api_user_orgs_test.go | ||
api_user_search_test.go | ||
api_user_star_test.go | ||
api_user_watch_test.go | ||
api_wiki_test.go | ||
attachment_test.go | ||
auth_ldap_test.go | ||
avatar.png | ||
benchmarks_test.go | ||
branches_test.go | ||
change_default_branch_test.go | ||
cmd_keys_test.go | ||
compare_test.go | ||
cors_test.go | ||
create_no_session_test.go | ||
csrf_test.go | ||
delete_user_test.go | ||
download_test.go | ||
dump_restore_test.go | ||
editor_test.go | ||
empty_repo_test.go | ||
eventsource_test.go | ||
explore_repos_test.go | ||
git_clone_wiki_test.go | ||
git_helper_for_declarative_test.go | ||
git_smart_http_test.go | ||
git_test.go | ||
goget_test.go | ||
gpg_git_test.go | ||
html_helper.go | ||
incoming_email_test.go | ||
integration_test.go | ||
issue_test.go | ||
lfs_getobject_test.go | ||
lfs_local_endpoint_test.go | ||
lfs_view_test.go | ||
links_test.go | ||
markup_external_test.go | ||
migrate_test.go | ||
mirror_pull_test.go | ||
mirror_push_test.go | ||
nonascii_branches_test.go | ||
oauth_test.go | ||
org_count_test.go | ||
org_project_test.go | ||
org_team_invite_test.go | ||
org_test.go | ||
private-testing.key | ||
privateactivity_test.go | ||
pull_compare_test.go | ||
pull_create_test.go | ||
pull_diff_test.go | ||
pull_merge_test.go | ||
pull_review_test.go | ||
pull_status_test.go | ||
pull_update_test.go | ||
README.md | ||
README_ZH.md | ||
release_test.go | ||
rename_branch_test.go | ||
repo_activity_test.go | ||
repo_branch_test.go | ||
repo_commits_search_test.go | ||
repo_commits_test.go | ||
repo_fork_test.go | ||
repo_generate_test.go | ||
repo_migrate_test.go | ||
repo_search_test.go | ||
repo_tag_test.go | ||
repo_test.go | ||
repo_topic_test.go | ||
repo_watch_test.go | ||
repofiles_change_test.go | ||
setting_test.go | ||
signin_test.go | ||
signout_test.go | ||
signup_test.go | ||
ssh_key_test.go | ||
timetracking_test.go | ||
user_avatar_test.go | ||
user_test.go | ||
version_test.go | ||
view_test.go | ||
webfinger_test.go | ||
xss_test.go |
Integration tests
Integration tests can be run with make commands for the appropriate backends, namely:
make test-sqlite
make test-pgsql
make test-mysql
make test-mysql8
make test-mssql
Make sure to perform a clean build before running tests:
make clean build
Run tests via local act_runner
Run all jobs
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest
Warning: This file defines many jobs, so it will be resource-intensive and therefor not recommended.
Run single job
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -j <job_name>
You can list all job names via:
act_runner exec -W ./.github/workflows/pull-db-tests.yml --event=pull_request --default-actions-url="https://github.com" -i catthehacker/ubuntu:runner-latest -l
Run sqlite integration tests
Start tests
make test-sqlite
Run MySQL integration tests
Setup a MySQL database inside docker
docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container)
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container)
Start tests based on the database container
TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-mysql
Run pgsql integration tests
Setup a pgsql database inside docker
docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container)
Start tests based on the database container
TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-pgsql
Run mssql integration tests
Setup a mssql database inside docker
docker run -e "ACCEPT_EULA=Y" -e "MSSQL_PID=Standard" -e "SA_PASSWORD=MwantsaSecurePassword1" -p 1433:1433 --rm --name mssql microsoft/mssql-server-linux:latest #(just ctrl-c to stop db and clean the container)
Start tests based on the database container
TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=gitea_test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql
Running individual tests
Example command to run GPG test:
For SQLite:
make test-sqlite#GPG
For other databases(replace mssql
to mysql
, mysql8
or pgsql
):
TEST_MSSQL_HOST=localhost:1433 TEST_MSSQL_DBNAME=test TEST_MSSQL_USERNAME=sa TEST_MSSQL_PASSWORD=MwantsaSecurePassword1 make test-mssql#GPG
Setting timeouts for declaring long-tests and long-flushes
We appreciate that some testing machines may not be very powerful and the default timeouts for declaring a slow test or a slow clean-up flush may not be appropriate.
You can either:
- Within the test ini file set the following section:
[integration-tests]
SLOW_TEST = 10s ; 10s is the default value
SLOW_FLUSH = 5S ; 5s is the default value
- Set the following environment variables:
GITEA_SLOW_TEST_TIME="10s" GITEA_SLOW_FLUSH_TIME="5s" make test-sqlite