Skip to content

Improve search query for adressbooks - #30195

Closed
butonic wants to merge 2 commits into
masterfrom
improve-search-query-for-adressbooks
Closed

butonic wants to merge 2 commits into
masterfrom
improve-search-query-for-adressbooks

Conversation

@butonic

@butonic butonic commented Jan 19, 2018

Copy link
Copy Markdown
Contributor

The current search query for addressbook entries is problematic:

MariaDB [core]> EXPLAIN
SELECT c.`carddata`, c.`uri`
FROM `oc_cards` c
WHERE c.`id` IN (
    SELECT DISTINCT cp.`cardid`
    FROM `oc_cards_properties` cp
    WHERE (
        ( (cp.`name` = 'CLOUD') AND (cp.`value`  COLLATE utf8_general_ci LIKE '%foo%') )
        OR ( (cp.`name` = 'FN') AND (cp.`value`  COLLATE utf8_general_ci LIKE '%foo%') )
    )
    AND (cp.`addressbookid` = 10)
) ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
| id   | select_type  | table       | type   | possible_keys                        | key             | key_len | ref  | rows   | Extra                              |
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
|    1 | PRIMARY      | c           | index  | PRIMARY,id_uri_index                 | uri_index       | 768     | NULL |      1 |                                    |
|    1 | PRIMARY      | <subquery2> | eq_ref | distinct_key                         | distinct_key    | 8       | func |      1 |                                    |
|    2 | MATERIALIZED | cp          | range  | card_name_index,card_contactid_index | card_name_index | 195     | NULL | 372808 | Using index condition; Using where |
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
3 rows in set (0.00 sec)

SELECT c.`carddata`, c.`uri` FROM `oc_cards` c WHERE c.`id` IN (     SELECT DISTINCT cp.`cardid`     FROM `oc_cards_properties` cp     WHERE cp.`addressbookid` = 10     AND cp.`name` IN ('CLOUD','FN')     AND cp.`value`  COLLATE utf8_general_ci LIKE '%filler%' ) ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
...
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
200 rows in set (0.29 sec)

0.29 sec. ouch. on large addressbooks (>100k cards) tables this query takes more than a second ... per addressbook. 20 addressbooks => 20 seconds until the user sees a result.

We can do better. Let us have a look at the query. Maybe we can simplify it? What about:

MariaDB [core]> EXPLAIN
SELECT c.`carddata`, c.`uri`
FROM `oc_cards` c
WHERE c.`id` IN (
    SELECT DISTINCT cp.`cardid`
    FROM `oc_cards_properties` cp
    WHERE cp.`addressbookid` = 11
    AND cp.`name` IN ('CLOUD','FN')
    AND cp.`value`  COLLATE utf8_general_ci LIKE '%filler%'
)
ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
| id   | select_type  | table       | type   | possible_keys                        | key             | key_len | ref  | rows   | Extra                              |
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
|    1 | PRIMARY      | c           | index  | PRIMARY,id_uri_index                 | uri_index       | 768     | NULL |      1 |                                    |
|    1 | PRIMARY      | <subquery2> | eq_ref | distinct_key                         | distinct_key    | 8       | func |      1 |                                    |
|    2 | MATERIALIZED | cp          | range  | card_name_index,card_contactid_index | card_name_index | 195     | NULL | 372808 | Using index condition; Using where |
+------+--------------+-------------+--------+--------------------------------------+-----------------+---------+------+--------+------------------------------------+
3 rows in set (0.00 sec)

SELECT c.`carddata`, c.`uri` FROM `oc_cards` c WHERE c.`id` IN (     SELECT DISTINCT cp.`cardid`     FROM `oc_cards_properties` cp     WHERE cp.`addressbookid` = 11     AND cp.`name` IN ('CLOUD','FN')     AND cp.`value`  COLLATE utf8_general_ci LIKE '%filler%' ) ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
...
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
200 rows in set (0.29 sec)

Well, nothing changed. But at least it is more readable ... so what about the indexes. Adding one for the materialized subquery became clearer:

MariaDB [core]> CREATE INDEX carddata_aid_n_v_idx ON oc_cards_properties (addressbookid, name, value);
Query OK, 0 rows affected (1.81 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [core]> EXPLAIN SELECT c.`carddata`, c.`uri` FROM `oc_cards` c WHERE c.`id` IN (     SELECT DISTINCT cp.`cardid`     FROM `oc_cards_properties` cp     WHERE cp.`addressbookid` = 12     AND cp.`name` IN ('CLOUD','FN')     AND cp.`value`  COLLATE utf8_general_ci LIKE '%filler%' ) ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+------+--------------+-------------+--------+-----------------------------------------------------------+----------------------+---------+----------------+------+---------------------------------+
| id   | select_type  | table       | type   | possible_keys                                             | key                  | key_len | ref            | rows | Extra                           |
+------+--------------+-------------+--------+-----------------------------------------------------------+----------------------+---------+----------------+------+---------------------------------+
|    1 | PRIMARY      | <subquery2> | ALL    | distinct_key                                              | NULL                 | NULL    | NULL           | 3962 | Using temporary; Using filesort |
|    1 | PRIMARY      | c           | eq_ref | PRIMARY,id_uri_index                                      | PRIMARY              | 8       | core.cp.cardid |    1 |                                 |
|    2 | MATERIALIZED | cp          | range  | card_name_index,card_contactid_index,carddata_aid_n_v_idx | carddata_aid_n_v_idx | 203     | NULL           | 3962 | Using index condition           |
+------+--------------+-------------+--------+-----------------------------------------------------------+----------------------+---------+----------------+------+---------------------------------+
3 rows in set (0.00 sec)

MariaDB [core]> SELECT c.`carddata`, c.`uri` FROM `oc_cards` c WHERE c.`id` IN (     SELECT DISTINCT cp.`cardid`     FROM `oc_cards_properties` cp     WHERE cp.`addressbookid` = 12     AND cp.`name` IN ('CLOUD','FN')     AND cp.`value`  COLLATE utf8_general_ci LIKE '%filler%' ) ORDER BY c.`uri` ASC LIMIT 200 OFFSET 0;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
...
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+
200 rows in set (0.03 sec)

Much better. We now have an index that correctly partitions the table.

@codecov

codecov Bot commented Jan 19, 2018 •

Copy link
Copy Markdown

Codecov Report

Merging #30195 into master will decrease coverage by 0.4%.
The diff coverage is 61.11%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #30195      +/-   ##
============================================
- Coverage     64.85%   64.44%   -0.41%     
- Complexity    19136    19138       +2     
============================================
  Files          1267     1271       +4     
  Lines         74895    74920      +25     
  Branches       1331     1328       -3     
============================================
- Hits          48575    48285     -290     
- Misses        25928    26245     +317     
+ Partials        392      390       -2
Flag Coverage Δ Complexity Δ
#javascript 54.18% <ø> (+0.04%) 0 <ø> (ø) ⬇️
#phpunit 65.59% <61.11%> (-0.47%) 19138 <3> (+2)
Impacted Files Coverage Δ Complexity Δ
...s/dav/appinfo/Migrations/Version20200114181454.php 0% <0%> (ø) 3 <3> (?)
apps/dav/lib/CardDAV/CardDavBackend.php 82.23% <100%> (-0.31%) 84 <0> (-1)
lib/private/DB/PostgreSqlMigrator.php 0% <0%> (-100%) 4% <0%> (ø)
...ilder/ExpressionBuilder/PgSqlExpressionBuilder.php 0% <0%> (-88.89%) 3% <0%> (ø)
lib/private/DB/AdapterOCI8.php 0% <0%> (-86.67%) 4% <0%> (ø)
lib/private/DB/AdapterPgSql.php 0% <0%> (-85.72%) 2% <0%> (ø)
...Builder/ExpressionBuilder/OCIExpressionBuilder.php 0% <0%> (-85.19%) 18% <0%> (ø)
lib/private/DB/OracleMigrator.php 0% <0%> (-76.85%) 10% <0%> (ø)
lib/private/DB/OracleConnection.php 0% <0%> (-66.67%) 12% <0%> (ø)
apps/files/lib/Helper.php 24.27% <0%> (-14.51%) 40% <0%> (+5%)
... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8cacea1...655be07. Read the comment docs.

@mrow4a

mrow4a commented Jan 23, 2018

Copy link
Copy Markdown
Contributor

I mean, essentially query is the same (it will be optimized to same thing), but ye avoiding table scan with indexes should help, but index on value makes no help here since you use %wildcard%. Can you check without index on value ?

@mrow4a

mrow4a commented Jan 23, 2018

Copy link
Copy Markdown
Contributor

Is 'filler' exactly the value, or sth it starts with like filler% ? Then it might work

$table = $schema->getTable("${prefix}cards_properties");
// Check for existing index spanning these columns
foreach ($table->getIndexes() as $index) {
// Check if we have a matching index already

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

/**
* Add index to oc_cards_properties to assist with searching with large numbers of rows
*/
class Version20180119134849 implements ISchemaMigration {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a cost estimation for this index - time expected given number of rows - given this could be released in a minor version we need to give useful information to the admins

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not worth mentioning

mysql> SELECT count(*) FROM oc_cards_properties;
+----------+
| count(*) |
+----------+
|   164306 |
+----------+
1 row in set (0.06 sec)

mysql> CREATE INDEX carddata_aid_n_v_idx ON oc_cards_properties (addressbookid, name, value);
Query OK, 0 rows affected (0.40 sec)
Records: 0  Duplicates: 0  Warnings: 0

Comment thread apps/dav/lib/CardDAV/CardDavBackend.php Resolved
@DeepDiver1975

Copy link
Copy Markdown
Contributor

needs rebase ....

@tomneedham

tomneedham commented Feb 14, 2018 •

Copy link
Copy Markdown
Contributor

Is 'filler' exactly the value, or sth it starts with like filler% ? Then it might work

We can use the same logic as the accounts table here if this helps. We have 'accounts.enable_medial_search' => true, which changes the search from %term% to term%

Separate PR imo

@tomneedham

Copy link
Copy Markdown
Contributor

Testing on my machine with 20k user accounts (provisioned and logged in at least once):

  • Running the query without the index ~180 - 200ms
  • With the index across just addressbookid, name ~1.8ms

I will adjust the PR to only cover those two fields - then we save some space in the index especially as value is VARCHAR 255.

@butonic

butonic commented Feb 21, 2018

Copy link
Copy Markdown
Contributor Author

well, medial search can only benefit if the index contains the value

@mrow4a

mrow4a commented Feb 21, 2018

Copy link
Copy Markdown
Contributor

@tomneedham so you just confirmed that only on 2 cols is enough? Can you try also with only addressbookid? And yes, maybe for medial search we can put full table in memory, but admins need to know that the table will be full in memory and each single update in that table will need to update that index structure.. each time..

@mrow4a

mrow4a commented Feb 21, 2018 •

Copy link
Copy Markdown
Contributor

@butonic @tomneedham for medial search, maybe better we put index only on value though with filler% - then it should be fast - but it will work only for filler%, not for %filler% which might only be solved with index on addressbookid.

@tomneedham

Copy link
Copy Markdown
Contributor

so you just confirmed that only on 2 cols is enough?

Yes - when not considering medial search.

but it will work only for filler%, not for %filler% which might only be solved with index on addressbookid.

That is fine - we should also use a config option to control this functionality like we have with 'accounts.enable_medial_search' => true,

@butonic

butonic commented Feb 22, 2018

Copy link
Copy Markdown
Contributor Author

@mrow4a please have a look at the explain output. Can you give me a reason for indexing only the addressbookid?

@mrow4a

mrow4a commented Feb 22, 2018 •

Copy link
Copy Markdown
Contributor

What is selectivity of unique addressbookid vs selectivity of unique addressbookid&cp.name pair vs full table in memory indexed. (i.e. how many rows returned for average unique value/pair for addresbookid vs addressbookid&cp.name)

@mrow4a

mrow4a commented Feb 22, 2018

Copy link
Copy Markdown
Contributor

Another problem I see is that it is hard to decide on where to put index, since we would need to know in advance if medial search is allowed or not. Honestyly I would go for %filler% and put index on addressbook alone if that works for that workload or addressbook&cp.name

@PVince81 PVince81 modified the milestones: development, backlog Mar 6, 2019
@DeepDiver1975

Copy link
Copy Markdown
Contributor

As discussed in #35777 the master branch will from now on hold the ownCloud 10 codebase.

This PR targetted ownCloud 11 which is postponed to a far distant future.

Because of that I'm closing this PR and kindly ask you to re-submit this PR in a few days.

Thanks a lot for your patience

@mrow4a

mrow4a commented Jan 14, 2020

Copy link
Copy Markdown
Contributor

@mrow4a mrow4a reopened this Jan 14, 2020
@mrow4a
mrow4a force-pushed the improve-search-query-for-adressbooks branch from 1fb983c to 8cc468d Compare January 14, 2020 18:02
@mrow4a

mrow4a commented Jan 14, 2020

Copy link
Copy Markdown
Contributor
  • medial search flag has been already added 3bc8057
  • rabase and test @butonic commits

@mrow4a mrow4a self-assigned this Jan 14, 2020
@mrow4a mrow4a removed this from the backlog milestone Jan 14, 2020
@mrow4a
mrow4a force-pushed the improve-search-query-for-adressbooks branch from 8cc468d to 43f46c7 Compare January 14, 2020 18:22
@mrow4a

mrow4a commented Jan 24, 2020

Copy link
Copy Markdown
Contributor

@micbar target 10.5

@mrow4a
mrow4a force-pushed the improve-search-query-for-adressbooks branch from 43f46c7 to 1e1112d Compare March 4, 2020 18:38
@mrow4a
mrow4a force-pushed the improve-search-query-for-adressbooks branch 2 times, most recently from 3b723a8 to 0b9d89a Compare March 4, 2020 21:27
Comment thread apps/dav/lib/CardDAV/CardDavBackend.php Outdated
}, $searchProperties)))
->andWhere($query->expr()->iLike('cp.value', $query->createNamedParameter($searchPattern)))
->setFirstResult($offset)->setMaxResults($limit)
->groupBy(['c.uri']) // groupby without aggregation is essentially distinct

@mrow4a mrow4a Mar 4, 2020 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@butonic in your commit, you did use ->groupBy(['c.uri', 'c.carddata']) that is not supported by ORACLE (you cannot groupby on BLOB). Is it possible that there are duplicates? If this is about VCARD version we can probably also try ->groupBy(['c.uri', 'size']) . Or maybe not groupby at all and filter on application level?

@mrow4a
mrow4a force-pushed the improve-search-query-for-adressbooks branch from 0b9d89a to 655be07 Compare March 21, 2020 05:59
@mrow4a

mrow4a commented Mar 21, 2020

Copy link
Copy Markdown
Contributor

Due to lack of knowledge about the app and legacy considerations regarding query, I decided not to continue this work in a given form in order not to break something. Just index has been added in #37152.

If someone decides to continue with this PR, it fails due to ->groupBy(['c.uri', 'c.carddata']), as c.carddata is BLOB and running groupby on BLOB is not allowed in Oracle. One solution is to groupby on different terms that give distinct results, but requires legacy considerations.

/drone/src/apps/dav/tests/unit/CardDAV/CardDavBackendTest.php:552
348	
349	Caused by
350	Doctrine\DBAL\Driver\OCI8\OCI8Exception: ORA-00932: inconsistent datatypes: expected - got BLOB

@mrow4a mrow4a removed their assignment Mar 21, 2020
@mrow4a mrow4a mentioned this pull request Aug 30, 2021
1 of 11 tasks
@butonic butonic closed this Jan 26, 2022
@butonic

butonic commented Jan 26, 2022 •

Copy link
Copy Markdown
Contributor Author

query has been changed since initially looking at this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants