Repository navigation
Conversation
Codecov Report
@@ Coverage Diff @@
## master #30195 +/- ##
============================================
- Coverage 64.85% 64.44% -0.41%
- Complexity 19136 19138 +2
============================================
Files 1267 1271 +4
Lines 74895 74920 +25
Branches 1331 1328 -3
============================================
- Hits 48575 48285 -290
- Misses 25928 26245 +317
+ Partials 392 390 -2
Continue to review full report at Codecov.
|
|
I mean, essentially query is the same (it will be optimized to same thing), but ye avoiding table scan with indexes should help, but index on |
|
Is 'filler' exactly the value, or sth it starts with like filler% ? Then it might work |
| $table = $schema->getTable("${prefix}cards_properties"); | ||
| // Check for existing index spanning these columns | ||
| foreach ($table->getIndexes() as $index) { | ||
| // Check if we have a matching index already |
| /** | ||
| * Add index to oc_cards_properties to assist with searching with large numbers of rows | ||
| */ | ||
| class Version20180119134849 implements ISchemaMigration { |
There was a problem hiding this comment.
We need a cost estimation for this index - time expected given number of rows - given this could be released in a minor version we need to give useful information to the admins
There was a problem hiding this comment.
not worth mentioning
mysql> SELECT count(*) FROM oc_cards_properties;
+----------+
| count(*) |
+----------+
| 164306 |
+----------+
1 row in set (0.06 sec)
mysql> CREATE INDEX carddata_aid_n_v_idx ON oc_cards_properties (addressbookid, name, value);
Query OK, 0 rows affected (0.40 sec)
Records: 0 Duplicates: 0 Warnings: 0
|
needs rebase .... |
We can use the same logic as the accounts table here if this helps. We have Separate PR imo |
|
Testing on my machine with 20k user accounts (provisioned and logged in at least once):
I will adjust the PR to only cover those two fields - then we save some space in the index especially as value is VARCHAR 255. |
|
well, medial search can only benefit if the index contains the value |
|
@tomneedham so you just confirmed that only on 2 cols is enough? Can you try also with only addressbookid? And yes, maybe for medial search we can put full table in memory, but admins need to know that the table will be full in memory and each single update in that table will need to update that index structure.. each time.. |
|
@butonic @tomneedham for medial search, maybe better we put index only on value though with filler% - then it should be fast - but it will work only for filler%, not for %filler% which might only be solved with index on addressbookid. |
Yes - when not considering medial search.
That is fine - we should also use a config option to control this functionality like we have with |
|
@mrow4a please have a look at the explain output. Can you give me a reason for indexing only the addressbookid? |
|
What is selectivity of unique addressbookid vs selectivity of unique addressbookid&cp.name pair vs full table in memory indexed. (i.e. how many rows returned for average unique value/pair for addresbookid vs addressbookid&cp.name) |
|
Another problem I see is that it is hard to decide on where to put index, since we would need to know in advance if medial search is allowed or not. Honestyly I would go for %filler% and put index on addressbook alone if that works for that workload or addressbook&cp.name |
|
As discussed in #35777 the master branch will from now on hold the ownCloud 10 codebase. This PR targetted ownCloud 11 which is postponed to a far distant future. Because of that I'm closing this PR and kindly ask you to re-submit this PR in a few days. Thanks a lot for your patience |
|
Reopening due to https://github.com/owncloud/enterprise/issues/3625. The work should add index for addressbookid_name_value and add medial_search flag that allows to switch from medial term wildcard (%term%) to end of term wildcard (term%) in order to utilize index on value. Targeting 10.5 (@micbar @butonic ) related:
|
1fb983c to
8cc468d
Compare
8cc468d to
43f46c7
Compare
|
@micbar target 10.5 |
43f46c7 to
1e1112d
Compare
3b723a8 to
0b9d89a
Compare
| }, $searchProperties))) | ||
| ->andWhere($query->expr()->iLike('cp.value', $query->createNamedParameter($searchPattern))) | ||
| ->setFirstResult($offset)->setMaxResults($limit) | ||
| ->groupBy(['c.uri']) // groupby without aggregation is essentially distinct |
There was a problem hiding this comment.
@butonic in your commit, you did use ->groupBy(['c.uri', 'c.carddata']) that is not supported by ORACLE (you cannot groupby on BLOB). Is it possible that there are duplicates? If this is about VCARD version we can probably also try ->groupBy(['c.uri', 'size']) . Or maybe not groupby at all and filter on application level?
0b9d89a to
655be07
Compare
|
Due to lack of knowledge about the app and legacy considerations regarding query, I decided not to continue this work in a given form in order not to break something. Just index has been added in #37152. If someone decides to continue with this PR, it fails due to |
|
query has been changed since initially looking at this. |
The current search query for addressbook entries is problematic:
0.29 sec. ouch. on large addressbooks (>100k cards) tables this query takes more than a second ... per addressbook. 20 addressbooks => 20 seconds until the user sees a result.
We can do better. Let us have a look at the query. Maybe we can simplify it? What about:
Well, nothing changed. But at least it is more readable ... so what about the indexes. Adding one for the materialized subquery became clearer:
Much better. We now have an index that correctly partitions the table.