Skip to content

Feat query support in list attributes & list indexes endpoint - #5885

Merged
abnegate merged 30 commits into
1.4.xfrom
feat-query-support-attributes-indexes
Aug 11, 2023
Merged

abnegate merged 30 commits into
1.4.xfrom
feat-query-support-attributes-indexes

Conversation

@fanatic75

@fanatic75 fanatic75 commented Jul 26, 2023 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds support for queries in the List Attributes endpoint and List Indexes endpoint

Test Plan

Adds E2E test case to verify list attributes and list indexes with queries

Related PRs and Issues

  • (Related PR or issue)

Checklist

  • Have you read the Contributing Guidelines on issues?
  • If the PR includes a change to an API's metadata (desc, label, params, etc.), does it also include updated API specs and example docs?

@fanatic75
fanatic75 requested a review from abnegate July 26, 2023 20:01
Comment thread app/controllers/api/databases.php
Comment thread app/controllers/api/databases.php Outdated
Comment on lines +1680 to +1684
foreach ($queries as $query) {
if ($query->getMethod() === Query::TYPE_SELECT) {
throw new Exception(Exception::GENERAL_QUERY_INVALID, 'Select queries are not valid.');
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we disallow select queries?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Select queries are not supported due to the limitation of no "key" in the metadata table, which is being used to validate select queries. Also, @TorstenDittmann said most of the keys are not optional so it's not worth it.
https://github.com/utopia-php/database/blob/a9f706034a12c7c87e6fd38eca69587fd3def0e0/src/Database/Database.php#L4509

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the error if they're enabled? I think the key attribute should be there since it's in the collections config here:

'$id' => ID::custom('key'),

Select queries might not particularly useful for these resources but I would suggest we allow them if easy enough, so we can remove this code and keep the controller complexity to a minimum

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, we cannot really use that because validate selections uses the property on line 245 above you have shared.
In this case, we need a property 'key' just like '$id'.
There's a property 'key' in the metadata table for all collections and databases but not for the ones that are created when a project is created.

Also, one more suggestion, for validating select queries, why don't we use $id instead of 'key' because seems to be they are similar? Then we can support select queries

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2023-08-01 at 4 21 21 PM Screenshot 2023-08-01 at 4 20 33 PM

I think if we use $id instead of $key in validating select queries then we can maybe use that because $id exists in all rows of metadata attributes and indexes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can try get the key and fallback to $id if it's not found, we're doing this a few other places I think

@fanatic75 fanatic75 Aug 3, 2023 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the changes. But sadly list attributes endpoint will have to be much more complex for adding select queries support, we need to have type attribute for validating the response model for Attributes list.

If select query does not include type support, we will have to add type in select query and then remove type property from output.

discussed two options with @christyjacob4, either we use validator to not support select queries or we support select queries but we add type in select query when it doesn't include it.

We decided to keep endpont consistent with select queries, we need to add type in select queries.

@fanatic75 fanatic75 changed the title Feat query support attributes indexes Feat query support in list attributes & list indexes endpoint Aug 3, 2023
@stnguyen90
stnguyen90 self-requested a review August 3, 2023 16:27
@fanatic75
fanatic75 changed the base branch from 1.4.x to cl-1.4.x August 7, 2023 15:04
@fanatic75
fanatic75 changed the base branch from cl-1.4.x to 1.4.x August 7, 2023 15:27
@fanatic75 fanatic75 closed this Aug 7, 2023
@fanatic75 fanatic75 reopened this Aug 7, 2023
Comment thread app/controllers/api/databases.php Outdated
Comment thread app/controllers/api/databases.php Outdated
Comment thread app/controllers/api/databases.php Outdated
Comment thread app/controllers/api/databases.php Outdated

//Add relationship data from options to attributes as it loses options during response setup
foreach ($attributes as $attribute) {
if ($attribute->getAttribute('type') === Database::VAR_RELATIONSHIP) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the type for the response models or just to check if it's a relationship here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic adds relationship attributes to each attribute object as the options key gets filtered out from Attribute Relationship Type as defined in the model.
I have moved this logic in the filter function of the Relationship Model so we can avoid this from controller logic.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we left it in the controller, would we be able to remove the type attribute after fetching from the database but before we call $response->dynamic? It would be much cleaner that way as opposed to calling $response->output then $response->static after filtering

@fanatic75 fanatic75 Aug 7, 2023 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section of the code is distinct from the $type attribute. Its purpose is to transition from the options key, which contains relationship data, to the appropriate keys that the models necessitate.

In terms of output and static, it is not feasible to eliminate the type attribute before the output method executes. As much as I dislike doing it, we must include the type property in the select query because the output method demands it.

@fanatic75
fanatic75 changed the base branch from cl-1.4.x to 1.4.x August 8, 2023 21:35
@fanatic75
fanatic75 changed the base branch from 1.4.x to cl-1.4.x August 9, 2023 15:38
@fanatic75
fanatic75 changed the base branch from cl-1.4.x to 1.4.x August 9, 2023 15:40
Comment thread tests/e2e/Services/Databases/DatabasesBase.php Outdated
Comment thread tests/e2e/Services/Databases/DatabasesBase.php Outdated
Comment thread tests/e2e/Services/Databases/DatabasesBase.php Outdated
Comment thread composer.json Outdated

@christyjacob4 christyjacob4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. please address the comments

@fanatic75
fanatic75 requested a review from christyjacob4 August 9, 2023 18:48
@abnegate

Copy link
Copy Markdown
Member

@fanatic75 Looks like we'll need to fix up some of the tests for other changes in the database library (select queries now no longer select internal attributes by default)

@fanatic75

Copy link
Copy Markdown
Contributor Author

@fanatic75 Looks like we'll need to fix up some of the tests for other changes in the database library (select queries now no longer select internal attributes by default)

I have fixed the tests,
One thing I find weird is, If you select only a single attribute that has a null value, we get an error 404 that document is not found.

@abnegate

Copy link
Copy Markdown
Member

One thing I find weird is, If you select only a single attribute that has a null value, we get an error 404 that document is not found.

@fanatic75 This will likely happen whenever $id is not selected, as the document empty check just looks at the $id value to determine if the document is empty. I've fixed this to check for any value instead here

@stnguyen90 stnguyen90 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Can this be merged into branch cl-1.4.x, though?

@abnegate

Copy link
Copy Markdown
Member

Can this be merged into branch cl-1.4.x, though?

@stnguyen90 There's a lot of conflicts there, so we will merge this into 1.4.x and resolve them later all together 👍

Comment thread src/Appwrite/Utopia/Database/Validator/Queries/Attributes.php Outdated
Comment thread tests/e2e/Services/Databases/DatabasesBase.php Outdated
Comment thread tests/e2e/Services/Databases/DatabasesBase.php Outdated
@abnegate
abnegate merged commit 6b915ba into 1.4.x Aug 11, 2023
@abnegate
abnegate deleted the feat-query-support-attributes-indexes branch October 25, 2023 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants