The test was using a `findIgnored` method in `AppProvider`, which only
existed for the purpose of testing. The test has been changed to instead
check for apps which would end up in the "can update" list (which is really
where the "ignored" apps are useful).
In the process, realised that using appId as a foreign key is worse than
packageName, because appId can get removed and added again, but it will
be different when the same app is inserted a second time. In order to
maintain the association of which apps have preferences stored against
them, they need to be stored against something with a bit more semantic
meaning. Thus, join onto package name instead.
This is a more concise syntax to say the same thing, and avoids an
OR clause in the where - which is often the cause of slowness in
many queries. Not sure if it was problematic in these cases, however
this COALESCE syntax is still more consise.
With no indexes at all, a join between X and Y tables would require a full
table scan of Y for each row in X. With an index on the relevant field in
Y, it would require an index lookup on the join field in Y for each row in
X, which contains a pointer to the row of interest in Y. This row is then
looked up and the relevant value extracted. By using a covering index (one
which includes all fields required to satisfy the query, with the first field
being the one which is looked up in the join), then once the index has been
searched, there is no need to then go to table Y because all the relevant
data is already in the index.
This offers a marginal performance improvement.
Merge download broadcast receivers
Previously, for all 4 states broadcast receivers were registered separately. These have now been merged into one receiver. IMHO this makes the code more readable and structured.
See merge request !368
Use AOSP Installer if permission check fails
If the old repo index is used and the permission check fails in ApkVerifier, this allows a fallback to the AOSP DefaultInstaller to show all permissions.
This has been discussed in https://gitlab.com/fdroid/fdroidclient/issues/704
Unfortunately, this shows our permission screen before download and then afterwards when the ``ApkVerifier`` fails the permission screen of the AOSP DefaultInstaller, i.e., the user sees two permission screen which she needs to acknowledge. This should only happen if an old repo format is used, thus I think this is okay. I don't know of any other solution to this problem.
See merge request !369
Make App and Apk parcelable and fix related installer NPEs
Installations fails currently due to
```
885 ACRA E Caused by: java.lang.NullPointerException
885 ACRA E at org.fdroid.fdroid.installer.InstallManagerService.getAppName(InstallManagerService.java:327)
885 ACRA E at org.fdroid.fdroid.installer.InstallManagerService.createNotificationBuilder(InstallManagerService.java:318)
885 ACRA E at org.fdroid.fdroid.installer.InstallManagerService.onStartCommand(InstallManagerService.java:158)
885 ACRA E at android.app.ActivityThread.handleServiceArgs(ActivityThread.java:2039)
885 ACRA E ... 10 more
```
This bug has been introduced in https://gitlab.com/fdroid/fdroidclient/merge_requests/359 where the packageName has been removed from the toContentValues() method.
The usage of ContentValues to send App/Apk objects to services was an hack in my opinion.
Thus, this PRs introduces proper parceling of App and Apk classes.
@pserwylo @eighthave @mvdan
See merge request !362
The usage of ContentValues to send App/Apk objects
to services was an hack in my opinion.
This hack broke in https://gitlab.com/fdroid/fdroidclient/merge_requests/359
where the packageName has been removed from the
toContentValues() method, which leads to NPEs in
the services.
Remove now unused package name from apk table
The package name is only stored in the `fdroid_app` table now, so we need to remove it form the `fdroid_apk` table. Under normal circumstances, I'd normally just leave unused fields in the DDL (the SQL which defines the tables) and never use it from within the Java code. However in this case, the package name formed part of the primary key of this table. Seeing as we are not inserting into that column any more, it isn't okay to leave it there but instead it must be removed so that we can put a more appropriate primary key on the table. In this case, the new primary key is `appId` + `vercode` + `repoId`.
I think this is the final merge request before I submit a MR with repo priorities.
See merge request !359
Wherever the "package name" of an apk is required, it can be requested by
asking for `Schema.ApkTable.Cols.App.PACKAGE_NAME`. Note the `App` which
indicates that it is in fact pulling this data from the `fdroid_app` table rather
than the `fdroid_apk` table.
Correctly identify the repo for a given URL to fix HTTP Auth.
**NOTE: Based on !355 (If that one gets merged first, I can rebase this, or else we can merge this one for both commits)**
When downloading arbitrary URLs using F-Droid (e.g. icons, .apk files, indexes) then it may be the case that the repo requires authentication. As such, we try to infer the repository based purely on the URL.
The old code took the basename of the URL, which means remove the last fragment (e.g. "index.jar") and use the remaining portion of the URL to lookup the repo.
This is broken for many reasons, partly because of the presence of a query string, partly because there are other things which are not just in the root directory of the repo (e.g. "/icons/*.png").
This new method iteratively peels off the right most segment of the URLs path, then looks to see if a repo exists at that address.
Note that this breaks down if you have nested repositories on a server, where one of the repositories is nested inside a directory that F-Droid knows about, such as "icons". In such a case, the following repositories:
* https://f-droid.org/repo (requires auth)
* https://f-droid.org/repo/icons (doesn't require auth)
will break down. If requesting something from the repo requiring auth:
* https://f-droid.org/repo/icons/org.fdroid.fdroid.png
Then it will lookup the database and find the repo which lives in "/icons" and doesn't require auth (or requires a different auth username/password). Not sure there is a lot that can be done about this without major refactoring. Such refactoring would require making sure a `Repo` is always given to a downloader for any HTTP request, and is probably a bit out of scope of this bug.
Also added tests for this behaviour.
Fixes#711.
See merge request !357
Firstly, this causes #721, possibly due to a bug in "Barcode
Scanner" whereby it seems to ignore the scheme when in caps,
assuming it is "http".
The relevant RFC is:
> RFC3986 (Uniform Resource Identifier (URI): Generic Syntax
In section 3.1, it describes the scheme:
> Although schemes are case-insensitive, the canonical form is
> lowercase and documents that specify schemes must do so with
> lowercase letters. An implementation should accept uppercase
> letters as equivalent to lowercase in scheme names (e.g., allow
> "HTTP" as well as "http") for the sake of robustness but should
> only produce lowercase scheme names for consistency.
Secondly, it is not valid to uppercase URLs at will. Although it
seems that there is some sort of more-compact-QR-generating-logic
that doesn't justify this. Funnily enough, I can't find anything
in RFC3986 about the case-insensitivity of URI paths. However
consider the following:
* https://i.imgur.com/fn33EcW.jpg
That is a valid path to an image. If we upper case it:
* HTTPS://I.IMGUR.COM/FN33ECW.JPG
or lower case it:
* https://i.imgur.com/fn33ecw.jpg
Then the server is entitled to treat it differently and indeed
it does. Both the upper case and lower case are no both 404's.
When downloading arbitrary URLs using F-Droid (e.g. icons, .apk files, indexes)
then it may be the case that the repo requires authentication. As such, we try
to infer the repository based purely on the URL.
The old code took the basename of the URL, which means remove the last fragment
(e.g. "index.jar") and use the remaining portion of the URL to lookup the repo.
This is broken for many reasons, partly because of the presence of a query string,
partly because there are other things which are not just in the root directory
of the repo (e.g. "/icons/*.png").
This new method iteratively peels off the right most segment of the URLs path,
then looks to see if a repo exists at that address.
Note that this breaks down if you have nested repositories on a server, where
one of the repositories is nested inside a directory that F-Droid knows about,
such as "icons". In such a case, the following repositories:
* https://f-droid.org/repo (requires auth)
* https://f-droid.org/repo/icons (doesn't require auth)
will break down. If requesting something from the repo requiring auth:
* https://f-droid.org/repo/icons/org.fdroid.fdroid.png
Then it will lookup the database and find the repo which lives in "/icons"
and doesn't require auth (or requires a different auth username/password).
Not sure there is a lot that can be done about this without major refactoring.
Such refactoring would require making sure a `Repo` is always given to a downloader
for any HTTP request, and is probably a bit out of scope of this bug.
Also added tests for this behaviour.
Fixes#711.
Prevent crash for servers that don't send etags with repo indexes
While working on #711, I noticed this bug using my [Mock HTTP Auth server](https://f-droid.org/wiki/page/PHP_Mock_Repository).
This fix always captures timestamps during repo updates, even if it is the same as last time. This is because we are dependent on it later on in the repo update process. Specifically, when updating from a HTTP server that doesn't send out etags with its responses, it will trigger a full blown repo update every time, even if all the values in the index are the same (name, description, etc). This is as distinct from better behaving servers that send etags, in which case we will only do a partial update (i.e. persist the "last updated time"). In such a case, the remainder of the update process will proceed, and ask for this timestamp.
See merge request !356
Always capture timestamps, even if it is the same. This is because we are dependent
on it later on in the repo update process. Specifically, when updating from a HTTP
server that doesn't send out etags with its responses, it will trigger a full blown
repo update every time, even if all the values in the index are the same (name,
description, etc). This is as distinct from better behaving servers that send etags,
in which case we will only do a partial update (i.e. persist the "last updated time").
In such a case, the remainder of the update process will proceed, and ask for this
timestamp.
Correctly delete single repo, not all repos.
In a recent commit, I cleaned up the code which deletes repo. At that point, instead of maybe concatenating strings together, sometimes with an `AND` statement,
it was changed to use the slightly better `QuerySelection`. This class is preferable because it doesn't need the developer to know whether there was
any previous constraints, and thus it knows whether to prepend an `AND`.
The problem arose because `QuerySelection` is effectively an immutable class. Calling `add()` on it returns a new copy with a different set of constraints.
The code which deleted the repo did not use this copy, and thus the resulting query had zero constraints.
The fix is to use the return value of `add()` correctly. It would've been easier to identify this bug if we had a lint check for "unused return values", though it is likely that that would get annoying very quickly.
Fixes#717.
See merge request !354
In a recent commit, I cleaned up the code which deletes repo. At that point,
instead of maybe concatenating strings together, sometimes with an `AND` statement,
it was changed to use the slightly better `QuerySelection`. This class is
preferable because it doesn't need the developer to know whether there was
any previous constraints, and thus it knows whether to prepend an `AND`.
The problem arose because `QuerySelection` is effectively an immutable class.
Calling `add()` on it returns a new copy with a different set of constraints.
The code which deleted the repo did not use this copy, and thus the resulting
query had zero constraints.
The fix is to use the return value of `add()` correctly. It would've been
easier to identify this bug if we had a lint check for "unused return values",
though it is likely that that would get annoying very quickly.
Fixes#717.
Use an integer primary key to join `fdroid_app` and `fdroid_apk` rather than the apps package name.
**Disclaimer:**
I realise this is a big change, but it needs to be done at some point, and it is not amenable to smaller changes, due to the fact that the app/apk relationship is so ingrained throughout F-Droid. Luckily, we have really quite comprehensive test coverage of the F-Droid `ContentProvider`s which helps to confirm that nothing should be majorly broken here.
**Some points of note:**
This is the first part of implementing #511, whereby the DB is refactored to better support multiple repositories.
Instead of joining `fdroid_app` and `fdroid_apk` tables using the package name, join based on an integer id autogenerated by sqlite. By default sqlite calls this `rowid` and it exists for every table, unless you've specified your own `NUMBER AUTO INCREMENT PRIMARY KEY` field. We have not done this for `fdroid_app`, so `rowid` is indeed the key we use in this MR. The package name was previously `id` in both the app and apk tables. Now `fdroid_app` makes use of `rowid` and `fdroid_apk` has a foreign key called `appId`.
The `ApkProvider` used to get away with only really querying the `fdroid_apk` table, and thus it didn't have to prefix any of the field names in the query with the table name. However now it always joins onto the `fdroid_app` table also, and as such, there are many places where field names needed to be prefixed with the table name (e.g. the `apk` alias or the `app` alias) to ensure the SQL is unambiguous when fields with the same name exist in both tables. The catch is, we want to reuse helper functions that build fragments of SQL, such as "Query based on package name". These helper functions are used both when updating and deleting apks (where field table prefixes are not allowed) and also in select statements (where they are required). Thus this changes comes with an `includeTableAlias` argument added to many of these methods (e.g. `ApkProvider.queryApp`).
There is still a package name column in the `fdroid_apk` table (the `id` field). This will be removed in future MRs and replaced with the package name from the joined `fdroid_app` table.
The `RepoPersister` used to dump apps in the db, then dump apks into the db. Now it needs to be a bit more nuanced, and dump apps into the db, _then ask the db what `rowid` was assigned to the apps_. This is then used when dumping the apks into the db. This also required some changes to how the `TempAppProvider` and `AppProvider` interact. In the interests of reusing code, both of these are able to provide operations on a similarly structured table but one is an in memory table (`temp_fdroid_app`) and the other is on disk (`fdroid_app`). In the past this was simpler, because the only interaction with the `TempAppProvider` was by using lists of `ContentOperation`s. Whereas now that we need to ask more substantial questions of the `TempAppProvider` other than "Insert this thing" or "update that thing", we needed to implement the `query` method in `TempAppProvider` similar to how it is in the base class `AppProvider`. As such, the common code for the base class and subclass `query` methods was extracted into `AppProvider.runQuery()`.
I tried to minimize the changes to the test suite as much as possible, so that it is possible to verify that they pass under the same conditions as before this change. However some changes were required to support the notion that apks depend on an app and its rowid, whereas this was not the case before. Thus there is some more boilerplate in the tests to ensure that inserting an apk ensures an app entry is present in the db too.
See merge request !345
Right now there is only one test in there anyway, so hopefully this is
a good tradeoff in terms of our time wasted vs not being able to run
those tests.
