Support extended <uses-permissions/> tags
`<uses-permission-sdk-23>` is a new tag for requesting permissions on _android-23_ and above. `<uses-permission/>` and `<uses-permission-sdk-23>` both can have optional _maxSdkVersion_ values, and these need to be fully supported in order for the Privileged Extension to be able to check the APK permissions against the index.xml permissions. This is needed so that it can prompt the user once, then download and install the APK after that. Its also needed for transparent background updates to check if the permissions have changed in the update.
This gets closer to the complete support, really the full permissions should be stored in the database. Then the only
conversion would happen ewhen parsing the XML. Right now, it still stores the old F-Droid permissions names, i.e. without
`android.permission.`. Then those are converted when they are loaded from the DB into an Apk instance.
See merge request !402
<uses-permissions/> tags can have min and max SDK to take effect. This is
not supported currently, and it necessary especially with the privileged
installer so it can properly represent the permissions that an APK is
requesting.
For example:
<uses-permission
android:name="android.permission.MANAGE_ACCOUNTS"
android:maxSdkVersion="22" />
<uses-permission-sdk-23
android:name="android.permission.CAMERA" />
<uses-permission-sdk-23
android:name="android.permission.CALL_PHONE"
android:maxSdkVersion="23" />
android.content.pm.PackageInfo is the Android class for representing data
about an APK/package. Since Apk.permission is the same thing, we should
use the same name.
Android won't protect us from other apps sending other Intents to these
receivers, so at least check that the action string matches what its
looking for. This is based on a lint recommendation.
Test all database migrations from db-version/42 onwards
While a bit arbitrary choosing 42, it is from more than two years ago. We can use this as our new baseline, in that this test will always check upgrades from 42 onwards, with each database bump.
Once the test was up and running, it immediately identified a bug in the database migration for v50 and v57. These have been fixed in this branch too.
Fixes#778.
See merge request !403
The migration resulted in a query being run which was broken. The query
was broken because it was dynamically generated by Java code. This Java
code resulted in a valid migration when until very recently when the
query was refactored to deal with a new DB structure. Now the query is
no longer suitable to be run against a DB_VERSION 49 database.
To resolve this, the migration now hard codes the query to a string
which is executable when the DB_VERSION is 49.
It was a little arbitrary to choose this date. However it was when the database
looked quite close to what it looks like now and it is from well over two years
ago. Going into the future, this test may as well always start out at 42 forever
more to ensure that database migrations from that point continue to work for
all future database migrations.
For upgrades from DB version earlier than 63, the whole table is recreated
by resetTransient() in migrateToPackageTable() so the upgrade method for
the OBB tables only needs to run when the database is at exactly version 63
This was mistakenly added to cd9582c9902dd4ac9218acfd69872f3eebcd3d93 when
it was rebased on !375.
support for APK Extension files aka "OBB"
OBB files are used by lots of apps like games and MAPS.ME to distribute large chunks of data. This adds basic support for distributing OBB files via F-Droid. The idea is that they are installed before the APK, so that once the APK is installed, the OBB files are already in place and ready to use. This also provides an F-Droid-specific Intent method for apps to fetch the OBB download URLs in case the app itself needs to handle the OBB download/update. That is similar to how it works in Google Play.
The fdroidserver changes are already merged: https://gitlab.com/fdroid/fdroidserver/merge_requests/143https://developer.android.com/google/play/expansion-files.html
See merge request !383
If a user clicks install, then uninstall on AppDetails, then there was not
yet a chance to refresh the App instance, and therefore app.installedApk
will still be null. This is really just a workaround for now, because
AppDetails needs a full refactoring.
This implements the APK Extension Files spec for finding, downloading, and
installing OBB files that are extension packs for APKs.
This needs WRITE_EXTERNAL_STORAGE since "installing" OBB files is just
copying them to a specific path on the external storage.
https://developer.android.com/google/play/expansion-files.html
This takes the APK file hash checker and turns it into a generic static
utility method for checking that a given file matches a given hash. This
will be needed as F-Droid handles other file types, like OBB and media.
By sending an Intent to F-Droid, it will reply with the full download URL
to the OBB file, if one exists for the currently installed version of the
requesting app.
This makes it easier to track the relationship between the index XML and
the database tables where that data is ultimately stored and used. There
are a few mismatches between the XML tag and database column names, so
those are just marked with a comment.
This makes it much easier to find all the spots in the code that need
changing when adding new columns/data to the APK table, like the OBB stuff.
In Android Studio, just Ctrl-Click on any table constant definition, and
then it lists all the places its used. Any new data will need to be added
in all of those locations.
OBB files are used in apps that need more than 100 megs to work well. This
is apps like MAPS.ME or games that put map info, media, etc. into the OBB
file. Also, OBB files provide a mechanism to deliver large data blobs that
do not need to be part of the APK. For example, a game's assets do not
need to change often, so they can be shipped as an OBB, then APK updates do
not need to include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
Properly support multiple repositories with the same app
Fixes#511.
Provides proper support for multiple repos in the database + updater (despite not being able to reorder them in the UI yet).
# Info
Previously, each app had only one row in the app table. This caused problems when its metadata was updated, because it would get overriden if two repositories provided the same app. This change:
* Creates a new `package` table which acts as the table that includes one row for each app (like the `app` table did previously)
* Re-purposes the existing `app` table as an `app_metadata` table, where each package can have multiple rows (one for each repository that provides that package)
This results in some queries which are slightly more complex, but the overall performance should not change too much. In the long term, it should have a net positive impact on performance, because we no longer need to join between tables based on a package name. Now we almost exclusively use integer IDs to join between tables. There are also appropriate indexes which make the queries avoid full table scans in all cases I'm aware of.
I realise this is a big MR, but it is as small as I could make it without submitting half finished stuff which breaks `master`. I've done my best to merge smaller MRs before hand, but I was unable to identify anything else to pull out of this MR as a separate thing.
## Migration
All app/apk metadata is dropped, and then the repos are asked to update themselves again next time F-Droid starts.
Additionally, existing repositories have their `priority` changed to something more meaningful. On current master, if you add two custom repositories in addition to the four default ones, you will get the following:
|rowid|address|priority|
|---------|-------|-------|
|1|https://f-droid.org/repo|10|
|2|https://f-droid.org/archive|20|
|3|https://guardianproject.info/fdroid/repo|10|
|4|https://guardianproject.info/fdroid/archive|20|
|5|http://10.0.0.6:8888/normal-repo/repo|10|
|6|http://10.0.0.6:8888/conflicting-repo/repo|10|
Note how the priority defaults to 10 for each additional repository which is added. This MR should change the priorities so they look like so:
|rowid|address|priority|
|-------|-------|--------|
|1|https://f-droid.org/repo|1|
|2|https://f-droid.org/archive|2|
|3|https://guardianproject.info/fdroid/repo|3|
|4|https://guardianproject.info/fdroid/archive|4|
|5|http://10.0.0.6:8888/normal-repo/repo|5|
|6|http://10.0.0.6:8888/conflicting-repo/repo|6|
Here is a snipped from the logcat from `DBHelper` during the migration:
```
Setting priority of repo https://f-droid.org/repo to 1
Setting priority of repo https://f-droid.org/archive to 2
Setting priority of repo https://guardianproject.info/fdroid/repo to 3
Setting priority of repo https://guardianproject.info/fdroid/archive to 4
Setting priority of repo http://10.0.0.6:8888/normal-repo/repo to 5
Setting priority of repo http://10.0.0.6:8888/conflicting-repo/repo to 6
```
All newly added repositories on this branch will get the next highest available priority.
# Future work
One thing which is not yet supported fully: Suggested version code.
Two repositories can end up with the same exact .apk file. If that .apk is the "suggested version", then we should eliminate the idea of "suggested version code" and instead have a "suggested apk" (which implicitly includes the repository it comes from, so we choose the one with the better priority). Right now, we kind of assume that it doesn't matter which repo provides the suggested apk, as long as one of them has an .apk with th ecorect version code and signing key.
I guess it doesn't _particularly_ matter from a security perspective, because a malicious repo wont be able to trick a user into installing an apk with a different signing key, but it would be good to iron this out.
See merge request !375
Clarify teminology (around providers) and misc cleanup
I extracted all of what I consider cosmetic changes from !375 into this changeset. There should be no behaviour change, except for:
* Bug fix in `DBHelper`
* Different handling of priorities for newly created repos
Everything else if method renames or other misc cleanups.
After this, I'll pull out a different set of changes from !375 and then rebase it on this again.
See merge request !400
Many times in the past, we would ask for an apk based on its package name
and its version code. However multiple repositories provide apks with the
same package name and version code, and such queries would (seemingly)
nondeterministically choose one of these matching apks. This clarifies the
wording in the code around when we explicitly ask for a given apk, and
when we kind of guess which one we want.
Most the time we have an `App` handy, which has a specific repo associated
with it. This allows us to be more specific about requesting `Apk`s.
The times we are kind of guessing is when we rely on the "suggested version
code" of an apk by clicking the misc "Install" or "Upgrade" button in
app details. In the future, we'll need to clear this up so that a more
specific apk is chosen when touching these buttons.
The query which dynamically figured out the preferred metadata based on
repo priority ended up being quite slow (although it did work). On lower
end devices, it has the potential to make F-Droid quite sluggish. By
optimistically precalculating the preferred metadata where possible, we
don't need to ask the question during the usual usage of F-Droid, only
when:
* Repo priorities are changed (there is not currently a UI for this, but
there are tests)
* Repos are enabled/disabled
* Repo updates are performed
Includes:
* One of the functions querying for apps did not correctly specify
the repository the repos came from.
* Fix deletion code which refered to incorrect field.
* Cleanup code style in some places.
Two repositories can (and always could) end up with the same exact .apk file.
If that .apk is the "suggested version", then we should eliminate the idea of
"suggested version code" and instead have a "suggested apk" (which implicitly
includes the repository it comes from, so we choose the one with the better
priority). Right now, we kind of assume that it doesn't matter which repo
provides the suggested apk, as long as one of them has an .apk with the correct
version code and signing key.
It shouldn't _particularly_ matter from a security perspective, because
a malicious repo wont be able to trick a user into installing an apk with a
different signing key, but it would be good to iron this out.
This commit adds a TODO explaining this for th ebenefit of any CRer.
The tests are in the .updater packate to make them easier to run as
a suite in Android Studio. Now the package can be right clicked and
ran to run all the tests to do with updating.
The index jar files were updated so as to include info in most
metadata fields (e.g. URLs/descriptions/summary/etc) to show that
that particular part of metadata came from a specific repo. This
will allow more specific tests to show that we can indeed query for
an app with metadata provided by the repo with the highest priority.
Required for future work which will be better able to deal with multiple repos
providing the same app.
Instead of migrating data into that table, we will drop and recreate the tables.
This is because before this feature is out, we'll need to do that anyway.
It is often helpful during debugging to be able to dump the contents
of an SQL result `Cursor` to the debug watch list. This is difficult
to do under normal circumstances. This adds a utility method really
only designed to be used during interactive debugging, which will do
its best to build a `Map` for each row in the `Cursor`. This can then
be used to test queries while the debugger is paused.
Even though this is not used yet, it will be a requirement in the
near future for the `RepoProvider` to be the one who decides what
the priority of new repositories is. This will prevent clients of
this provider from specifying wrong priorities that result in gaps
For example, if we accidentally ended up with priorities of
1, 2, 4, and then 5, this would cause problems if the user tried to
drag the second repo to the position of the 4th repo. It is easier
to do these priority shuffles if we can assume that the priorities
are contiguous.
Originally, I hoped that the arguments a method took would help enough
to differentiate the intent of that method. This was the case for methods
such as `getContentUri()` and `find()`. However they are a little confusing
to work with, so this change renames a bunch of methods to be more specific.
In addition, it makes some renames from app -> package which will help with
the upcoming change to add a `package` table to the database.
