support for APK Extension files aka "OBB"
OBB files are used by lots of apps like games and MAPS.ME to distribute large chunks of data. This adds basic support for distributing OBB files via F-Droid. The idea is that they are installed before the APK, so that once the APK is installed, the OBB files are already in place and ready to use. This also provides an F-Droid-specific Intent method for apps to fetch the OBB download URLs in case the app itself needs to handle the OBB download/update. That is similar to how it works in Google Play.
The fdroidserver changes are already merged: https://gitlab.com/fdroid/fdroidserver/merge_requests/143https://developer.android.com/google/play/expansion-files.html
See merge request !383
If a user clicks install, then uninstall on AppDetails, then there was not
yet a chance to refresh the App instance, and therefore app.installedApk
will still be null. This is really just a workaround for now, because
AppDetails needs a full refactoring.
This implements the APK Extension Files spec for finding, downloading, and
installing OBB files that are extension packs for APKs.
This needs WRITE_EXTERNAL_STORAGE since "installing" OBB files is just
copying them to a specific path on the external storage.
https://developer.android.com/google/play/expansion-files.html
This takes the APK file hash checker and turns it into a generic static
utility method for checking that a given file matches a given hash. This
will be needed as F-Droid handles other file types, like OBB and media.
By sending an Intent to F-Droid, it will reply with the full download URL
to the OBB file, if one exists for the currently installed version of the
requesting app.
This makes it easier to track the relationship between the index XML and
the database tables where that data is ultimately stored and used. There
are a few mismatches between the XML tag and database column names, so
those are just marked with a comment.
This makes it much easier to find all the spots in the code that need
changing when adding new columns/data to the APK table, like the OBB stuff.
In Android Studio, just Ctrl-Click on any table constant definition, and
then it lists all the places its used. Any new data will need to be added
in all of those locations.
OBB files are used in apps that need more than 100 megs to work well. This
is apps like MAPS.ME or games that put map info, media, etc. into the OBB
file. Also, OBB files provide a mechanism to deliver large data blobs that
do not need to be part of the APK. For example, a game's assets do not
need to change often, so they can be shipped as an OBB, then APK updates do
not need to include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
Properly support multiple repositories with the same app
Fixes#511.
Provides proper support for multiple repos in the database + updater (despite not being able to reorder them in the UI yet).
# Info
Previously, each app had only one row in the app table. This caused problems when its metadata was updated, because it would get overriden if two repositories provided the same app. This change:
* Creates a new `package` table which acts as the table that includes one row for each app (like the `app` table did previously)
* Re-purposes the existing `app` table as an `app_metadata` table, where each package can have multiple rows (one for each repository that provides that package)
This results in some queries which are slightly more complex, but the overall performance should not change too much. In the long term, it should have a net positive impact on performance, because we no longer need to join between tables based on a package name. Now we almost exclusively use integer IDs to join between tables. There are also appropriate indexes which make the queries avoid full table scans in all cases I'm aware of.
I realise this is a big MR, but it is as small as I could make it without submitting half finished stuff which breaks `master`. I've done my best to merge smaller MRs before hand, but I was unable to identify anything else to pull out of this MR as a separate thing.
## Migration
All app/apk metadata is dropped, and then the repos are asked to update themselves again next time F-Droid starts.
Additionally, existing repositories have their `priority` changed to something more meaningful. On current master, if you add two custom repositories in addition to the four default ones, you will get the following:
|rowid|address|priority|
|---------|-------|-------|
|1|https://f-droid.org/repo|10|
|2|https://f-droid.org/archive|20|
|3|https://guardianproject.info/fdroid/repo|10|
|4|https://guardianproject.info/fdroid/archive|20|
|5|http://10.0.0.6:8888/normal-repo/repo|10|
|6|http://10.0.0.6:8888/conflicting-repo/repo|10|
Note how the priority defaults to 10 for each additional repository which is added. This MR should change the priorities so they look like so:
|rowid|address|priority|
|-------|-------|--------|
|1|https://f-droid.org/repo|1|
|2|https://f-droid.org/archive|2|
|3|https://guardianproject.info/fdroid/repo|3|
|4|https://guardianproject.info/fdroid/archive|4|
|5|http://10.0.0.6:8888/normal-repo/repo|5|
|6|http://10.0.0.6:8888/conflicting-repo/repo|6|
Here is a snipped from the logcat from `DBHelper` during the migration:
```
Setting priority of repo https://f-droid.org/repo to 1
Setting priority of repo https://f-droid.org/archive to 2
Setting priority of repo https://guardianproject.info/fdroid/repo to 3
Setting priority of repo https://guardianproject.info/fdroid/archive to 4
Setting priority of repo http://10.0.0.6:8888/normal-repo/repo to 5
Setting priority of repo http://10.0.0.6:8888/conflicting-repo/repo to 6
```
All newly added repositories on this branch will get the next highest available priority.
# Future work
One thing which is not yet supported fully: Suggested version code.
Two repositories can end up with the same exact .apk file. If that .apk is the "suggested version", then we should eliminate the idea of "suggested version code" and instead have a "suggested apk" (which implicitly includes the repository it comes from, so we choose the one with the better priority). Right now, we kind of assume that it doesn't matter which repo provides the suggested apk, as long as one of them has an .apk with th ecorect version code and signing key.
I guess it doesn't _particularly_ matter from a security perspective, because a malicious repo wont be able to trick a user into installing an apk with a different signing key, but it would be good to iron this out.
See merge request !375
Clarify teminology (around providers) and misc cleanup
I extracted all of what I consider cosmetic changes from !375 into this changeset. There should be no behaviour change, except for:
* Bug fix in `DBHelper`
* Different handling of priorities for newly created repos
Everything else if method renames or other misc cleanups.
After this, I'll pull out a different set of changes from !375 and then rebase it on this again.
See merge request !400
Many times in the past, we would ask for an apk based on its package name
and its version code. However multiple repositories provide apks with the
same package name and version code, and such queries would (seemingly)
nondeterministically choose one of these matching apks. This clarifies the
wording in the code around when we explicitly ask for a given apk, and
when we kind of guess which one we want.
Most the time we have an `App` handy, which has a specific repo associated
with it. This allows us to be more specific about requesting `Apk`s.
The times we are kind of guessing is when we rely on the "suggested version
code" of an apk by clicking the misc "Install" or "Upgrade" button in
app details. In the future, we'll need to clear this up so that a more
specific apk is chosen when touching these buttons.
The query which dynamically figured out the preferred metadata based on
repo priority ended up being quite slow (although it did work). On lower
end devices, it has the potential to make F-Droid quite sluggish. By
optimistically precalculating the preferred metadata where possible, we
don't need to ask the question during the usual usage of F-Droid, only
when:
* Repo priorities are changed (there is not currently a UI for this, but
there are tests)
* Repos are enabled/disabled
* Repo updates are performed
Includes:
* One of the functions querying for apps did not correctly specify
the repository the repos came from.
* Fix deletion code which refered to incorrect field.
* Cleanup code style in some places.
Two repositories can (and always could) end up with the same exact .apk file.
If that .apk is the "suggested version", then we should eliminate the idea of
"suggested version code" and instead have a "suggested apk" (which implicitly
includes the repository it comes from, so we choose the one with the better
priority). Right now, we kind of assume that it doesn't matter which repo
provides the suggested apk, as long as one of them has an .apk with the correct
version code and signing key.
It shouldn't _particularly_ matter from a security perspective, because
a malicious repo wont be able to trick a user into installing an apk with a
different signing key, but it would be good to iron this out.
This commit adds a TODO explaining this for th ebenefit of any CRer.
The tests are in the .updater packate to make them easier to run as
a suite in Android Studio. Now the package can be right clicked and
ran to run all the tests to do with updating.
The index jar files were updated so as to include info in most
metadata fields (e.g. URLs/descriptions/summary/etc) to show that
that particular part of metadata came from a specific repo. This
will allow more specific tests to show that we can indeed query for
an app with metadata provided by the repo with the highest priority.
Required for future work which will be better able to deal with multiple repos
providing the same app.
Instead of migrating data into that table, we will drop and recreate the tables.
This is because before this feature is out, we'll need to do that anyway.
It is often helpful during debugging to be able to dump the contents
of an SQL result `Cursor` to the debug watch list. This is difficult
to do under normal circumstances. This adds a utility method really
only designed to be used during interactive debugging, which will do
its best to build a `Map` for each row in the `Cursor`. This can then
be used to test queries while the debugger is paused.
Even though this is not used yet, it will be a requirement in the
near future for the `RepoProvider` to be the one who decides what
the priority of new repositories is. This will prevent clients of
this provider from specifying wrong priorities that result in gaps
For example, if we accidentally ended up with priorities of
1, 2, 4, and then 5, this would cause problems if the user tried to
drag the second repo to the position of the 4th repo. It is easier
to do these priority shuffles if we can assume that the priorities
are contiguous.
Originally, I hoped that the arguments a method took would help enough
to differentiate the intent of that method. This was the case for methods
such as `getContentUri()` and `find()`. However they are a little confusing
to work with, so this change renames a bunch of methods to be more specific.
In addition, it makes some renames from app -> package which will help with
the upcoming change to add a `package` table to the database.
optionally keep install history
This adds the functionality for keeping the install/uninstall history along with a preference to enable it as a custom build option for allowing another app to read it. Keeping the history has many uses, including:
* "popularity contest"
* displaying install history locally in F-Droid
* reporting to IT device manager for tracking activity for malware, etc.
This is ready to be merged, but it is based on !386, so its marked WIP.
@dschuermann this touches some of your architecture, please review :)
See merge request !392
Since e69a6d5a8f24e7745516001f58bee49e05f2ea9e, the Apk instance is
provided in the constructor and is available as a final instance variable.
No need to pass it around. Thanks to @pserwylo for spotting this.
ACTION_INSTALL_STARTED was being sent twice per transaction with the
default installer. Also, it should be sent as the first step of the install
process.
For now, this is disabled by default and hidden in the expert preferences
since it doesn't do anything yet inside of F-Droid. It is useful now for
whitelabel builds to fetch the install history from another app. #396
This allows a designated app to read the install history from F-Droid via a
ContentProvider. The app is designated by the packageName defined in the
string install_history_reader_packageName.
The install and uninstall history has lots of uses, including displaying
to the user in the app itself, reporting to the Device Administrator to
enable tracking of installs/uninstalls from the admin's app repo, etc. It
can also be used as part of a "popularity contest" #396
Now that the packageName is included in the Installer broadcast Intents,
it can be used to fetch the app name from the database, if all other ways
fail.
If F-Droid or InstallManagerService get killed while an install is in
progress, that install will ultimately broadcast back to
InstallManagerService to manage the notifications. The state is gone
since things have been killed, so include the Apk instance in the
Intent that is included in the broadcasts so that
InstallManagerService can fetch all required info from the database.
closes#698
Push requests for install/uninstall
This is the first basic implementation of the idea of "push requests" from the server. This implements two requests that the server can ask of all clients: `install` and `uninstall`. A default repository in the client can be marked to `ignore` or `always` accept push requests in this merge request. There is also the sketch of a third option called `prompt` which gives the user the standard "Just Once/Always" choice; that is not yet implemented.
This allows central management of app installs/uninstalls for a pool of devices, as well as other ideas.
A use case for this feature is documented here:
https://f-droid.org/wiki/page/Whitelabel_Builds
See merge request !386
F-Droid shouldn't crash if a push request includes a bad package name. This
just makes it silently ignore those push requests. If its a debug build,
it will send a message to logcat. I'm not sure this is best way to handle
this, but this is better than crashing the app. This will make it harder
for repo operators to debug issues with push requests.
This allows whitelabel versions of apps to specify built-in app repos that
have push requests accepted by default. This is useful for the case where
there is a central manager of the core apps that are installed.
https://gitlab.com/fdroid/fdroidserver/issues/177
