Also use a newer version of testy.at.or.at index for the index-v1 test,
because it includes apks with "KnownVuln" anti features whereas the
older version did not.
This improves performance when we need to decide whether or not apps are
installed or not while scrolling through large lists.
Fixes#1143.
Also change Jackson tests to properly ignore App#isApk.
While investigating #1086 which was about swap being busted, I
discovered that we recently introduced a worse bug when working with
multi sig stuff. The swap process, when installing an app (or even when
listening for if a user started installing - before they even did
anything), would ask for an apk from any repo. This is wrong, because we
should only ask for the apks from the swap repo when presented with a
swap dialog.
By fixing this so that it asks for a specific apk, this may also
fix the issue in #1086, because that was about us not asking for enough
info from the database for each Apk which was returned. Now we just
return all columns, because the performance overhead should be minimal,
but it prevents this class of bugs, where we didn't fully populate
our value object. However, I'm not confident that it is fixed, because I
was unable to reproduce it due to the other crash fixed in this change.
Relevant crash:
```
java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String org.fdroid.fdroid.data.Apk.getUrl()' on a null object reference
at org.fdroid.fdroid.views.swap.SwapAppsView$AppListAdapter$ViewHolder.setApp(SwapAppsView.java:311)
at org.fdroid.fdroid.views.swap.SwapAppsView$AppListAdapter.bindView(SwapAppsView.java:422)
at org.fdroid.fdroid.views.swap.SwapAppsView$AppListAdapter.newView(SwapAppsView.java:414)
at android.support.v4.widget.CursorAdapter.getView(CursorAdapter.java:269)
at android.widget.AbsListView.obtainView(AbsListView.java:2349)
at android.widget.ListView.makeAndAddView(ListView.java:1864)
at android.widget.ListView.fillDown(ListView.java:698)
...
```
Now that we need only "insert" new apps rather than"
* Identify if an app exists
* If so, update
* If not, insert
There is much less code required for all of this stuff.
When preparing a temp database to write to, don't copy all apps/apks.
Instead, only copy those _not_ belonging to the repo we are updating.
In an ideal world, we'd not even need to copy them, but we need
their IDs to be in the temp database so that we don't accidentally
use the same auto-generated ID as the main database.
This also means that we can drop the check for "does this app exist,
and hence should we UPDATE it instead of INSERTing it?" and always
just insert it.
Then, when copying the temp table back to disk, first delete all
apps/apks _belonging to the repo being updated_. Then, copy back the
apks/apps we found in the repo. This again improves performance because
we no longer need to bopy back and forth data which we know
wont change (as evidenced by the fact it belongs to a differen trepo).
I don't think this was possible earlier before we did the work to
support repo priorities properly. That is because we had a single app
which was serviced by several repositories. Now, we have multiple
entries in the `fdroid_app` table, for each repo which supports
that app.
The test files used in Issue763MultiRepo.java are signed using MD5, which
is now considered invalid. So if that test is run on any recent Java, it
will fail with a signature error. This updates the test files to be signed
with SHA1 instead.
* index.microg.jar is a new version fetched from https://microg.org
* index.antox.jar was resigned with the testy.at.or.at key, since antox
website doesn't exist anymore.
RepoUpdater.prepareRepoDetailsForSaving() was broken here because the Repo
properties were being set before calling it, and then the Repo instance was
passed to it for comparison. So the comparison was always saying the value
was unchanged. In IndexV1Updater, the flow doesn't need those checks.
This also fixes the bug where added repos never had their name/description/
icon/etc show up in ManageRepos and RepoDetails.
@cde found this bug working on mirror support, thanks!
related to #35closes#1016
The install process automatically sanitizes filenames to avoid exploits
that put attack code in the filename. Media files are also installed using
this logic, so the installed check needs to use sanitized file names to be
accurate.
Before mult-signature support, the process of marking an app as
installed in the `InstalledAppProvider` didn't have any side effects
beyond its own table. Now, it is also responsible for calculating the
`suggestedVersionCode` of the associated app as well.
This means old tests around suggested versions no longer work. This is
because they would insert an App, and set the
`Cols.SUGGESTED_VERSION_CODE` using a `ContentValues`. This was then
overwritten by the `InstalledAppProvider` asking for the real
calculation for suggested versions. That is - it would check for
relevant apks and figure out which was best.
To make the old tests correct, they need:
* To be able to "install" apps with the correct signature.
* To insert the relevant apks into the database, not just depend on the
presence of an `app`.
Previously, it was only done on repo update.
Now it is done whenever an app is installed or unisntalled. The query to
update the suggested version for each app is quite slow when run at the
end of a repo update. However in this change, we are limiting the query
to only update a single app, which means that performance should not be
a problem.
When a single repo provides apks with multiple signatures, then we need
to be able to select the preferred one. This adds tests for this which
fail, because that feature has not yet been implemented.
This is really the intention of the method, given it used to accept
a version code and a package name. Now it optionally accepts a sig
also. If present, it will restrict the query to apks with that sig.
Also added to the multi-sig tests to ensure this method takes it into
consideration.
There is some magic conversions going on so that booleans get
converted into integers, but they are only on Android. Under
robolectric, it throws a class cast exception instead.
Some were removed and left removed if they were run during tests,
because the tests are supposed to be automated and the noise they added
would not have helped diagnose a failure.
Also removed the dead code around "uses-feature" which will never
get implemented, especially as it is in the XML index.
gitlab's diff views wrap badly when lines are longer than 118. Android
Studio places a grey line in the UI at 120.
@SuppressWarnings("LineLength") is added to a bunch of files to prevent
making this commit huge. People can remove that as they work on those
files, and fix the issues then.
I also ran Android Studio's default Ctrl-Alt-L code formatter, where it was
easy to do, and I was already in the file.
This allows us to test "installing" Adaway, which has a native code
dependency that the default Robolectric setup doesn't support (defaults
to armeabi-v7a).
fdroidserver currently only supports a single WhatsNew field that
comes from the CurrentVersionCode of the app. Google Play and
fastlane supply support a WhatsNew field per-release, but we don't use
that data anywhere, and implementing that in the data structures would
add a lot of complexity since Apk would then need to have its own
"localized" section like App does.
The "Video" field is just a URL pointing to a video.
closes#910
We no longer prefix the list of categories with "Whats New", "Recently
Updated", and "All". The new UI doesn't require this. The only place
they were being used now were in the tests.
This is based on @pserwlyo's work. The App and Apk classes currently
need just the public instance variables auto-filled by Jackson, so
everything else is considered opt-in, via @JsonProperty declarations.
This is currently only used for setLocalized(), setUsesPermission(),
and setUsesPermissionSdk23().
# Conflicts:
# app/src/test/java/org/fdroid/fdroid/updater/IndexV1UpdaterTest.java
The parser should accept additional elements to each
uses-permission entry, in case more XML attributes is added to
<uses-permission>. <uses-permission> has had two attributes
since the beginning.
Other changes to this JSON data structure are bad index-v1
format, and will cause crashes:
* Removing an element e.g. null from a uses-permission entry
would be invalid index-v1 JSON, since that structure mirrors
the uses-permission AndroidManifest.xml element, which has a
long standing fixed definition of name/maxSdkVersion. That
should crash so that fdroidserver authors know they are
generating invalid index-v1.
* setting versionCode to anything but an int is invalid index-v1
JSON, and should crash. versionCode has been defined as an
32-bit signed integer value since the beginning of Android.
* <uses-permission android:name=""> has been defined as a string
since the beginning of Android.
https://developer.android.com/guide/topics/manifest/uses-permission-element.html
repoId is used in Repo, App, and Apk instances to point to the Repo data
in the database. It does not come from the index files, but rather the
client database.
An important security protection is erroring when the index-v1.jar is
older than what is currently in the database. If the current or older
jar is allowed to be parsed, then a malicious server or
Man-In-The-Middle could replay old version of the index-v1.jar to
prevent the clients from learning about updates that fix security issues
Tell the Jackson JSON parser to ignore App/Apk fields that should never
come from the index, but instead are set locally to keep track of the
current state of things on the device.
There are two forms of tests to enforce that the proper things get
ignored. It is not possible to do this with decorators alone, so I
chose to use @JsonIgnore and leave the variables we want filled in
undecorated. Also, all of the instance variables in Apk/App/Repo
should come directly from the index metadata so that they are pure
data classes. Currently some state info is stored in them, those are
decorated with @JsonIgnore.
The tests then include lists of accepted and ignored properties, and
anything that is not in those lists will cause the tests to fail. So
if someone is adding a new instance variable, they will get a fail
until the tests are updated. One set of tests actually writes blank
instances out as JSON since that's the easiest test to write, and
Jackson treats @JsonIgnore the same in both directions. Then there is
another test that reads a JSON file with added, unsupported values to
make sure that they are properly ignored.
This adds support for parsing the new index-v1.json data as defined in
fdroidserver!221. This new index metadata format is required to
support localization, graphics, screenshots, etc.
refs #15
This is in response to identifying a bug with the way priorities work
with categories. Two repos may both specify different categories for
the same package. In this case, F-Droid should only select the
categories from the highest priority repo. Well, it is not to say that
this is the most preferable option, but it is consistent with other ways
that repo priorities are used.
