Move code causing verify error into separate helper class
Fixes#748.
I'm not 100% sure on how the `@TargetApi` and `VerifyError` work
together. However it is something along the lines of:
* Class loader needs `CleanCacheService`.
* At this point, it loads the bytecode for that class and verifies
that it all makes sense.
* The bytecode within the method targeted at API 21 is not understood
by earlier APIs, because the entire `Os` class was introduced in 21.
* By putting it into a different class, that class is only loaded
at runtime on devices with API of 21 or higher.
Previously, `@TargetApi` + the relevant guard condition to check
the build version at runtime suffices to prevent this. However it seems
that if the entire class does not even exist on earlier APIs, then it
is no longer good enough.
See merge request !379
I'm not 100% sure on how the `@TargetApi` and `VerifyError` work
together. However it is something along the lines of:
* Class loader needs `CleanCacheService`.
* At this point, it loads the bytecode for that class and verifies
that it all makes sense.
* The bytecode within the method targeted at API 21 is not understood
by earlier APIs, because the entire `Os` class was introduced in 21.
* By putting it into a different class, that class is only loaded
at runtime on devices with API of 21 or higher.
Previously, `@TargetApi` + the relevant guard condition to check
the build version at runtime suffices to prevent this. However it seems
that if the entire class does not even exist on earlier APIs, then it
is no longer good enough.
At time of writing (and for some time before), fdroidserver has forced
a description of "No description available" for apps which don't have
descriptions at all:
* https://gitlab.com/fdroid/fdroidserver/blob/0.6.0/fdroidserver/metadata.py#L876
However, if the description is not set for whatever reason, it should not
crash the client.
The icon files are downloaded for each version of the app. Over time, old
versions will pile up. This cleans out the ones that have not been used in
over a year.
On < android-21, this will delete icons that were downloaded over a year
ago even if they are still in use because it is only possible to check
mtime, not atime.
If CleanCacheService runs while an APK is being installed, it should not
delete the APK that is in the process of being installed. This does that
by only deleting those files if they are older than an hour. Same goes for
the index files.
#738
It was passing the wrong time value in the recursion, which made for a
really old "olderThan" time. This also then flipped the logic on the
next round through the recursion, causing files to be deleted even if
"Keep Cache Time" was set to "Forever".
closes#719closes#736
Before, CleanCacheService was only scheduled at app start for once a day.
If the user selects a time less than a day, then CleanCacheService should
run more frequently.
closes#719
In android-21, they exposed the formerly internal method for getting stat
structs of files. From that, we can get the last access time, which is a
much better way to determine which files to delete rather than last
modified time.
closes#644
By catching the exception here and returning null, the problem is then
passed on further down the line where it is harder to debug. The hash is
required wherever this method is called, so this should fail immediately.
#699
ApkVerifier Tests
This are some tests for ApkVerifier. More will follow when we merge https://gitlab.com/fdroid/fdroidserver/merge_requests/150 and implement parsing of permissions with min and max sdk versions.
NOTE: This androidTest cannot run as a Robolectric test because the required methods from PackageManger are not included in Robolectric's Android API.
The corresponding exception by robolectric:
```
org.fdroid.fdroid.installer.ApkVerifierTest > testVerifier FAILED
00:31:18.241 [DEBUG] [TestEventLogger] java.lang.NoClassDefFoundError: java/util/jar/StrictJarFile
00:31:18.241 [DEBUG] [TestEventLogger] at java.lang.Class.getDeclaredMethods0(Native Method)
00:31:18.241 [DEBUG] [TestEventLogger] at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
00:31:18.241 [DEBUG] [TestEventLogger] at java.lang.Class.getDeclaredMethod(Class.java:2128)
00:31:18.241 [DEBUG] [TestEventLogger] at org.robolectric.util.ReflectionHelpers.callStaticMethod(ReflectionHelpers.java:224)
00:31:18.241 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.RobolectricInternals.performStaticInitialization(RobolectricInternals.java:54)
00:31:18.241 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.ShadowWrangler.classInitializing(ShadowWrangler.java:119)
00:31:18.241 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.RobolectricInternals.classInitializing(RobolectricInternals.java:18)
00:31:18.241 [DEBUG] [TestEventLogger] at android.content.pm.PackageParser.<clinit>(PackageParser.java)
00:31:18.241 [DEBUG] [TestEventLogger] at android.content.pm.PackageManager.getPackageArchiveInfo(PackageManager.java:3545)
00:31:18.241 [DEBUG] [TestEventLogger] at org.fdroid.fdroid.installer.ApkVerifier.verifyApk(ApkVerifier.java:56)
00:31:18.241 [DEBUG] [TestEventLogger] at org.fdroid.fdroid.installer.ApkVerifierTest.testVerifier(ApkVerifierTest.java:78)
00:31:18.242 [DEBUG] [TestEventLogger] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[...]
00:31:18.244 [DEBUG] [TestEventLogger]
00:31:18.244 [DEBUG] [TestEventLogger] Caused by:
00:31:18.245 [DEBUG] [TestEventLogger] java.lang.ClassNotFoundException: java.util.jar.StrictJarFile
00:31:18.245 [DEBUG] [TestEventLogger] at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
00:31:18.245 [DEBUG] [TestEventLogger] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
00:31:18.245 [DEBUG] [TestEventLogger] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
00:31:18.245 [DEBUG] [TestEventLogger] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
00:31:18.245 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.InstrumentingClassLoader.loadClass(InstrumentingClassLoader.java:124)
00:31:18.245 [DEBUG] [TestEventLogger] at java.lang.Class.getDeclaredMethods0(Native Method)
00:31:18.245 [DEBUG] [TestEventLogger] at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
00:31:18.245 [DEBUG] [TestEventLogger] at java.lang.Class.getDeclaredMethod(Class.java:2128)
00:31:18.245 [DEBUG] [TestEventLogger] at org.robolectric.util.ReflectionHelpers.callStaticMethod(ReflectionHelpers.java:224)
00:31:18.245 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.RobolectricInternals.performStaticInitialization(RobolectricInternals.java:54)
00:31:18.245 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.ShadowWrangler.classInitializing(ShadowWrangler.java:119)
00:31:18.245 [DEBUG] [TestEventLogger] at org.robolectric.internal.bytecode.RobolectricInternals.classInitializing(RobolectricInternals.java:18)
00:31:18.245 [DEBUG] [TestEventLogger] at android.content.pm.PackageParser.<clinit>(PackageParser.java)
00:31:18.245 [DEBUG] [TestEventLogger] at android.content.pm.PackageManager.$$robo$$getPackageArchiveInfo(PackageManager.java:3545)
00:31:18.245 [DEBUG] [TestEventLogger] at android.content.pm.PackageManager.getPackageArchiveInfo(PackageManager.java)
00:31:18.245 [DEBUG] [TestEventLogger] at org.fdroid.fdroid.installer.ApkVerifier.verifyApk(ApkVerifier.java:56)
00:31:18.246 [DEBUG] [TestEventLogger] at org.fdroid.fdroid.installer.ApkVerifierTest.testVerifier(ApkVerifierTest.java:78)
00:31:18.246 [DEBUG] [TestEventLogger] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[...]
```
See merge request !367
More misc code cleanup around database code
I'm pulling out the final bit of unrelated code from my database refactor branch in the hope of making the final diff easier. This cleans up a few switch statements with only one option, closes some cursors, and removes some dead code. Comments in the commits explain the dead code.
See merge request !374
AS picked up that the statement is always false, so the body of the if is
never executed. This is indeed the case, because the constructor assigns
the object which is being checked for null.
The code only existed so that it could be used in a test. Subsequently,
a further test was written to test this code (used by the first test).
Since none of the code is actually used in the app, it has been removed.
See #511 for details. This is in prepration for having an even more normalized
`fdroid_package` table. That table will be the authoritative reference of what
"packages" are known about in the client. The "app" table (now thought of as "app metadata") will
be specific to each repository which provides different metadata about that app.
The test was using a `findIgnored` method in `AppProvider`, which only
existed for the purpose of testing. The test has been changed to instead
check for apps which would end up in the "can update" list (which is really
where the "ignored" apps are useful).
In the process, realised that using appId as a foreign key is worse than
packageName, because appId can get removed and added again, but it will
be different when the same app is inserted a second time. In order to
maintain the association of which apps have preferences stored against
them, they need to be stored against something with a bit more semantic
meaning. Thus, join onto package name instead.
This is a more concise syntax to say the same thing, and avoids an
OR clause in the where - which is often the cause of slowness in
many queries. Not sure if it was problematic in these cases, however
this COALESCE syntax is still more consise.
With no indexes at all, a join between X and Y tables would require a full
table scan of Y for each row in X. With an index on the relevant field in
Y, it would require an index lookup on the join field in Y for each row in
X, which contains a pointer to the row of interest in Y. This row is then
looked up and the relevant value extracted. By using a covering index (one
which includes all fields required to satisfy the query, with the first field
being the one which is looked up in the join), then once the index has been
searched, there is no need to then go to table Y because all the relevant
data is already in the index.
This offers a marginal performance improvement.
Merge download broadcast receivers
Previously, for all 4 states broadcast receivers were registered separately. These have now been merged into one receiver. IMHO this makes the code more readable and structured.
See merge request !368
The usage of ContentValues to send App/Apk objects
to services was an hack in my opinion.
This hack broke in https://gitlab.com/fdroid/fdroidclient/merge_requests/359
where the packageName has been removed from the
toContentValues() method, which leads to NPEs in
the services.
Wherever the "package name" of an apk is required, it can be requested by
asking for `Schema.ApkTable.Cols.App.PACKAGE_NAME`. Note the `App` which
indicates that it is in fact pulling this data from the `fdroid_app` table rather
than the `fdroid_apk` table.
Firstly, this causes #721, possibly due to a bug in "Barcode
Scanner" whereby it seems to ignore the scheme when in caps,
assuming it is "http".
The relevant RFC is:
> RFC3986 (Uniform Resource Identifier (URI): Generic Syntax
In section 3.1, it describes the scheme:
> Although schemes are case-insensitive, the canonical form is
> lowercase and documents that specify schemes must do so with
> lowercase letters. An implementation should accept uppercase
> letters as equivalent to lowercase in scheme names (e.g., allow
> "HTTP" as well as "http") for the sake of robustness but should
> only produce lowercase scheme names for consistency.
Secondly, it is not valid to uppercase URLs at will. Although it
seems that there is some sort of more-compact-QR-generating-logic
that doesn't justify this. Funnily enough, I can't find anything
in RFC3986 about the case-insensitivity of URI paths. However
consider the following:
* https://i.imgur.com/fn33EcW.jpg
That is a valid path to an image. If we upper case it:
* HTTPS://I.IMGUR.COM/FN33ECW.JPG
or lower case it:
* https://i.imgur.com/fn33ecw.jpg
Then the server is entitled to treat it differently and indeed
it does. Both the upper case and lower case are no both 404's.
When downloading arbitrary URLs using F-Droid (e.g. icons, .apk files, indexes)
then it may be the case that the repo requires authentication. As such, we try
to infer the repository based purely on the URL.
The old code took the basename of the URL, which means remove the last fragment
(e.g. "index.jar") and use the remaining portion of the URL to lookup the repo.
This is broken for many reasons, partly because of the presence of a query string,
partly because there are other things which are not just in the root directory
of the repo (e.g. "/icons/*.png").
This new method iteratively peels off the right most segment of the URLs path,
then looks to see if a repo exists at that address.
Note that this breaks down if you have nested repositories on a server, where
one of the repositories is nested inside a directory that F-Droid knows about,
such as "icons". In such a case, the following repositories:
* https://f-droid.org/repo (requires auth)
* https://f-droid.org/repo/icons (doesn't require auth)
will break down. If requesting something from the repo requiring auth:
* https://f-droid.org/repo/icons/org.fdroid.fdroid.png
Then it will lookup the database and find the repo which lives in "/icons"
and doesn't require auth (or requires a different auth username/password).
Not sure there is a lot that can be done about this without major refactoring.
Such refactoring would require making sure a `Repo` is always given to a downloader
for any HTTP request, and is probably a bit out of scope of this bug.
Also added tests for this behaviour.
Fixes#711.
Always capture timestamps, even if it is the same. This is because we are dependent
on it later on in the repo update process. Specifically, when updating from a HTTP
server that doesn't send out etags with its responses, it will trigger a full blown
repo update every time, even if all the values in the index are the same (name,
description, etc). This is as distinct from better behaving servers that send etags,
in which case we will only do a partial update (i.e. persist the "last updated time").
In such a case, the remainder of the update process will proceed, and ask for this
timestamp.
