Bombarded with Apple-app-site-association requests
I wanted to measure the traffic that our different Rails controller actions were getting, I used InfluxDB to store the data, and Grafana to visualise it.
Here is how our Rails hook looks like:
# http://guides.rubyonrails.org/v5.0/active_support_instrumentation.html
ActiveSupport::Notifications.subscribe('process_action.action_controller') do |_name, _started, _finished, _id, payload|
begin
hostname = payload[:headers].fetch('SERVER_NAME', 'N/A')
payload_data = {
values: {
value: 1,
view_runtime: payload[:view_runtime].to_f,
db_runtime: payload[:db_runtime].to_f,
}.compact,
tags: {
controller_action: [payload[:controller], payload[:action]].join('#'),
http_status: payload[:status],
http_method: payload[:method],
hostname: hostname,
}.compact,
}
payload_data[:tags][:http_status] = 500 if payload[:exception_object].present?
Rails.configuration.billetto_metrics.write_point('rails.pageviews', payload_data, 'ns')
rescue => doh
ErrorReporting.notify(doh) rescue nil
end
end
After having run this for a few hours I could make some nice graphs:
SELECT sum("value") FROM "rails.pageviews" WHERE $timeFilter GROUP BY time($__interval), "controller_action" fill(0)
The above query would generate me a graph of which controller actions were hit the most:
The green line is requests hitting our AppleController#app_site_association action that lives at /.well-known/apple-app-site-association, I decided to focus on only that traffic, and make a graph grouped per domain, here we can see that the traffic is evenly spread out across all of our domains:
The traffic looks strangely organic, high during the day, low during the evening, this let me believe that it wasn’t bot-traffic.
What are apple-app-site-association files? #
Apple-app-site-association (AASA) files takes care of defining a mapping between website URLs and views inside of iOS applications.
Lets say that you visit event with an event at https://billetto.dk/e/10000, if the AASA-file specifies "/e/*", then it instructs the iOS browser that if the Billetto application is installed, this view is also available as a non-web version.
This allows for a customer to browse our website on their iOS device, and if they have our application installed, they would be asked to see the content using the native application instead of the website.
But why were we getting all these requests?
Looking at the documentation #
When does an iOS device make a request? The article “Supporting Associated Domains in Your App” from Apple tells us this:
When your app is installed, the system attempts to download and verify the association file from the domains listed in your entitlement. For each domain, the system appends /.well-known/apple-app-site-association to its name. With this new URL, the system downloads the contents and ensures the file includes the app’s application identifier.
(…)
A validation may fail and the association will be denied if:
The JSON file is invalid or doesn’t contain the application identifier.
- The server returns a 400-499 code.
- Redirects to something other than secure HTTP will fail.
If the server returns a 500-599 code, the system assumes that the file is temporarily unavailable and may retry again.
After an app successfully associates with a domain, it remains associated until the app is deleted from the device. During development, delete your app from your testing device each time you update the association file to immediately see your changes.
We definitely don’t have that many installs of our application, so that couldn’t explain the requests alone.
Asking the tools #
Maybe the response we were giving the iOS devices were, somehow, malformed or incorrect, and that lead the devices keep pinging our servers.
By putting in our URL on several apple app site association testers, everything looked fine:
(From http://branch.io/resources/aasa-validator/)
(From https://search.developer.apple.com/appsearch-validation-tool)
So no real answer here, everything looked fine.
First idea: No caching #
I then thought that it could be that devices weren’t caching the response they received because our endpoint didn’t specify how long a user-agent should cache the response, I added a 1 hour cache:
expires_in 60.minutes, public: true
That didn’t change anything.
Looking at the response #
I decided then to look at the actual response to see if there was anything that looked odd:
$ curl -vs https://billetto.dk/.well-known/apple-app-site-association
GET /.well-known/apple-app-site-association HTTP/1.1
Host: billetto.dk
User-Agent: curl/7.54.0
Accept: /HTTP/1.1 200 OK
server: nginx
date: Sun, 02 Feb 2020 15:38:21 GMT
content-type: application/pkcs7-mime
transfer-encoding: chunked
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-download-options: noopen
x-permitted-cross-domain-policies: none
referrer-policy: strict-origin-when-cross-origin
cache-control: max-age=3600, public
etag: W/“39d81bc8709b591a39c2f37b6b0f576d”
x-request-id: fb8affcf-fa85-4e21-93b7-be4300ba3c94
x-runtime: 0.004255
strict-transport-security: max-age=31536000; includeSubDomains
x-clacks-overhead: GNU Terry Pratchett* Connection #0 to host billetto.dk left intact
The content looked like this:
{
"applinks": {
"apps": [],
"details": [
{
"appID": "ZGBHJ986UM.com.billetto",
"paths": [
"/",
"/en",
"/da",
"/sv",
"/no",
"/*/events/*",
"/events/*",
"/*/events/*/tickets",
"/events/*/tickets",
"/*/e/*",
"/e/*",
"/*/e/*/select",
"/e/*/select",
"/*/search",
"/search",
"/*/plan_feed",
"/plan_feed",
"/*/tickets",
"/tickets",
"/*/users/*",
"/users/*",
"/*/users/tickets"
]
}
]
}
}
The only thing that puzzled me a bit was the content-type: application/pkcs7-mime, when I read about it, it looked like it was some leftover implementation for old iOS versions for websites that wasn’t fully https-enabled, you could instead encrypt your AASA-file to serve it through http:
If your app runs in iOS 8, the file must have the MIME type application/pkcs7-mime and it must be CMS signed by a valid TLS certificate.
Because we are https-enabled we can just set the content-type to JSON, however that didn’t get rid of the requests.
Looking at the server logs #
I dived into our server logs, and noticed that the same IPs would hit our webserver multiple times over a minute:
This was indeed a bit strange, so I dived into our iOS application and found this piece of configuration:
<key>com.apple.developer.associated-domains</key>
<array>
<string>applinks:billetto.dk</string>
<string>applinks:billetto.se</string>
<string>applinks:billetto.co.uk</string>
<string>applinks:billetto.no</string>
<string>applinks:billetto.eu</string>
<string>applinks:billetto.nl</string>
<string>applinks:billetto.de</string>
<string>applinks:billetto.it</string>
<string>applinks:billetto.es</string>
<string>applinks:billetto.fi</string>
<string>applinks:billetto.fr</string>
<string>applinks:billetto.ie</string>
<string>applinks:billetto.nl</string>
<string>applinks:billetto.pt</string>
<string>applinks:billetto.at</string>
<string>applinks:billetto.be</string>
<string>applinks:billet.to</string>
<string>applinks:www.billetto.dk</string>
<string>applinks:www.billetto.se</string>
<string>applinks:www.billetto.co.uk</string>
<string>applinks:www.billetto.no</string>
<string>applinks:www.billetto.eu</string>
<string>applinks:www.billetto.nl</string>
<string>applinks:www.billetto.de</string>
<string>applinks:www.billetto.it</string>
<string>applinks:www.billetto.es</string>
<string>applinks:www.billetto.fi</string>
<string>applinks:www.billetto.fr</string>
<string>applinks:www.billetto.ie</string>
<string>applinks:www.billetto.nl</string>
<string>applinks:www.billetto.pt</string>
<string>applinks:www.billetto.at</string>
<string>applinks:www.billetto.be</string>
<string>applinks:www.billet.to</string>
</array>
It would make sense that an iOS application upon installation would try to connect to the whole list to fetch the individual AASA-configurations.
I noticed a couple of mistakes:
Duplicate .nl #
billetto.nl and www.billetto.nl was both there on the list twice, so that was fixed, perhaps Apple is smart enough to remove duplicates before making requests either before compile or at run-time on the devices, but we don’t know.
Duplicates were removed.
Wrong domain for Germany #
We do not own Billetto.de, and use Billetto.eu for our German customers, however the list included Billetto.de and not Billetto.eu.
Could a rogue domain in the com.apple.developer.associated-domains-list invalidate the whole configuration and be the reason for all of these requests?
We removed Billetto.de from the list, and added Billetto.eu.
Redirects from www. to non-www. #
In the image we can see a couple of 301 redirects, these are redirects from our www.-version to our non-www.-version of our site, Apple does not allow us to use redirects as the documentation says:
You must host the file using https:// with a valid certificate and without using any redirects.
Fixed! #
After applying these 3 changes the extra apple-app-site-association requests went away and the traffic returned to normal!
This shows the power of monitoring, without it, we might not have caught any extra traffic to a strange endpoint.