Skip to content

  Pipeline

Audio analytics are derived from parsing server log files. These log files are generated when stream or podcast data is served to listeners and then pushed through a pipeline to ultimately become available as analytics data.

graph LR
    A1(Content servers) --> A2(Log processing) --> A3[(Bigquery<br>database)] --> A4(API) --> A5(Dashboard)

Access logs

Content servers only log raw data requests. Every audio listening session causes one or more access log entries, depending on the content type being served.

Example log file entry
"Session",/stationname.mp3, 51.252.70.34, 2023-Apr-04 13:12:00, 300, [32] Broken pipe, GET //stationname.mp3 HTTP//1.1 / Connection: Keep-Alive / Host: live.stationname.com:443 / User-Agent: Dalvik//2.1.0 (Linux; U; Android 13; V2131 Build//TP1A.220624.014) / Icy-MetaData: 1 / Accept-Encoding: identity /  / , 642aaf7801bbf283_89161_0yeLTboO_OTEuMTkzLjI0NS4xODI!_000000005dt, , 0, /stationname.mp3

For example, a podcast file listened in a browser is typically downloaded in segments as the user listens, resulting in multiple access logs for the same session. Icecast sessions consist of one continuous connection, resulting in one log entry, but connections may drop and resume automatically - causing multiple logs for the same "session".

A log shipper runs on each content server to send batches of new access logs into the processing pipeline.

graph LR
    A1(Log files<br>*AIS,Icecast*) --> A2(Log shipper <br> *fluentbit*) --> |HTTPS| A3[(Log<br>processor)]

Log processing

The log processor receives raw log entries in batches. Using stateful processing and measurement rules it enriches the data and extracts meaningfull metrics like users, sessions and total listen hours. Log processing will:

  • Discard invalid log entries or ones with IP addresses in the reserved range,
  • Use a third-party database to lookup geographical information for the IP,
  • Use internal and third-party databases to decode the User-Agent string,
  • Identify bots or automated traffic coming from servers,
  • Combine multiple access logs into coherent listening session data,
  • Lookup all object identifiers (provider,channel,stream,episode) from the content platform using the stream mountpoint or episode file name,
  • Store the results to Google BigQuery.
graph LR
    A1(Log entries) --> A2(Filter) --> A3(Enrich) --> |HTTPS| A4[(Store)]

Processing Geographic data

Geographic location is inferred from the listener's IP address using a commercial third-party database. It performs well to establish broad audience location, but can misidentify locations of listeners using a VPN or other proxy masking their real IP address. Accuracy at the country level is typically very high, with regional accuracy somewhat less and city location the lowest.

Note

Geographic location via IP lookup is an approximation only and not similar in accuracy to GPS data.

This also includes other information that can be retrieved using IP address, such as the autonomous system number and registered organisation that owns the IP.

Processing Technology data

Most applications accessing audio content include a User Agent header in the request to identify themselves and the platform they are running on. This field plays a critical role in generating unique users and is also used to identify the device, platform and application making the request.

Example User-Agent for Samsung Galaxy S22 Android browser
Mozilla/5.0 (Linux; Android 12; SM-S906N Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.119 Mobile Safari/537.36

There is no defined standard for what this string should contain. iono.fm combines open-source and commercial libraries with custom rules to best attempt identification. These rules and libraries are periodically updated to cater for changes and new applications.

Info

Audio broadcasters with custom mobile applications can provide their own User-Agent for inclusion.

Database storage

Analytics data is stored to Google BigQuery in the EU zone. BigQuery is a columnar datastore, ideal for storing and querying on high volumes of time-series data. Once data is stored inside the database, it becomes available to API queries.