Skip to content

  Availability

Several aspects of the processing pipeline affects how quickly analytics results become available to API endpoints.

Logging delay

Log files are created by content servers only after data served. For continuous requests like Icecast streams, this implies a minimum latency for a session equal to it's duration. For example, a listening session of 2 hours will only create a log once it ends, at which point it will go into the processing pipeline. For stream sessions this is the largest contributor to delayed availability.

Warning

Session data is back-filled to the actual listening period it occured in and audio sessions can be up to 24 hours long. This means data for the past ±30 hours may still continuously be updated.

Database

Bigquery introduces delays in data availability into two ways:

  • Input buffer lag. Data batches sent into BigQuery uses processing to perform load jobs which has variable capacity. Batches can become available within a minute or take up to 90 minutes to become available in the database after ingestion. This is affected by incoming volume, so during peak listening times when a lot of logs are ingested this delay may be longer than during off-peak times.
  • Materialised view refresh. The API uses materialised views to optimise both query speed and cost. Our materialised views refresh between 30 and 60 minutes after data changes in the base tables.

On average, new historical data should become available within an hour after processing.

Response caching

API responses are cached by the server and the same query will receive a cached result for up to 10 minutes.

All responses contain the X-Cache HTTP header to indicate a HIT (served from cache) or MISS (new response loaded from database).

Rate limiting

At the moment the API does not perform any global or per-user rate limiting.