Github Polling

How to poll for new artifacts

GitHub like many public API providers puts limits on how often you can poll the API. At the time of writing:

• Unauthenticated requests are limited to 60 per hour
• Authenticated requests are “limited” to 5000 per hour See GitHub rate limits

A handy thing to know is that conditional requests that return 304 Not-Modified do not count towards your rate limit. You can read more about conditional requests here:

• GitHub conditional requests
• Mozilla - HTTP Conditional Requests

I’ve written a handy bash script that uses conditional requests to poll for the latest artifacts:

#!/usr/bin/env bash

### Parameters ###
OWNER="{owner}"
REPO="{repo}"
TOKEN="{Optional GitHub user token - for private repos}"
OUT="latest.zip"
##################

baseUrl="https://api.github.com/repos/$OWNER/$REPO"
headerFile="/tmp/$OWNER-$REPO-hdr"
curlParams=(-s -D "$headerFile")
etagRegex='ETag: "(.+)"'

# include user auth token if provided
if [ ! -z "$TOKEN" ]; then
    curlParams+=(-u "$TOKEN")
fi

# include etag header from previous request
if [[ -f "$headerFile" && $(cat "$headerFile") =~ $etagRegex ]]; then
    etag="${BASH_REMATCH[1]}"
    curlParams+=(-H "If-None-Match: \"${etag}\"")
fi

# get latest artifact url
artifact_list=$(curl "${curlParams[@]}" "$baseUrl/actions/artifacts")
echo "$artifact_list"
if [ ! -z "$artifact_list" ];then
    artifact_url=$(echo "$artifact_list" | jq -j '.artifacts | sort_by(-.id) | first | .archive_download_url')
fi

# download artifact
if [ ! -z "$artifact_url" ];then
    curl -s -L -u "$TOKEN" "$artifact_url" -o "$OUT"
else
    echo "No new artifact found"
fi