How to Download The Straits Times Print Edition in PDF for Free

The Straits Times is an English language daily newspaper based in Singapore. They provide access to the full print editions in PDF format for both iPad and Android tablets, but this requires a subscription.

Without a subscription, you will not be able to view the print edition of the newspaper on your tablet, but there’s a way to download The Straits Times print edition in PDF for free using your desktop computer.

For this hack, you will need the following tools.

A Linux PC workstation. You can also run Linux on a virtualised environment. Else, install Cygwin on Windows.
curl utility – to retrieve the JSON data.
jq utility – a lightweight JSON processor. This is to decode the JSON output.
wget utility – to download the PDF files.

Open a shell terminal, and run curl as follows.

$ today=$(date +"%Y-%m-%d")
$ curl -s -k -H "projectName: straitstimesapp" -H "Authorization: Bearer ZHVtbXk=" "https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today"

1 2	$ today=$(date +"%Y-%m-%d") $ curl -s -k -H "projectName: straitstimesapp" -H "Authorization: Bearer ZHVtbXk=" "https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today"

The variable “today” will be set to today’s date in “yyyy-mm-dd” format and this variable will in turn be used by the curl utility to retrieve the data for the current day. If you try to access the URL from your browser directly, you will get a blank page unless you are able to send the additional custom headers (specified after the -H parameter in curl) in the HTTP request.

Now, let’s anaylse the JSON data. The links to the PDF files are in the “pdfs” array section of the JSON output. An example is shown below.

"pdfs" : [
         {
            "cover" : "http://cdn.phoenixplatform.com.sg/Media/14106/248761/b72fd64b-5a64-429f-91a0-d37332668771.pdf",
            "Id" : 248761,
            "ProjectId" : 14106,
            "name" : "ST20150812-PRI-001-00",
            "isNoPublicationDay" : false,
            "rank" : {
               "fileRank" : "0",
               "sectionName" : "Top of the News",
               "Headings" : [],
               "sectionRank" : "0"
            },
            "pub_date" : "2015-08-12T00:00:00Z"
         },

"pdfs" : [

{

"cover" : "http://cdn.phoenixplatform.com.sg/Media/14106/248761/b72fd64b-5a64-429f-91a0-d37332668771.pdf",

"Id" : 248761,

"ProjectId" : 14106,

"name" : "ST20150812-PRI-001-00",

"isNoPublicationDay" : false,

"rank" : {

"fileRank" : "0",

"sectionName" : "Top of the News",

"Headings" : [],

"sectionRank" : "0"

"pub_date" : "2015-08-12T00:00:00Z"

The link to the PDF file can be found in the “cover” field, but if you notice, the filename does not make any sense. The “name” field gives a better indication about the contents of the PDF file – it gives the date, section and the page number. It will definitely make more sense to name the PDF file as “ST20150812-PRI-001-00.pdf” than as “b72fd64b-5a64-429f-91a0-d37332668771.pdf”. So, we will need to rename the PDF files with the names found in the “name” field. This is where the jq utility will play a part.

To download all the PDFs in one go, try running the shell script below. The script will generate URLs for all the PDFs and will use the wget utility to download the files.

#!/bin/sh

today=$(date +"%Y-%m-%d")
header1="projectName: straitstimesapp"
header2="Authorization: Bearer ZHVtbXk="
st_url="https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today"

curl -s -k -H "$header1" -H "$header2" "$st_url" | \
jq -r '.[].pdfs[] | .cover + " " + .name' | \
while IFS= read line ; do
    set -- $line
    infile=$1
    outfile="${2}.pdf"
    wget $infile -O "$outfile"    
done

#!/bin/sh

today=$(date +"%Y-%m-%d")

header1="projectName: straitstimesapp"

header2="Authorization: Bearer ZHVtbXk="

st_url="https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today"

curl -s -k -H "$header1" -H "$header2" "$st_url" | \

jq -r '.[].pdfs[] | .cover + " " + .name' | \

while IFS= read line ; do

set -- $line

infile=$1

outfile="${2}.pdf"

wget $infile -O "$outfile"

done

The script will not download PDFs in the classified section. ~~If you are able to decode the classified section and download the PDF files, you can share your code in the comments below and I will update the article.~~ If you want to download the PDFs in the classified section as well, use the script below. Thanks to Ng Wee Kiat for sharing the jq filter.

#!/bin/sh

today=$(date +"%Y-%m-%d")
header1="projectName: straitstimesapp"
header2="Authorization: Bearer ZHVtbXk="
st_url="https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today" 

curl -s -k -H "$header1" -H "$header2" "$st_url" | \
jq -r '.[].pdfs[], .[].headings[].pdfs[] | .cover + " " + .name' | \
while IFS= read line ; do
    set -- $line
    infile=$1
    outfile="${2}.pdf"
    wget $infile -O "$outfile"    
done

#!/bin/sh

today=$(date +"%Y-%m-%d")

header1="projectName: straitstimesapp"

header2="Authorization: Bearer ZHVtbXk="

st_url="https://Tigerspike-Fuse-api-SIN-LIVE.stapi.straitstimes.com/api/documents?pdfDate=$today"

curl -s -k -H "$header1" -H "$header2" "$st_url" | \

jq -r '.[].pdfs[], .[].headings[].pdfs[] | .cover + " " + .name' | \

while IFS= read line ; do

set -- $line

infile=$1

outfile="${2}.pdf"

wget $infile -O "$outfile"

done

Update (24-Dec-2013): Somehow the original URL http://st.buuuk.in/ipad_layout no longer works. Use the new URL http://stpdf.buuuk.in/ipad_layout instead.

Update (23-July-2014): Added a shell script to download all the PDF files.

Update (15-Aug-2014): Use the URL http://stapp.straitstimes.com/ipad_layout?device=android-tablet&v=1&day=0&device_name=tablet if the existing URL does not work.

Update (24-Aug-2015): Article updated with new procedure. Previous URLs and methods no longer work.

Update (26-Aug-2015): Shell script updated to download PDFs from the classified section.

Note (23-Apr-2016): This article will no longer be updated.

Mohamed Ibrahim

ibrahim = { interested_in(unix, linux, android, open_source, reverse_engineering); coding(c, shell, php, python, java, javascript, nodejs, react); plays_on(xbox, ps4); linux_desktop_user(true); }