Ever since I finished migrating my blog to a real hosting provider (no more random downtime!), I’ve been itching for a new side project. Alexa skills have been fun, but I kind of wanted to stand up something new, while also maintaining my frugal lifestyle. I recently secured a free non-commercial license from MySportsFeeds.com, which, by the way, is an awesome company. Needing something to test out their API (and showcase my eternal Cubs obsession), I turned to create a website that serves a single purpose: to inform the visitor how many home runs (or “dingers”) the Cubs have hit in the current year. I then started thinking about implementation options, and ultimately landed on the Heroku PEAN Stack.
REQUIREMENTS
I liked the idea of going client-side to practice some Javascript web calls, but the MSF API has some specific requirements around providing an API private key, which I can’t leave exposed to an end user. So, there would have to be at least some server-side options to protect the private key. Additionally, although not a hard requirement, MSF requests that you only call the API as often as your subscription updates (in my case, every five minutes). So, there would have to be some kind of data-caching happening on said server as well. Additionally, my bonus objective for this project was to find a free hosting provider – I didn’t want to pay for anything past the custom domain name.
IMPLEMENTATION
This is where things got especially tricky. Given that I’d need at least minimal access to data storage and I’d need to run some server-side code, many free providers (like Github Pages) were out. I could use the AWS free tier for a year – but that felt like cheating, as I’d have to pay for some resources after the first 12 months. So, remembering my DePaul Senior Project, I turned to Heroku.
HEROKU
Heroku offers, within reasonable limitations, a free service level for hosting web applications with server-side code and PostgreSQL usage. This is exactly what I was looking for. Plus, the automated build and deployment integration was a huge plus for Heroku. Dinge.rs runs on a PEAN (PostgreSQL, Express, Angular, Node) stack, similar to what’s outlined in this blog post. I created a new repo on GitHub (with private access – remember, the API key needs to be in a config file), generated a new Node project with Express and Supervisor, and turned to the custom backend logic.
CUSTOM API
Next, I started creating my own API (here’s the “N” in “PEAN”) to fetch the number of dingers for the Cubs from the MSF API. For now, let’s just implement it as a simple HTTP get, and ignore our data caching requirement.
const express = require('express'); const router = express.Router(); const pg = require('pg'); pg.defaults.ssl = true; const config = require('../config'); const https = require('https'); const btoa = require('btoa'); const connectionString = process.env.DATABASE_URL || config.EXTERNAL_DB_CONNECTION_STRING; /* GET home page. */ router.get('/', function (req, res, next) { res.sendFile('index.html'); }); router.get('/api/v1/dingers', (req, res, next) => { var results = []; // Call MSF API to check number of home runs for the cubs this year const httpOptions = { hostname: 'www.mysportsfeeds.com', port: '443', path: '/api/feed/pull/mlb/current/overall_team_standings.json?teamstats=HR', method: 'GET', headers: { "Authorization": "Basic " + btoa(config.API_USERNAME + ":" + config.API_PASSWORD) } }; httpOptions.headers['User-Agent'] = 'node ' + process.version; const request = https.request(httpOptions, (response) => { let responseBufs = []; let responseStr = ''; // Process response response.on('data', (chunk) => { if (Buffer.isBuffer(chunk)) { responseBufs.push(chunk); } else { responseStr = responseStr + chunk; } }).on('end', () => { responseStr = responseBufs.length > 0 ? Buffer.concat(responseBufs).toString('utf8') : responseStr; // Got our response, parse out the number of dingers for the cubs let dingers; JSON.parse(responseStr).overallteamstandings.teamstandingsentry.forEach((teamEntry) => { if (teamEntry.team.Abbreviation === 'CHC') { dingers = teamEntry.stats.Homeruns['#text']; } }); done(); return res.json(dingers); }) }) .setTimeout(0) .on('error', (error) => { console.log(error); }); request.write(""); request.end(); });
ANGULAR
Now that we have our API created with a route in Express (/api/v1/dingers), let’s set up our front-end code (brought to you by the letter “A”).
angular.module('dingersApp', []) .controller('mainController', ($scope, $http, $window) => { $scope.data = {}; $scope.data.currentYear = (new Date()).getFullYear(); $http.get('/api/v1/dingers') .success((dingers) => { $scope.data.dingers = dingers; }) .error((error) => { console.log('Error:',error); }); });
<html ng-app="dingersApp"> <head> <title>Dingers!</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet" media="screen"> <link href="stylesheets/style.css" rel="stylesheet" media="screen"> <link rel="icon" type="image/x-icon" href="../images/favicon/favicon.ico"> <link rel="apple-touch-icon-precomposed" href="../images/favicon/baseball-152-185882.png"> <meta name="msapplication-TileColor" content="#ffffff" /> <meta name="msapplication-TileImage" content="../images/favicon/baseball-144-185882.png"> <link rel="apple-touch-icon-precomposed" sizes="152x152" href="../images/favicon/baseball-152-185882.png"> <link rel="apple-touch-icon-precomposed" sizes="144x144" href="../images/favicon/baseball-144-185882.png"> <link rel="apple-touch-icon-precomposed" sizes="120x120" href="../images/favicon/baseball-120-185882.png"> <link rel="apple-touch-icon-precomposed" sizes="114x114" href="../images/favicon/baseball-114-185882.png"> <link rel="apple-touch-icon-precomposed" sizes="72x72" href="../images/favicon/baseball-72-185882.png"> <link rel="apple-touch-icon-precomposed" href="../images/favicon/baseball-57-185882.png"> <link rel="icon" href="../images/favicon/baseball-32-185882.png" sizes="32x32"> <script src="//code.jquery.com/jquery-2.2.4.min.js" type="text/javascript"></script> <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" type="text/javascript"></script> <script src="//cdnjs.cloudflare.com/ajax/libs/angular.js/1.5.6/angular.min.js"></script> <script src="javascripts/app.js"></script> </head> <body ng-controller="mainController"> <div class="container dingers-main-container"> <ng-include src="'views/indexMain.html'"></ng-include> <ng-include src="'views/footer.html'"></ng-include> </div> </body> </html>
You’ll notice that my index.html file doesn’t really contain anything other than references to CSS, Javascript, and ng-includes. If you read my post about Angular display templates in SharePoint, you may have noticed this is just my style. Feel free to adopt your own style. Or don’t – I’m a software developer turned blog writer, not your mom.
RUNNING WITH NPM AND EXPRESS
Now that we have our frontend and backend code in place, it seems like a good time to fire up our app locally, check for errors, and push it to source control. We can run the app locally by first building the app with npm install, and then starting Express with npm start. If everything was successful, we can navigate to localhost:3000 and we’ll see a page that looks like this:
Looks like we’ve got our year calculated from the Javascript Date class, and we’re pulling back the number of dingers from our API now.
AUTOMATED DEPLOYMENTS
Sounds like a good time to commit this to our GitHub repo using git commit -a and git push. Once everything’s in Git, let’s create a new Heroku project and connect our GitHub repo for automated deployments. I loosely followed this guide to get things up and running. But, instead of creating a remote branch for Heroku, I prefer to just have my builds pulled directly from my GitHub master branch. It’s really simple to do this via the UI – just log into your Heroku account, go under the Deploy tab, connect your GitHub account and select the master branch. I thought this approach made more sense – you can adopt it or not. Again, I’m not your mom.
Once we do this, Heroku should start pulling in our latest commit and building it. Give that a minute or two, and we can navigate to view our app live on Heroku.
POSTGRESQL
Alright, so now we’ve got our basic environment set up. There’s just one thing left – that pesky “P” in PEAN. First, let’s add a PostgreSQL instance to our app in Heroku. Navigate to the Resources tab on Heroku, and search for the Postgres add-on. Make sure you select the Hobby Dev – Free instance, unless you’d like to pay for something with more capabilities. When you’re done, your Resources dashboard will look similar to mine:
Now, let’s get our connection string for PostgreSQL. We’ll need this to run our initialization script. Click on the Heroku Postgres :: Database link on your Resources dashboard. On the new tab, expand Database Credentials and copy the value under URI.
Let’s paste that in our config, so our Node app can use it. We can now create our initialization script named database.js.
const pg = require('pg'); const config = require('../config'); const connectionString = process.env.DATABASE_URL || config.EXTERNAL_DB_CONNECTION_STRING; pg.defaults.ssl = true; const client = new pg.Client(connectionString); client.connect(); console.log('Connected to DB! Creating table...'); const query = client.query(`CREATE TABLE dingers(id SERIAL PRIMARY KEY, dingers VARCHAR(40) not null, lastFetched TIMESTAMP WITH TIME ZONE not null)`); query.on('end', () => { client.end(); }); console.log('Table created successfully.');
Now, let’s run the initialization script. We’ll run the npm command locally – it’ll connect to the Postgres instance remotely using the connection string. Enter the command npm run database.js.
DATA CACHING
Okay, so now that we’ve got our DB initialized with a item inserted, we can add our data caching logic to our app.js Node file. See my implementation below.
const express = require('express'); const router = express.Router(); const pg = require('pg'); pg.defaults.ssl = true; const config = require('../config'); const https = require('https'); const btoa = require('btoa'); const connectionString = process.env.DATABASE_URL || config.EXTERNAL_DB_CONNECTION_STRING; /* GET home page. */ router.get('/', function (req, res, next) { res.sendFile('index.html'); }); router.get('/api/v1/dingers', (req, res, next) => { var results = []; // Get a postgres client from the connection pool pg.connect(connectionString, (err, client, done) => { // Handle connection errors if (err) { done(); console.log(err); return res.status(500).json({ success: false, data: err }); } const selectQuery = 'SELECT * FROM dingers ORDER BY id ASC'; // SQL Query > Select Data client.query(selectQuery) .on('row', (row) => { results.push(row); }) .on('end', () => { // Check to see if the last fetched time is more than 5 minutes ago if (results[0] && results[0].lastfetched) { if (new Date(results[0].lastfetched) <= new Date(Date.now() - 60 * 5 * 1000)) { console.log('Cached data in DB is old - getting new data...'); // Call MSF API to check number of home runs for the cubs this year const httpOptions = { hostname: 'www.mysportsfeeds.com', port: '443', path: '/api/feed/pull/mlb/current/overall_team_standings.json?teamstats=HR', method: 'GET', headers: { "Authorization": "Basic " + btoa(config.API_USERNAME + ":" + config.API_PASSWORD) } }; httpOptions.headers['User-Agent'] = 'node ' + process.version; const request = https.request(httpOptions, (response) => { let responseBufs = []; let responseStr = ''; // Process response response.on('data', (chunk) => { if (Buffer.isBuffer(chunk)) { responseBufs.push(chunk); } else { responseStr = responseStr + chunk; } }).on('end', () => { responseStr = responseBufs.length > 0 ? Buffer.concat(responseBufs).toString('utf8') : responseStr; // Got our response, parse out the number of dingers for the cubs let dingers; JSON.parse(responseStr).overallteamstandings.teamstandingsentry.forEach((teamEntry) => { if (teamEntry.team.Abbreviation === 'CHC') { dingers = teamEntry.stats.Homeruns['#text']; } }); console.log('Got new dingers from API! Saving to DB...'); // Save the new data from MSF API to DB client.query('UPDATE dingers SET dingers=$1, lastfetched=$2', [dingers, (new Date()).toISOString()]); // return number of dingers and exit done(); return res.json(dingers); }) }) .setTimeout(0) .on('error', (error) => { console.log(error); }); request.write(""); request.end(); } else { // Cached data is less than 5 minutes old let minsAgo = ((new Date() - new Date(results[0].lastfetched)) / (60 * 1000)).toFixed(2); console.log('Cached data is still recent (last fetched', minsAgo, 'minutes ago)'); // return cached number of dingers and exit done(); return res.json(results[0].dingers); } } else { // Defensive validation above failed, return an error console.log('No results found in DB.', JSON.stringify(results)); done(); return res.json('Error: no results found in DB.', JSON.stringify(results)); } }); }); }); module.exports = router;
Let’s test this out locally to make sure we didn’t break anything. Assuming all is well, push these changes to the master branch. Or, if you’re a little more anal about Git strategies this early in a project, create a new branch, push there, create a pull request, and merge it back to master. This should kick off an automatic build in Heroku – wait a minute or two, and your changes should be live.
CUSTOM DOMAIN
Finally, I wanted to add a sweet domain name to my app so I should show it off to my friends. Yeah, if you really wanted to make this a 100% free effort, you could stick with the Heroku subdomain or get a free .tk TLD (which is actually a pretty bad idea), but I thought I’d “treat-yo-self” a little and drop $22/year on a domain name I really liked. We can actually point that domain right to our Heroku instance by updating our DNS settings on the domain registrar and updating the incoming hostnames on Heroku. I won’t go into too much detail on how this is done because it’s already well documented by Heroku.
CONCLUSION
And we’re done! There are a few UI-related updates I’ll probably make to this when I’m bored, but otherwise it’s a *mostly* finished product. Hopefully this aided you in your journey to master the F-PEAN stack (that’s Frugal-PEAN, for those of you following along at home) and you can get next cost-free app up and running quickly.
So what’s next? I’m thinking about allowing the ability to change MLB teams and/or setting up this API with an Alexa skill, just for the sake of completeness. Stay tuned!