Fetch and Render GitHub Markdown without CORS

I wanted to embed the contents of my GitHub project on another website, but the path to get there wasn’t straightforward. Here are the roadblocks I hit, and how I got around them.

Skip to the end if you just want the final solution.

Idea 1: Render in an iframe

I’d love to just create an iframe to show everything in my GitHub project on another website. An iframe seemed like a beautiful solution, but….

<iframe src="https://github.com/handstandsam/ShoppingApp"></iframe>

Roadblock: GitHub blocks iframes for its content.

Idea 2: Fetch the HTML, and Render it Manually

I thought I could just scrape the content from the website for my GitHub project and then render it on my site. However, I couldn’t pull arbitrary content from another web host due to CORS.

Note: If I had a server I could do this because I wouldn’t have CORS issues, but I was trying to do this completely in a frontend web page without a server.

Roadblock: CORS Browser Security Policies

Idea 3: Use the GitHub API to fetch the README File

GitHub has an awesome API that we can use to access the contents of a repository! I can’t use it to get the rendering of the entire project page, but I can access individual files like my README.md.

https://api.github.com/repos/handstandsam/ShoppingApp/contents/README.md

This allows me to pull down the contents of the file. The problem is that I can’t do that just in the frontend browser itself due to CORS.

Are you sensing a theme? Doing things in a browser is hard, but it helps make us safer on the web, so I can’t argue with that.

Roadblock: CORS Browser Security Policies

Idea 4: Use JSONP with the GitHub API

JSONP (JSON with Padding) is a workaround for CORS. You basically load an arbitrary bit of JavaScript from a 3rd party site, and have it call an arbitrary function that you know the name of.

Well, GitHub has support for JSONP! We will load JavaScript into our page from https://api.github.com/repos/handstandsam/ShoppingApp/contents/README.md?callback=myCallback and when the loading is done, it will invoke myCallback(results) assuming that the remote server has support for JSONP.

This exposes us to so many security vulnerabilities so PLEASE only do this with trusted sites. They could arbitrarily execute code within the context of your website and you wouldn’t know.

Note: JSONP uses the same functionality of loading in 3rd party javascript to do things like Analytics tracking, or fancy animations with JS libraries like BootStrap JS. It’s just programatically creating a <script> tag.

Roadblock: The response to the API contains Base64 encoded content.

Idea 5: Decode Base64 File Contents and Show README

I was able to define my callback for the GitHub API and then render the text into a pre (preformatted text) element. I did this by Base64 Decoding the GitHub API response’s “content” field, and then programmatically creating a pre element and setting its textContent.

function myCallback(response) {
    // Decode the Base64 Encoded Content
    let decodedContent = atob(response.data.content);

    // Create a "pre" HTML tag and render the content
    let pre = document.createElement("pre");
    pre.textContent = decodedContent;
    document.getElementsByTagName('body')[0].append(pre);
}

Roadblock: The contents weren’t formatted, just plain markdown.

Idea 6: Use a JS Library to Render the Markdown

There is a JavaScript library for everything. In this case I found markedjs/marked. I just give it a string of Markdown, and it’ll give me back the rendered HTML.

<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
function githubMarkdownCallback(response) {
    // Decode the Base64 Encoded Content
    let rawMarkdown = atob(response.data.content);

    // Use the markedjs library to transform markdown -> html
    let markdownHtml = marked.parse(rawMarkdown)

    // Add the new div to the body of the html page
    let div = document.createElement("div");
    div.innerHTML = markdownHtml;
    document.getElementsByTagName('body')[0].append(div);
}

FINAL SOLUTION!

Fetch my project’s README.md contents from GitHub’s public API using JSONP and use markedjs to render the Markdown into HTML.

<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script type="text/javascript">
function myCallback(response) {
    // Decode the Base64 Encoded Content
    let rawMarkdown = atob(response.data.content);

    // Use the markedjs library to transform markdown -> html
    let markdownHtml = marked.parse(rawMarkdown)

    // Add the new div to the body of the html page
    let div = document.createElement("div");
    div.innerHTML = markdownHtml;
    document.getElementsByTagName('body')[0].append(div);
}

let script = document.createElement('script');
script.src = 'https://api.github.com/repos/handstandsam/ShoppingApp/contents/README.md?callback=myCallback';
document.getElementsByTagName('head')[0].appendChild(script);
</script>

Intro to Kotlin Multiplatform JavaScript

One of the compilation targets of Kotlin Multiplatform is JavaScript. It’s pretty awesome that I can take the same Kotlin code and tools I write Android with, and write code that runs in a browser (or in Node.js).

Here is the video I’ve created to walk you through the creation of a Kotlin Multiplatform Project using the templates in Intellij IDEA, and what you can do to get your Kotlin running as JavaScript in the Browser!

Here are the code snippets mentioned in the video:

./gradlew build
 js(LEGACY) {
     browser {
         webpackTask {
             output.libraryTarget = "this" // Will add to window
         }
         binaries.executable()
     }
 }
 fun printHi() {
     println("Hello World Sam!")
 }
 fun main(args: Array) {
     printHi()
 }
 helloworldsam.printHi()

Exercise Your Mobile App’s API with AngularJS and a Simple HTTP Proxy

Capturing your mobile app’s API responses typically very arduous:

  • Run your app, and run through a scenario to generate the calls you want while you grab the logs.
  • or… run a bunch of API request with Postman or curl, but having to run the calls in order and copy/paste Authentication headers, etc to make it work.

You would think building a AngularJS web interface for your API to do the tedious parts for you would be easy… which it technically is… but not in a way that you can trick the browser into thinking it’s secure.  That’s because a web browser uses security policies that typical block CORS (Cross-Origin Resource Sharing).  That means that if you are on http://myui.com and want to make a request to your API (which is on http://myapi.com) and you API doesn’t have JSONP (JSON with Padding) support, you are in a hard spot and need something server-side to make those calls for you, and then return the results.

A trivial example of the pain CORS will cause is shown here with my GitHub profile.  If I want to pull this data, and add it to my website, I can’t do it because of CORS blocking.

curl https://api.github.com/users/handstandsam

{
  "login": "handstandsam",
  "id": 264948,
  "avatar_url": "https://avatars.githubusercontent.com/u/264948?v=3",
  "gravatar_id": "",
  "url": "https://api.github.com/users/handstandsam",
  "html_url": "https://github.com/handstandsam",
  "followers_url": "https://api.github.com/users/handstandsam/followers",
  "following_url": "https://api.github.com/users/handstandsam/following{/other_user}",
  "gists_url": "https://api.github.com/users/handstandsam/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/handstandsam/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/handstandsam/subscriptions",
  "organizations_url": "https://api.github.com/users/handstandsam/orgs",
  "repos_url": "https://api.github.com/users/handstandsam/repos",
  "events_url": "https://api.github.com/users/handstandsam/events{/privacy}",
  "received_events_url": "https://api.github.com/users/handstandsam/received_events",
  "type": "User",
  "site_admin": false,
  "name": "Sam Edwards",
  "company": "http://github.com/handstandtech",
  "blog": "http://handstandsam.com",
  "location": null,
  "email": null,
  "hireable": null,
  "bio": null,
  "public_repos": 7,
  "public_gists": 2,
  "followers": 10,
  "following": 58,
  "created_at": "2010-05-04T22:15:11Z",
  "updated_at": "2016-02-26T23:08:36Z"
}

GitHub is smart enough to provide the ability to use JSONP (invokes a local javascript method with the result of your API request), which allows us to actually do this.

curl https://api.github.com/users/handstandsam?callback=mycallback

/**/mycallback({
  "meta": {
    "Content-Type": "application/javascript; charset=utf-8",
    "X-RateLimit-Limit": "60",
    "X-RateLimit-Remaining": "55",
    "X-RateLimit-Reset": "1457061648",
    "Cache-Control": "public, max-age=60, s-maxage=60",
    "Vary": "Accept",
    "ETag": "\"5dd9556a9a8f8067c294a028b1f1a21d\"",
    "Last-Modified": "Fri, 26 Feb 2016 23:08:36 GMT",
    "X-GitHub-Media-Type": "github.v3",
    "status": 200
  },
  "data": {
    "login": "handstandsam",
    "id": 264948,
    "avatar_url": "https://avatars.githubusercontent.com/u/264948?v=3",
    "gravatar_id": "",
    "url": "https://api.github.com/users/handstandsam",
    "html_url": "https://github.com/handstandsam",
    "followers_url": "https://api.github.com/users/handstandsam/followers",
    "following_url": "https://api.github.com/users/handstandsam/following{/other_user}",
    "gists_url": "https://api.github.com/users/handstandsam/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/handstandsam/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/handstandsam/subscriptions",
    "organizations_url": "https://api.github.com/users/handstandsam/orgs",
    "repos_url": "https://api.github.com/users/handstandsam/repos",
    "events_url": "https://api.github.com/users/handstandsam/events{/privacy}",
    "received_events_url": "https://api.github.com/users/handstandsam/received_events",
    "type": "User",
    "site_admin": false,
    "name": "Sam Edwards",
    "company": "http://github.com/handstandtech",
    "blog": "http://handstandsam.com",
    "location": null,
    "email": null,
    "hireable": null,
    "bio": null,
    "public_repos": 7,
    "public_gists": 2,
    "followers": 10,
    "following": 58,
    "created_at": "2010-05-04T22:15:11Z",
    "updated_at": "2016-02-26T23:08:36Z"
  }
})

Most of us aren’t that lucky however, and JSONP is not supported by the API we are using.  And while this is great in many cases, it still doesn’t help us with anything other than GET requests.  POST, PUT and DELETE commands are all blocked from CORS so we are very limited.

If you are just interested in GET requests, and JSONP is not a possibility, you could instead modify the server code to return the header “Access-Control-Allow-Origin: *”. Most modern browsers would allow you to do a CORS GET request. Unfortunately in a lot of cases, GET requests are not enough or we don’t have access to the server code.

Well, how do I get around this to make REST calls to our server, like POST?

Well, write some server code that can do the requests for you since it won’t have security restrictions like the browser does.  But, most of us don’t have time or that.  The solution I came up with on the latest project I’m learning the API for is “php-guzzle-http-proxy“. This is as dirt simple as I could get and navigate around all these CORS problems I have been mentioning without having to write ANY server-side code.

Just plop this repo in a directory of your PHP server, and start making requests (/proxy in this case).  You’ll just add in a special header to your request I’ve named “X-Proxy-Url”.  This is the URL you REALLY want to call, but CORS won’t let you.  You can do any type of request you want: GET, POST, PUT or DELETE.

In curl it would look like:

curl -H "X-Proxy-Url: https://myapi.com/login" -H "Accept: application/json" -X POST -d '{"username": "sam", "password": "notsecure"}' http://myui.com/proxy

In AngularJS it would look like:

$http({
  method: 'POST',
  url: '/proxy,
  headers: {
    'Accept': 'application/json',
    'X-Proxy-Url': 'https://myapi.com/login'
  },
  data: {"username": "sam", "password": "notsecure"}
}).success(function(response){
  console.log(response.data);
});

That’s it. No server-side code required, just this one tiny PHP proxy script to allow you to make all types of REST calls including GET, POST, etc.

The tool I wrote (that unfortunately I can’t share) used Angular and Bootstrap.  When using the two in combo, when you get your JSON response back, display it with the following tag to make it formatted and easy to read.

<pre>{{data | json}}</pre>

Let me know if you found this helpful on Twitter at @HandstandSam

———————–

Related Links:

Continuous Integration at LesFurets.com – Richmond Java Users Group (RJUG)

“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” – 1st principal of the Agile Manifesto

This presentation by Raphaël Brugier (@rbrugier) at the Richmond Java Users Group last night (Feb 17th) was really good and highlighted:

  • Continuous Integration.  A case study of LesFurets.com. 40,000 unit tests running in 3 minutes.  Selenium tests running in 15 minutes.  Passing tests == shippable code.
  • They went from shipping a feature in a month, to as soon as ready (possibly only a few days).
  • A different kind of Git Flow using git octopus. – https://github.com/lesfurets/git-octopus
  • They have new developers hit the deploy button their first day.  Share deployment responsibilities within the dev team.
  • “If it hurts, do it more often.” – In regards to doing a deployment/release.
  • Work on feature branches.  Ship features as ready, not based on sprints. (git octopus facilitates this)
  • Detect feature branch merge conflicts after every push and make sure it’s resolved ASAP.
  • QA is owned by developers and sign off is given by product when dev demos it.
  • and more…

I would highly recommend watching the recording of the event.  Heads up: The speaker had a French accent and the video is from my MacBook’s camera, however, you get to watch it and otherwise you wouldn’t be able to :-p.

YouTube Link: https://www.youtube.com/watch?v=uTOPvC3lV1Q

Slides: http://www.slideshare.net/raphaelbrugier/continuous-delivery-journey-at-lesfuretscom

Meetup Info: http://www.meetup.com/Richmond-Java-Users-Group/events/226909145/

Enabling Cross-Origin Requests in Chrome from a HTML file:///

To allow cross-origin AJAX request using POST/PUT/DELETE requests to occur from a local html file that you open in your browser such as:

file:///Users/handstandsam/index.html

On your Mac, quit Chrome completely, then re-launch it using the following Terminal command:

open /Applications/Google\ Chrome.app/ –args –disable-web-security

That’s it.

WARNING: This does disable web security, so be cautious about what sites you access while Chrome is launched in this mode.

More info on Stack Overflow: http://stackoverflow.com/questions/3102819/disable-same-origin-policy-in-chrome