Go from Simple to Scalable: downloading multiple web pages

My impressions after some programming in Go can be summed up in two words: simple and scalable.

Say I want to download the content of example.com. I
can write a simple main function to do just that

package main

import (
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    response, err := http.Get("http://www.example.com")

    if err != nil {
        log.Fatal("Something went wrong.n")
    }

    defer response.Body.Close()
    body, err := ioutil.ReadAll(response.Body)
    log.Println(len(string(body[:])))
}

Going concurrent

Ok, all is fine and dandy, but say I want to download multiple pages (or hit multiple REST apis or whatever). Go simply scales! Let’s extract the function from
the previous example for requesting a webpage and returning the length of the response body.

package main

import (
    "io/ioutil"
    "log"
    "net/http"
)

func get_page(url string) int {
    response, err := http.Get(url)

    if err != nil {
        log.Fatal("Something went wrong.n")
    }

    defer response.Body.Close()
    body, err := ioutil.ReadAll(response.Body)

    if err != nil {
        log.Fatal("Something went wrong while reading the response body.n")
    }

    return len(body)
}


func main() {
    log.Println(get_page("http://www.example.com"))
}

We can then call this function with the go keyword, spawning a goroutine. We
will then face the problem of waiting for the spawned goroutine to finish.

In go, we share by communicating. This happens through a channel. It acts as a queue we can use to send results back to our main goroutine (main). Channels block while waiting for data, so the effect is kinda like joining multiple threads, and main will not exit until all is done.

package main

import (
    "io/ioutil"
    "log"
    "net/http"
)

func get_page(url string) int {
    response, err := http.Get(url)

    if err != nil {
        log.Fatal("Something went wrong.n")
    }

    defer response.Body.Close()
    body, err := ioutil.ReadAll(response.Body)

    if err != nil {
        log.Fatal("Something went wrong while reading the response body.n")
    }

    return len(body)
}


func main() {
    incoming_responses := make(chan int)

    pages := [...]string{"http://www.example.com",
                         "http://www.example.com",
                         "http://www.example.com"}

    for _, page := range pages {
        go func () {
            incoming_responses <- get_page(page)
        }()
    }

    i := len(pages)

    for i != 0 {
        log.Println(<-incoming_responses)
        i--
    }

    log.Println("done")
} 

In a future post I will go into detail about things that are wrong/error prone with this example and how we can fix them. Now Go create something!

Published by pgk

Person

%d bloggers like this: