Krdytkyu

Goproxy is a Go package implementing an HTTP proxy.
I am using it to store requests from a client (eg: a web browser) for further processing.

In an http.Request struct, the Body is an io.ReadCloser, it can only be read once.

The problem I faced is that I need to read the Body to store it, but still make it accessible to goproxy so that it can be sent to the server.

I solved it by reading the body with ioutil.ReadAll, and then creating a ReadCloser with the byte slice I got. I then assign this ReadCloser to the request Body.

package main



import (

    "bytes"

    "flag"

    "fmt"

    "io/ioutil"

    "log"

    "net/http"

    "net/url"

    "regexp"

    "sync"



    "github.com/elazarl/goproxy"

)



var requests map[string]Request

var mutex sync.Mutex



type Request struct {

    Method        string

    URL           url.URL

    Proto         string // "HTTP/1.0"

    ProtoMajor    int    // 1

    ProtoMinor    int    // 0

    Header        http.Header

    Body          byte

    ContentLength int64

    Host          string

    PostForm      url.Values

    RequestURI    string

}



type RequestBody struct {

    *bytes.Reader

}



func (r RequestBody) Close() error {

    return nil

}



func storeRequest(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {

    body, err := ioutil.ReadAll(r.Body)

    if err != nil {

        // only print error for now

        log.Println(err)

        return r, nil

    }

    r.Body.Close()

    r.Body = RequestBody{bytes.NewReader(body)}

    fmt.Println(string(body))

    mutex.Lock()

    requests[r.URL.Host] = append(requests[r.URL.Host], Request{

        Method:        r.Method,

        URL:           *r.URL,

        Proto:         r.Proto,

        ProtoMajor:    r.ProtoMajor,

        ProtoMinor:    r.ProtoMinor,

        Header:        r.Header,

        Body:          body,

        ContentLength: r.ContentLength,

        Host:          r.Host,

        PostForm:      r.PostForm,

        RequestURI:    r.RequestURI,

    })

    mutex.Unlock()

    return r, nil

}



func main() {

    proxy := goproxy.NewProxyHttpServer()

    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*$"))).

        HandleConnect(goproxy.AlwaysMitm)

    proxy.OnRequest().DoFunc(storeRequest)



    requests = make(map[string]Request)



    addr := flag.String("addr", ":8080", "proxy listen address")

    flag.Parse()

    log.Fatal(http.ListenAndServe(*addr, proxy))

}

I would like to know if this is a correct way of solving this problem.

The hardest part of your question:

Is it allowed to change Body field value of http.Request struct?

Part of http.Request documentation:

// Body is the request's body.

//

// For client requests a nil body means the request has no

// body, such as a GET request. The HTTP Client's Transport

// is responsible for calling the Close method.

//

// For server requests the Request Body is always non-nil

// but will return EOF immediately when no body is present.

// The Server will close the request body. The ServeHTTP

// Handler does not need to.

Body io.ReadCloser

The documentation says nothing about it. So we need to find some top level policy on such things. As far as I know there is no one in Go documentation.

Body field type is io.ReadCloser interface, the actual struct type is hidden behind it.

• the actual type may have non exported fields which work in conjuction with Close method,


• the hidden struct may be part of some reusable pool and will be lost,


• 
  HTTP trailer headers may be somewhere behind EOF.

You may take a look at current net/http sources to find the answer, but it can change in next Go release.

So I guess your best bet is to ask the Go community or even propose a change to Body documentation that will clear things out.

Another approach would be to wrap original Body and read it together with request handler.