I am running a Go binary that has an http server with systemd. I have it setup so that Caddy has a reverse-proxy to this go http server.
http: Accept error: accept tcp [::]:8002: accept4: too many open files;
dial tcp 192.85.2.4:443: socket: too many open filesWhen I look at the open files of the process, I get 1025 or less, though my ulimit is set to a much larger limit:
$ lsof -p 1243 | wc -l
1025$ ulimit -Sn
200000$ ulimit -Hn
1048576I'm not sure if that's the problem but seems that it could be? Seems like the Go server should be spawning new goroutines or take care of that somehow.
EDIT: Here's my server script:
package main
import ( "fmt" "time" "net/http"
)
type Config struct{}
func (c *Config) testerHandler(w http.ResponseWriter, r *http.Request) { r.Body.Close() time.Sleep(1 * time.Second) fmt.Fprintf(w, "hello\n")
}
func main() { c := &Config{} http.HandleFunc("/tester", c.testerHandler) fmt.Println("listening on ") http.ListenAndServe(":8000", nil)
}EDIT: And here's a script I use to spam my server:
package main
import ( "log" "net/http"
)
func main() { number := 10000 log.Printf("spamming %d numbers\n", number) ch := make(chan interface{}) client := http.Client{} for i:=0; i<number; i++{ go func(number int) { u := "" rsp, err := client.Get(u) if err != nil { ch <- err return } rsp.Body.Close() ch <- rsp }(number) } var errs int m := make(map[int]int) for i:=0; i<number; i++ { rsp := <-ch switch rsp.(type) { case *http.Response: code := rsp.(*http.Response).StatusCode m[code]++ default: log.Println(rsp.(error)) errs++ } } log.Println("errs:", errs) for k, v := range m { log.Printf("%d:%d\n", k, v) }
}3 Answers
This is a problem that can be solved by modifying some parameters of your system.
I recommend that you follow the following thread: Too many open files
2I was able to fix this by either Disabling Keep Alives on the ReverseProxy or by setting a low "ReadTimeout" on the Go Server.
Simple example
func main(){
srv := &http.Server{ Addr: ":8080", ReadTimeout: 2 * time.Second, Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { go fmt.Printf("handling request\n") fmt.Fprintln(w, "Hello, client") }),
}
log.Fatal(srv.ListenAndServe())
}Systemd takes over ulimits and sets the actual (=soft) runtime limit for max number of files to 1024. It is expected that the process takes care of requesting resources it needs explicitly - that is how most "professional" applications are written. For home grown ones, the resource can be requested in the dot-service file, e.g.,:
[Service]
LimitNOFILE=10240
LimitNOFILESoft=10240
