Qfyilyi

im using curl lib to perform some webscraping on a swedish recipe database page(i have of course asked for permission). However im getting different results when i run curl in the commandline and curl lib with the same options.

curl "https://kokboken.ikv.uu.se/sok.php" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3" --compressed -H "Referer: https://kokboken.ikv.uu.se/sok.php" -H "Content-Type: application/x-www-form-urlencoded" -H "Connection: keep-alive" -H "Upgrade-Insecure-Requests: 1" --data "search_text=tomat&dummy=&search_type=all&rec_cats"%"5B"%"5D=all&submit_search=S"%"F6k&recid=&offset=0&searchid="

this curl command returns the correct response which is a html page with a list of recipes.

but when i run it in with the help of curllib(i use a c# wrapper called CurlThin) It returns the blank search page without any results as i my parameters where not passed. The expected behavior if i would enter the page via a browser.

C# code

private CURLcode _global;

private SafeEasyHandle _easy;

private readonly string HEMMETS_ROOT = "https://kokboken.ikv.uu.se/";



_global = CurlNative.Init();

        // curl_easy_init() to create easy handle.

        _easy = CurlNative.Easy.Init();



        CurlNative.Easy.SetOpt(_easy, CURLoption.CAINFO, CurlResources.CaBundlePath);

        CurlNative.Easy.SetOpt(_easy, CURLoption.USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0");



        //Add headers to the request 

        var headers = CurlNative.Slist.Append(SafeSlistHandle.Null, "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");

        // Add one more value to existing HTTP header list.

        CurlNative.Slist.Append(headers, "Accept-Language: sv-SE,sv;q=0.8,en-US;q=0.5,en;q=0.3");

        CurlNative.Slist.Append(headers, "Referer: https://kokboken.ikv.uu.se/sok.php");

        CurlNative.Slist.Append(headers, "Content-Type: application/x-www-form-urlencoded");

        CurlNative.Slist.Append(headers, "Connection: keep-alive");

        CurlNative.Slist.Append(headers, "Upgrade-Insecure-Requests: 1");



        // Configure libcurl easy handle to send HTTP headers we configured.

        CurlNative.Easy.SetOpt(_easy, CURLoption.HTTPHEADER, headers.DangerousGetHandle());



                    CurlNative.Easy.SetOpt(_easy, CURLoption.URL, HEMMETS_ROOT + "sok.php");



        var postData = "search_text=tomat&dummy=&search_type=all&rec_cats"%"5B"%"5D=all&submit_search=S"%"F6k&recid=&offset=0&searchid=";





        // This one has to be called before setting COPYPOSTFIELDS.

        CurlNative.Easy.SetOpt(_easy, CURLoption.POSTFIELDS, postData);

        CurlNative.Easy.SetOpt(_easy, CURLoption.POST, 1);

        CurlNative.Easy.SetOpt(_easy, CURLoption.SSL_VERIFYHOST, 0);

        CurlNative.Easy.SetOpt(_easy, CURLoption.SSL_VERIFYPEER, 0);



        var stream = new MemoryStream();

        CurlNative.Easy.SetOpt(_easy, CURLoption.WRITEFUNCTION, (data, size, nmemb, user) =>

        {

            var length = (int)size * (int)nmemb;

            var buffer = new byte[length];

            Marshal.Copy(data, buffer, 0, length);

            stream.Write(buffer, 0, length);

            return (UIntPtr)length;

        });



        var result = CurlNative.Easy.Perform(_easy);

        html = Encoding.GetEncoding(1252).GetString(stream.ToArray());



        return html;

I Suspect A: that my request is not performing a post as it is behaving as i opened the site with a get request. Or B: That im missing some kind of curl option that the commandline specifies automatically.

Edit
I believe im using libcurl 1.1.1