gmane.lisp.drakma.devel

Re: Changing content-type seems to have no effect

Michael Minerva — 2012-03-15T23:24:37

Hey Edi,

Thank you so much for your response.  

I am now thinking my problem may have something to do with the encoding will the file be encoded using base64 (with multipart/form-data)?  Is this something you can set?  Is this what external-format-out is for?

Thanks again,

--Mike
On Mar 15, 2012, at 3:08 PM, Edi Weitz wrote:

Re: Changing content-type seems to have no effect

Edi Weitz — 2012-03-15T21:08:21

If you want to use the content type
"application/x-www-form-urlencoded", you can only use strings as
values in the parameter list.  You could, if you insist, read your
file into a string if it's not too big.  I have to say that this all
sounds wrong, though.

Edi.

On Thu, Mar 15, 2012 at 6:53 PM, Michael Minerva
<minerva< at >agentsheets.com> wrote:

Changing content-type seems to have no effect

Michael Minerva — 2012-03-15T17:53:16

I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma.  

So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day).  Here is the code I use to do that:

(with-output-to-string (stream)
  (setf drakma:*header-stream* stream)
(let ((cookie-jar (make-instance 'drakma:cookie-jar)))
  (drakma:http-request
   "http://localhost:8888/?q=rest/user/login"
    :method :post
    :cookie-jar cookie-jar
    :parameters 
    '(("username" . "login")
      ("password" . "password"))
    )
  (drakma:http-request
   "http://localhost:8888/?q=rest/file"
    :method :post
    :cookie-jar cookie-jar
    :content-length t
    :parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif"))
    :content-type "application/x-www-form-urlencoded" 
    ;:parameters 
    )))


This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors).  I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma).  This is why I added the line :content-type "application/x-www-form-urlencoded"  to my second request (posted above) but it seems to have no effect.  

Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data.  Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file).  
 
"POST /?q=rest/user/login HTTP/1.1
Host: localhost:8888
User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M  (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 34

HTTP/1.1 200 OK
Date: Thu, 15 Mar 2012 17:49:51 GMT
Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
X-Powered-By: PHP/5.3.6
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: \"1331833791\"
Vary: Accept
Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly
Content-Length: 115
Connection: close
Content-Type: text/yaml

POST /?q=rest/file HTTP/1.1
Host: localhost:8888
User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M  (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ
Connection: close
Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm
Content-Length: 244958

HTTP/1.1 200 OK
Date: Thu, 15 Mar 2012 17:49:52 GMT
Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
X-Powered-By: PHP/5.3.6
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: \"1331833792\"
Vary: Accept
Content-Length: 4
Connection: close
Content-Type: text/yaml

"

Thanks,

--Michael

Re: Unwanted url-encoding of GET parameters.

Edi Weitz — 2012-03-11T14:57:53

On Sun, Mar 11, 2012 at 6:50 AM, William Halliburton
<whalliburton< at >gmail.com> wrote:

Bastards... :)

Re: Unwanted url-encoding of GET parameters.

William Halliburton — 2012-03-11T05:50:22

Thanks much. After some wiresharking, I found that google doesn't like the
drakma user-agent and changing it to firefox did the trick.

On Sat, Mar 10, 2012 at 2:46 AM, Edi Weitz <edi< at >agharta.de> wrote:

_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

Re: Unwanted url-encoding of GET parameters.

Edi Weitz — 2012-03-10T10:46:49

You are not allowed to send arbitrary characters in the request line.
FWIW, I just tried your example with Firefox and this is what the
browser sends according to LiveHttpHeaders:

  http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B

And Google returns the requested audio file.

Edi.

On Sat, Mar 10, 2012 at 1:08 AM, William Halliburton
<whalliburton< at >gmail.com> wrote:

_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

Unwanted url-encoding of GET parameters.

William Halliburton — 2012-03-10T00:08:22

Hello folks,

I'm trying to use drakma to fetch urls that contain utf8 characters but
HTTP-REQUEST automatically url encodes any non latin-1 ascii characters.

On my cursory reading of the RFCs, this seems conforming behavor, but in
this case it is definitely unwanted.

For example, the following url

  http://translate.google.com/translate_tts?tl=ru&q=вы

if entered directly into the browser correctly returns the text-to-speech
audo file but

when attempting to use HTTP-REQUEST, the url is being url encoded into

  http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B

of which google does not url-decode and fails to return the correct data.

So, for this case, the url-encoding is unwanted.

I am willing to submit patch an additional argument into HTTP-REQUEST to
disallow the encoding.

Thoughts?

Thank you,
William
_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

Re: drakma and non-ASCII content

Austin Haas — 2012-03-09T15:41:40

You can also explicitly supply the value of the content-length as a
keyword argument to http-request. I use a wrapper that adds this:

  :content-length (if content
                      (babel:string-size-in-octets content :encoding :utf-8)
                      0)

I agree, though, that the default behavior can be surprising. It has
bitten me, too, and I've seen at least one other library built-on
Drakma that didn't account for it, either.

-austin

drakma and non-ASCII content

Robert Brown — 2012-03-09T15:24:44

A co-worker of mine had some problems today using Drakma to POST a STRING
containing non-ASCII characters encoded as UTF-8.  He was doing something
equivalent to:

(http-request "http://zappa.com/favicon.ico"
              :method :post
              :content (concatenate 'string
                                    "hello" (string #\white_square) "world")
              :external-format-out :utf-8)

which ends up setting the Content-Length header value to 11, which is the
LENGTH in characters of the string being sent.

The documentation says that "if content is a sequence, Drakma will use LENGTH
to determine its length and will use the result for the 'Content-Length' header
sent to the server," so Drakma is working as documented.

Also, I think I understand why this behavior is the default.  You don't want to
scan a content string in order to determine what its length will be in octets
once it has been encoded.  My co-worker could have used a vector of type
(unsigned-byte 8) to hold his UTF-8 encoded data.

However, I think the current Drakma behavior may be a mistake.  People who want
high performance are probably manipulating encoded strings as vectors of
(unsigned-byte 8).  It's the casual users, those sending strings, who are the
ones most likely to be bitten by the default behavior, but only when their
strings contain non-ASCII characters.

bob

http redirect following url-encodes location

Willi Nesensohn — 2012-02-27T19:20:29

I'm using drakma to GET data over http. It turns out that when following 
redirects, drakma uses the location in the redirection header, which it 
url-encodes. Usually this might not be a problem, but when the location 
redirected to is already url-encoded this method fails.

Is this intended behaviour?

I'm not sure what the clean solution to this would be, as decoding the 
location before using it just doesn't feel right. For the lack of a 
better solution that's what i'm doing right now though.

Any pointers in the right direction would be greatly appreciated.

Best regards,
Wilfried Nesensohn

Re: A simple Cookie and viewing the actual Post

Ryan Davis — 2012-02-13T15:51:14

It looks like you are wanting a session cookie
(http://en.wikipedia.org/wiki/HTTP_cookie#Session_cookie). These are a
bit of a special case usage of cookies; your server issues each HTTP
client (usually a web browser) a unique ID that is used to track some
state on the server. This is generally impossible to know in advance.

If you trace the HTTP traffic in a browser, I bet you make two requests;
one where the server issues the session cookie (frequently a login
page), and then another that POSTs to node/create.

In drakma you can do the same thing, make two requests, and use the same
'drakma:cookie-jar in both requests. Some psuedo-code:

(let ((cookie-jar (make-instance 'drakma:cookie-jar)))
  (drakma:http-request
   "http://path/to/create/session"
   :method :post
   :cookie-jar cookie-jar)
 
  (drakma:http-request
   "http://path/to/create/node"
   :method :post
   :cookie-jar cookie-jar)
  )

The drakma:cookie-jar will accept the session cookie from the server in
the first http-request, and then pass it back to the server in the
second http-request. Of course, you'll need to fill out the http-request
parameters to match what your server is asking for.

HTH,

Ryan Davis
Acceleration.net
Director of Programming Services
2831 NW 41st street, suite B
Gainesville, FL 32606

Office: 352-335-6500 x 124
Fax: 352-335-6506


On 2/9/2012 4:32 PM, Michael Minerva wrote:

Re: A simple Cookie and viewing the actual Post

Michael Minerva — 2012-02-09T21:32:05

I'm sorry if my question is unclear but I just started using Drakma today and I am not very experienced with http-requests in general.   

I am trying to create an http-request that looks something like this:

POST /?q=node/create HTTP/1.1
Host: localhost:8888
Cookie: SESSdc0685ed01f285dab628a3700259e6bc=rCfmWLUeugRQNAQTs14yox_DWqZyGE2fYUS8W24j4co
Content-Length: 0
Content-Type: application/x-www-form-urlencoded

I tried doing this with a cookie-jar and a cookie but the make-instance of cookie required parameters such as name which I do not have (and did not need when I made the http-request by other means).  I wanted to see the actual http-post that was being created so I could compare it to other successful http-posts and try and find my problem (is this what header-stream might help with)?  

--Mike
On Feb 9, 2012, at 2:09 PM, Edi Weitz wrote:

Re: A simple Cookie and viewing the actual Post

Edi Weitz — 2012-02-09T21:09:20

You know about this, don't you?

  http://weitz.de/drakma/#*header-stream*

Otherwise, I don't really understand your question.  Perhaps you could
tell us what exactly you tried, what happened, and what you expected
to happen.

Cheers,
Edi.


On Thu, Feb 9, 2012 at 9:41 PM, Michael Minerva <minerva< at >agentsheets.com> wrote:

A simple Cookie and viewing the actual Post

Michael Minerva — 2012-02-09T20:41:17

I just started using drakma today and so far it seems really great but I had a couple quesitons.

First, is it possible to send a simpler for example i want my http-post to have something like this:

Cookie: SESSdc0685ed01f285dab628a3700259e6bc=rCfmWLUeugRQNAQTs14yox_DWqZyGE2fYUS8W24j4co

I tried making a drakma cookie and adding it to a cookie-jar but none of that seemed to work.

Also, is there a way to print the actual post I am sending so I can see what might be wrong with it?

Thanks a lot,

--Mike

Re: Drakma and client certificates?

Rob Blackwell — 2012-01-14T18:41:24

Okay, I think it was a typo in the Drakma util.lisp file - I have issued a pull request on github to fix it.

https://github.com/RobBlackwell/drakma/commit/149e445e39faa940fb211b7dd627161b7fd0c5df 

Thanks,

Rob.

Drakma and client certificates?

Rob Blackwell — 2012-01-14T12:15:11

Hi,

I'm trying to use client certificates with Drakma to call an existing web based API.

I have the latest code from https://github.com/edicl/drakma installed in my quicklisp local-projects directory. I'm using SBCL on OS X and Linux.

The client certificate is a .pfx file, and I've converted it to .pem format using

openssl pkcs12 -in robblackwellmanage.pfx -out robblackwellmanage.pfx.pem

I have verfied that this works as a server certifcate, using hunchentoot, thus:

(hunchentoot:start (make-instance 'hunchentoot:ssl-acceptor :ssl-privatekey-file "/Users/reb/certs/robblackwellmanage.pfx.pem" :ssl-certificate-file "/Users/reb/certs/robblackwellmanage.pfx.pem" :ssl-privatekey-password "password" :port 4343))

When I try to use this as a client certificate:

 (drakma:http-request resource
 :certificate "/Users/reb/certs/robblackwellmanage.pfx.pem"
 :key "/Users/reb/certs/robblackwellmanage.pfx.pem"
 :certificate-password "password"
 :method :get
 :content ""
 :content-type "application/xml"
 :additional-headers headers)

I get the following error

SSL initialization error: Can't load certificate passwordSSL error queue is empty.
   [Condition of type CL+SSL::SSL-ERROR-INITIALIZE]

Interestingly, if I deliberately supply the wrong password then I get a different error

SSL initialization error: Can't load RSA private key file /Users/reb/certs/robblackwellmanage.pfx.pemSSL error queue is empty.
   [Condition of type CL+SSL::SSL-ERROR-INITIALIZE]

I guess I'm doing something really silly - please excuse my ignorance, but any advice, suggestions or guidance on how to proceed would be very much appreciated.

Thanks!

Rob

Re: cl-oauth and drakma interoperability issue

mohegan-skunkworks — 2012-01-08T02:28:13

got it,

 thanks for the follow up. I appreciate it !


On Jan 7, 2012, at 8:31 PM, Edi Weitz wrote:

Re: cl-oauth and drakma interoperability issue

Edi Weitz — 2012-01-08T01:31:31

On Sat, Jan 7, 2012 at 10:50 PM, mohegan-skunkworks
<fh< at >mohegan-skunkworks.com> wrote:

It means that I currently don't have the time to maintain it.  But you
can check out the github repository and see what's happening there:

  http://netzhansa.blogspot.com/2011/08/ediware-moving-to-github.html

Cheers,
Edi.

Re: cl-oauth and drakma interoperability issue

mohegan-skunkworks — 2012-01-07T22:24:32

thanks for the feedback...
I'll take a closer look at cl-oauth then...

On Jan 7, 2012, at 5:06 PM, Zach Beane wrote:

Re: cl-oauth and drakma interoperability issue

Zach Beane — 2012-01-07T22:06:54


For what it's worth, I had a similar issue with ZS3, but rewriting ZS3
to pass parameters as :parameters instead of in the URL worked to fix
it.

Zach

Re: cl-oauth and drakma interoperability issue

mohegan-skunkworks — 2012-01-07T21:50:11

ok, does this mean that this library is no longer maintained ?

On Jan 4, 2012, at 9:54 AM, Edi Weitz wrote: