gmane.lisp.drakma.devel

Changing content-type seems to have no effect

Michael Minerva — 2012-03-15T17:53:16

I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma.  

So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day).  Here is the code I use to do that:

(with-output-to-string (stream)
  (setf drakma:*header-stream* stream)
(let ((cookie-jar (make-instance 'drakma:cookie-jar)))
  (drakma:http-request
   "http://localhost:8888/?q=rest/user/login"
    :method :post
    :cookie-jar cookie-jar
    :parameters 
    '(("username" . "login")
      ("password" . "password"))
    )
  (drakma:http-request
   "http://localhost:8888/?q=rest/file"
    :method :post
    :cookie-jar cookie-jar
    :content-length t
    :parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif"))
    :content-type "application/x-www-form-urlencoded" 
    ;:parameters 
    )))


This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors).  I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma).  This is why I added the line :content-type "application/x-www-form-urlencoded"  to my second request (posted above) but it seems to have no effect.  

Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data.  Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file).  
 
"POST /?q=rest/user/login HTTP/1.1
Host: localhost:8888
User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M  (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Connection: close
Content-Type: application/x-www-form-urlencoded
Content-Length: 34

HTTP/1.1 200 OK
Date: Thu, 15 Mar 2012 17:49:51 GMT
Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
X-Powered-By: PHP/5.3.6
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: \"1331833791\"
Vary: Accept
Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly
Content-Length: 115
Connection: close
Content-Type: text/yaml

POST /?q=rest/file HTTP/1.1
Host: localhost:8888
User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M  (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/)
Accept: */*
Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ
Connection: close
Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm
Content-Length: 244958

HTTP/1.1 200 OK
Date: Thu, 15 Mar 2012 17:49:52 GMT
Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6
X-Powered-By: PHP/5.3.6
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000
Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0
ETag: \"1331833792\"
Vary: Accept
Content-Length: 4
Connection: close
Content-Type: text/yaml

"

Thanks,

--Michael

Unwanted url-encoding of GET parameters.

William Halliburton — 2012-03-10T00:08:22

Hello folks,

I'm trying to use drakma to fetch urls that contain utf8 characters but
HTTP-REQUEST automatically url encodes any non latin-1 ascii characters.

On my cursory reading of the RFCs, this seems conforming behavor, but in
this case it is definitely unwanted.

For example, the following url

  http://translate.google.com/translate_tts?tl=ru&q=вы

if entered directly into the browser correctly returns the text-to-speech
audo file but

when attempting to use HTTP-REQUEST, the url is being url encoded into

  http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B

of which google does not url-decode and fails to return the correct data.

So, for this case, the url-encoding is unwanted.

I am willing to submit patch an additional argument into HTTP-REQUEST to
disallow the encoding.

Thoughts?

Thank you,
William
_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

drakma and non-ASCII content

Robert Brown — 2012-03-09T15:24:44

A co-worker of mine had some problems today using Drakma to POST a STRING
containing non-ASCII characters encoded as UTF-8.  He was doing something
equivalent to:

(http-request "http://zappa.com/favicon.ico"
              :method :post
              :content (concatenate 'string
                                    "hello" (string #\white_square) "world")
              :external-format-out :utf-8)

which ends up setting the Content-Length header value to 11, which is the
LENGTH in characters of the string being sent.

The documentation says that "if content is a sequence, Drakma will use LENGTH
to determine its length and will use the result for the 'Content-Length' header
sent to the server," so Drakma is working as documented.

Also, I think I understand why this behavior is the default.  You don't want to
scan a content string in order to determine what its length will be in octets
once it has been encoded.  My co-worker could have used a vector of type
(unsigned-byte 8) to hold his UTF-8 encoded data.

However, I think the current Drakma behavior may be a mistake.  People who want
high performance are probably manipulating encoded strings as vectors of
(unsigned-byte 8).  It's the casual users, those sending strings, who are the
ones most likely to be bitten by the default behavior, but only when their
strings contain non-ASCII characters.

bob

http redirect following url-encodes location

Willi Nesensohn — 2012-02-27T19:20:29

I'm using drakma to GET data over http. It turns out that when following 
redirects, drakma uses the location in the redirection header, which it 
url-encodes. Usually this might not be a problem, but when the location 
redirected to is already url-encoded this method fails.

Is this intended behaviour?

I'm not sure what the clean solution to this would be, as decoding the 
location before using it just doesn't feel right. For the lack of a 
better solution that's what i'm doing right now though.

Any pointers in the right direction would be greatly appreciated.

Best regards,
Wilfried Nesensohn

A simple Cookie and viewing the actual Post

Michael Minerva — 2012-02-09T20:41:17

I just started using drakma today and so far it seems really great but I had a couple quesitons.

First, is it possible to send a simpler for example i want my http-post to have something like this:

Cookie: SESSdc0685ed01f285dab628a3700259e6bc=rCfmWLUeugRQNAQTs14yox_DWqZyGE2fYUS8W24j4co

I tried making a drakma cookie and adding it to a cookie-jar but none of that seemed to work.

Also, is there a way to print the actual post I am sending so I can see what might be wrong with it?

Thanks a lot,

--Mike

Drakma and client certificates?

Rob Blackwell — 2012-01-14T12:15:11

Hi,

I'm trying to use client certificates with Drakma to call an existing web based API.

I have the latest code from https://github.com/edicl/drakma installed in my quicklisp local-projects directory. I'm using SBCL on OS X and Linux.

The client certificate is a .pfx file, and I've converted it to .pem format using

openssl pkcs12 -in robblackwellmanage.pfx -out robblackwellmanage.pfx.pem

I have verfied that this works as a server certifcate, using hunchentoot, thus:

(hunchentoot:start (make-instance 'hunchentoot:ssl-acceptor :ssl-privatekey-file "/Users/reb/certs/robblackwellmanage.pfx.pem" :ssl-certificate-file "/Users/reb/certs/robblackwellmanage.pfx.pem" :ssl-privatekey-password "password" :port 4343))

When I try to use this as a client certificate:

 (drakma:http-request resource
 :certificate "/Users/reb/certs/robblackwellmanage.pfx.pem"
 :key "/Users/reb/certs/robblackwellmanage.pfx.pem"
 :certificate-password "password"
 :method :get
 :content ""
 :content-type "application/xml"
 :additional-headers headers)

I get the following error

SSL initialization error: Can't load certificate passwordSSL error queue is empty.
   [Condition of type CL+SSL::SSL-ERROR-INITIALIZE]

Interestingly, if I deliberately supply the wrong password then I get a different error

SSL initialization error: Can't load RSA private key file /Users/reb/certs/robblackwellmanage.pfx.pemSSL error queue is empty.
   [Condition of type CL+SSL::SSL-ERROR-INITIALIZE]

I guess I'm doing something really silly - please excuse my ignorance, but any advice, suggestions or guidance on how to proceed would be very much appreciated.

Thanks!

Rob

cl-oauth and drakma interoperability issue

mohegan-skunkworks — 2011-12-29T18:01:45

I just filled an issue on github...

The text is below...
Hi,
I'm using cl-oauth to authenticate with twitter (it's in the cl-twitter package, also on github).

I received a complaint from a quicklisp user that he couldn't authenticate with twitter.

After looking into this a bit I realized that drakma is encoding the uri on get requests. Now, cl-oauth passes in an encoded uri so this uri gets encoded twice. This was never a problem for me because I was using v 1.2.3 of drakma where this didn't happen.

Here's the code diff (plus my proposed change :)
========================== v 1.2.3 ==================================
(when (and (not parameters-used-p)
parameters)
(setf (uri-query uri)
;; append parameters to existing query of URI
(format nil "~< at >[~A~]~:*~:[~;&~]~A"
(uri-query uri)
(alist-to-url-encoded-string parameters external-format-out))))

===========================current github/v1.2.4=============
(when-let (all-get-parameters
(append (dissect-query (uri-query uri))
(and (not parameters-used-p) parameters)))
(setf (uri-query uri)
(alist-to-url-encoded-string all-get-parameters external-format-out)))
===============================my proposed change===================================================
(when (and (not parameters-used-p)
parameters)
(when-let (all-get-parameters
(append (dissect-query (uri-query uri))
(and (not parameters-used-p) parameters)))
(setf (uri-query uri)
(alist-to-url-encoded-string all-get-parameters external-format-out))))

A few more comments :

This is the hand-off in cl-oauth :

(defun http-request (uri &key (request-method :get) parameters drakma-args)
  ;; TODO handle redirects properly
  (let* ((param-string-encoded (alist->query-string parameters :include-leading-ampersand nil :url-encode t)))
    (case request-method
      (:get 
       (apply #'drakma:http-request 
      (uri-with-additional-query-part uri param-string-encoded)
      :method request-method
      drakma-args))
      (:post
        (apply #'drakma:http-request
               uri
               :method request-method
               :content param-string-encoded
               drakma-args))

As you can see url-encode is set to t. That was because (I think !) previous versions required encoding and drakma wasn't providing any. Now, ideally tis flag s/b nil. However the issue the becomes the string splitting in dissect-query. This splits on "=" which is also the terminating symbol for the authentication string...
=======================session===================
(drakma::split-string "oauth_signature=oq37d1/qm[....]fIKb778=&include_entities=T&oauth_consumer_key=9[....]cYBg&oauth_token=206[...]Tt5SwRvCJqQWgR3ajEQpk&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1325002586&oauth_nonce=3680613621135035286&oauth_version=1.0" "&")
("oauth_signature=oq37d1/qmFX0YuQUwxsgfIKb778=" "include_entities=T"
[....])

CL-USER> (drakma::split-string "oauth_signature=oq37d1/qmFX0YuQUwxsgfIKb778=" "=")
; compiling (DEFUN HTTP-REQUEST ...)
STYLE-WARNING: redefining DRAKMA:HTTP-REQUEST in DEFUN

("oauth_signature" "oq37d1/qm[...]778")
(I've elided some of the strings...).

I think my proposal resolves the issue. However, it would require a bit more work to get to what I think is the 'right' solution...

At this stage I'd like to get some feedback on whta you consider the right course of action before proceeding to submit a patch_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

HTTP Put and form-data

Ben Holm — 2011-11-13T21:54:44

Hi all. I started using drakma the other day to do some client work with
reviewboard, and I had a little trouble. Reviewboard requires http puts
using form-data for updates (
http://www.reviewboard.org/docs/manual/1.6/webapi/2.0/overview/). Drakma
explicitly disallows this.

I modified request.lisp to let my client do what I needed. Since this is my
first time writing a webservice client, I'm not sure if reviewboard or
drakma is wrong, or if I misunderstood something. Below is the diff of what
I did.

Any thoughts?

Thanks,
-Ben

Index: request.lisp
===================================================================
--- request.lisp (revision 4683)
+++ request.lisp (working copy)
< at >< at > -410,7 +410,7 < at >< at >
     (parameter-error "Don't know how to handle scheme ~S." (uri-scheme
uri)))
   (when (and close keep-alive)
     (parameter-error "CLOSE and KEEP-ALIVE must not be both true."))
-  (when (and form-data (not (eq method :post)))
+  (when (and form-data (and (not (eq method :put)) (not (eq method
:post))))
     (parameter-error "FORM-DATA makes only sense with POST requests."))
   (when range
     (unless (and (listp range)
< at >< at > -432,7 +432,7 < at >< at >
     (when (and file-parameters-p (not (eq method :post)))
       (parameter-error "Don't know how to handle parameters in ~S, as this
is not a POST request."
                        parameters))
-    (when (eq method :post)
+    (when (or (eq method :put) (eq method :post))
       ;; create content body for POST unless it was provided
       (unless content
         ;; mark PARAMETERS argument as used up, so we don't use it
_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

read-status-line is seeing the HTTP request line!?

Daniel Weinreb — 2011-03-17T15:30:21

Has anybody seen a behavior in which read-status-line
signals a syntax-error, and it turns out that the value
of "line" looks like an HTTP request line rather
than an HTTP status (response) line?  The following
is the stack trace.

 From my old C++ days (may they never return), I
conjectured that some string buffer is being reused
or something like that.; We are running on CCL
(formerly openmcl), and nothing like that is
going on at the CCL level.; I don't know much
about flexi-streams or gray-streams. Anyway
that's just a rather wild guess. Thanks!

<READ-STATUS-LINE at pc +669
     Arguments: (STREAM &amp;OPTIONAL DRAKMA::LOG-STREAM)
       STREAM: <FLEXI-STREAMS:FLEXI-IO-STREAM #x30200938EB3D>
       DRAKMA::LOG-STREAM: NIL
     Local bindings:
       CHUNGA:CURRENT-ERROR-MESSAGE "While reading status line:""
       DRAKMA::LINE: ""POST /stat/ping HTTP/1.1""
       DRAKMA::FIRST-SPACE-POS: 4
       DRAKMA::SECOND-SPACE-POS: 15

Thanks!

- Dan

http-request error

André Schütz — 2011-02-28T15:50:06

Hello,

I loaded the drakma system successfully into my clisp environment and got the following error when I tried to make a http-request:

(http-request "http://lisp.org/")

Fehler: NODELAY in SOCKET-CONNECT is unsupported.

Any ideas?

Thanks in advance
Andre Schütz

Failure to load under SBCL on MS Windows 7 (64-bit)

Duncan Bayne — 2011-01-21T04:41:37

Hi All,

I'm using Drakma via Quicklisp on SBCL.  This works fine under Ubuntu
10.10, but when I attempt to quickload Drakma on MS Windows Ultimate
64-bit, I get 'Error opening shared object "libssl32.dll"'.

Here's a REPL session that demonstrates the problem:

=====
* (load (merge-pathnames "quicklisp/setup.lisp" (user-homedir-pathname)))

T
* (ql:quickload '("drakma"))
To load "drakma":
  Load 1 ASDF system:
    drakma
; Loading "drakma"
................
debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR:
  Unable to load foreign library (LIBSSL).
  Error opening shared object "libssl32.dll":
  126.

Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY          ] Try loading the foreign library again.
  1: [USE-VALUE      ] Use another library instead.
  2: [TRY-RECOMPILING] Recompile reload and try loading it again
  3: [RETRY          ] Retry loading component ("cl+ssl" "reload").
  4: [ACCEPT         ] Continue, treating loading component ("cl+ssl" "reload")
                       as having been successful.
  5: [ABORT          ] Give up on "drakma"
  6:                   Exit debugger, returning to top level.

(CFFI::FL-ERROR "Unable to load foreign library (~A).~%  ~A")[:EXTERNAL]
0]
=====

I'm quite new to Lisp, but very happy to help in any way necessary
(running debug builds, etc.) with tracking this down.

Yours,
Duncan Bayne

[Patch]: better file-names when sendingmultipart/form-data from streams.

Stas Boukarev — 2010-12-26T14:01:45

The attached patch does two things:
* :filename option takes precedence of automatically determined names
 (it was the case previously only for pathnames, and not for streams or functions).
* if stream is a file-stream, which means it's a pathname designator,
use file-namestring to get the filename.

Patch for disabling SSL in drakma

Marko Kocić — 2010-10-06T09:47:52

Hi all,

I'm sending a quick patch to disable CL+SSL dependency, as well as
https support in drakma when :drakma-no-ssl feature is present.
It works in a similar way as :hunchentoot-no-ssl for hunchentoot.
It also added quick info about it in "Download and installation"
section of the documentation.

The motivation is to be able to use drakma on windows where working
libssl32 is not easy to be found.

Regards,
Marko Kocic
_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

drakma patch: Don't try to funcall keywords

Faré — 2010-09-28T15:41:08

This bit me when updating Drakma. Note that another function in same
file uses the same colloquialism as this one: testing if a symbol is
fbound before trying to funcall it, otherwise just print it - which
works great for keywords.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Don't forget your daily prayer to Baah-kup,
the God of data storage and recovery!
_______________________________________________
drakma-devel mailing list
drakma-devel< at >common-lisp.net
http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel

SOCKS5 proxy support

Red Daly — 2010-08-06T23:29:44

I have modified Drakma to use a SOCKS5 proxy if desired.  Before I
submit a patch I'd like to know what is the preferred interface for
this:

1.  Modify http-request with a SOCKS-PROXY option analogous to the
PROXY option.  Introduce SOCKS5 protocol code  (about 200 lines) into
hunchentoot and rely on no external libraries

2.  Modify http-request with a SOCKS-PROXY option analogous to the
PROXY option.  Introduce a dependency on an external SOCKS5 library.

3.  Modify Drakma in some way that allows a user to establish a
socks-proxy connection without explicitly supporting this in Drakma
itself.

The implementation requires that instead of establishing a socket
connection to HOST/PORT, the socket connects to SOCKS-HOST/SOCKS-PORT
instead and then sets up the socket to forward to HOST/PORT.

The implementation is available here:
http://github.com/gonzojive/drakma/tree/socks

socks5 spec: http://www.faqs.org/rfcs/rfc1928.html

Red

New release 1.2.3 [Was: duplicate cookies problemwith proposed patch]

Edi Weitz — 2010-08-05T19:32:48

Thanks for the patch.  I'm back from vacation now and I've just
released 1.2.3 which fixes this.

Edi.


On Thu, Jul 29, 2010 at 11:22 AM, Vsevolod Dyomkin <vseloved< at >gmail.com> wrote:

Regarding URL encoding...

Herbert Snorrason — 2010-07-21T13:28:42

Although I understand why it's not done by default, it seems slightly
odd to me that Drakma doesn't provide a URL-ENCODE function whereas
Hunchentoot does. I would've thought the need was considerably more
immediate in Drakma...

With greetings,
  Herbert Snorrason

Patch: Unclosed `node in doc/index.html`


Walter GR — 2010-07-11T23:29:36
Patch attached.

An unclosed <code> node in doc/index.html causes all text after the 
documentation for *allow-dotless-cookie-domains-p* to render in a 
monospaced font.

This is my first patch ever, so let me know if I did it wrong.

Thanks,

Walter



New Drakma release 1.2.2
Edi Weitz — 2010-07-10T10:06:11
There's a new release of Drakma out at weitz.de.  This is a bugfix release.

ChangeLog: http://weitz.de/drakma/CHANGELOG.txt



New version 1.2.1
Edi Weitz — 2010-05-19T19:41:45
Yeah, sorry, I should have been more careful.  At least this was
reported by three persons in a couple of minutes which gives me the
warm and fuzzy feeling that there are enough users to catch
mistakes... :)

Thanks.  Bugfix release online.

Edi.


On Wed, May 19, 2010 at 8:36 PM, Zach Beane <xach< at >xach.com> wrote:



New Chunga version 1.1.1
Edi Weitz — 2010-05-19T15:25:03
And, again on weitz.de, there's also a new release of Chunga.



Version 1.1.1
2010-05-19
Read quoted cookie values (Red Daly)