gmane.comp.lang.ruby.modules.mechanize.user

Error trying to reload serialized page

Jack Royal-Gordon — 2012-05-23T19:40:15

What is the correct way to serialize an HTML page gotten from Agent::get, and then to reload the serialized page back into Mechanize?  I tried the following code:
agent = Mechanize.new
p1 = agent.get(url)
=> #<Mechanize::Page … >

p2 = Mechanize::Page.new(p1.uri.to_s, p1.response.to_s, p1.body)

and got the error:
Mechanize::ContentTypeError Exception: Mechanize::ContentTypeError

I also tried:
p2 = Mechanize::Page .new(p1.uri, p1.response, p1.body)

and got the error:
INTERNAL ERROR!!! undefined method `follow_meta_refresh' for nil:NilClass
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/activesupport-3.1.3/lib/active_support/whiny_nil.rb:48:in `method_missing'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:350:in `meta_refresh'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:139:in `block (2 levels) in pretty_print'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:138:in `block in pretty_print'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:164:in `object_group'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/mechanize-2.4/lib/mechanize/page.rb:134:in `pretty_print'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:154:in `block in pp'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:199:in `block (2 levels) in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:225:in `nest'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:198:in `block in group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:210:in `group_sub'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/prettyprint.rb:197:in `group'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:154:in `pp'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:77:in `block in pp'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:121:in `guard_inspect_key'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:77:in `pp'
/Users/jackrg/.rbenv/versions/1.9.2-p290/lib/ruby/1.9.1/pp.rb:51:in `pretty_inspect'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:47:in `block in execute'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:19:in `run_with_binding'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/commands/eval.rb:46:in `execute'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:280:in `one_cmd'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:261:in `block (2 levels) in process_commands'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:260:in `each'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:260:in `block in process_commands'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:253:in `catch'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:253:in `process_commands'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug19-0.11.6/cli/ruby-debug/processor.rb:173:in `at_line'
(eval):5:in `block in at_line'
<internal:prelude>:10:in `synchronize'
(eval):3:in `at_line'
/Users/jackrg/Documents/Novelty-Stats/vendor/local/ruby/1.9.1/gems/ruby-debug-base19-0.11.25/lib/ruby-debug-base.rb:55:in `at_line'
/Users/jackrg/Documents/Novelty-Stats/prototype/test4.rb:61:in `mechanize_page'
(irb):4:in `irb_binding'


_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

way to get response headers only?

shig odani — 2012-04-30T14:21:12

Hello, I'm interested in grabbing the final uri of a given uri after any
redirects without retrieving the response body of this final uri. For
example, if uri1 redirects to uri2, which in turn redirects to uri3, I'd
like to just grab the response header of uri3.

I know that I can get the full response (header and body) by doing
Mechanize.new.get(uri1).uri, but again, I'm looking to make just a header
request.

Thanks.
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

Resource time out error handling

Ross Cameron — 2012-03-27T05:25:27

Hi

I have a long running process and eventually get the following error:

/home/linker/.rvm/gems/ruby-1.8.7-p358/gems/net-http-persistent-2.5.2/lib/net/http/persistent.rb:821:in 
`request': too many connection resets (due to Resource temporarily 
unavailable - Timeout::Error) after 0 requests on 1777511481920, last 
used 1332824337.78363 seconds ago (Net::HTTP::Persistent::Error)

I have not found a way as yet to trap Net:HTTP errors through Mechanize.

Would someone please point me in the right direction for dealing 
/handling this situation.

Thanks
Ross



_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

dealing with Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

Robert Poor — 2012-01-27T20:41:25

I'm 99% certain that this is what's being discussed in

https://github.com/tenderlove/mechanize/issues/168

But I'm not sure what the remedy is.  My production app is running on
Heroku, so I need a fix that I can push there.  Can someone post
instructions that this neophyte can follow?  (I gather I'm supposed to fork
and add the current master version from github to my config, but I need
simple little steps for muddy little feet.)  Thanks...

I'm running Rails 3.2, Mechanize 2.1.  Mechanize is failing with

        Encoding::UndefinedConversionError: "\xA2" from ASCII-8BIT to UTF-8

with this (partial) stack trace:

from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:807:in
`block in response_read'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:387:in `call_block'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:378:in `<<'
from /usr/local/lib/ruby/1.9.1/net/protocol.rb:88:in `read'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2445:in `read_chunked'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2419:in `read_body_0'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2379:in `read_body'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:795:in
`response_read'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:267:in
`block in fetch'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1194:in `block in
transport_request'
from /usr/local/lib/ruby/1.9.1/net/http.rb:2342:in `reading_body'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1193:in `transport_request'
from /usr/local/lib/ruby/1.9.1/net/http.rb:1177:in `request'
from
/app/vendor/bundle/ruby/1.9.1/gems/net-http-persistent-2.3.3/lib/net/http/persistent.rb:621:in
`request'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize/http/agent.rb:264:in
`fetch'
from
/app/vendor/bundle/ruby/1.9.1/gems/mechanize-2.1/lib/mechanize.rb:319:in
`get'
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

Reload previously retrieved page

Jack Royal-Gordon — 2012-01-15T02:45:58

I'm a newbie with Mechanize, and I have what seems like a simple question: I'm trying to save a page that has been loaded and then reload it later for further processing.  I see that Mechanize has a method parse(uri, response, body) that (I presume) can load the DOM without physically accessing the site over the internet.  If I save the URI and body from a previous Mechanize.get(), how do I get the "response" from the page that is also required in order to reload the page?  Or is there a better way to save and restore the page without incurring the latency of a second internet access?  I've also tried Mechanize::Page.new() but I have the same problem in that I need a "response" for that call (nil gives an error).

500 error in tdiary using mechanize

三輪晋( Miwa Susumu — 2012-01-07T15:36:38

hi all

i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/ ).

i am trying to update the diary with the mechanize.

i run the code (see below), mechanize returns
Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

Googled Treats, that it's a server-side error.

(e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
program doesn't work, but it works in the browser.
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser.
)

However, there are no errors on the server side.

Do people know the cause?

In addition, web browser( Firefox ) can safely update my diary.

[Environment]
% ruby --version
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]

% gem list

*** LOCAL GEMS ***

domain_name (0.5.1)
mechanize (2.1)
net-http-digest_auth (1.2)
net-http-persistent (2.3.3)
nokogiri (1.5.0)
ntlm-http (0.1.1)
tmail (1.2.7.1)
unf (0.0.4)
unf_ext (0.0.4)
webrobots (0.0.12)


 * NetBSD 5.99.58
 * tdiary 3.1.0


[code]
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
  uri = "http://localhost/~rin/diary/update.rb"
  user = "test"
  pass = "test"
  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

  text = "!test\nhello tdiary!\n"

  agent = Mechanize.new
  agent.log = Logger.new($stdout)
#  agent.log.level = Logger::INFO
  agent.auth(user, pass)                 <= i use digest authentication
  agent.get(uri, nil, referer)
  agent.page.form_with(:action => "update.rb") { |form|
    form['body'] = text
    form.submit(form.button_with(:name => "append"))           <=
seems to be an error here.
  }

end

main


[mechanize log]
% ./tdwrite.rb
I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=100
D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="c304545256e7e4d0a03c3e2d6ae4ec80"
D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
content-type => text/html; charset=UTF-8
D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
content-length => 7160
D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=99
D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
=> User-Agent,Accept-Encoding
D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160 total)
D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
"old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=98
D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="a903c6e42ed5896cd1da05516fc5b93d"
D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
Net::HTTPInternalServerError 1.1 500 Internal Server Error
D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
content-type => text/html
D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
connection => close
D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
content-length => 416
D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
/usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
`fetch': 500 => Net::HTTPInternalServerError
(Mechanize::ResponseCodeError)
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
`response_authenticate'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
`fetch'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
`post_form'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
`submit'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
`submit'
        from ./tdwrite.rb:24:in `main'
        from (eval):23:in `form_with'
        from ./tdwrite.rb:22:in `main'
        from ./tdwrite.rb:29


[apache web server access_log]

127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
/~rin/diary/update.rb HTTP/1.1" 200 7160
127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
/~rin/diary/update.rb HTTP/1.1" 500 416


[apache web server error_log]

(In particular there are no errors)

500 error in tdiary using mechanize

三輪晋( Miwa Susumu — 2012-01-07T15:36:38

hi all

i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/ ).

i am trying to update the diary with the mechanize.

i run the code (see below), mechanize returns
Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

Googled Treats, that it's a server-side error.

(e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
program doesn't work, but it works in the browser.
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser.
)

However, there are no errors on the server side.

Do people know the cause?

In addition, web browser( Firefox ) can safely update my diary.

[Environment]
% ruby --version
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]

% gem list

*** LOCAL GEMS ***

domain_name (0.5.1)
mechanize (2.1)
net-http-digest_auth (1.2)
net-http-persistent (2.3.3)
nokogiri (1.5.0)
ntlm-http (0.1.1)
tmail (1.2.7.1)
unf (0.0.4)
unf_ext (0.0.4)
webrobots (0.0.12)


 * NetBSD 5.99.58
 * tdiary 3.1.0


[code]
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
  uri = "http://localhost/~rin/diary/update.rb"
  user = "test"
  pass = "test"
  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

  text = "!test\nhello tdiary!\n"

  agent = Mechanize.new
  agent.log = Logger.new($stdout)
#  agent.log.level = Logger::INFO
  agent.auth(user, pass)                 <= i use digest authentication
  agent.get(uri, nil, referer)
  agent.page.form_with(:action => "update.rb") { |form|
    form['body'] = text
    form.submit(form.button_with(:name => "append"))           <=
seems to be an error here.
  }

end

main


[mechanize log]
% ./tdwrite.rb
I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=100
D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="c304545256e7e4d0a03c3e2d6ae4ec80"
D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
content-type => text/html; charset=UTF-8
D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
content-length => 7160
D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=99
D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
=> User-Agent,Accept-Encoding
D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160 total)
D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
"old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=98
D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="a903c6e42ed5896cd1da05516fc5b93d"
D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
Net::HTTPInternalServerError 1.1 500 Internal Server Error
D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
content-type => text/html
D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
connection => close
D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
content-length => 416
D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
/usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
`fetch': 500 => Net::HTTPInternalServerError
(Mechanize::ResponseCodeError)
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
`response_authenticate'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
`fetch'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
`post_form'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
`submit'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
`submit'
        from ./tdwrite.rb:24:in `main'
        from (eval):23:in `form_with'
        from ./tdwrite.rb:22:in `main'
        from ./tdwrite.rb:29


[apache web server access_log]

127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
/~rin/diary/update.rb HTTP/1.1" 200 7160
127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
/~rin/diary/update.rb HTTP/1.1" 500 416


[apache web server error_log]

(In particular there are no errors)

How to download file after button click with noURL

Piotr Polok — 2012-01-04T06:33:16

HI,

I'm trying to download file with no success using below code:

  $mech->get( $search_result );
  $mech->form_number( 4 );

  $mech->current_form()->click("eksport_do_pliku");
  print $mech->res();

This is part of HTML source:
"
<button type="submit" name="eksport_do_pliku" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px">Eksportuj dane do pliku 
</button>
"

The result of "$mech->res();" is "HTTP::Response=HASH(0x8a49d10)"

Please help :)

500 error in tdiary using mechanize

三輪晋( Miwa Susumu — 2012-01-07T15:36:38

hi all

i use tdiary( The web diary system http://sourceforge.net/projects/tdiary/ ).

i am trying to update the diary with the mechanize.

i run the code (see below), mechanize returns
Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

Googled Treats, that it's a server-side error.

(e.g. but perl... WWW::Mechanize::FAQ - search.cpan.org My Mech
program doesn't work, but it works in the browser.
http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/FAQ.pod#My_Mech_program_doesn%27t_work,_but_it_works_in_the_browser.
)

However, there are no errors on the server side.

Do people know the cause?

In addition, web browser( Firefox ) can safely update my diary.

[Environment]
% ruby --version
ruby 1.8.7 (2011-02-18 patchlevel 334) [i386-netbsdelf]

% gem list

*** LOCAL GEMS ***

domain_name (0.5.1)
mechanize (2.1)
net-http-digest_auth (1.2)
net-http-persistent (2.3.3)
nokogiri (1.5.0)
ntlm-http (0.1.1)
tmail (1.2.7.1)
unf (0.0.4)
unf_ext (0.0.4)
webrobots (0.0.12)


 * NetBSD 5.99.58
 * tdiary 3.1.0


[code]
#!/usr/pkg/bin/ruby -Ku

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

def main
  uri = "http://localhost/~rin/diary/update.rb"
  user = "test"
  pass = "test"
  referer = "http://www.area51.gr.jp/~rin/diary/update.rb"

  text = "!test\nhello tdiary!\n"

  agent = Mechanize.new
  agent.log = Logger.new($stdout)
#  agent.log.level = Logger::INFO
  agent.auth(user, pass)                 <= i use digest authentication
  agent.get(uri, nil, referer)
  agent.page.form_with(:action => "update.rb") { |form|
    form['body'] = text
    form.submit(form.button_with(:name => "append"))           <=
seems to be an error here.
  }

end

main


[mechanize log]
% ./tdwrite.rb
I, [2012-01-08T00:25:20.594807 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595147 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.595197 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.595264 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.595308 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.595352 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.595395 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.595434 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.598261 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.598313 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.598359 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.598397 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.598433 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.598469 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.598504 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.598538 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.598573 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=100
D, [2012-01-08T00:25:20.598794 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.598965 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.599088 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.605070 #9165]  INFO -- : Net::HTTP::Get:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605298 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.605371 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.605417 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.605462 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.605499 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="4NEBxvG1BAA=22e28cac4aac01662c270678d2355fbbfbe46395",
nc=00000000, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="c304545256e7e4d0a03c3e2d6ae4ec80"
D, [2012-01-08T00:25:20.605545 #9165] DEBUG -- : request-header:
referer => http://www.area51.gr.jp/~rin/diary/update.rb
D, [2012-01-08T00:25:20.605581 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.605616 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.755405 #9165]  INFO -- : status: Net::HTTPOK 1.1 200 OK
D, [2012-01-08T00:25:20.755802 #9165] DEBUG -- : response-header:
content-type => text/html; charset=UTF-8
D, [2012-01-08T00:25:20.755878 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.755925 #9165] DEBUG -- : response-header:
authentication-info => rspauth="af4628a938bc6d2b04bf73ba60fb4a59",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000000, qop=auth
D, [2012-01-08T00:25:20.755962 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.755998 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.756032 #9165] DEBUG -- : response-header:
content-length => 7160
D, [2012-01-08T00:25:20.756067 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.756102 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=99
D, [2012-01-08T00:25:20.756137 #9165] DEBUG -- : response-header: vary
=> User-Agent,Accept-Encoding
D, [2012-01-08T00:25:20.756250 #9165] DEBUG -- : Read 7160 bytes (7160 total)
D, [2012-01-08T00:25:20.756403 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.772181 #9165]  INFO -- : form encoding: ISO-8859-1
D, [2012-01-08T00:25:20.774358 #9165] DEBUG -- : query:
"old=20120108&year=2012&month=1&day=8&title=&body=%21test%0Ahello+tdiary%21%0A&plugin_ping_send=true&api=0&plugin_tags_filter=on&plugin_tb_url=&plugin_tb_section=p01&plugin_tb_excerpt=&append=+%E8%BF%BD%E8%A8%98+"
I, [2012-01-08T00:25:20.775397 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.775641 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.775879 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.776086 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.776280 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.776507 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.776697 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.776922 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.777128 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.777381 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.778814 #9165]  INFO -- : status:
Net::HTTPUnauthorized 1.1 401 Authorization Required
D, [2012-01-08T00:25:20.779127 #9165] DEBUG -- : response-header:
content-type => text/html; charset=iso-8859-1
D, [2012-01-08T00:25:20.779357 #9165] DEBUG -- : response-header:
connection => Keep-Alive
D, [2012-01-08T00:25:20.779569 #9165] DEBUG -- : response-header:
www-authenticate => Digest realm="tDiary",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
algorithm=MD5, qop="auth"
D, [2012-01-08T00:25:20.779818 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.780037 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.780231 #9165] DEBUG -- : response-header:
content-length => 290
D, [2012-01-08T00:25:20.780439 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.780628 #9165] DEBUG -- : response-header:
keep-alive => timeout=5, max=98
D, [2012-01-08T00:25:20.780847 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.781100 #9165] DEBUG -- : Read 290 bytes (290 total)
D, [2012-01-08T00:25:20.781460 #9165] DEBUG -- : gzip body
I, [2012-01-08T00:25:20.787175 #9165]  INFO -- : Net::HTTP::Post:
/~rin/diary/update.rb
D, [2012-01-08T00:25:20.787408 #9165] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5
D, [2012-01-08T00:25:20.787652 #9165] DEBUG -- : request-header: accept => */*
D, [2012-01-08T00:25:20.787889 #9165] DEBUG -- : request-header:
content-type => application/x-www-form-urlencoded
D, [2012-01-08T00:25:20.788136 #9165] DEBUG -- : request-header:
user-agent => Mechanize/2.1 Ruby/1.8.7p334
(http://github.com/tenderlove/mechanize/)
D, [2012-01-08T00:25:20.788393 #9165] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity
D, [2012-01-08T00:25:20.788710 #9165] DEBUG -- : request-header:
authorization => Digest username="diary", realm="tDiary", qop=auth,
uri="/~rin/diary/update.rb",
nonce="6ZYExvG1BAA=98d27d517f0275958fb5065ff69c8088ea850008",
nc=00000001, cnonce="a6a1370113ae4db8792628a3e87927e8",
response="a903c6e42ed5896cd1da05516fc5b93d"
D, [2012-01-08T00:25:20.789087 #9165] DEBUG -- : request-header:
content-length => 211
D, [2012-01-08T00:25:20.789302 #9165] DEBUG -- : request-header:
referer => http://diary:diary< at >localhost/~rin/diary/update.rb
D, [2012-01-08T00:25:20.789520 #9165] DEBUG -- : request-header: host
=> localhost
D, [2012-01-08T00:25:20.789711 #9165] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
I, [2012-01-08T00:25:20.842091 #9165]  INFO -- : status:
Net::HTTPInternalServerError 1.1 500 Internal Server Error
D, [2012-01-08T00:25:20.842477 #9165] DEBUG -- : response-header:
content-type => text/html
D, [2012-01-08T00:25:20.842718 #9165] DEBUG -- : response-header:
connection => close
D, [2012-01-08T00:25:20.842961 #9165] DEBUG -- : response-header:
authentication-info => rspauth="f92b2000ce402faad6c351d19abdc07c",
cnonce="a6a1370113ae4db8792628a3e87927e8", nc=00000001, qop=auth
D, [2012-01-08T00:25:20.843216 #9165] DEBUG -- : response-header:
server => Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.9-dev DAV/2
D, [2012-01-08T00:25:20.843419 #9165] DEBUG -- : response-header: date
=> Sat, 07 Jan 2012 15:25:20 GMT
D, [2012-01-08T00:25:20.843644 #9165] DEBUG -- : response-header:
content-length => 416
D, [2012-01-08T00:25:20.843865 #9165] DEBUG -- : response-header:
content-encoding => gzip
D, [2012-01-08T00:25:20.844076 #9165] DEBUG -- : response-header: vary
=> Accept-Encoding
D, [2012-01-08T00:25:20.844377 #9165] DEBUG -- : Read 416 bytes (416 total)
D, [2012-01-08T00:25:20.844808 #9165] DEBUG -- : gzip body
/usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:303:in
`fetch': 500 => Net::HTTPInternalServerError
(Mechanize::ResponseCodeError)
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:655:in
`response_authenticate'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/http/agent.rb:300:in
`fetch'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:1036:in
`post_form'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize.rb:426:in
`submit'
        from /usr/pkg/lib/ruby/gems/1.8/gems/mechanize-2.1/lib/mechanize/form.rb:178:in
`submit'
        from ./tdwrite.rb:24:in `main'
        from (eval):23:in `form_with'
        from ./tdwrite.rb:22:in `main'
        from ./tdwrite.rb:29


[apache web server access_log]

127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "GET /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "GET
/~rin/diary/update.rb HTTP/1.1" 200 7160
127.0.0.1 - - [08/Jan/2012:00:25:20 +0900] "POST /~rin/diary/update.rb
HTTP/1.1" 401 290
127.0.0.1 - diary [08/Jan/2012:00:25:20 +0900] "POST
/~rin/diary/update.rb HTTP/1.1" 500 416


[apache web server error_log]

(In particular there are no errors)

How to download file after button click with noURL

Piotr Polok — 2012-01-04T06:33:16

HI,

I'm trying to download file with no success using below code:

  $mech->get( $search_result );
  $mech->form_number( 4 );

  $mech->current_form()->click("eksport_do_pliku");
  print $mech->res();

This is part of HTML source:
"
<button type="submit" name="eksport_do_pliku" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px">Eksportuj dane do pliku 
</button>
"

The result of "$mech->res();" is "HTTP::Response=HASH(0x8a49d10)"

Please help :)

how to click button on the form with no names

Piotr Polok — 2012-01-03T04:54:39

HI,

I need to click second button on below form and download a file, could 
you please give a advice how to do that :)

HTML source:

             <form method="POST" target="_self">
                  <p style="margin-bottom: 
5px;margin-left:45px;margin-top:-40px;"><input id="radio_5" 
checked="checked" name="format" value="1" 
type="radio"style="vertical-align:middle;"><label for="radio_5"> liniowy 
</label></p></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_6" name="format" value="2" type="radio" 
style="vertical-align:middle;"><label for="radio_6"> xml</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_7"  name="format" value="3" 
type="radio"style="vertical-align:middle;"><label for="radio_7"> 
MT942</label></p>
                  <p style="margin-bottom: 5px;margin-left: 45px"><input 
id="radio_8" name="format" value="4" type="radio" 
style="vertical-align:middle;"><label for="radio_8"> 
MT942/Multicash</label></p>
             </td>
          </tr>
          <tr>
              <td colspan="2"><hr width="98%" color="#acacac" 
noshade="noshade" size="1" style="vertical-align: bottom"/></td></tr>
  </tr>
           <tr>
               <td class="wyciagi_w_td1">
                 <button type="submit" class="buty" 
onclick="document.getElementById('cod').value='1';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Zapisz 
parametry</button>
               </td>
               <td style="text-align: left;">
                 <input id="cod" name="cod" value="" type="hidden">
                 <button type="submit" class="buty" 
style="margin-left:20px" 
onclick="document.getElementById('cod').value='2';"><img 
src="gfx/przyciski/akceptuj.png" 
style="vertical-align:middle;margin-right:5px" >Eksportuj dane do pliku 
</button>

                 <a class="nw" 
href='javascript:openWindow("start.php?page=pdf&pdf=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/pdf.gif" 
style="vertical-align:middle;margin-right:5px;height:28px">Wydruk 
.pdf</a>&nbsp;
                 <a class="nw" 
href='javascript:openWindow("start.php?page=wydruk&typdruk=wirtualne&szukaj=wy_oddo&data_new1=2011-12-03&data_new2=2012-01-03&sort_typ=data&rows=100&rach=1&idnrb=5&rach_wir=1&rach_wirtualne=&types=oddo")' 
style="margin-left:30px"><img src="gfx/przyciski/drukarka.png" 
style="vertical-align:middle;margin-right:5px;">Przygotuj wydruk</a>
              </td>
  </tr>
</table>
     </FORM>

deobfuscating javascript?

shig odani — 2011-12-29T15:33:19

Not sure if this post is quite appropriate for this list but wasn't sure
where else to ask. I'm using mechanize to visit some pages that have
obfuscated javascript and I'm wondering if there's some ruby or mechanize
way to deobfuscate it or otherwise interact with the html elements which
are being obfuscated by javascript.

For more background, please see
http://www.labnol.org/software/deobfuscate-javascript/19815/
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

creating a Mechanize::Form from scratch...

Robert Poor — 2011-12-21T04:41:10

I need to POST a form that doesn't originate from a GET[*].  I know
what belongs in the form.  It's not clear to me how to create a
Mechanize::Form from scratch -- the only way I can see to do it is GET
a page that has a form on it, strip its contents and repopulate it.
But is there an easier (or less surprising) way to do this?

- ff

[*] If you must know, the site I'm working with uses javascript to
construct and post forms, so Mechanize doesn't have an obvious form to
work with.

Mechanize GETting twice without redirect?

Robert Poor — 2011-12-20T18:27:44

[Cross posted to Ruby on Rails Forum and Mechanize mailing list.]

I'm using Mechanize for page scraping (Ruby 1.9.2 / Rails 3.0.5 /
Mechanize 2.0.1).  I'm seeing a case where a single

  agent.get(url)

generates two HTTP GETs.  Why is this happening?

The response to the first GET is a 200 (no redirect) and doesn't have
any meta-refresh.  I don't see why Mechanize is issuing the second GET
(which happens to be failing with an EOFError with Content-Length /
body length mismatch).

Details: I'm using the nifty Charles web proxy debugger to monitor
browser / server interactions.

=====
In the original browser + server exchange, I see:

Req: POST /login/Login HTTP/1.1
Rsp: sets two cookies + HTTP/1.1 302 Moved Temporarily =>
        https://online.nationalgridus.com/eservice_enu/

Req: GET /eservice_enu/ HTTP/1.1
Rsp: set a cookie + HTTP/1.1 200 OK
The body contains onLoad Javascript to set this.location =
'start.swe?SWECmd=Start'

Req: GET /eservice_enu/start.swe?SWECmd=Start HTTP/1.1
Rsp: sets four cookies + HTTP/1.1 200 OK

=====
In the mechanize = server exchange:

My code: page2 = agent.submit(login_form)
Req: POST /login/Login HTTP/1.1
Rsp: set two cookies + HTTP/1.1 302 Moved Temporarily =>
        https://online.nationalgridus.com/eservice_enu/

Req: GET /eservice_enu/ HTTP/1.1
Rsp: set a cookie + HTTP/1.1 200 OK
The body contains onLoad Javascript to set this.location =
'start.swe?SWECmd=Start', but Mechanize can't follow that
automatically.  So I do an agent.get() to emulate it:

My code:   page3 =
agent.get("https://online.nationalgridus.com/eservice_enu/start.swe?SWECmd=Start")
Req: GET /eservice_enu/start.swe?SWECmd=Start HTTP/1.1
Rsp: sets four cookies + HTTP/1.1 200 OK

Note that at this point both the user driven and mechanize driven
interactions appear to be identical.  But Mechanize appears to
generate another GET all by itself:

Req: GET /eservice_enu/start.swe?SWECmd=Start HTTP/1.1
Rsp: sets four cookies + HTTP/1.1 200 OK

... and this response throws an EOFError:
Content-Length (536) does not match response body length (524) - EOFError

=====
So: Why did Mechanize generate that last GET without me asking it to?
 Was the EOFError actually in the first GET and it's doing a retry?
If so, how do I work around the length mismatch?

mechanize dinamic iframe

Adis Osmonov — 2011-10-11T17:11:14

Hi,
could anyone give me an idea how can I get any element inside iframe,
which created through javascript,
using mechanize?

I know that Mechanize doesn't handle Javascript, but there mb a solution?
Thanks in advance for any help.
Sorry for my english – I'm a russian

Takanori 9303807 Oogaru は不在にしております。

ohgaru takanori — 2011-09-17T13:02:15

2011/09/17 から不在にしております。*2011/09/24 に帰社いたします。*

Mail to you after back to the office.

Mechanize authentication

Skurt — 2011-09-17T08:01:47

Hello,
I'm trying to access authenticated web using mechanize 2.0.1.
Hopefully, Im not mistaken somewhere else, Im not really an IT expert,
so I'll be verbose, just to be sure.

I was able to break through what I believe is BASIC AUTHENTICATON with
this code:
require "rubygems"
require 'mechanize'
agent = Mechanize.new
agent.basic_auth('admin:', 'kloktat5x')
agent.get('http://10.0.0.138:8080/')

it's accessing my router administration and it works fine.
Where I have problems is with web-based or cookie-based
authentication, it is this kind where you enter your usename and
password into a html form and hit some process or submit button.

Well, my code is this:
(please mind confidentiality of my personal information and be so kind
and do not misuse it)

require 'rubygems'
require 'logger'
require 'mechanize'

agent = Mechanize.new {|a| a.log = Logger.new(STDERR) }
#agent.set_proxy('a-proxy', '8080')
page = agent.get 'http://www.hofyland.cz/'


form = page.forms.first
form.hlnick = 'skurt'
form.hlpsw = 'sptnkn34'

page = agent.submit form

output = File.open("output.html", "w") { |file|  file << page.body }

and you can try what the output.html will look like afterwards, It'll
be body of the login page - the script doesnt authenticate against the
web, it simply reloads the page and saves the body.

Is someone who have similar experiences, who may be able to solve this
kind of problem out there?

With my best regars
Jan Kadera

setting a form field creates duplicate fields?

Robert Poor — 2011-09-08T17:20:52

I receive a form, I set some fields, I post the form.  It all seems to
work, along these lines:

    form['__EVENTTARGET'] = 'objTimePeriods$rlPeriod$1'
    form['__EVENTARGUMENT'] = ''
    form['objTimePeriods$rlPeriod'] = 2
    agent.submit(form)

But when I look at what's going out over the wire (thanks to the
delightful www.charlesproxy.com), I see the following being posted:

objTimePeriods$rlPeriod: 2
__EVENTTARGET: objTimePeriods$rlPeriod$1
__EVENTARGUMENT:
objTimePeriods$rlPeriod: 1

Notice that the 'objTimePeriods$rlPeriod' field appears twice -- the
'1' was the previous value of the form before I set it to '2'.  I've
quadruple-checked the spelling and punctuation (even in the raw
transmitted byte stream), and they really do have the same field
names.

My code seems to work, but this may be just luck.   Am I doing
something wrong?  Do I need to check for the presence of a form field
before I set it?

weird encoding issue

boghra — 2011-08-01T09:09:41

Hi,

Just first day with Mechanize and I am trying to access facebook fan page
wall posts i.e. facebook.com/walmart?sk=wall, example:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
agent.user_agent_alias = 'Windows Mozilla'
page = agent.get('http://www.facebook.com')

form = page.form_with(:id => 'login_form')
# just removed actual username password
form.email = 'fb-user-name'
form.pass = 'fb-password'
page = agent.submit(form)

page = agent.get("/walmart?sk=wall")
span  = page.search "//span[< at >class='messageBody']"
puts span

puts span doesn't print anything (and actual page has spans with given class
name) and if try to print page.body, for all html opening tag '<' are
printed as \u and html gets escaped, not sure how to tell it to use utf
encoding ? also not sure why things get escaped ? please advice !

Thanks,
Boghra
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

Get a page without css/js ?

Sandy Reid — 2011-07-17T11:46:00

When getting a page using mechanize is it possible to avoid downloading
css/js ?
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users

Mechanize doesn't add cookies to request

Alex Egg — 2011-07-11T03:11:51

These are my cookies: pp agent.cookies

   [SESS0d5421b77d41416bd1d57db5eba42384=aaa86QA6Nf_ISbkcjkxet,

 bbsessionhash=b5af88de76a01ee20340a88dc433793d,

 bblastvisit=1310353559,

 bblastactivity=1310353559,

 bbuserid=6425,

 bbpassword=4afd254205b86456d8e4d7ce969f634d,

 JSESSIONID=aaaJhfsv_RHIWfp2ikxet]


However when I make request: page = agent.get("https://asdf.com/fdsa")


The cookies aren't added to the request:


D, [2011-07-10T20:07:35.523103 #47281] DEBUG -- : saved cookie:
SESS0d5421b77d41416bd1d57db5eba42384=aaaXsYX8MGC8XdPzGkxet

D, [2011-07-10T20:07:35.523477 #47281] DEBUG -- : saved cookie:
SESS0d5421b77d41416bd1d57db5eba42384=aaaqEBLpnek9AgkBGkxet

D, [2011-07-10T20:07:35.523822 #47281] DEBUG -- : saved cookie:
bbsessionhash=bd5f66648bbfeb3e6ad09d3cba6d0a29

D, [2011-07-10T20:07:35.524147 #47281] DEBUG -- : saved cookie:
bblastvisit=1310353655

D, [2011-07-10T20:07:35.524466 #47281] DEBUG -- : saved cookie:
bblastactivity=1310353655

D, [2011-07-10T20:07:35.524788 #47281] DEBUG -- : saved cookie:
bbuserid=6425

D, [2011-07-10T20:07:35.525111 #47281] DEBUG -- : saved cookie:
bbpassword=4afd254205b86456d8e4d7ce969f634d



I, [2011-07-10T20:07:35.525216 #47281]  INFO -- : follow redirect to:
https://asdf.com/fdsa

I, [2011-07-10T20:07:35.529606 #47281]  INFO -- : Net::HTTP::Get: /fdsa

D, [2011-07-10T20:07:35.529671 #47281] DEBUG -- : request-header: accept =>
*/*

D, [2011-07-10T20:07:35.529712 #47281] DEBUG -- : request-header: user-agent
=> Mechanize/2.0.1 Ruby/1.9.2p180 (http://github.com/tenderlove/mechanize/)

D, [2011-07-10T20:07:35.529751 #47281] DEBUG -- : request-header:
accept-encoding => gzip,deflate,identity

D, [2011-07-10T20:07:35.529790 #47281] DEBUG -- : request-header:
accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7

D, [2011-07-10T20:07:35.529828 #47281] DEBUG -- : request-header:
accept-language => en-us,en;q=0.5

D, [2011-07-10T20:07:35.529866 #47281] DEBUG -- : request-header: cookie =>
JSESSIONID=aaa8ieMR1av0S-XwGkxet

D, [2011-07-10T20:07:35.529903 #47281] DEBUG -- : request-header: host =>
asdf.com

I, [2011-07-10T20:07:35.854720 #47281]  INFO -- : status: Net::HTTPForbidden
1.1 403 Forbidden


The login fails b/c the 7 cookies aren't in the request. Why is this
happening?


Alex
_______________________________________________
Mechanize-users mailing list
Mechanize-users< at >rubyforge.org
http://rubyforge.org/mailman/listinfo/mechanize-users