gmane.comp.python.imdbpy

Re: Episodes production_year - imdbpy2sql

Davide Alberani — 2012-04-15T09:18:14

Hi!
Hmmm... never noticed it. :-/
Unfortunately it really seems that the data in the plain text data files
are unreliable, on this.  From movies.list.gz:
  "House M.D." (2004) {Dying Changes Everything (#5.1)}   2004

Which doesn't make much sense, for:
  http://www.imdb.com/title/tt1274484/

(and it's not even that every episode use the year of the first season... :-/ )

I fear there's not much that can be easily done in imdbpy2sql.py,
but I'll have a look.
Maybe you can speed up your stored procedure adding some indexes
(on title.episode_of_id, perhaps), but I'm not sure.


Thanks for the report!

Re: new small features

Davide Alberani — 2012-04-15T08:52:08

Hi and pardo for the delay: very busy weeks.


Yes, that's the idea.  Not much tested, as usual. :-P


The IMDbPY exceptions are organized in a small hierarchy:
- IMDbError : Base class for every exception raised by the imdb package.
        |
        +-> IMDbDataAccessError: unable to access some data
        +-> IMDbParserError : unable to parse something

You can import them from the imdb._exceptions module.
If you want to catch them all, use IMDbError.

Episodes production_year - imdbpy2sql

darklow — 2012-04-12T10:03:22

Hi,

  Would it be possible to improve imdbpy2sql script so that episodes 
production_year field contains correct episode year, which is calculated 
by episode movie_info release date?

  I am not sure how episode production year is calculated, but it is 
wrong and incomplete for almost every TV show.

  Here you can run this select (example House M.D.) and see that 
production_date differs a lot from real episode production date:

SELECT
         title.id, production_year, (select max(SUBSTRING(info, 
'([0-9]{4})')::int) from movie_info where movie_id=title.id and 
info_type_id=16) as real_year
FROM    title
WHERE   episode_of_id=(select id from title where title='House M.D.')
ORDER BY real_year DESC

  Right now to fix this after every update i run following SQL script, 
however it takes few hours to update, so i was wondering maybe this 
could be fixed in imdbpy2sql.

  Thanks.
  PS. Thanks for the gender field. Just tested and it works perfectly!

  Here is the update function i run after update, if inte

Re: new small features

Albert Claret — 2012-04-10T13:59:16

Hi,

Both features seem very interesting, will reraiseExceptions allow us to 
catch Exceptions from our own programs?

I haven't figured out a way to properly intercept an Exception (for 
example, a connection error) raised by the imdbpy module from my own code 
to, allow the operation to be retried, for example. I assume that setting 
reraiseExceptions and catching them from my own code should work, right?

Regards,
Albert

On Saturday, March 24, 2012 4:34:41 PM UTC+1, Davide Alberani wrote:
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev_______________________________________________
Imdbpy-devel mailing list
Imdbpy-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
https://lists.sourceforge.net/lists/listinfo/imdbpy-devel

Re: Compilation under Win7 (VS2010)

Damien — 2012-04-03T04:12:00


Hi David,

Yep, removing that option builds the object file. Poked around a little more. I 
removed all reference to '-mno-cygwin' from 
C:\Python27\Lib\distutils\cygwinccompiler.py - it's a bit of a hack but I 
successfully built your latest git release this way.

Unfortunately, didn't help with my SQL Server connection woes... :(

Damien.




------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev

Re: Gender for Actors / imdbpy2sql

Davide Alberani — 2012-04-01T11:53:06

Shame on me!
As H. Turgut Uyar pointed to me, it should have been "gender".

I've just renamed it, pardon for the trouble.

Re: Gender for Actors / imdbpy2sql

Davide Alberani — 2012-04-01T10:46:03

On Wed, Mar 28, 2012 at 14:14, Davide Alberani
<davide.alberani-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org> wrote:

Introduced a "genre" column in the "name" table, defined as a nullable
string of length 1: 'm' for male, 'f' for female, NULL for entries that came
from files which contain both males and females.


HTH,

Re: Compilation under Win7 (VS2010)

Davide Alberani — 2012-03-31T15:55:46

Uh, that's a problem. :(

I wonder if there's any way to tell the build process to not introduce
that option...

Bye the way, what happens if you run that command (minus the --mno-cygwin
argument) from the command line?  If it compile, you should re-run the build
process and it _could_ work...

Besides that, I'm officially out of ideas. :-/

Re: Gender for Actors / imdbpy2sql

Davide Alberani — 2012-03-28T12:14:35

Yep, that would be very useful, and you're right: I think it will be very easy
to add it.

I hope to introduce it very soon, thanks for the idea!

Gender for Actors / imdbpy2sql

darklow — 2012-03-28T09:52:03

Hi,

Sometimes there is need to know is actor male or a female.

And only way to detect is by looking at cast_info role_id which is very 
un-handful.
I added column sex for name table, however following SQL update runs for 
a very long time while at import level it would cost almost anything.

update name set sex=2 where id in
     (select person_id from cast_info where role_id=2 group by 1 order by 1)

Re: Compilation under Win7 (VS2010)

Damien — 2012-03-26T07:45:13


Um, nope, the build fails due to --mno-cygwin :(

http://bugs.python.org/issue12641

I tried it against the basic, 32bit Python 2.7.2 setup using the latest MinGW 
release (GCC 4.6.2). I spent ages last night trying to work out the older 
dependencies for a 4.5.2 MinGW GCC, but there's no quick and easy way of retro 
installing an older version. I tried manually downloading TDM snapshots but that 
didn't really work as I missed some files.

Here's the output when trying to compile your latest version in git (as of last 
night):


d:\install\IMDb\IMDbPY\alberanid-imdbpy-509b15b20f94>python27 setup.py install
Created locale for: en it tr.
running install
running bdist_egg
running egg_info
writing requirements to IMDbPY.egg-info\requires.txt
writing IMDbPY.egg-info\PKG-INFO
writing top-level names to IMDbPY.egg-info\top_level.txt
writing dependency_links to IMDbPY.egg-info\dependency_links.txt
reading manifest file 'IMDbPY.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-in

Re: small improvements: JSON

Davide Alberani — 2012-03-25T07:32:09

That's an excellent idea!  JSON support is for sure something we've to add.
We already have XML output; try something like: movie.asXML()

Probably the best solution would be to derive the JSON from that XML (at
least, it should be less code to maintain).

I have to think a little about it (maybe we can convert the xml to
python objects
using lxml or beautifulsoup, and then use simplejson for the output)

Thanks for the idea!

Re: Compilation under Win7 (VS2010)

Davide Alberani — 2012-03-25T07:04:12

Hmmm... I may be wrong, but I don't think MinGW can compile 64 bit
executables.
Probably http://mingw-w64.sourceforge.net/ can be of help.
Or maybe not. :-)


In this environment (Python 32 bit, MinGW), you can surely build IMDbPY
without too much problems.

Let us know if you find something useful.

Thanks!

Re: Compilation under Win7 (VS2010)

Damien — 2012-03-25T03:46:10

http://www.lfd.uci.edu/~gohlke/pythonlibs/

I went down that path but then you hit the whole --no-cygwin issue with the 
latest MinGW release... :(
 

In the end I uninstalled the 64bit Python setup, replacing it with the 32bit 
edition then downloaded the binary on the webpage. So far so good, but hitting 
problems trying to dump the IMDb data to SQL Server. I'll start another thread 
for that.

However, has anyone managed to get IMDbPy working under Windows against 64bit 
Python? What compiler version did you use?

Cheers,

Damien.



------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Imdbpy-devel mailing list
Imdbpy-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/imdbpy-devel

small improvements: JSON

James Rubino — 2012-03-24T20:57:56

one more very small possible improvement
I have a generic wrapper to convert all person, movie, company classes into
json.

Where Movie.data() and the likes would have outputted class data it simply
takes the name and id and turns it into a string.
How can we best implement JSON output within all of the IMDbPy package
(perhaps it is already)?.

simple wrapper with tests
https://gist.github.com/2187476
------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure_______________________________________________
Imdbpy-devel mailing list
Imdbpy-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
https://lists.sourceforge.net/lists/listinfo/imdbpy-devel

new small features

Davide Alberani — 2012-03-24T15:34:41

Two new small features in the mercurial/git repository:
- "reraiseExceptions" option (can be used programmatically -
instancing IMDb - or
   in the imdbpy.cfg configuration file) to re-raise any caught exceptions.
   Default to False, could be useful in some circumstances.
- "timeout" option (can be used like "reraiseExceptions") to set a
timeout in seconds
   for the http connections.  By default I've set it at 30 seconds.
Can be disabled
   entirely setting it to False or None.

And an improvement:
- better SQL statements for the creation of temporary tables (re)storing
   imdbIDs... it should work gracefully with postgresql and possibly
other databases.

As usual, please report any problem and strange behavior: I'm not too sure
about the consequences of these changes. :-)

Re: Compilation under Win7 (VS2010)

Davide Alberani — 2012-03-23T08:01:22

You can always exclude it with --without-cutils


Not sure about it.
I have an environment with mingw32 installed, and then, in
C:\Python27\Lib\distutils  I've put a distutils.cfg file like this:
[build]
compiler = mingw32

(I'm sure you can specify it on the command line, but I'm lazy...)

If you find a way to compile using VS2010 let me know. :-)

Compilation under Win7 (VS2010)

Damien — 2012-03-23T02:27:34

Hi there,

I'm trying to get IMDbPy 4.8.2 going under a Windows 7 environment.
 
I begun with a completely new Python 2.7.2 environment, 
with manual installation  of lxml from http://www.lfd.uci.edu/~gohlke/pythonlibs/

However, I cannot seem to compile the cutils bit of the sql parser. 
I have VS2010. 
I tried setting the VS90COMNTOOLS variable to point to the tools directory 
but mt.exe bombs with an error that it cannot locate the manifest file.

Anyone manage to get this going?

Regards,

Damien.


------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure

Re: [sql] indexes

Emmanuel Tabard — 2012-02-25T15:45:40

Le 25 févr. 2012 à 16:09, Davide Alberani a écrit :


This is a reasonable approach. Sometimes too much indexes could be worse than no index at all :D



Fair enough ! I can help you with sql nightmares. To be honest, I only use the parsing and sql importing part of your project, which is by itself awesome :p



I saw that your github repo is just a mirror ? Is it possible to use pull requests on github or this is a read only repo ?
Point is, the process is easier on github to propose path and discuss it. 

If it's not possible, i'll create a sourceforge account :)


I don't have much time, my startup eats a lot of my time ... But your project has an important part in mine, so if I can share my optims be sure I'll will commit them !
I'll probably make one of my own project public one day. It's python, it imports imdb with imdbpy add wikipedia synospys and bio(you can target language), freebase, themoviedb, thetvdb (etc) to have an universal and automatized movie database. For now, the code is too dirty t

Re: [sql] indexes

Davide Alberani — 2012-02-25T15:09:21

Yep, I kept the indexes at the minimum... maybe too much. :-)


No problems at introducing them, but... would not it be easier if you just join
the IMDbPY project? :-P
If it's ok for you, give me your sourceforge and bitbucket usernames, and
I'll grant you write permission to the repository.
It's ok even if you don't really have much time to dedicate to it: when
you need something, just ask to me (to be sure that it will not create some
strange problems that only me can know) and commit. :-)

Point is, IMDbPY was always a playground, for me (come on... we didn't
REALLY need to support both SQLObject and SQLAlchemy ORMs ;-) and I'm not
an expert on databases.

On the db, actually, I'd like to see this work done:
1. an overall evaluation of the current status: is the structure still
valid/meaningful?
   (keeping in mind that many choices were due to performance reasons - maybe
   with wrong assumptions on my side).
2. introduce indexes/foreign keys were needed.
3. analyze if it's possible to get rid of the mo

[sql] indexes

Emmanuel Tabard — 2012-02-25T14:12:26

Hi ! 

Your sql schema misses a few essentials indexes : 

 - title.imdb_id
 - title.episode_nr
 - title.season_nr

 - name.imdb_id

You maybe don't need those indexes :)

I just wanted to know if you're ok pulling this upstream. In the other case I'll maintain a patch on my side !

Bests,

Emmanuel
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/