gmane.comp.lang.racket.devel

Re: proposal for moving to packages: repository

Matthew Flatt — 2013-05-24T20:54:43

You're right --- I misunderstood your example.

Still, I'm happy enough with the result in your example. The conversion
does preserve `git log --follow' results for the files that survive,
which was my intended spec. And maybe it's better to explain my
interest as `git blame', since my main interest in the history of a
file is often how/why a particular bit of code ended up as it is.


Just to confirm, my experiment on the main repo completed in right at
20 hours. (The `git log --follow's and `git blame's that I tried look
good to me.)


I don't see how that works. Since my script leaves each file in its
original location for old commits, I expect a subdirectory
`filter-branch' to still drop history for the old locations. In any
case, I'm happy to sort out that detail later.

If we agree that `git mv' before splitting is practical, though, that's
all I need for now.

From my perspective, the important thing is to have the ability to just
edit and move files around to sort out packages, instead of having the
indirection of a script that edits and moves files around.

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: proposal for moving to packages: repository

Eli Barzilay — 2013-05-24T16:44:35

I'm really surprised.  Given that you consider this a *lot* easier,
and that I consider it (reorganization + split) a lot messier, I think
that I'm still not getting something.



That's what it looks like, but I'd double-check to make sure that it
happens.



I don't believe that it's *not* doing this, so I did the double-check
in the form of a test.  When I run it, I see these bad things (which I
expected to happen, so wrote it as a test):

* The "c" file got completely lost (this is the pre-reorganization
  file deletion scenario)

* The "b" file got lost too (post-reorg deletion)

* The history of "e" during the "A" days got lost, since it was not
  recognized as a rename in the A->B move due to being edited too.

=> The first two are things that a script can deal with doing some
   kind of scan like I mentioned (go over the full history of the full
   tree).

=> The third one is something that requires human judgment *but* if
   the A/e historic file is considered as deleted, and if deleted
   files from the original directories are included with doing the
   above, then it should still be there in the rewritten repo.

Test file attached; probably need to do very little other than
adjusting the paths to the two racket scripts.




What I'm saying is that if filter-branch using your script takes 20
hours, and you want to use it to split the repo to 5 packages, and if
a simple filter-branch with a subdirectory filter takes a few minutes,
then instead of:

  * filter-branch using your script 5 times to create each repository
  Total runtime: more than 4 days

you do this:

  * filter-branch one time using your script to reorganize the files
    according to packages
  * use filter-branch with a subdirectory filter 5 times to create
    each repository
  Total runtime: about 21 hours

This latter use would end up with the final tree being exactly the
same (since you're talking about doing the reorganization within git),
but the history would be different since it's as if the files were
there the whole time.

Re: Constructing an identifier to an unexported binding

Matthias Felleisen — 2013-05-24T16:14:47

Assuming that the lookup function for type environments is isolated and in one place, it looks to me like the effort is equivalent: create one function. 

But since we have an expensive way of forging these identifiers, we could also argue we should get a cheap one, just in case someone else needs this ability.

Re: Constructing an identifier to an unexported binding

Matthew Flatt — 2013-05-24T15:49:49

Adding an operation to construct the identifier directly makes sense to
me, and I can see how it might be more convenient to construct an
identifier instead of changing the comparisons.

At Thu, 23 May 2013 18:08:09 -0700, Eric Dobson wrote:
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: proposal for moving to packages: repository

Matthew Flatt — 2013-05-24T12:26:52

Great! Let's do that, because I remain convinced that it's going to be
a lot easier.



I believe the file-lifetime computation in "slice.rkt" takes care of that.


Ditto.


There's no need to move anything while extracting a repository slice;
the movements happen before.


Yes, that's what I suggested.

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: proposal for moving to packages: binary vs source

Eli Barzilay — 2013-05-24T07:53:20

[Note subject change...]

Two days ago, Eric Dobson wrote:

There have been a bunch of concerns expressed about the question of
distributing sources or not -- but I think that generally speaking,
there shouldn't be any problems at all.  Here's a list of things that
contribute to not having such concerns:

1. The eventual goal would be to have very easy selection of packages
   that you want to install.  Either with (a) a bunch of installers,
   (b) possibly doing this by just a different URL that will have the
   installers listed in it as arguments, or (c) with a post-install
   dialog that will ask you for additional packages to install.  (In
   the (c) case, it could also detect packages that you had decided to
   install previously, and re-use the same list.)

   The bottom line is that if *you* want to get the sources, then it
   should be extremely easy to just have them installed (c), or create
   installers that include the sources (a;b) which you'll use.  The
   main point here is that using packages will make such variations
   very easy to implement, and make it easy for you to add sources or
   provide popular options based on demand.

2. With the geiser/drracket concern about reduced functionality
   because there are no sources: the information about the source of
   bindings is still there.  (Ie, things work fine if you remove a
   random source file from a current installation -- the only
   difference is that the actual source file is not there.)

   Now, I'm assuming that there is some way with the package system to
   know for any given file which package it came from.  With this
   information, I think that it would be easy to do something like
   this:

     * In drr, if you try to jump to a definition for a function whose
       source is not included, you get a popup telling you that you
       don't have the source, and list an on-line URL where the source
       can be found (which is inferrable from the package information)
       as well as a one-button-click option to install the source and
       then open the file.

     * Geiser could do exactly the same, and also use something like
       `url-handler-mode' to visit the source file directly from the
       on-line source in addition to offering to install the sources.

3. I think that there should be an option for package owners to decide
   how their package gets installed, so for example, if realm must be
   distributed with its sources, it can just specify that and avoid
   the stripping that other packages would go through.

Re: proposal for moving to packages: repository

Eli Barzilay — 2013-05-24T07:26:45

(In this context this is clear; the problem in Carl's post is that it
seemed like he was suggesting keeping the whole repository and doing
the split by removing material from clones -- which is and even fuller
history, but one that has large parts that are irrelevant.)



If that can be done reliabely, then of course it makes it possible to
do the split reliabley after the first restructure.  It does come with
a set of issues though...


Here are a bunch of things that I thought about as I went over this.
In no particular order, probably not exhaustive, and possibly
repetitive:

* Minor: better to use `find-executable-path' since it's common to
  find systems (like mine) with an antique git in /usr/bin and a
  modern one elsewhere.  (In my case, both scripts failed since
  /usr/bin has an antique version.)

* There is an important point of fragility here: you're relying on git
  to be able to find all of the relevant file movements (renames and
  copies), which might not always be correct.  On one hand, you don't
  want to miss these operations, and on the other you don't want to
  have a low-enough threshold to identify bogus copies and renames.

* Because of this, I think that it's really best to inspect the
  results manually.  The danger of bogus copies, for example, is real,
  especially with small and very boilerplate-ish files like "info.rkt"
  files.  If there's a mistaken identification of such a copy you can
  end up with a bogus directory kept in the trimmed repo.  In
  addition, consider this information that the script detects via git
  for a specific commit:

    A/f1.ss renamed to B/f1.rkt
    A/f2.ss renamed to B/f2.rkt
    ...
    A/f47.ss renamed to B/f47.rkt
    A/f48.ss renamed to B/f48.rkt
    A/f49.ss deleted
    A/f50.ss deleted
    B/f49.rkt created
    B/f49.rkt created

  For a human reviewer, it's pretty clear that this is just a
  misidentification of two more moves (likely to happen with the kind
  of restructures that we did in the past, where a single commit both
  moves a file, and changes its contents).  This is why on one hand I
  *really* like to use such scripts (to make sure that I don't miss
  such things), but OTOH I want to review the analysis results to see
  potential problems and either fix them manually or figure out a way
  to improve the analysis and run it again.

* Also, I'd worry about file movements on top of paths that existed
  under a different final path at some point, and exactly situations
  like you described, where a file was left behind, but that file is
  completely new and should be considered separate (as in the case of
  a file move and a stub created in its place).

* The script should also take care to deal with files that got removed
  in the past.  For example, the drscheme collection has some file
  which gets removed, and later (completely unrelated) most of the
  contents migrated to drracket.  If the result of the analysis is
  that most of the material moved this way, and because of that you
  decide to keep the old drscheme collection -- you'd also want to
  keep that file that disappeared before the move, since it's part of
  the relevant history.

  So I'd modify this script to run on the *complete* repository -- the
  whole tree and all commits -- and generate information about
  movements.  Possibly do what your script is does for the whole tree,
  then add a second step that runs and looks for such files that are
  unaccounted for in the results, and decide what to do with them.

  I think that this also means that it makes sense to create a global
  database of all file movements in a single scan, instead of running
  it for each package.

* Technical: I thought that it might make sense to use a racket server
  (with netcat for the actual command), or have it "compile" a /bin/sh
  script to do the actual work instead of using `racket/kernel' for
  speed.  However, when I tried it on the plt tree, it started with
  spitting out new commits rapidly, but eventually slowed down to more
  than a second between commits, so probably even the kernel trick is
  not helping much...

* Actually, given the huge amount of time it's running (see next
  bullet), it's probably best to make it do the movements from all
  paths at the same time.  In this specific context, this means that
  it scans the package-restructured repo (from the first step) into a
  package-restructured repo (possibly with the same toplevel names)
  with all the files moved to their correct places, and the resulting
  repo can now be conveniently split into the sub-repos with a simple
  subdirectory filter.

* And speaking about the time: what I saw is about 19k commits (I
  think, I killed it and speaking from memory now), where it started
  to work very fast, then slowed down considerabely.  After about 5
  hours it was about half-way through the 19k commits, and the rate
  reached about 1.5 seconds per commit.  Assuming a linear growth in
  time per commit, this means that the whole operation is something
  that would take about 20 hours.  (I didn't leave it running, since I
  don't want it to disturb the nightly build that will start soon.)

  This is not making it impossible -- just very hard to do reliabely,
  so I really wouldn't want to see this going without a human
  supervising eye as I described above.

* Much better would be to run this to generate human readable and
  editable output: then not only go over this output manually and make
  sure that it all makes sense, but also identify points of
  optimization.  For example, knowing that all of drscheme/* moved to
  drracket/* is going to work out much better than dealing with each
  file separately on each commit re-doing.

* Re the commit messages being edited with "--msg-filter": one thing
  to note is that there are already such lines in the history portion
  that was ported from subversion.  Even for those commits, you
  probably want to add the sha1s still, since there might be
  references to the sha1s of an svn-imported commit, ending with the
  svn revision, then the original sha1 that was the first translation
  of the svn commit.

* It's not clear to me what you want to do at this point, but you
  originally described two filter-branch steps: one to restructure the
  repository soon, and another to split into packages.  If this is
  still the plan, then each of these steps would need to add a
  historical sha1 behind.

  Alternatively, do the first restructure with in-repo moves instead,
  but then it would be a good idea to run the slice script and make
  sure that it succeeds in finding all of *these* renames correctly,
  as a good first-level sanity test.

* Still, I consider all of this a huge amount of work and I still
  don't see any benefit for doing it.  Just the time spent on these
  explanations is more than what I'd spend on what I suggested
  yesterday wrt starting a separate setup step for the core and for
  the rest.

  BTW, the script is still useful, of course -- I'd probably do
  something similar, except that I'd use some shell scripts to inspect
  the history of all files, and refine it as I described above.  The
  thing is that this can be done without any effect on current
  progress, since the split during the build is made on current
  directories.

Re: proposal for moving to packages: repository

Eli Barzilay — 2013-05-24T06:41:22

Yes, although the difference between "entire repository" and
"individual files" is mostly theoretical.  The main point is that the
log history is made from changes to *content* -- you can't have some
made up history planter artificially for a file.  (And it is the same
for most CMSs if not all; the main difference here is that git doesn't
keep meta information about copying and renaming.)



Exactly.  (I don't get the rebase comment though -- even without
rebasing what you get is this filtered history.)



Yes, but it can get a bit subtle.  Like I said, the `filter-branch'
tool is basically replaying the entire history, giving you points to
inject hooks that can modify the tree or the commits, etc.  Note that
in all uses that were mentioned, there was a "--prune-empty" flag,
which means that commits that didn't have any change are dropped.  I'm
mentioning this because some people might have an illusion that it's
better to *not* do that and keep these commits.  Here's an example why
this is not useful: say that you have this edit sequence:

  foo< at >somewhere creates A/x, with log message "created x"
  bar< at >somewhere edits it, with log message "edited x"
  baz< at >somewhere moved it into B/x, with log "renamed A to B"

If at this point you use any git tools, they can see the real history.
For exmaple, you can use `blame' to see which lines were written by
which user.  Also, assuming that these are all the changes, a git log
will show the three commits as they appear above.

Now, if you you use filter-branch to modify the repository and keep
only the "B" directory, but you *don't* use the "--prune-empty" flag:
the fact that you want to keep these other commits won't help -- the
full history would have the three commits with the same three
messages, but doing a log for just the file would show only the
commits for the file, so the first two commits won't be shown.
Similarly, blame can't show anything useful -- you'll only see
baz< at >somewhere as the author of the entire file.  And the reason this
makes sense is that the full commit history has the first two commits,
but they had no change -- so there's nothing that ties them to the
file in the trimmed repository, let alone something that relates them
to specific lines in the file...

(Two notes: (a) This is just a demonstration -- obviously, this is a
trimming that is done in a bad way since it dropped A even though it's
part of the history of B.  (B) Actually, it looks like the
"--subdirectory-filter" drops empty commits anyway, but the above
explains why it makes sense to do that.)



The point is that every such messing-with-history should be done very
carefuly and checked thoroughly, since the chance to mess things up is
very real.  In the above, it's obvious that I should have not droped A
in the filter -- but if it's some random single file which you had in
the framework collection, out of tons of other files in the drracket
package, then it's unlikely that I will catch it -- which is why I
prefer using tools for these things and resolve all such issues with
the people who know about the code.



The thing is that having two such filters (one to restructure the big
repository and one to split it) is both increasing chances for making
mistakes, and making the job of the second restructure much harder to
do.  To the point where doing it manually is infeasible, which is why
I said that it will guarantee losing history.

(And I'll reply to Matthew's suggested tool next.)

Re: Constructing an identifier to an unexported binding

Eric Dobson — 2013-05-24T01:08:09

Right, but why cannot we forge an identifier easily? I'm happy getting
an armed identifier. What are the reasons for preventing such a
construction?

On Thu, May 23, 2013 at 6:04 PM, Carl Eastlund <cce-1vnkWVZi4QaVc3sceRu5cw< at >public.gmane.org> wrote:
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Carl Eastlund — 2013-05-24T01:04:35

Essentially yes.  It doesn't do anything else, but it needs an identifier
to do it.  Currently, TR starts with a module and a symbol, goes through an
expensive process to forge an identifier from them, just to call
free-identifier=? to compare based on the module and the symbol after all.
Doing the comparison directly, without ever forging the identifier, would
be quicker.

On Thu, May 23, 2013 at 8:43 PM, Eric Dobson <eric.n.dobson-Re5JQEeQqe8AvxtiuMwx3w< at >public.gmane.org>wrote:

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Eric Dobson — 2013-05-24T00:43:06

Isn't that exactly what free-indentifier=? is checking for on
identfiers with a module level binding? Or is there something else it
does?

On Thu, May 23, 2013 at 3:13 PM, Carl Eastlund <cce-1vnkWVZi4QaVc3sceRu5cw< at >public.gmane.org> wrote:
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: proposal for moving to packages: repository

Matthew Flatt — 2013-05-23T22:47:25

The history I want corresponds to `git log --follow' on each of the
files that end up in a repository.


That's true if you use `git filter-branch' in a particular way. I'll
suggest an alternative way, which involves filtering the set of files
in a commit-specific way. That is, the right set of files to keep for
each commit are not the ones in the final place, but the ones whose
history we need at each commit.


To make sure I'm not confused, I've implemented this idea. My
implementation is unlikely to be exactly right, yet, but I think it
works as a proof of concept.


The enclosed "slice.rkt" script takes a subdirectory and a destination
directory. Run it in the top directory of a git repository, and it
finds all the files in the given subdirectory, and then it closes over
the history of each file via `git log --follow'.

From that point, we could use the computed set of paths as the ones to
keep during a `git filter-branch' on every commit, but that's not
ideal. For example, a file in collection "a" that is destined for
package "a" may have originated in "b" (think "mzlib"), where the
same-named file sticks around in "b" after the copy. It's nicer and
cleaner to have irrelevant files disappear after the relevant copy/move
is made.

So, I took one more step: "slice.rkt" constructs a range of commits
during which the file should exist, based on when it was moved or
copied. (Forks and merges are a minor obstacle, which the script works
around by enlarging ranges to hit commits in the `--first-parent'
traversal.) Conceptually, the result is a mapping from commit ids to
paths, but that would be a big table to read on every `filter-branch'
step, so it's reported as a table of commits with enter/leave
transitions. The output of "slice.rkt" is files: "state.rktd" for the
set of files to be kept in the initial commit, and "actions.rktd" to
specify the transitions.

The enclosed "prune.rkt" script works with `git filter-branch
--index-filter'. It uses "actions.rktd" (read-only) and "state.rktd"
(which it updates via transitions).


The Racket git repo is large, so I've only tried the `git
filter-branch' step so far on smaller repos, such as the "iplt"
repository. In my clone of "iplt", I `git mv'ed "web/internal" to
"ex/internal". Then, with the scripts in "/tmp",

 racket /tmp/slice.rkt ex /tmp
 git filter-branch --index-filter "racket /tmp/prune.rkt /tmp" --prune-empty

leaves the repo with only the files of "ex", and `git log --follow'
on various files looks right.

I'll try on a clone of the Racket repo and report back.

FWIW, before doing this for real, I'd want to add a `--msg-filter' that
extends each commit message to add the original commit id, since we
have references to the old ids in various places (and so it would be
handy to have them in the new repos).
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Carl Eastlund — 2013-05-23T22:13:12



I would not have thought that'd work, but apparently identifier-binding
will give one that information.  Nice going, Ryan!

--Carl
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Ryan Culpepper — 2013-05-23T20:13:21

There's another way around this issue, which is to avoid creating these 
identifiers at all. In other words, change the representation of the 
type environment to something that supports symbol+module pairs as keys 
in addition to identifiers. The easiest way to do that is to add in a 
hash table behind the current free-id-table, since the two tables would 
handle disjoint sets of identifiers.

Ryan

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: proposal for moving to packages: repository

Robby Findler — 2013-05-23T15:44:08

Hi Eli: I'm trying to understand your point. Do I have this right?

Background: The git history consists of a series checkpoints in time of the
entire repository, not a collection of individual files. So, when I do "git
log x.rkt" then what I get is essentially a filtered list (except where
people didn't properly rebase, but lets ignore that) of those checkpoints:
all the ones where "x.rkt" changed.

Big Question: The issue is, then, when we split up the current repo into
smaller repos, what are the series of checkpoints that we're going to "make
up" for the individual repos? Right?

Your Advice: And, IIUC, you're suggesting that the best way to deal with
this question is to defer it until we are more sure of the actual split we
want to make. So we don't mess with the history at all and instead just
work at the level of some script that we can run to just use "mv" and
company to move things around. When we know exactly what ends up going
where, then we can figure out how to make up a new, useful history for the
separate repositories.

Is that the point?

Robby



On Thu, May 23, 2013 at 4:41 AM, Eli Barzilay <eli-oSK4jVRJLyZg9hUCZPvPmw< at >public.gmane.org> wrote:

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket] Parens/string quotes automatic behavior

Laurent — 2013-05-23T15:26:58

+1 indeed, that way you can follow easily with typing a paren, thus
enclosing it again.

Laurent
Le 23 mai 2013 17:17, "John Clements" <clements-USm5ewCNdr/ECKqllllIWg< at >public.gmane.org> a écrit :

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket] Parens/string quotes automatic behavior

John Clements — 2013-05-23T15:15:53

On May 23, 2013, at 8:13 AM, Robby Findler wrote:


+1

John

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Robby Findler — 2013-05-23T15:13:22

This sounds like the right solution to me too.

Robby

On Thursday, May 23, 2013, Matthias Felleisen wrote:

_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: [racket] Parens/string quotes automatic behavior

Robby Findler — 2013-05-23T15:13:21

Oh: another possibility: keep the selection where it is (or, perhaps
better, select the open quote, the def, and the close quote).


_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Parens/string quotes automatic behavior

Nadeem Abdul Hamid — 2013-05-23T14:15:06

Hello Racket devs,

I'm working on tweaking how typing a double quote is handled in strings
when DrRacket's auto parens mode is on, per recent post on the users list.
If any of you use the mode and can offer feedback on the following, it'd be
appreciated: In addition to handling Laurent's initial feature request (see
message at bottom), I'm setting it up so that if the following string is in
the DrRacket window:
  "abcdefghi"
and you select the _def_ and press " (double quote), then it places
double-quotes around the "def" with additional quotes to ensure that the
other two portions of the string are still validly delimited, i.e.
  "abc""def""ghi"

One question is where to put the cursor following this operation? Should it
be right inside the beginning of the lifted string, or in front of its
double quotes, i.e.:
  "abc""|def""ghi"
or "abc"|"def""ghi"
?

Another question is whether to space off the "def" string that is created
from the surrounding ones, i.e.
  "abc" "def" "ghi"
instead of "abc""def""ghi"?

(As a side note: the Emacs paredit mode handles the situation of a double
quotes typed inside a string by inserting an escaped \". It uses a separate
key combination to handle splitting both strings and parenthesized
expressions into two pieces.)

Thanks,

--- nadeem

On May 22, 2013, at 7:07 AM, Laurent wrote:

there is one problem with string quotes.
string-quote symbol ", it places a quote which cuts the current string and
leaves the right part in a bad syntax.
want to split the string in two parts.
the extra string-quote added by the paren-match behavior), and left to go
back between the two strings, which is mildly annoying.
behavior for strings that splits the string instead of inserting a single
string-quote, possibly unless the left symbol is a backslash?
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev

Re: Constructing an identifier to an unexported binding

Matthias Felleisen — 2013-05-23T13:51:45

On May 23, 2013, at 9:42 AM, Sam Tobin-Hochstadt <samth-1vnkWVZi4QaVc3sceRu5cw< at >public.gmane.org> wrote:



Perhaps that's a mistake. 
_________________________
  Racket Developers list:
  http://lists.racket-lang.org/dev