gmane.lisp.asdf.devel

Various issues

Tobias C Rittweiler — 2012-05-24T09:13:35

Hi there!

Thanks to the ASDF maintainers. I just visited ASDF's website
on common-lisp.net since a long time, and it makes a nicely
maintained impression! Well done.

I have the following issues:

  * The rather old ASDF version that I'm using ("2.010") does
    not seem to be able to cope with multiple :depends-on clauses.
    (Only one clause seems to be used.)

    Is this intended behaviour? Might it be fixed in more recent
    versions?

  * :STATIC-FILE (and :FILE for that matter) are not discussed
    or explained in the manual.

  * We generate a 'version.lisp' with a (DEFPARAMETER ...) form
    in it when using "make" to build a Lisp binary using buildapp.
    
    Is there a component that loads a file if it exists and
    if it does not exist just skips the clause?

Thanks,

T

Tests on Mac OS X

Robert Goldman — 2012-05-17T14:26:17

I tested on the following lisps

ccl clisp sbcl ecl abcl allegro allegromodern

with success.  Haven't tried on CMUCL yet, because of prior problems.  I
will see about picking up a snapshot sometime.

lisp source debugger

Yuan Luo — 2012-05-12T22:08:46

Hi,
This maybe an irrelevant question, if so, please point me to the best list
to ask. But I am wondering whether you guys can recommend some source level
debugger that can POINT ME PRECISELY to where the error was. For example,
Allegro CL cannot because many times it tells me that there is an error in
a function, but it doesn't tell me which line even if I set their debug
flag to 3 the highest level and suppressed everything else, speed,
optimization etc. (my function has hundreds of lines then I am frustrated
...)

Best,
Yuan
_______________________________________________
asdf-devel mailing list
asdf-devel-F1HGIaG5STRyXAeb93iumQ< at >public.gmane.org
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

:default-component-class and custom module classes

Nikodemus Siivola — 2012-05-07T04:22:29

I was trying to make a custom module class, which would have provided
its own :DEFAULT-COMPONENT-CLASS via
:DEFAULT-INITARGS, but

        (setf (module-default-component-class ret)
              (or default-component-class
                  (and (typep parent 'module)
                       (module-default-component-class parent))))

means that does work as I hoped it would. The closest I can get is by
overriding ASDF::MODULE-DEFAULT-COMPONENT-CLASS method, but that seems
fairly icky.

Is there a better way?

Cheers,

 -- Nikodemus

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Reloading a .asd

Stelian Ionescu — 2012-05-06T19:27:01

I'm using the elisp snippet below(courtesy of Stas Boukarev with some
modifications) to reload a .asd file. Is asdf::load-sysdef the best way
to trigger a reload(but not recompilation) ?

Use case: I want to work on a system that isn't in the ASDF path, it
being one of the many development trees I have around(e.g. CFFI), so I
want to be able to say that "this cffi.asd is the CFFI to use during
this session". Any ideas ?

(defvar slime-custom-file-loaders
  '(("asd"  . slime-load-asdf)
    ("lisp" . slime-load-file)))

(defun slime-load-asdf (filename)
  (slime-eval-with-transcript
   `(asdf::load-sysdef ,(file-name-sans-extension filename)
                       ,(slime-to-lisp-filename (expand-file-name filename)))))

(defun slime-custom-load-file ()
  (interactive)
  (unless buffer-file-name
    (error "Buffer %s is not associated with a file." (buffer-name)))
  (check-parens)
  (when (and (buffer-modified-p)
             (y-or-n-p (format "Save file %s? " (buffer-file-name))))
    (save-buffer))
  (let* ((type (file-name-extension buffer-file-name))
         (loader (cdr (assoc-string type slime-custom-file-loaders t))))
    (funcall (or loader
                 'slime-load-file)
             buffer-file-name)))

(define-key slime-mode-map (kbd "C-c C-l") 'slime-custom-load-file)

asdf-bundle

Faré — 2012-05-03T15:55:06

asdf-bundle is now working (hopefully) on SBCL and CCL as well as ECL,
by concatenating fasls. It won't work for systems where CFFI creates .so.

Try it!
(require :asdf)
(require :asdf-bundle)
(asdf:oos 'asdf::load-fasl-op :your-favorite-system)

It creates a your-favorite-system.system.fasl in the fasl cache.

Actually, I'm not sure about ECL anymore - load-fasl-op fails at the last minute
(after the fasl-op itself) because it tries doing something with a .fasb
even though I didn't enable the bytecode compiler. Since I leveraged that part
of the code from asdf's asdf-ecl.lisp, I suspect it is broken there already.

Juanjo: can you give a look when you have time?

Others: you're welcome to contribute support for your favorite implementations.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Don't have good ideas if you aren't willing to be responsible for them.
        — Alan Perlis

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Tool for transcoding lisp source files and addingcoding file options.

Douglas Crosher — 2012-04-25T03:39:38

A tool to add the coding file option to lisp source files is now available. This can also be used to transcode all the files to
UTF-8, or to attempt transcoding to ISO-8859-1 or US-ASCII with loss in comments. There is an example for the current Quicklisp
releases with a few required exceptions. This is experimental, particularly the transcoding to ISO-8859-1 or US-ASCII. Please
backup your code before giving this a try! See:
http://www.scieneer.com/files/transcode.lisp

With the coding file option in place, the developmental version of ASDF is able to pass the correct external-format to 'load and
'compile-file.

Note the lambda-reader is handled by substituting references to the lambda character back to "(lambda".

Since the :encoding file option should become unnecessary with the coding file option, and also because it would be invalid upon any
transcoding, it is removed from .asd files. I don't think it would be practical to try and update the :encoding declarations.

With the coding file option on all the files, all that asdf-encodings needs to do is read this file option. The UTF-8 and other
detection could be removed and moved into a tools such as this.

If some of the CL implementations would add the reading of the file option as an external-format then the coding file option could
be used by 'load and 'compile-file outside of ASDF - then for these CL implementations asdf-encodings would not be necessary and
ASDF could pass on the encoding-file-option reading external-format which may well be the :default. I will look at implementing
this for the Scieneer CL, and perhaps take a look at CMUCL, but it would be best optimized for each CL implementation.

Regards
Douglas Crosher

ASDlite - light-weight excerpt from ASDF

Dmitriy Ivanov — 2012-04-23T07:55:40

Hello folks,

Excuse my bothering if this is not a correct place to announce.
Nonetheless...

ASDlite Design Goals

1. Small footprint.
2  Ease of embedding into applications and systems not related to
   "compile-and-load Lisp files" tasks.
   For example: YstokHelp (http://lisp.ystok.ru/yhelp/)
3. ASDF compatibility for basic needs.
4. Operation arguments specification in dependencies.

Observation and Rationale

 1. The ASDF built-in operation hierarchy is actually of two-level depth.
    Original ASDF code does not exploit operation inheritance much
    (though something can be found in asdf-ecl.lisp).

 2. The operation slots are rather useless:
    original-initargs
      Is only referred in print-object
    parent
      In principle, indicates the target operation that required this one.
      But due to reusing operation objects in make-sub-operation,
      this is not the case.
      Also used for marking along with other visiting slots during traverse
      but we follow another approach.

    Adding entirely new operations, i.e. on the first level, is fine.
    But there is comfortable way to refine existing operations: the easiest
way
    is to add slots to base operation classes as only those properties
    do get passed into dependency operations.

    There is a more simple way pass arguments from operate to
    operation functions - by means of key arguments!

 3. The :do-first initarg is actually ignored by ASDF - its always set to
     ((compile-op (load-op ,< at >depends-on))))

 4. Avoid inline methods, which are rather inelegant in ASDF:
    - they rely on eval,
    - ASDF tries to remove a method from a generic function of a different
name.
    Due to non-standard behavior of remove-method in LW 4.4, system
redefinition
    intrusively signals about this.

 5. Referring to features in component definition is more useful than for
    dependency rules.

Operations in ASDlite

  operation ::= keyword | operation-instance
  operation-type ::= keyword | type-symbol
  operation-designator ::= keyword | (keyword . plist)
       | type-symbol | operation-instance

Operations are passed to perform and other operation-specific  methods.
Operation designators can be used in the right-hand side of rules.

We encourage using simple keywords like :compile or :load.
For these, ASDlite defines corresponding methods with eql specializers.
The plist allows you to pass key arguments to the operation methods.
In the "normal" mode, ASDlite accepts only keyword-based forms.

If you feel these are not enough and need "full-fledged" ASDF operation
classes, you can switch to the ASDF compatibility mode as follows:
- push :asdf-compat into *features* before compiling asdlite.lisp,
- refer to asdf:compile-op, asdf:load-op and the like,
- define your own subclasses of the operation class
  etc.
In the compatibility mode, ASDlite accepts all the above forms of operations
and designators.

Downloads and Details
http://lisp.ystok.ru/asdlite/

--
Sincerely,
Dmitriy Ivanov
lisp.ystok.ru

Faster loading for deployed systems?

Erik Pearson — 2012-04-22T20:14:32

Hi,
I am trying to implement a method of using a "batteries included" lisp
image (ccl, linux, 64bit) which can be used for running a variety of lisp
tasks in a batch mode. I chose to pre-load the image because it was just
too slow to load the individual fasl files at run time, either via asdf or
through a generated load-list.

I'm using ASDF to load a small system at run-time, but finding that even
with small system (say, one file), performance is sluggish, over 2 seconds.

It looks like the problem crops up when there are many dependencies. Poking
around the asdf code this appears to be because operate always traverses
the system dependencies, and pokes the asd files to find if they have
changed.

This makes sense for asdf in development mode, since it needs to discover
changed systems. Not so good for a deployed system.

ASDF assumes that since the system def file has not changed, the system
hasn't been changed either (I know there has been discussion of this
assumption.) But lets assume that at least part of the reason for that is
because it is very time consuming to check the entire set of files every
time the developer wants to recompile and load just a single file.

Why not assume in a deployment mode that NO system will have changed if it
is already loaded in memory, and unless the application has asked that it
be reloaded? This would be for performance as well as for safety -- we
really don't want dependency checking to result in any filesystem checks,
etc.

I feel that this problem is really pushing my grokking of asdf, but I'm
delving ahead anyway.

I haven't tested it extensively, but I did just waste a perfect sunday
morning fiddling with this:

extended module with a slot "up-to-date-p", but surely there is a better
name, this I just chose while in early play mode:

added these methods:

(defmethod operation-done-p ((o load-op) (c module))
  (module-up-to-date-p c))

(defmethod mark-operation-done ((operation load-op) (c module))
  (setf (module-up-to-date-p c) t))

tweaked the operate method so that it exits early if the module has been
marked as up-to-date

 (if (operation-done-p op system)
  (return-from operate (values op nil)))

and do-traverse to only collect "kids" if it is not done

(unless (operation-done-p operation c)
  (while-collecting (internal-collect)


Note that this code goes in the special section of operate for handling
modules.

I think that is all that I did.

I found that the time to handle the require for my current project dropped
from about 2 1/2 to about 1/2 second.

Thoughts?
_______________________________________________
asdf-devel mailing list
asdf-devel-F1HGIaG5STRyXAeb93iumQ< at >public.gmane.org
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Please test asdf-encodings

Faré — 2012-04-21T23:38:22

Dear all,

asdf-encodings now includes a version of
Douglas Crosher's encoding detection algorithm.
It is also automatically disabled on implementations without unicode support.

As compared to Douglas's version, the detection algorithm
uses :ascii or :latin1 instead of :default as a fallback
when no declaration was found and no UTF-n encoding was detected.
It also uses a 1024-byte buffer rather than 320-byte buffer,
to imitate what Emacs does with respect to the beginning of a file.

I suggest that asdf-encodings is now ready for testing,
and invite you to test it.

An example package that uses it is my lambda-reader,
heavily modified from Brian Mastenbrook's original
to make it hopefully portable to all implementations
with or without utf-8 support.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Sin lies only in hurting other people unnecessarily. All other "sins" are
invented nonsense. (Hurting yourself is not sinful — just stupid.)
        — Robert Heinlein, "Time Enough For Love"

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

ASDF and character encodings

Daniel Herring — 2012-04-20T23:43:27

This topic came up at the Boston Lisp meeting last night.  Here was my 
takeaway from the discussion.


ASDF does not need to support arbitrary encodings in system definition 
files.  UTF-8 is the simplest and most portable solution for non-ASCII 
characters in system definitions.

Thus ASDF can simply declare that all system definitions are read using 
UTF-8.  On systems where unicode support is not available, system 
definitions will be read as "extended ASCII", and the non-ASCII characters 
will simply print incorrectly.  Filenames should only be specified using 
ASCII characters to avoid portability issues.


Authors do need a way to support arbitrary encodings in other source 
files, but this should be restricted to cases where there is an actual 
need to use a non-UTF-8 character set.  Example use cases are 
locale-specific libraries, a toolchain that converts between character 
sets, or possibly a character-set-detection library.  Again, other 
character sets can be a major portability issue.


There are only a handful of libraries that use something other than ASCII 
or UTF-8.  These should be easy to fix, given approval from their authors.


Hopefully that's an accurate record of the general consensus and this 
writeup will be helpful to others in some way.

- Daniel

Update: encoding file options version.

Douglas Crosher — 2012-04-18T14:24:47

A revised version of ASDF replacing the system definition :encoding support with support for reading the encoding from file options
is now available at: http://www.scieneer.com/files/asdf.lisp

Changes:

* Can read the file options in a wider range of files. Also supports C, Python, XML, and probably others. It's more robust and
may be of some utility in other code.

* EBCDIC support removed to save on the line count. Can add this back at a cost of about 40 lines.

* UTF-8 detection removed to save on the line count. Still detects a UTF-8 BOM, and reads a UTF-8 encoding file option. Adding a
UTF-8 encoding file option will help other tools too anyway.

* Removed the lengthy external-format translation table to save on the line count. It should be easy for CL implementations to
include more aliases so move the burden of maintaining the aliases to the CL implementation. CLISP and the Scieneer CL already
support an extensive range of codes and aliases, and an update set of aliases has been sent for CMUCL. SBCL, ECL, and CCL all have
a useful set of aliases, but could consider adding a few more. A hook is include for translating the external-formats just in case
it is needed, and it is a list of functions to call in turn so multiple projects could add hooks.

* Changed the asdf.lisp coding file option to us-ascii from utf-8 - it's a highly portable tool and this should be adequate.

* This version is now down to just 114 lines more than the standard version, and the bulk of the code is just one function.

* Tested on CCL, CLISP, CMUCL, ECL, SBCL, and Scieneer CL,

* Added a large set of test files that exercise the reading of the encoding file option and try to include enough characters from
each character set to check that the encoding option has been successful mapped to an appropriate external-format. This includes
all the encoding in linux 'iconv -l' that include support for the characters needed by CL, but excludes the EBCDIC codes. All 629
tests pass on the Scieneer CL, 628 on CLISP, and a much smaller but useful sent on CCL, CMUCL, ECL, and SBCL due to their limited
aliases and limited encoding support. See: http://www.scieneer.com/files/coding-tests.tar.bz2

Keep in mind that even if a library author chooses an encoding that your CL implementation does not support, if they have added the
encoding file option then the files can be reliably and automatically converted to something supported such as UTF-8. There may be
a tool out there already in the Emacs or Python community to do this, or something can be written and could run on CLISP which is
free and has extensive coding support.

Regards
Douglas Crosher

Testing with RMCL....

Robert Goldman — 2012-04-17T19:36:17

I am willing to run the ASDF tests using RMCL, since my Mac still has
Snow Leopard, instead of Lion.

*However*, as a non-RMCL user, I have no idea how to run it from the
shell, so that we can run the test suite on RMCL.

Can someone please offer me a clue?

I can't be trying to run this from the GUI every time it needs a test.
I had to give up testing LispWorks for the same reason.

thanks,
r

Ran the tests on Mac OS X

Robert Goldman — 2012-04-16T19:09:09

...all have passed.

rpg% echo $ASDF_TEST_LISPS
ccl clisp sbcl ecl abcl allegro allegromodern

Cheers,
r

Please test 2.20.16

Faré — 2012-04-16T16:29:36

OK, so I propose we freeze the current state of ASDF for a release 2.21,
hopefully some time next week.
Please test 2.20.16.

Changes since 2.20:
* New features:
* Most importantly, encodings.
ASDF now lets you specify an :encoding for a system, module or component,
that is used when loading or compiling Lisp files.
See the documentation.
By default, the only useful value is :utf-8, and
we recommend you use UTF-8 everywhere.
We intend to make it the default in the future
(current default is the legacy behavior of using whichever implicit
default your underlying implementation is currently configured to use).
An extension asdf-encodings is available that supports more encodings,
including autodetection of encoding in the near future.
* You can now specify :hostname in your asdf-output-translations,
so you can easily share a home directory yet split its fasl cache
between several subtly different machines.
* Bug Fixes:
* lp#982285. since 2.014.4, the default source-registry
was skipping anything from /usr or XDG_DATA_DIRS,
because of a missing test in getenv-absolute-pathnames. Fixed in .15.
* In 2.019.9 aka 2.20, ECL support was broken by using
a function in the wrong package (defined in cl-user). Fixed in .1.
* logical-pathname support was somewhat broken since 2.017.6,
due to the way most implementations fail to read physical namestrings
when *default-pathname-defaults* is a logical-pathname.
Fixed, together with other logical-pathname issues,
and a test case was added to the test suite to ensure no further regression.
Works great modulo quirks around implementation bugs on CLISP and Allegro.
* Minor tweaks:
* Use :unspecific in pathname components on more implementations.
* export and/or document more utilities.
* add a few missing compatfmt for Genera.

I do not intend to make any further changes to ASDF before the 2.21 release,
except for fixes to any bug found, of course.
Experiments with encodings and autodetection are redirected to
the asdf-encodings system for the time being.
Discussion of a possible merge of asdf-encodings into ASDF
will happen only after asdf-encodings itself is reasonably stable.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
The mice which helplessly find themselves between the cats teeth acquire no
merit from their enforced sacrifice.— Mahatma Gandhi

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Tested all on 2.20.12 Mac OS X 10.6

Robert Goldman — 2012-04-15T21:26:51

Seems to work fine.  Still not testing on CMUCL, waiting for patch to that.

cheers,
r

logical pathname fixes

Faré — 2012-04-12T20:51:28

I added a more functional test for logical pathnames,
and made a round of fixes for logical pathnames.
Some things are still broken on CLISP and Allegro, but I blame the
implementations
(see test/test-logical-pathname.script for details).

p-cos, janderson, pjb: I believe you are the three known users of
logical pathnames.
Can you test the latest ASDF in your respective setups? (currently 2.20.10).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Think you can, or think you can't — either way, you'll be right. — Henry Ford

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Latest commit

Robert Goldman — 2012-04-12T15:41:01

I just ran the noupgrade tests (with CMUCL disabled) on Mac OS X, and
all passed.

I assume that I don't need to bother with Linux testing....

r

testing new ASDF

Robert Goldman — 2012-04-11T21:56:32

All tests pass on Mac OS X until I get to CMUCL 20c, which still fails
for reasons that seem unrelated to ACL.

These are test-all-noupgrade.  Will test with upgrade later.

Note that I am still running 10.6, not the latest 10.7.  It would
probably be a Good Thing to find someone who will run the tests on 10.7.

For the moment, I'm purging CMUCL from my test suite.


Cheers,
r

LOCALAPPDATA vs APPDATA

Faré — 2012-04-01T22:37:44

ASDF has favoring LOCALAPPDATA over APPDATA since 2.015.6. But I see
quicklisp is still using 2.014.6.

Now, considering the ECL bug in 2.20 (aka 2.019.9), I recommend that
ECL and/or Quicklisp should upgrade to 2.019.8 (if they upgrade),
until 2.21 is ready (which it won't be until we fix those 20 systems
that depend on something else than UTF-8).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
(map()(lambda(n l)(princ(subseq (format nil "(~{~A~^
~})~%"(let(r)(do-external-symbols(x :cl)(push x r))(sort r #'string<
:key #'string))) n (+ n l)))(finish-output))'(875 949 8 6010 4863 371
11)'(4 3 5 5 3 2 2))

_______________________________________________
asdf-devel mailing list
asdf-devel< at >common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/asdf-devel

Test results on Mac OS X

Robert Goldman — 2012-03-30T02:36:49

As before, all these pass:

ccl clisp sbcl ecl abcl allegro allegromodern

cmucl fails on test-around-compile.script

no idea why....

cheers,
r