gmane.comp.lang.felix.general

Dictionary type implemented

john skaller — 2012-05-26T03:44:14

I have implemented a basic dictionary type.

Here's how to use it. First for convenience we need these:


fun dflt : opt[string] -> string =
| Some ?x => x
| None => "NONE"
;

fun dflt : opt[string * string ] -> string * string =
| Some ?x => x
| None => "NONE","NONE"
;


Now we can do:

/////////
var x = strdict[string] ();
add x "Hello" "World";
println$ dflt$ get x "Hello"; 
println$ "Missing .." + dflt (get x "silly");
////////

you can also write:

x.add "Hello" "World" 

using the operator . style. We get:

~/felix>flx jj
World
Missing ..NONE

The "dflt" function is  a way to map:

dflt: opt[T] -> T

so you don't have to keep matching against Some/None.

You can delete entries:

/////////
var found = del x "Hello";
println$ "Found="+ str found;
found = del x "Hello";
println$ "Found="+ str found;
//////

which gives:

Found=true
Found=false

Remember, "del" is a generator, you cannot ignore the returned value!
Or Felix will silently remove the result variable, and thence the deletion!

You can make a dictionary from an association list:

///////
var y = strdict$ list $ ("A","1"), ("D","2"), ("B","3");
/////////
and we have iterators:

////////
match ?k,?vv in y do
  println$ k,vv;
done
////////////

We also have the functions:

dict.first 
dict.get_ge key
dict.get_gt key
dict.get_le key
dict.get_lt key

There are many more functions to add, but this should get you going!

You can make a dictionary of any value type.

CAVEAT: The keys are actually NTBS: null terminated byte strings.
If you have a string key with a null character in it, only the text up to the
first null will be counted. This is a restriction imposed by the implementation,
JudySLArray.

CAVEAT: there's a limit of about 10,000 characters for the key length.

JudySLArray is an order of magnitude faster than STL Map.
It is roughly the same speed as a STL unordered map (hash table).
However it is a superior data structure because it is ordered: you can
get keys out in order, which you cannot do with a hash table.
It is therefore an ideal data structure .. much better than
dictionary types used in Perl, Python, C++, or most other languages.

CAVEAT: the naive implementation using C++ strings pays a penalty,
since Judy uses char arrays, and modifies them on sequential searches.
For functional behaviour using C++ strings we're forced to copy up/copy down
the string data to a buffer which has to be malloced and freed on every call.

Faster variants of the search functions using a C buffer are available and
used in the iterator.

TODO: a lot more nice functions can be added, such as filtering keys
on regexps, a specialised version mixing strdict with dsarray to give
a JSON data structure, converting to association lists, etc etc.

Also a comprehension would be nice.

The implementation is in src/lib/std/strdict.flx.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

plugin data

john skaller — 2012-05-25T00:47:42

At present the plugins don't hyperlink correctly because the data
format is wrong. They're being fed the server_config.cfg file
which has stuff like:

# Configuration file for webapp
delay=0.05
port=8080
server_root=.
document_root=$server_root/html
C_PATH = $INSTALL_ROOT/lib/rtl,/usr/local/include,/usr/include,/usr/include/c++/4.6.2,/usr/local/include,/usr/include,/usr/include/c++/4.2.1,/usr/include/c++/4.2.1/x86_64-apple-darwin10,/usr/include/c++/4.4.3,/usr/include/c++/4.4.3/x86_64-linux-gnu,/usr/lib/gcc/x86_64-linux-gnu/4.4.3/include
FLX_PATH=$INSTALL_ROOT/lib
FDOC_PATH=$DOCS,$DOCS/doc,$DOCS/web
FLX_PKGCONFIG=$INSTALL_ROOT/config
DB = $server_root/db/wiki.db

in it. The plugins require

C_PATH += /usr/include/c++/4.2.1
C_PATH += /usr/include/c++/4.2.1/x86_64-apple-darwin10
C_PATH += /usr/include/c++/4.4.3
C_PATH += /usr/include/c++/4.4.3/x86_64-linux-gnu
C_PATH += /usr/lib/gcc/x86_64-linux-gnu/4.4.3/include

instead. Although the wiki config does try to build a native Felix
representation of this data, it's no use to the plugins, which require
a string they parse themselves.

The string format is "cheap untyped and unsafe polymorphism".
Plugins have trouble with static data types (it can be done but
it isn't quite so easy to say, pass a list, we have to note there's
no type checking because the interfaces are all extern "C"
instead of C++).

Passing a string has the advantage that it doesn't require any knowledge
of what plugins exist. This layout:

extension flx -> flx2html::xlat_felix
extension flxh -> flx2html::xlat_felix
extension c     -> cpp2html::xlat_cpp
extension cpp   -> cpp2html::xlat_cpp
extension cxx   -> cpp2html::xlat_cpp
extension h     -> cpp2html::xlat_cpp
extension hpp   -> cpp2html::xlat_cpp
extension fpc  -> fdoc2html::xlat_fpc
extension fdoc  -> fdoc2html::xlat_fdoc
extension ml    -> ocaml2html::xlat_ocaml
extension mli   -> ocaml2html::xlat_ocaml
extension py    -> py2html::xlat_py

is not currently properly used by my webserver or the wiki: the file
extension is just ignored. Both programs have hard coded mappings
from the file extensions to inbuilt handlers which dispatch to fixed
plugins to do the colourisation. The plugins themselves (felix and
fdoc plugins) also do this.

None of this is optimal, but for the moment the hyperlinking problem
needs to be fixed so the wiki displays dispatch properly to
library files, etc.

I am going to change more things slowly regarding configuration.

For example, the initial build system calculates a C compiler.
The way to use this is hard coded into plat/config.flx,
which is linked to by flx so it can compile the C++ generated
by flxg.

But the right way to do this is to use flx_pkgconfig to get the
required fields from a database. This allows you to just point
to a different compiler package e.g to switch from gcc to
clang without having to recompile flx.

Another way to do it .. is to point flx at a plugin which has the
appropriate pre-compiled flx code. This approach is needed
when "string parameters" aren't enough (eg colourising code).

One thing to look at now is that since we have "OO" in Felix
by way of interfaces and objects, and we can have records,
of anything (eg config parameters) with statically fixed
fields, it's time to consider a dictionary type.

It is possible to do this: given a Python or JS type of dictionary,
string to string, it's possible to "automagically" populate a
record using it. In other words you have a record type:

interface X {
mydata: string;
yourdata: string;
};

and you have a dictionary:

var d = dict[string, string] (list ("mydaya","value","yourdata":"other". ...);

var x : X = object X() end;
populate x from d;

Non-trivial, but the compiler knows the names of the fields in the
object, and what type they are, so it can easily load up the record
from the dictionary (well it's a lot of hackery in Scheme action codes
in the parser but it's all doable in user space, no mods to the
compiler required).

Of course we HAVE dictionaries already: JudySL arrays map
(null terminated) strings to pointers. They're just not convenient
to use (no iterators, no simple syntax for setting values etc).
but we can just copy Python there until we have a suitable
typeclass. We can also use STL map for string/string
(though there's probably no point since Judy is sure to be
10x faster).

The important thing here is to provide a dynamic <--> static
typing bridge.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Extending flx_pkgconfig

john skaller — 2012-05-24T10:53:08

I'm extending the capabilities of flx_pkgconfig, so it can query pkgconfig (*.pc)
databases.

pkgconfig is a seriously bad program. It is very badly written,
it is full of bugs, and it has limited capabilities (which is why
flx_pkgconfig was written). Indeed, it has a very specialised application
and it can't even handle that properly.

I have extended flx_pkgconfig (next commit) to add;

--help

well it should always have had that ..

--extension=fpc

to set the extension to look for, if you set pc it will find pkgconfig files
instead of flx_pkgconfig files (will play with this to see if both can
be queried at once later).

I have added variables as used by pkgconfig:

prefix = /usr/local
lib = ${prefix}/lib

Libs: -L${lib} -lmylib

Also blank lines are tolerated, and lines starting with # are skipped.
pkgconfig allows trailing # comments which are not handled yet.

Pkgconfig also doesn't allow comma separators: some options go
like this:

-W1,linkeropt,-W1

and have embedded commas, so I'll have to remove that feature.
Some convenience flags such as --cflags can be added later.

A couple of stupid features not supported yet: splitting linker switches
into -L  and -l. Probably support that by filtering values based on
a regexp (although at this point, flx_pkgconfig is very carefully built
to avoid using any regexps).

One of the biggest problems is version control. In pkgconfig, a Version
field provides the version, and you can write:

Requires: fred < 1.2.11
Requires: joe = 2.3.4

This makes the "Requires" field special (since the version requirement isn't in the
correct form of a general field). Also breaks the dumb parsing I'm using for variables
(which would think the last line is an assignment of 2.3.4 to variable "Requires: joe" :)

However with enough hacks and tweaking, flx_pkgconfig can probably be made
a "drop in" replacement for pkgconfig.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

ncurses on OS, a note

john skaller — 2012-05-24T04:29:30

I'm writing some ncurses stuff to perhaps help manage Felix
installations etc. 

I found the installed version on OSX doesn't work. I found this note:

http://www.uponmyshoulder.com/blog/2010/fix-ncurses-in-os-x-10-6-3/

and I'm trying it out. Nope. Press an arrow key and still get the f'ing escape
sequence, not interpreted by ncurses as it should be.

Unix is such .... Terminals. Ye gads, how utterly archaic is this concept?
We've had actual keyboards and screens on PCs for a while.

Ah well. No idea how to proceed, except perhaps make a Felix
resource data base to replace the crap. I note, the terminal emulators
all work fine.



--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

windows utils

john skaller — 2012-05-23T12:42:23

gnuwin32.sourceforge.net/

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

automagical record coercions (record subtyping)

john skaller — 2012-05-22T00:07:35

At present if you have a function accepting a record with
the set of fields F, you can call it with a record with
the set of fields G, provided F is subset of G, and,
you *explicitly* coerce G to F:

typedef F = (a:int);
typedef G = (a:int, b:int);
fun f(x:F)=>x.a;
var g = (a=1, b=2);
println$ f( g :>> F);

If you leave off the coercion, you will get this:

Client Error binding expression (f g)
CLIENT ERROR
[lookup_name_with_sig] Can't find f of struct  {a:(int);b:(int);}
In /Users/johnskaller/felix/./ft.flx: line 5, cols 9 to 10
4: var g = (a=1, b=2);
5: println$ f( g);
           **
6:


There's an argument to provide "some kind of subtyping" here:
a G "is an F" (it just has extra fields). The coercion we use has
a common generic name in algebra: "the forgetful functor",
which is a structure preserving map that simply "forgets" some
of its domains structure.


If we could do automagical coercions, you could write "generic"
functions and just call them with any record which extended
the parameter type, without having to bother with the coercion.

If there were multiple matches, then we can use the "best match"
rule we currently use, but applied to record specialisations instead
of type variable specialisations: if one coercion forgets fields X
to make a match, and another forgets fields X \cup Y we'll pick the
one that forgets the least fields, namely the one that merely forgets X.

Note that X is a strict subset of X \cup Y: two coercions H and K are
only comparable if, of the set of fields they forget, one is a subset
of the other.

Now, the problem I have implementing this is it appears at first
glance to require a "hack" to the unification algorithm. Unification
solves for type variables. Specialisation is a modification of unification
that specifies some type variables are independent, and some dependent.
Independent types variables are regarded as existential, that is,
they're actually fixed but unknown types, and so we're not allowed
to replace them. For example

parameter = int * v 
argument = w * int

unification generally would provide a solution 

v = int
w = int

but we do not wish to allow that, we only want to allow specialisation,
we don't want to allow w to be replaced, only v. Roughly in the equation:

int * v = w * int

we only want to allow variables on the LHS to be replaced, so in this case
there's no solution.

Because of the way the unification algorithm works, it isn't possible to just
ask for "only substitutions of variables on the LHS" because sometimes
variables on the RHS get put on the LHS by a substitution, so we have
to list the dependent variables we want to solve for.

Ok, so, the point is just saying "we want to allow the LHS to unify with the
RHS if the LHS is a record and the RHS is a record with more fields" will
not work. In fact all hell would break loose.

So I'm thinking of this solution: if the LHS is a record, we just replace
it with a new dependent variable. Then, unification will match ANY type.

Then, AFTER we have got the result back from unification, we apply the
subtyping rule as a constraint. So, suppose we have

typedef F = (a:int);
typedef G = (a:int, b:int);
fun f(x:F)=>x.a;
var g = (a=1, b=2);
println$ f( g );

then we match f[v] of (v) instead of f of F.
We get a match with most general unifier (mgu) 

v -> G

Now we apply the constraint:

F is subtype of G

which in this case passes: f is a candidate.

It's vital to know that this subtyping doesn't necessarily apply in general.
It *could* apply for records nested in records and tuples provided we're
willing to reconstruct the whole top level record or tuple with a sliced back
component.

More on this later. However the proposal is to hack the overloading
algorithm to change record argument to a variable and retrofit
the record subtyping rule as a constraint.


--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Objects in Felix

john skaller — 2012-05-17T17:38:21

Felix has been able to do OO for some time, it just isn't sugared.
Here's an example:

//////////////////
// An OO experiment

fun X (var x:int, var y:int) = {
  proc setx(a:int) { x = a; } 
  proc sety(a:int) { x = a; } 
  fun getx() => x;
  fun gety () => y;
  fun sum () => x + y;
  return (setx=setx, sety=sety, getx=getx, sum=sum);
}

var a = X(2,3);
println$ a.getx();
a.setx(22);
println$ a.getx(), a.sum();

var b = a :>> (sum:1->int);

proc f(v:(sum:1->int)) 
{
  match v with
  | (sum= ?pp)=> println$ pp();
  endmatch;
}

f b;
/////////////


Here, X can be viewed as a constructor for an object.
The type of the object is:

(
  setx:int->void,
  sety:int->void,
  getx: unit -> int,
  gety: unit -> int,
  sum: unit -> int
)

that is, it's a record of functions. Notice that the "data" is 100% encapsulated
by the constructor function in this case (but, you could return a pointer
if you wanted a variable, or a value if you didn't).

Now the line

var b = a:>> (sum: 1-> int);

is an upcast (coercion) to a type with only the sum field. So we have 
subclassing based on "forgetting some fields". There's a more
anonymous way forget fields using a match. This will work too:

  match a with
  | (sum= ?pp)=> println$ pp();
  endmatch;

you should note that record coercions have nothing to do with
objects (coercions and pattern matches work with all records).

At present the overloading system requires exact matches so you
must explicitly coerce a record to strip fields to match a function
taking a record. [It may be possible to relax that].

Clearly, you can easily implement extensions:

fun Y(a:X, b:int) { .... }

The most important thing to note about this style of OO is that
the subtyping notion is NOT based on extension. You can
upcast to a "base" consisting of ANY subset of fields without
pre-declaring any base types. This is similar to Ocaml's object
system and is conceptually VASTLY superior to conventional
inheritance: a type with a set of fields Y "isa" X for X with every subset
of the fields of Y.

I wonder if this is worth sugaring ..

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

install layout

john skaller — 2012-05-17T12:48:18

Im still struggling with the install layout.

This makes sense:

felix/share/felix-version/lib/std/*.flx ... etc etc

Each version has a new standard library, tool sources, etc.
This would also cover C++ sources.

Now, the platform dependent stuff is in two parts:

(a) configuration data: *.fpc files, specs for calling C++,
C++ config headers.

(b) compiled binaries: executables, static libs, shared libs

There will be a standard one of these: host, an abstract name
for the current desktop config. Sharable between users.

Other configs could be installed for cross compiling.

Each user currently has $HOME/felix with personal data.
Presently this is just cache data PLUS if the user is a felix
developer, initialisation data for the host config.

In principle, the config data depends on the platform
not on the Felix version. It may change with compiler
upgrades (independently of the Felix version).

So one problem here is whether to version the config data.
It will surely change with Felix version. On the other hand,
if we do that each new install will lose the local tailoring
(where various third partly libraries are kept, for example).

At present persistence come from the builders HOME/felix.
Better than nothing, but still not really right.

Now, at present we have

flx --test=install-point

and that only works because there's a single install point.

If we split shared stuff from the config, it won't work.
We'd need TWO variables: one to version the shared
library code and one for the config.

The idea would be

flx --config=configdir

which defaults to host. --test defaults to /usr/local/lib/felix/
share/version. But where is host? Do we need to repeat
the version in the config?

Maybe the way forward is to include the share/version IN 
the config? But then an upgrade would require duplicating
the configs.

I'm confused :) Need help thinking on this.


--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

xml2

john skaller — 2012-05-16T01:23:13

How much does the webserver (mikes) and wiki
depend on xml2? 

It's another dependency ..

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

reading unit

john skaller — 2012-05-14T22:10:18

The following never works and can't be made to work:

var ch = ischannel[unit] ();
var u = read ch;

The reason is: you cannot use the unit value u, so the read is
optimised away. Even

C_hack::ignore(u);

fails with a diagnostic, because Felix tries to optimise away all units,
and there's a diagnostic in the code generator to report when the
front end failed to do this. The only way to do it is to use
a supervisor call, and it isn't clear even that will work.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

catch variable name

john skaller — 2012-05-10T11:10:03

So here's the new code:

////
proc mythrow[T] : T  = "throw $1;";
const fred: int = "fred";

try 
  var x = 1;
  mythrow 20;
  x = 10;
catch fred : int =>
  x = 2;
  println$ "Exception " + str fred;
endtry

println$ x;
////

The reason you still have to type the variable name is ...
I don't know how else to do it at the moment.

An obvious implementation is to convert the catch code into a procedure:

proc x73735 (fred:int) { 
  x = 2;
  print$ ..
}

and then:

catch (int &exn) { x73735 exn; }

The problem with this .. and in many other similar cases is that

return

won't work.  Felix has no frontend block concept.
This issue also impacts yield. It also changes the kind
of gotos, but Felix can handle non-local gotos.

Which suggests .. non-local returns.

Actually a block would be nice, which allowed local variables
without interfering with returns and perhaps supervisor calls,
as well as goto's into the block: the variables would be lifted out,
eliminating the block, and renamed to prevent a conflict.

Felix already has (AST level) identifier renaming routines,
the main problem with them is it makes diagnostics "down the track"
hard to understand (since you get told about identifiers whose names
were invented by the compiler .. :)

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Install tree

john skaller — 2012-05-10T10:35:05

I am thinking of changing the install tree to this

/usr/local/lib/felix/host/felix-latest/....

[replace / with \ on windows]

or perhaps 

felix-latest/host/...

not sure whether to put the target or the version first.
Might have to change the cache address in a similar way.

The idea is that you can say:

flx --target=platform ....

where the default platform is host. The host platform is configured
to edit, compile, and run on your desktop machine (as detected by
the build system).

The platform name would be arbitrary, although it may pay to copy
the idea gcc uses assuming gcc is halfway reasonable about 
cross compilation. I think that's:

i686-apple-darwin10-gcc-4.2.1

on my box:

processor-vendor-os-product-version


On my box at least I want to test:

* both 32 and 64 bit targets
* both gcc and clang

which is 4 distinct platforms .. all of which are technically still "host"
platforms. 

Actually, since most of the code is platform independent it would probably be more like:

/usr/local/lib/felix/felix-latest/share/...
/usr/local/lib/felix/felix-latest/i686-apple-darwin10-gcc-4.2.1

or something. I would like to remove platform configuration code from the
Felix library (plat/config.flx in particular).

Most of the C++ code for the RTL is platform independent, and each 
file includes a platform configuration file, which should live as

config/*.hpp


However, to make it easy to "ship" the default config could still be built in (hard
coded), otherwise the client needs to provide extra files somewhere just
to run the compiler.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

semantics

john skaller — 2012-05-10T04:08:10

I'm still struggling to lock down Felix semantics.

As stuff like try/catch shows, certain code has to be stackless,
and other code doesn't.

It's hard to know what's what due to the optimiser.
Current "inline" makes a difference: non-flat code can be
flattened by inlining. You can only do supervisor calls from
flat code (zero stack). If you make C blocks like
try { .. } .catch { .. }  (and sometimes while () { .. } or for () { .. } etc)
you're introducing a stack which prevents returning to the 
supervisor, as well as jumping into the block (you can jump
into a while or for provided there are no variable initialisations
missed)

I need to carefully sort out what's what, and provide annotations
for enforcement. Eg

stack proc f (..) ...

ensures the proc f is on the stack or inlined not the heap, 

heap proc f () .. 

ensures inlining or on the heap or something :)

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Exception experiment

john skaller — 2012-05-09T14:20:52

The following code now works:

///////////////
proc mythrow[T] : T  = "throw $1;";
const exn : int = "_exn";

try 
  var x = 1;
  mythrow 20;
  x = 10;
catch int :>
  x = 2;
  println$ "Exception " + str exn;
endtry

println$ x;
/////////////////////
~/felix>flx --test=build/release ex
Exception 20
2
/////////////////

Note: this is experimental DO NOT USE in production code.
For playing only. It generates the "obvious" C++:

try { ....}
catch (int &_exn) { .. }

Note temporary hack: the exception caught is named _exn.
This is mainly because I couldn't figure out how to allow the user
to name it: it can be done, and the syntax will probably change to

catch (x:int) => ....

The catch thing is basically a procedure, but it isn't declared in the
normal way, so I have to actually think about how the
variable gets a name. The implementation is a hack: try just
generates "try {" and endtry just generates "}" and catch generates
"} catch (" + type + " &_exn) {": naming the variable should imply
it has a scope from the start of the catch handler to its end, but the
implementation doesn't make any scope. C++ does though!

Ok, so: you can NOT use this construction if there is a supervisor call
inside the try block. Supervisor calls work by returning control,
which implies there must not be any machine stack. After the call
we resume, and you can't jump into the middle of a block in C++.

It's worse though: you cannot use this construction if you call
a procedure closure in the try block either, because that also works by
returning control. However this doesn't mean you can't call
a procedure in there, it just means the procedure has to be
inlined. Most directly called procedures will be inlined unless they're too big,
the caller is too big, or the procedure is recursive.

Procedures called via closures can't be inlined.

Furthermore, "throws" from procedures cannot propagate to their
callers, because procedure calls don't use the machine stack.
You can throw from inside a function though,  since they do.

It's rather unfortunate, but in Felix you have to KNOW what
the implementation of code will be. Inlined or not matters.
Functions also get inlined, and what's more, many of them
are actually converted to procedures (by passing a pointer to the
result variable, and replacing return x with *result = x;)

The spaghetti stack is a way to swap stacks quickly, without
having to swap machine stacks (which cannot be done portably
other than by using threads). The machine stack is used for
functions, for speed and C compatibility.

The resulting system is a hybrid with somewhat difficult characteristics,
but there's no real alternative if we want to generate C++ which can
also use existing C++ code.

The right way to handle this is probably to use the type system to
prevent incorrect guesses as to the final optimised implementation:
optimisation shouldn't change the semantics of correct code,
but it does allow technically "incorrect" code to work, which is bad
because down the track it may fail for unknown reasons.

I dislike exceptions. But it's reality that C++ code such as library functions
might throw them in circumstances where we actually want to catch them
rather than terminate.

Also, they're useful for quick exits. For example searching an array or list
for a value using an iter function should be able to stop as soon as it
succeeds.

Anyhow, the RULE for proper use of exceptions is to keep them localised,
and make sure the try block contains simple algorithmic code.



--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

sugar

john skaller — 2012-05-09T08:19:53

Just some quickies: Mike has this:

  fun get_suffix(fn:string) = {
    var fname = fn;
    var suffix = "";
    match rfind (fname, "?") with
      | Some ?pos => { suffix = fname.[pos + 1 to]; fname=fname.[0 to pos]; }
      | None[int] =>
      {
        match rfind (fname, ".") with
          | None[int] => {}
          | Some ?pos => { suffix = fname.[pos + 1 to]; }
        endmatch; 
      }
    endmatch;
    return suffix;
  }

which is perfectly valid: calculate a closure then execute it. But Felix
now has statement matches so you can write this too:

  fun get_suffix(fn:string) = {
    var fname = fn;
    var suffix = "";
    match rfind (fname, "?") with
      | Some ?pos => suffix = fname.[pos + 1 to]; fname=fname.[0 to pos]; 
      | None[int] =>
      
        match rfind (fname, ".") with
          | None[int] => {};
          | Some ?pos =>  suffix = fname.[pos + 1 to]; 
        endmatch; 
      
    endmatch;
    return suffix;
  }

But then, we could get more functional again:

  fun get_suffix(fn:string) =>


    match rfind (fn, "?") with
      | Some ?pos => fn.[pos + 1 to];
      | None[int] =>
      
        match rfind (fn, ".") with
          | None[int] => ""
          | Some ?pos =>   fn.[pos + 1 to] 
        endmatch
      
    endmatch
  ;

Noting the assignment to fname in the first match case is useless,
we're able to drop the fname variable and just use fn.

As a curiosity, this could be done as so:

  match find_last_of(fn,".?") with
  | Some ?pos => fn.[p[os + 1 to]
  | None => ""

since find_last_of searches for any of the characters in the second
argument, whereas rfind searches for the whole string as a substring.
[The C++Standard is clear but the description is a bit hard to deciper]

BTW: I'm not a style nazi, but the recommended style for Felix is
2 character indent, including for continuation lines. This is useful
in narrow format documentation such as books, web pages, etc.

If you want to line function arguments up do this:

fun long_name (
  arg1:t1,
  arg2:t2
)

rather than this:

fun long_name (arg1:t2
                             arg2,t2) 

The latter style cannot work unless the font is monospaced.
[I think it looks ugly too, because you have blocks starting
all over the place]


Another useful hint: for long string quotations:

  val x =
"""the quick brown
fox jumped over
the greasy hog""";

looks horrible. You can do this:


  val x =
    "the quick brown \n" 
   "fox jumped over \n"
   "greasy hog"
  ;

without  performance hit (the strings are folded by the compiler
early in the compilation process). You can also do this:

macro val nl = "\n";
val x =
  "the quick brown " nl
  "fox ....

with the same result. If you do this:

val nl = "\n";

instead, the result will be the same but the concatenation might
happen at run time (not sure!).

Another useful hint:

If you want to write code with prefix functional notation .. try Scheme :)
C uses a mix of prefix, postfix, and infix notation, and so does Felix.
However Felix is cleaner:

* you don't need to write f(x) to call a function, f x will do
for the innermost application. In Felix operator whitespace
is left associative so f g x means (f g) x and you may need
to write f (g x) if that's what you want.

* if you can write f x, you can write x.f which again reduces the
need for parens:

f x.g

or even

x.g.f

as if they were method calls. This works with tuples and (should work
with almost) everything else too:

f (1,2) ==> (1,2).f

Operator . has a higher precedence that whitespace.

If you need to factor an expression:

val k = x + y * z

you can write:

val a = y * z;
val k = x + a;

but this leaves the temporary accessible. Another way that doesn't require
any statements is:

val k = let ?a = y * z in x + a;

The LHS of a let binder, ?a here, is an arbitrary pattern, so you can write

val two = 1,2;
val second = let _,?second = two in second;

as well. You can even write:

val maybe = Some 100;
val cent = let Some ?x = maybe in x;

if you're sure the None case can't occur (if it does you get
a run time error). let is just a single case match:

val cent = match maybe with | Some ?x => x endmatch;

You can also leave the "endmatch" off in an expression if there's
no ambiguity (but NOT in a statement match):

val cent = match maybe with | Some ?x => x;

Similarly you can leave off the endif if there's no ambiguity:

   var y = if x == 2 then 1 else 20;


--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

backing up the wiki

john skaller — 2012-05-09T06:53:34

I'm a unix ignoramus, need help figuring out how to store the wiki data.

1. The plan to start is to manually push/pull from a github repository.

2. The name of the directory containing the wiki must match the repository name
AFAIK.

3. because we're using git hub, the tutorial etc can be part the wiki,
and still edited offline. Very nice. Though we don't know how to handle
conflicts.

4. For security reasons, the wiki is run under account felixweb, which
does not have a login shell (I think the home directory is mine though :)

5. Currently the data is own felixweb group felixweb.
The wiki account cannot push/pull because it has no shell
and no keys.

6. I can't push/pull either because my account is an ordinary one,
I can't modify another users files.

7. root can't use ssh for security reasons.

8. I am guessing .. note *guessing* a possible solution is to make
myself and mike and felixweb members of group felixweb
then we can use our personal accounts to push/pull the data,
though I don't know who the owner would be, provided all the
files had r/w permission for group members, it should all work.

Is that right??

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

try/catch

john skaller — 2012-05-06T13:32:51

I'm playing with this:

var n = 1;
var d = 0;

union numeric_error = | overflow of string | divide_by_zero of int;
type std_exception = "::std::exception";
fun msg: std_exception -> string = "$1.msg";

var r = 
  try n/d
  catch numeric_error with
  | overflow ?s => -1
  | divide_by_zero ?n => n
  catch std_exception with
  | ?x => #{ println$ "exception " + x.msg; throw x; return 0; }
  endtry
;

The idea is basically to be able to catch a (C++) exception
thrown by an expression and return an alternate value.

Perhaps a better example:

try fold_left (fun (acc, elt) => if elt == 0 then throw 0 else acc * elt) lst
catch int with | ?i => i 
entry

Basically, you can have a series of catch clauses,
which catch particular types, and then pattern match
the caught value and return some expression. This allows
default values and shortcuts. The pattern match is an
ordinary match on the caught exception. The match handlers
must all return a value of the primary expression type
(the type any should be allowed as well, so you can rethrow).

It's a but clumsy to "do" something in an expression though.

The syntax is related to functions:

fun f: int -> string = | ?x => str x;

which of course means:

fun f(x:int) => str x;

Another way to do this is match "catch" the same syntax as "fun",
that is, any of the forms of fun.

catch (x:int) => ....
catch (x:long) => ...
catch (x:std_exception) => ..

The removes the need to put the match cases in as part of the syntax,
however the return type, if specified on any of these functions,
would have to equal the primary expression type, a bit redundant.

Still thinking. Comments appreciated.

Note: the try/entry is an expression. It has to return a
consistent value (or not return normally).
It can't be used by statements, because procedural
code has no machine stack.

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

linkage names

john skaller — 2012-04-30T19:19:18

I have been playing with the new naming system.

It works kinda like this:

//PRIMITIVE 8473 INSTANCE 83822: string
typedef ::std::basic_string<char> _a8473t_83822;
typedef int _us2;

//PRIMITIVE 3795 INSTANCE 83825: int
//typedef int int;

//TYPE 83827: string^2
typedef _at<_a8473t_83822,2> _at83827;

//TYPE 83828: string * list[string]
typedef _tt2<_a8473t_83822,void*> _tt83828;

//TYPE 83829: list[string]^2
typedef _at<void*,2> _at83829;

//PRIMITIVE 5809 INSTANCE 83830: ostream
typedef ::std::ostream* _a5809t_83830;

//TYPE 83831: ostream * string
typedef _tt2<_a5809t_83830,_a8473t_83822> _tt83831;

...

//TYPE 83827: string^2
template <> struct _at<_a8473t_83822,2> {
  static size_t const len = 2;
  typedef _a8473t_83822 element_type;
  _a8473t_83822 data[2];
  _at<_a8473t_83822,2>() {}
  _at<_a8473t_83822,2>(_a8473t_83822 a0, _a8473t_83822 a1) {
    data[0] = a0;
    data[1] = a1;
  }
};

//TYPE 83828: string * list[string]
template <> struct _tt2<_a8473t_83822,void*> {
  _a8473t_83822 mem_0;
  void* mem_1;
  _tt2<_a8473t_83822,void*>(){}
  _tt2<_a8473t_83822,void*>(_a8473t_83822 a0, void* a1):
    mem_0(a0), mem_1(a1){}
};

Basically, we have a template _tt2<X,Y> for tuple names,
we never define it generally, but we do specialise it.
Similarly _at for array types, _ft for function types.

The executable code continues to use the aliases (typedefs).
Each specialisation's argument are also using the aliases,
the each appearance of a structure name only uses one level
of template syntactically.

The idea here is that any Felix code needing, say, an
array of two strings or a list of ints or whatever will name
them the same for the purpose of external linkage,
so C++ functions can be linked across translation units
and DLL boundaries using C++ type safe linkage.

Without this, it is hard to "export" a function that takes
a structurally typed value (tuple, array, variant, etc):
we can put the argument type in a header file easily,
but the name will be 

_a8473t_83822

or something which could change with any minor modification
to any part of the library, so we cannot then create a Felix binding
to this C name in out importation script.

A name like:

_tt2<string, int>

however is deterministic, based only on the names of the tuple
argument types.

nominally typed enties (structs for example) can be given their Felix
names as their C names. And we can wrap such names in namespaces
corresponding to the Felix class containing them to ensure they're unique.

but I forgot about abstract names:

type metres  = "int";
type feet = "int";

and now metres * metres and feet * feet both have type:

  _tt<int, int> // actually _at<int,2>

This actually saves on duplicated definitions but it makes it impossible
to export both

f of (metres * metres)
f of (feet * feet)

in C++ since the linkage names are

f (int, int)

in both cases. In fact the problem has nothing at all to do with tuples!!!!
It's a problem for

export fun f(x:metres)=> ...
export fun f(x:feet)=>

too. So, we can create, for primitives, a special tag name:

struct metres;
struct feet;

and use that for the tuple definition instead of "int": the actual
type used in the definition is still int, just the name of the tuple
type itself uses the tag name (the tag name is never defined).

this is cool because it solves another problem: what do we name
a polymorphic type instance?

Clearly from

type vect[T] = "::std::vector<?1>";

we need to generate a tag name like this:

template <typename T> struct vect;

Again, this is not a type. It's just a unique name
used when defining the actual type.

However, this got me thinking: how does Felix actually handle
recursive types now?

The distinction between the tag/structure names and the actual types
is that the tag names can put a dummy where recursion occurs:
in fact the internal Felix representation uses a weird encoding
for fixpoints already, a list is like:

1 + T * (fix -2)

where the -2 means the fixpoint binder is up two levels in the term:

(1 + T * x) as x

One level for the * and another for the +. The obvious implementation for
tags is just

_fix<2>

The idea was the definitions of types could do this:

struct list {
..
list *next;
};

and it would work because C supports types recursion via incomplete types.
The BIG problem with C++ templates is that they don't: you can't give a template
class definition an incomplete type argument, even if it is only used as the target
of a pointer in the definition. [This is a serious BUG in C++ and is half the reason
Felix exists in the first place. It's intolerable]

However, Felix hardly every uses this. The actual list definition uses a void *
and casts. It does this because that's how variants (sums, unions, etc) are
encoded: we use universal types and casts on usage.

It does use it sometimes though:

fun f(x:int): (int -> t) as t=> f;

//TYPE 79711: (int -> fix-1_a as fix-1_a )
struct _ft79711 {
  typedef _ft79711* rettype;
  typedef int argtype;
  virtual _ft79711* apply(int const &)=0;
  virtual _ft79711 *clone()=0;
  virtual ~_ft79711(){};
};


Another example:

var x = (1,&x);

//TYPE 79705: (int * &fix-2_a as fix-2_a )
struct _tt79705 {
  int mem_0;
  _tt79705* mem_1;
  _tt79705(){}
  _tt79705(int a0, _tt79705* a1):
    mem_0(a0), mem_1(a1){}
};

These show that it is NOT enough to just use a template
to generate a tuple .. in the tuple case here, the type of
the second argument is a pointer to the structure type.

BTW: this type is USEFUL!!! It's an infinite list.
And those things can be used in iterators (streams) to good effect.

The tuple above, however can be defined by

template<> struct _tt<int, fix<2>*>
{
  ...
}

and the name is canonical (unlike _tt79705)!!

So unfortunately .. we cannot use template definitions for tuples
(or functions or arrays or whatever, which is very sad).

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

grrr

john skaller — 2012-04-30T05:34:34

Why do I hate unix so much ... grrr ...

This works fine on my OSX box,
but on Linux:

skaller< at >felix:~/felix$ build/release/bin/flx_ls . '.*flx.*'
src/demux/flx_demux.hpp
src/rtl/flx_main.cpp
src/rtl/flx_compiler_support_bodies.hpp
src/rtl/flx_ioutil.cpp
src/rtl/flx_eh.hpp

[There are hundreds of files that match the pattern]

Actually I' just trying to find where Mike put the Wiki so I can
make a Git repo for the contents. Locate doesn't find it.

The pain is: flx_ls is designed for this job, to replace the 
unix file system and tools handling of directories (which is
exceptionally bad).

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

External linkage problem may be solved

john skaller — 2012-04-28T04:47:04

One of the major problem facing Felix is the lack
of a canonical name for its types. It's ok to require a user
to export a function or procedure. However exporting
types is horrible and it doesn't really work.

The problem is partly avoided by C functions with used defined
names because C linkage does not provide type safe linkage.
But that also introduces a problem!

Consider this function:

fun sum(x:list[int]) => fold_left + of (int*int) 0 x;

We'd like to just say:

 export fun sum(x:list[int]) => fold_left + of (int*int) 0 x;

and export it as C++. We have to use C++ linkage because there could
be many functions called sum. We would also like to gain the benefits
of type safe linkage.

The problem here is: what is the C++ name of the type list[int]?

The current answer is something like:

_tt936478

in one compilation and for another file

_tt223456

Felix just uses a fresh integer whenever it needs to invent the name of a 
an anonymous/structural type (such as a list).

Now a list is defined (in effect) by:

(1 + T * self) as self

which means "empty or T and list". It is easy to name tuples:

tup2<int, long>

would suffice. Similarly variants:

var2<double, string>

and there's a way to synthesise names for unions based variants (since the union and the
components all have names, at least unique in their scope). But how to do the
recursion?

Here's the answer, this actually works on g++:

    // core combinators
    struct unit;
    struct point;
    template<class T,class U> struct fix;
    template<class T, class U> struct tup2;
    template<class T, class U> struct var2;

    template <> struct   
      fix<
        point,
        var2<unit, tup2<int,point> > 
      > 
    { 
      // definition goes here
    };


using the fix and point types to represent recursion. 

--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

Os abstraction model

john skaller — 2012-04-25T02:18:12

Now we're trying to get stuff working on Windows, we need a model
for portable code. Note there's a trickiness needed here:

a) Cygwin can do both posix and windows functions,
so it may not be "one or the other"

b) We still need to write platform specific code

c) All OS .. especially posix .. have extensions and variations,
indeed some lameness is found in compilers, especially gcc,
when trying to get *their* extensions too, without conflicting
with standards (some _GNU_SOURCE routines conflict
with Posix and gcc folks haven't been smart enough to provide
a way to get the Posix standard stuff without losing the desired
gnu extensions .. grrr .. )

At present the goal is to get "flx" to work. This is probably not
that hard. Windows actually provides most of the C std library
and a lot of posix emulation stuff already.

For example, there's no "stat()" function in windows but there
is a set of _stat functions. The stat buffer even has gid in it,
even though it's always 0.

So a core model is something like:

std/filestat.flx               // portable
std/posix/filestat.flx    // posix code
std/win32/filestat.flx   // win32 code

where the portable code goes in std/filestat.flx PLUS 
conditional compilation. That will, unfortunately, also have stuff like:

if WIN32 do
include "std/win32/filestat";
else
include "std/posix/filestat";
done

The problem with this is that you also get Posix or Win32 specific functions
you can accidentally call. If you consider:

POSIX:
  pod type stat_t = "struct stat" requires Posix_headers::sys_stat_h;
  proc stat: string * &stat_t * &int = "*$3=stat($1.c_str(),$2);";

WIN32:
  pod type stat_t = "struct __stat64" requires Posix_headers::sys_stat_h;
  proc stat: string * &stat_t * &int = "*$3=_stat64($1.c_str(),$2);";
 
however you see the Felix interface is the same. However there's actually
a completely different (and better) way to do this on Windows using
GetFileInfoEx or something.


--
john skaller
skaller-Rn4VEauK+AKRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
http://felix-lang.org




------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/