gmane.comp.security.openxpki.devel

Development tracks

Martin Bartosch — 2012-05-24T19:37:00

Hi folks,

I like the momentum Oxi has been getting lately - possibly due to moving from SVN to git, which allows us to pursue experimental and potentially disruptive development.

The develop branch will soon see the integrating of the following development tracks. This will probably result in a huge construction site, but - hey - it's the develop branch for a reason ;-)

- database unification and improvements (Michael's submission for indexing LOBs) -> schema change necessary
- Dieter's submission to the WF engine (paused workflows) -> schema change necessary
- watchdog daemon keeping track of stalled workflows
- inclusion of the NICE API (Oliver)
- inclusion of the new Connector infrastructure we are currently working on (and successive migration of the core configuration to Connector)

Some things to come later this year:
- redesign of the SCEP workflow: many more configuration options, allowing for flexible policies for renewal and support for initial enrollment via SCEP (challenge password)
- inclusio

paused workflows,proc state and more: first implementation

Dieter Siebeck — 2012-05-18T13:20:08

Hi,

my name is Dieter Siebeck and I am quite new to the OpenXPKI project. I 
am working with Oli Welter since nearly 3 months now, mainly on his 
idea/draft of "paused workflows" (see his mail to the list from 2012-02-14).

Now I have published my first implementation of this feature. You can 
see and download it here:

https://github.com/dsiebeck/openxpki/tree/feature%2Fworkflow

(this branch has been rebased on the devel branch 22 days ago.)

POD  Documentation  of most of the new methods is still missing, but I 
wanted to publish the current state, to give other interested people the 
opportunity to see and maybe correct the big picture.
Also missing are test files/scripts. I would appreciate some 
recommendation, which place to look for the most up2date state-of-the 
art testing files for this.

Now the idea of the implementation: basic need was to gain relatively 
low-level control to the execution of workflows and activities. To 
achieve this, the best (and maybe only) way seemed, to install an "own"

Removal of obsoleted database entries

Joachim Astel — 2012-05-03T10:50:55

We're doing some database cleanup within our database via cronjob for
our OpenXPKI databases, especially for the audittrail log:

DELETE FROM audittrail WHERE DATEDIFF(NOW(),CONVERT(logtimestamp,DATETIME))>365

Unfortunately, generated CRL (via openxpkicmd) are growing within our
workflow table. We would like to remove old workflows with workflow_type
CRL_ISSUANCE, but backend database entries which are associated with it
would also have to be deleted -- to avoid database garbage.
Are there already scripts around in other projects addressing this issue?

Greetings
    -Achim

PS: I've already deleted workflows with 
"DELETE FROM WORKFLOW WHERE WORKFLOW_TYPE='I18N_OPENXPKI_WF_TYPE_CRL_ISSUANCE'"
I suppose a garbage collector would be good for this problem too. :-)

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discuss

SQL DYNAMIC patch incomplete ?

Oliver Welter — 2012-04-26T17:24:55

Hi Micha,

I am running into some problems using the datapool after merging your 
dynamic-patch.

my $msg = CTX('api')->get_data_pool_entry( {
   KEY => $chipid ,
   NAMESPACE => 'smartcard.smartchipid'
} );

results in an exception (I added ref and column)

'LABEL' => 
'I18N_OPENXPKI_SERVER_DBI_SQL_SELECT_DYNAMIC_PARAM_NO_HASH_REFERENCE',
PARAMS' => {
     '__REF__' => '',
    '__COLUMN__' => 'DATAPOOL_KEY',
    '__TABLE__' => 'DATAPOOL'
}

I added some more debug in SQL.pm lin 970

##! 1: ' DYNAMIC arg ' . Dumper $args->{DYNAMIC}

OpenXPKI::Server::DBI::SQL::select (line 970):  DYNAMIC arg $VAR1 = {
           'DATAPOOL_KEY' => 'chipid1234',
           'NAMESPACE' => 'smartcard.smartchipid',
           'PKI_REALM' => 'I18N_OPENXPKI_DEPLOYMENT_TEST_DUMMY_CA'
         };


I found the  {value => xx } brackets are missing in the statement in 
Server/API/Object.pm, Line 604ff , but even after adding them, the error 
remains the samem, having this debug output:

OpenXPKI::Server::DBI::SQL::select (line 970):  D

Management of pull requests

Michael Bell — 2012-04-12T15:24:33

Hi,

could somebody please review my last pull request? I think it is
generally a good idea to ask for a review and do not merge my own stuff.

So any volunteers?

Best regards, Michael

git: fixing wrong branching

Michael Bell — 2012-04-06T19:36:33

Hi,

I wrote the code to add the missing indexes but I created the new branch
for this feature on another feature branch and not on develop. Is there
a chance to move the start of the new feature branch to the develop branch?

feature/sql_multiple_index_per_table was created from
feature/server_dbi_explicit_sql_like_usage. You can visit the situation
at github. The fork bellmich/openxpki is public.

Sorry for the trouble

Michael

P.S. next time I remove my old fork and create a fresh one.

Git and OpenXPKI … a short primer for people new to git and GitHub

Andreas Leibl — 2012-04-02T08:43:59

Dear all,

Changing the version control tool for a project is never easy and I want to say a big thank-you to all the developers that weren't git users before and still agreed to the move ... it can be daunting at times. Following some remarks and issues on this mailing list I'd like to provide a little write-up that should help everyone get going without spending too much time learning git. 

For those inclined to dig deeper into git there are lots of tutorials on the web but I'd also like to recommend the following O'Reilly book and videos to you (they helped me a lot on the way):

http://shop.oreilly.com/product/0636920022862.do
http://shop.oreilly.com/product/0636920017462.do
http://shop.oreilly.com/product/0636920024774.do

Now, without further ado, the promised primer. This is my understanding of how we want to do things, please correct me if I got it wrong. Hope you find it useful.

Kind regards

  Andreas

P.S.: I've added the primer to the repo. Can someone please have a look at the pull request?

Changes to Connector to make live

Oliver Welter — 2012-04-02T08:24:38

Hi,

I want to suggest some things that would make the whole connector 
monster easier to handle (at least at experienced while moving the 
Smartcard API).

1) Return of a hash containing all leaves under one node.
It is very often the case, that you expect a set of config values at a 
node. Retrieving them requires a loop construct each time on the 
programmers side. Besides, at least in most cases, the Connector has 
access to the hash which will make the access a lot faster then first 
exporting the keys and then getting them one by one.

2) Return of an array of values instead keys, when the node is an 
ordered list
Same problem as above.

3) Fixed naming convention and supporting method for the "multiple 
resolver problem".
In Smartcard Personalisation there are mulitple places that require the 
following:
* have a list of alternative "resolvers" (Proxy-Connectors)
* walk them in a defined order
* stop looping as soon as a resolver returns a result
* store the id of the used resolver to reuse it in the

Contexts (CTX) and OpenXPKI::Config

Oliver Welter — 2012-03-30T14:50:11

Hi,

I hit a strange problem that I cannot explain...

I use CTX('config') from inside a Workflow Action to access the new 
config layer (Instance of OpenXPKI::Config). I added a cache property to 
the config class which works fine as long as I am processing the same 
"workflow execution". If I now come back with another workflow action, 
the cache is empty again.
I added a "create" and "lastaccess" timestamp to the class -  the create 
time is always the same, the last access date is empty on every new 
workflow, so it looks like I get a new copy of the object on each new 
connect to the server.

As far as I understand the Context class, its just a singleton holding a 
reference to the instance which should not get copied on access.

Anybody has an idea whats going on?

Oliver

License

Sergei Vyshenski — 2012-03-30T12:51:33

Hi,

The FreeBSD project became anxious about licensing of 3 party products
associated with FreeBSD.
An automated system for license control of each bit of software is being
introduced.
They want me to state formally, which type of license does the openxpki
have.
This info will be checked by the FreeBSD officer through the internet
search.

At the moment pages

https://sourceforge.net/projects/openxpki/reviews/

and

http://www.openxpki.org/foundation/sw_license.txt

both say it is "AL v2.0".

But it seems to me that somewhere on SF or github I have recently seen
some words about dual license, like the license of perl.

Is it ok to use "AL v2.0" as a license for openxpki?
Before you answer, beware: page http://www.openxpki.org/ is in the
read-only mode (together with SVN), so nobody can change the content there.

Regards, Sergei


------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sf

exploding load after search queries

Joachim Astel — 2012-03-29T15:21:42

I have recognized a load issue when doing parallel searches within the
openxpki database.

When I try ~ 20 parallel queries to the OpenXPKI database by dowing
a database search after certificates - each witch an increased number
of entries to be shown per page to 100, the load of the machine is
going up very fast because the machine is waiting for I/O.

Is there some way to finetune this (restrict maximum number of parallel
API connections, optimize I/O with search queries et al.)?

I'm using openxpki.1520 and MySQL as a backend database.

Greetings
-Achim

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure

Develop Head now working / New Debian Packages

Oliver Welter — 2012-03-26T09:20:11

Hello Developers,

the current development head on github was fixed and should now work 
with the new config layer inside (but its not used in any place at the 
moment).

You can grab the current version including the new NICE workflows from 
the package repository (http://packages.openxpki.org/debian/).

If you want to build from source, you need to fetch the Connector Module 
from https://github.com/mrscotty/connector (use "develop" branch!).

For further information read the perldoc of OpenXPKI::Config ad the 
commit log of 39390c92

Oliver

Log4perl

Michael Bell — 2012-03-23T16:05:57

Hi,

I see the following message really often during "make test":

Log4perl: Seems like no initialization happened. Forgot to call init()?

The init CA test fails because "log" is empty in the server context. Any
ideas what I am missing?

Best regards

Michael

P.S. the SQL SELECT code is complete but I don't want to commit it if
the basic tests are failing.

Missing perl module

Michael Bell — 2012-03-23T11:17:19

Hi,

I am missing the perl module Connector::Proxy::Config::Versioned. Do you
know where I can find it?

I only discovered Config::Versioned 0.02 on CPAN.

Best regards

Michael

Project member status/permissions update

Martin Bartosch — 2012-03-07T16:56:51

Hi,

I just wanted to let you know that I have updated the project member permissions. I have removed SVN permissions for all members (see my earlier mail) and generally cleaned up permissions and roles to reflect *current* involvement in the project as known to us.

If anybody is missing anything (permission or role wise), please let us know.

Regards,

Martin


------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Repositories and development: git vs svn

Martin Bartosch — 2012-03-07T16:41:01

Dear colleagues,

We have reached the crossroads regarding svn and git. The histories have diverged due to recent commits on both. For technical reasons, there is no straightforward method of merging them in a clean manner.

Therefore, we suggest that the svn repository be officially switched to read-only and all further work be committed to git on SF.
If there are no objections we will completely disable the SF Subversion repository in a few weeks, eventually leaving only the git repositories.

We understand that there are still open issues regarding the nightly builds for freebsd and the replication of the www mirrors. Future efforts should be put on resolving these issues rather than coming up with a workaround to maintain both svn and git simultaneously.

***

Official repositories:

Main repository on SourceForge:
git://openxpki.git.sourceforge.net/gitroot/openxpki/openxpki

On GitHub:
git://github.com/openxpki/openxpki.git

Secondary (developer) repositories exist on GitHub.

***

Some notes on branch

utf-8 Ansatz

Sergei Vyshenski — 2012-02-24T14:07:13

Hi,

We have finished testing of
(1) a "trivial" utf-8 patch and
(2) a perl 5.14 related patch

Two of them together are tested with perl 5.8, 5.10, 5.12, and 5.14.
That is 5.14 related patch is backward compatible.

(
Please consider: versions 5.8 and 5.10 are already in the "End of life"
status, and we are expecting a new stable 5.16 this year.
http://www.cpan.org/src/README.html
)

The utf-8 patch was invented after major reading about perl deltas:

http://perldoc.perl.org/index-history.html

It works with FF, and Chrome at the client side, and does not depend
on language settings in these browsers.
(Remember that IE8 does not work with or without these patches.)
Hope that Julia will commit both patches later today.

1. "Trivial" utf-8 hack for perls 5.8, 5.10, and 5.12.

Main (but not the only) idea here is to explicitly call an operator
Encode::_utf8_on() before each call of the Serialization module.

This operator has a very useful property. If applied twice it has the
same effect as if applied once: A

Idea for implementation of paused workflows

Oliver Welter — 2012-02-13T23:11:14

Hi All,

as a follow up to the proposal on enhanced Workflow managment from last 
october, here are some ideas on implementing it.

Current situation: The actions in the workflows are marked either as 
autorun or end up in a state that expects a user interaction and are 
routed to frontend tasks (e.g. approval steps). There is no method to 
pause/resume a workflow o recover from a temporary failure.

Proposed Enhancements: All Activities should be able to catch 
foreseeable failures (e.g. availability of tokens or network connected 
resources) and provide a retry logic if the failure is expected to be 
temporary. Instead of making an obscure internal sleep/loop construct, 
the activity should be able to use the workflow engine to say "please 
execute me again in 5 Minutes".

The baseclass OpenXPKI::Server::Workflow::Activity is extended by a 
helper method "pause", that can be called from within the execute method 
of each activity. By default, the parameters are provided by the XML 
configuration as paramet

git and svn

Sergei Vyshenski — 2012-02-10T09:20:02

Hi Martin,

Sorry for hitting this point again, but lastmidnight generator is stuck:

http://www7.openxpki.org/lastmidnight/index.html

For use with lastmidnight generator
I have a working copy of a "git snapshot" which was created with a command:

git clone git://openxpki.git.sourceforge.net/gitroot/openxpki/openxpki

To update this snapshot nightly from a SF repository I used to issue
command:

git svn rebase

which now says:

Unable to determine upstream SVN information from working tree history

What is a methodical way to update my git snapshot?

All the best, Sergei

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Problems with Umlaut / UTF8

Oliver Welter — 2012-02-08T10:56:53

Hi All,

I was trying to issue certificates using german Umlaute and run into 
several problems...

The main issue is, that the Serialization system crashes with 
I18N_OPENXPKI_SERIALIZATION_SIMPLE_READ_HASH_KEY_LENGTH_FORMAT_CORRUPTED 
when you try to transmit umlauts. Remarkably, the message with the 
umlauts is received and properly decoded (as war as I see) , but the 
next following one throws the error.

However, in the database the umlaut chars show up as two characters and 
when you view such information on the frontend again, it gets decoded to 
clumsy boxes. Besides, the database (mysql 5.1) is by default 
initialised in  latin1, but changing it to utf8 does not help either.

I use a debian squeeze with de/en utf8 locale, the webpages are also 
delivered using utf8 as charset, so I dont see anything "outside" to 
create this problems.

Anybody here has positive experience with Umlauts and can help out?

Oliver

Issues with DBI/SQL abstraction layer

Oliver Welter — 2012-02-03T16:49:58

Hi,

I have a (severe?) issue with the SQL abstraction layer. I am using 
mysql as backend and unfortunatly no clue about other rdbms - therefore 
I need some feedback if the issue is mysql only or not.

Problem:
I am searching a certificate by its subject, the subject I am looking 
for contains a backshlash character.

my $issuer_dn = "OU=Trustcenter,O=Snakeoil\, Inc.,C=US";
CTX('dbi_backend')->first(
   TABLE   => 'CERTIFICATE',
   COLUMNS => [ 'IDENTIFIER' ],
   DYNAMIC => {
     'SUBJECT' => $issuer_dn,
     'PKI_REALM' => $pki_realm
});

I get an empty result, which I tracked down to:

According OpenXPKI::Server::DBI::SQL, Line 979 the query uses a "like" 
statement on the condition. In mysql the backslash is an escape 
character when used in like statements:
http://dev.mysql.com/doc/refman/5.0/en/string-comparison-functions.html

Workaround: I ended up now in adding slashes to this special query, 
which does the job, but I expect that can make some headache one day in 
any other query, too. Besides I d