gmane.comp.security.openxpki.devel

Development tracks

Martin Bartosch — 2012-05-24T19:37:00

Hi folks,

I like the momentum Oxi has been getting lately - possibly due to moving from SVN to git, which allows us to pursue experimental and potentially disruptive development.

The develop branch will soon see the integrating of the following development tracks. This will probably result in a huge construction site, but - hey - it's the develop branch for a reason ;-)

- database unification and improvements (Michael's submission for indexing LOBs) -> schema change necessary
- Dieter's submission to the WF engine (paused workflows) -> schema change necessary
- watchdog daemon keeping track of stalled workflows
- inclusion of the NICE API (Oliver)
- inclusion of the new Connector infrastructure we are currently working on (and successive migration of the core configuration to Connector)

Some things to come later this year:
- redesign of the SCEP workflow: many more configuration options, allowing for flexible policies for renewal and support for initial enrollment via SCEP (challenge password)
- inclusion of the current Smartcard personalization development: a complete, user friendly and highly customizable Smartcard personalization and certificate life cycle management application
- multi node and cluster support (active-active setup with shared database)
- a VMWare appliance as a showcase for Oxi (hopefully including the new Smartcard personalization with all required external dependencies, such as mock LDAP and Active Directory servers)

cu

Martin


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

paused workflows,proc state and more: first implementation

Dieter Siebeck — 2012-05-18T13:20:08

Hi,

my name is Dieter Siebeck and I am quite new to the OpenXPKI project. I 
am working with Oli Welter since nearly 3 months now, mainly on his 
idea/draft of "paused workflows" (see his mail to the list from 2012-02-14).

Now I have published my first implementation of this feature. You can 
see and download it here:

https://github.com/dsiebeck/openxpki/tree/feature%2Fworkflow

(this branch has been rebased on the devel branch 22 days ago.)

POD  Documentation  of most of the new methods is still missing, but I 
wanted to publish the current state, to give other interested people the 
opportunity to see and maybe correct the big picture.
Also missing are test files/scripts. I would appreciate some 
recommendation, which place to look for the most up2date state-of-the 
art testing files for this.

Now the idea of the implementation: basic need was to gain relatively 
low-level control to the execution of workflows and activities. To 
achieve this, the best (and maybe only) way seemed, to install an "own" 
object  of class OpenXPKI::Server::Workflow, which replaces (and of 
course extends) the CPAN Workflow class (the file existed already, but 
only for documentation purposes). Now its a matter of fact, that the 
CPAN workflow Module isn't *really* designed for extendability. You have 
no way to tell Workflow::Factory that you want "My::Workflow" instead of 
"Workflow". And the Workflow class itself has not many access- or hook 
points  to control the execution of code.(for example, in method 
Workflow::set(), the name of workflow-class is hardcoded as the one and 
only class, which is allowed to use the private set-Method).

So I had to use some perl tricks to get around this limitations (if we 
dont want to copy the hole Workflow-Module and extend it directly):

1) OpenXPKI::Workflow::Factory returns always an object of class 
OpenXPKI::Server::Workflow. This is achieved via a friendly capturing 
maneuver:
# OpenXPKI::Workflow::Factory
sub fetch_workflow {
     my ( $self, $wf_type, $wf_id ) = < at >_;
     my $wf = $self->SUPER::fetch_workflow($wf_type, $wf_id);
    ...
    my $oxiWf = OpenXPKI::Server::Workflow->new($wf);
    return $oxiWf;

#OpenXPKI::Server::Workflow
use base qw( Workflow )
sub new{
     my ( $class, $BaseWorkflow ) = < at >_;
     my $self = bless {}, $class;
     #take over all properties from (original) base workflow
     while ( my ( $key, $val ) = each %{$BaseWorkflow} ) {
         $self->{$key} = $val;
     }
     ...
    return $self;
}

not the most beautiful code at all, but - thanks to perls open oop 
structure - it works. OpenXPKI::Server::Workflow has all methods and 
properties from the original workflow.

Another limitation: Workflow::Factory::fetch_workflow() uses only 
certain data (two hardcoded keys) of the returned db data hash of the 
persister.
Since there is no entry point/hook to alter this behaviour, a second 
DB-fetch is needed to retrieve other data from the workflow-table (and 
pass it to the workflow-object):

     my $persister = $self->get_persister( $wf_config->{persister} );
     my $wf_info   = $persister->fetch_workflow($wf_id);
     my $oxiWf = OpenXPKI::Server::Workflow->new($wf,$self,$wf_info);

So the full signature of the worklow constructor is:
     my ( $class, $BaseWorkflow, $Factory, $wfData ) = < at >_;
(The reference of the factory is used to call the 
Factory::save_workflow() method several time to store the current states)

2) OpenXPKI::Server::Workflow overrides the execute_action() - method. 
As a wrapper method, it executes some code before and after 
SUPER::execute_action(). Here is the place to determine/set the 
appropriate "proc state" (which is a new db field and can have the values :

  * init
  * wakeup
  * resume
  * running
  * manual
  * finished
  * pause
  * exception
  * retry_exceeded

The first 3 are not expected to see in "normal" circumstances - because 
they are quite volatile by design. "running" will be set, before 
SUPER::execute_action/Activity::run is called. After eexecution of one 
or more Activities, either "manual" (waiting for interaction)  or 
"finished" will be set.
If an exception occurs, the proc state "exception" is set. Also the 
message code (not translation) will be saved in WF context (key 
"wf_exception")

The two states "pause" and "retry_exceeded" concern the new "pause" feature.

3) OpenXPKI::Server::Workflow::Activity has some new methods and utility 
features:

first the pause() method: every derived concrete activity-class can now 
(somewhere in its "run"-method call:

$self->pause('some cause description');

This will immediately end the execution of the "run"-method (this is 
achieved via OpenXPKI::Server::Workflow::Pause, an "pseudo" exception 
class) .

Additionally, $workflow->pause() is called, which does the hole work: 
check the count of allowed retries, sets the "wake up" timestamp, 
notifiy observers, write history entries etc. The workflow state remains 
unchained (this is achieved via overriding the 
Workflow::_get_next_state() method). If "max_retry" is exceeded, an 
special exception will be thrown, which results in the special proc 
state "retry_exceeded". The given "cause" of the pause will also be 
stored in workflow context, key "wf_pause_msg".

For convenience, the Activity objects can now always "speak" to their 
workflow via $self->{CURRENT_WORKFLOW}. Also I have a new utility method 
"_get_wf_action_param($key)", which retrieves the param $key for the 
current action in the xml-conf of current workflow.

4) Wake up: if an paused workflow is executed again (via command line, 
the new "watcher" demon etc ), the proc state changes to "wakeup" and  
the hook method Activity::wake_up() will be called. Afterwards, 
proc-state changes to "running" and the normal execution begins.

5) Resume: if a workflow with proc state "exception" is executed again, 
the proc state "resume" is set and the corresponing hook method is 
called. Afterwards, proc-state changes to "running" and the normal 
execution beginns.

6) The settings of "count try" and "resume_at" can be set/configured (in 
this order)
manual set via $self->set_count_try/max_retry()
in action definition of the xml config of current workflow
in the xml config of the activity

Additionally, the retry-intervall can be set as second parameter of 
$self->pause('cause', $retry_interval), which has the highest priority.

These are the main points of the implementation. Any existing Activity 
class should run without any modifications and problems.

Of course, the workflow table has obtained some new fields: 
workflow_proc_state, workflow_wakeup_at, workflow_count_try and 
workflow_reap_at (the later corresponds to the "reap at"-Feature, which 
is not implemented yet. Here's the full layout of my workflow table:

CREATE TABLE IF NOT EXISTS `workflow` (
   `workflow_id` decimal(49,0) NOT NULL,
   `pki_realm` varchar(255) DEFAULT NULL,
   `workflow_type` varchar(255) DEFAULT NULL,
   `workflow_version_id` decimal(49,0) DEFAULT NULL,
   `workflow_state` varchar(255) DEFAULT NULL,
   `workflow_last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP 
ON UPDATE CURRENT_TIMESTAMP,
   `workflow_proc_state` varchar(20) DEFAULT NULL,
   `workflow_wakeup_at` timestamp NULL DEFAULT NULL,
   `workflow_count_try` int(11) DEFAULT NULL,
   `workflow_reap_at` timestamp NULL DEFAULT NULL,
   PRIMARY KEY (`workflow_id`)
);


Next steps are:

 1. documentation
 2. write some tests (suggestions welcome, see above)
 3. setting the "reap at" timestamp (as proposed by oli)
 4. implementing the "watcher" process, which runs in background and
    periodically checks, if it finds some paused workflows to "wake up".

Any comments, suggestions and corrections are highly appreciated!

Another question: where would be the best place to put an explanation 
like the one in this email here? In one perl class? Or is there an extra 
place for such docs, which give an overview over the interplay of 
several classes?

have a nice weekend,

Dieter

PS:

Since test files are missing, this is the way to test the feature 
(relying on I18N_OPENXPKI_WF_TYPE_TESTING 
(OpenXPKI::Server::Workflow::Activity::WorkflowTest):

openxpkictl restart --debug 
OpenXPKI::Server::Workflow:128,OpenXPKI::Server::Workflow::Activity:20

openxpkicmd   I18N_OPENXPKI_WF_TYPE_TESTING (expected output: "Workflow 
created (ID: xxxxx), State: STEP2")

openxpkicmd  --wfid xxxxx --param action=pause  --wfaction 
I18N_OPENXPKI_WF_ACTION_WORKFLOWTEST I18N_OPENXPKI_WF_TYPE_TESTING 
(expected output: "New Workflow State: STEP2")


execute last line more than 4 times, and the "max retry exceeded" event 
occurs.

Alternatively, you can set "--param action=crash" to provoke an exception.
to continue normally (till SUCCES) you need to give an arbitrary value 
for "action", e.g. "abc" (due to some limitation in the parameter 
parsing here)












------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
OpenXPKI-devel mailing list
OpenXPKI-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f< at >public.gmane.org
https://lists.sourceforge.net/lists/listinfo/openxpki-devel

Removal of obsoleted database entries

Joachim Astel — 2012-05-03T10:50:55

We're doing some database cleanup within our database via cronjob for
our OpenXPKI databases, especially for the audittrail log:

DELETE FROM audittrail WHERE DATEDIFF(NOW(),CONVERT(logtimestamp,DATETIME))>365

Unfortunately, generated CRL (via openxpkicmd) are growing within our
workflow table. We would like to remove old workflows with workflow_type
CRL_ISSUANCE, but backend database entries which are associated with it
would also have to be deleted -- to avoid database garbage.
Are there already scripts around in other projects addressing this issue?

Greetings
    -Achim

PS: I've already deleted workflows with 
"DELETE FROM WORKFLOW WHERE WORKFLOW_TYPE='I18N_OPENXPKI_WF_TYPE_CRL_ISSUANCE'"
I suppose a garbage collector would be good for this problem too. :-)

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

SQL DYNAMIC patch incomplete ?

Oliver Welter — 2012-04-26T17:24:55

Hi Micha,

I am running into some problems using the datapool after merging your 
dynamic-patch.

my $msg = CTX('api')->get_data_pool_entry( {
   KEY => $chipid ,
   NAMESPACE => 'smartcard.smartchipid'
} );

results in an exception (I added ref and column)

'LABEL' => 
'I18N_OPENXPKI_SERVER_DBI_SQL_SELECT_DYNAMIC_PARAM_NO_HASH_REFERENCE',
PARAMS' => {
     '__REF__' => '',
    '__COLUMN__' => 'DATAPOOL_KEY',
    '__TABLE__' => 'DATAPOOL'
}

I added some more debug in SQL.pm lin 970

##! 1: ' DYNAMIC arg ' . Dumper $args->{DYNAMIC}

OpenXPKI::Server::DBI::SQL::select (line 970):  DYNAMIC arg $VAR1 = {
           'DATAPOOL_KEY' => 'chipid1234',
           'NAMESPACE' => 'smartcard.smartchipid',
           'PKI_REALM' => 'I18N_OPENXPKI_DEPLOYMENT_TEST_DUMMY_CA'
         };


I found the  {value => xx } brackets are missing in the statement in 
Server/API/Object.pm, Line 604ff , but even after adding them, the error 
remains the samem, having this debug output:

OpenXPKI::Server::DBI::SQL::select (line 970):  DYNAMIC arg $VAR1 = {
           'DATAPOOL_KEY' => 'chipid1234',
           'NAMESPACE' => {
                            'VALUE' => 'smartcard.smartchipid'
                          },
           'PKI_REALM' => {
                            'VALUE' => 
'I18N_OPENXPKI_DEPLOYMENT_TEST_DUMMY_CA'
                          }
         };


I dont know what I am overlooking but I cant get it to work =(

Oli

Management of pull requests

Michael Bell — 2012-04-12T15:24:33

Hi,

could somebody please review my last pull request? I think it is
generally a good idea to ask for a review and do not merge my own stuff.

So any volunteers?

Best regards, Michael

git: fixing wrong branching

Michael Bell — 2012-04-06T19:36:33

Hi,

I wrote the code to add the missing indexes but I created the new branch
for this feature on another feature branch and not on develop. Is there
a chance to move the start of the new feature branch to the develop branch?

feature/sql_multiple_index_per_table was created from
feature/server_dbi_explicit_sql_like_usage. You can visit the situation
at github. The fork bellmich/openxpki is public.

Sorry for the trouble

Michael

P.S. next time I remove my old fork and create a fresh one.

Git and OpenXPKI … a short primer for people new to git and GitHub

Andreas Leibl — 2012-04-02T08:43:59

Dear all,

Changing the version control tool for a project is never easy and I want to say a big thank-you to all the developers that weren't git users before and still agreed to the move ... it can be daunting at times. Following some remarks and issues on this mailing list I'd like to provide a little write-up that should help everyone get going without spending too much time learning git.

For those inclined to dig deeper into git there are lots of tutorials on the web but I'd also like to recommend the following O'Reilly book and videos to you (they helped me a lot on the way):

http://shop.oreilly.com/product/0636920022862.do
http://shop.oreilly.com/product/0636920017462.do
http://shop.oreilly.com/product/0636920024774.do

Now, without further ado, the promised primer. This is my understanding of how we want to do things, please correct me if I got it wrong. Hope you find it useful.

Kind regards

Andreas

P.S.: I've added the primer to the repo. Can someone please have a look at the pull request?

Git and OpenXPK ... a short primer for people new to git
========================================================

Introduction
------------

Some background info first, then onto the command line. OpenXPKI is based on UNIX so familiarity with the shell (korn, bash, ...) and SSH is expected.

Authoritative repository
------------------------

https://github.com/openxpki/openxpki

Register (for free) at github.com and follow this project. The website will provide the URLs needed to perform a git clone (see below).

Development model
-----------------

Git does not force you to do things in a certain way. That's good because you can do things your own way. It's also bad because if different contributors use different approaches there will be chaos.

There are two official branches:

master The stable, production branch containing officially-released versions
develop The unstable, development branch that may or may not work

Additional branches may be created by integrators (see below for a definition of what an integrator is).

If you want the code for the latest (or any) release: checkout the master branch.

If you want to do the latest development build: checkout the develop branch.

For some philosophical insight into the branching and naming philosophy used here please see git-flow (https://github.com/nvie/gitflow)

Roles
-----

Note: any person can have more than one role, i.e. act as both an integrator and developer.

- Integrator:
An integrator's job is to take code provided by developers and integrate it into other branches. In this role (s)he does not write any code but simply puts things together. Integrators have write access to the authoritative repository.

- Developer:
Writes/changes code and pass them on to the integrators. They do not need access to the authoritative repository.

Before we get started
---------------------

Workflow for followers
----------------------

If you just want to follow the project (e.g. to do stable and development builds) you can pull directly from the authoritative repository.

git clone git://github.com/openxpki/openxpki.git

To follow the official releases, do

git checkout master

and then

git pull

whenever you want to update your local copy.

To follow the (unstable) development (e.g. for your nightly builds), do

git checkout develop

and then

git pull

whenever you want to update your local copy.

Workflow for developers
-----------------------

Note: There is more than one way to do it. This describes the workflow based on GitHub's pull request feature. Using this feature is not a requirement, so if you're familiar with git feel free to do it your own way.

First, you need to create your own repository on GitHub.

- Login to GitHub.
- Go to https://github.com/openxpki/openxpki and click on "Fork" on the upper right hand side. This will create your own, public repository based on the official one. Click on "Fork to <your username>".

Next, you need your own local copy of that repository.

git clone git-9UaJU3cA/F/QT0dZR+AlfA< at >public.gmane.org:<username>/openxpki.git
cd openxpki (the newly created directory containing your copy)

Work is performed in your own, private branches, which are based on the develop branch. An integrator will integrate your work into the develop branch later after you've finished your piece of work.

- Switch to the development branch:

git checkout develop

- Important: create your own branch, e.g. "feature/git-primer"

git branch feature/git-primer

- Important: switch to your new branch:

git checkout feature/git-primer

- Important: push your new branch to your own GitHub repository

git push origin feature/git-primer

Code ... (or write documentation ... or both).

After completing a meaningful piece of work you may want to commit your changes to your private branch.

- To remove files, use git rm file(s).
- To add new files in the next commit, use git add file(s).
- To stage modified files for the next commit, use git add file(s).

Files you have created or modified but not staged using "git add" will not be committed. Now use

git commit

to commit the staged changes. The format of the log message should be:

- One line containing a brief summary of the commit.
- A blank line.
- Further explanations of the commit if necessary.

Whenever you want others to see the work you are doing in your own branch you can push your commits to your own GitHub repo using

git push

Your work is happening on a separate branch, so it is not affected by any commits to the master or develop branch. Hence it is safe to do regular

git pull

requests to stay in sync with the upstream repo.

Let's say you created your own feature branch when the develop branch was as commit b and you created your first commit x:

develop a - b
\
feature/git-primer - x

Now someone else's changes may get added to develop and your changes are no longer based on the latest commit of the develop branch:

develop a - b - c - d
\
feature/git-primer - x

This may or may not create problems when integrating your changes into the develop branch. To facilitate integration, you should make sure that your changes are based on the latest commit of the develop branch before sending a pull request. This is done by rebasing your commits (sometimes also referred to as rewriting history).

If your work does not conflict with the changes in develop (i.e. you haven't changed the same lines) the command

git rebase develop feature/git-primer

will result in a new graph like this:

develop a - b - c - d
\
feature/git-primer - x'

Note: if "git rebase" does result in conflicts you will need to resolve these before you can proceed. Ask your friendly git guru for help if necessary. :-)

Note: if you have "git push"-ed your previous commits to your GitHub repository your local copy and the one on GitHub will have diverged. To fix that, you will need to

git push -f

once to force your local changes into (your own) GitHub repo.

Let's say you complete your work with one final commit

develop a - b - c - d
\
feature/git-primer - x' - y

and pushed everything to GitHub. You can now ask an integrator to pull your changes into the develop branch.

To do that, log on to GtHub and visit your own openxpki repo fork.

- Click on the branches tab.
- Click on your branch that you want merged upstream.
- Click on "Pull Request" in the upper right hand corner.

IMPORTANT: make sure you adjust the settings on the following screen. By default, it will suggest merging into the master branch. Change this to the develop branch and click on "Update commit range".

Click on "Send pull request".

A few final notes:

- You can have (and work on) as many private branches as you like.
- For different features or bug fixes create different branches.
- If in doubt, branch. You can easily merge later. Splitting up commits is much more difficult.

Workflow for integrators
------------------------

The job of an integrator is to act as a gatekeeper and quality assurance for changes to the project. When a developer sends a merge request all integrators will recieve a message. One of them will review the changes and either

- accept the changes and merge the pull request or
- write comments and ask for further information and/or changes or
- decline and close the pull request without merging.

Note: the integrator may be the same person as the developer that sent the pull request.

To act on pull requests simply log on to GitHub and click on the pull request in your "Public Activity" timeline. If you can't find it there look at your notifications (the little radio tower at the top right). The web form is pretty self-explanatory.

Famous last words
-----------------

If you feel something is incorrect or missing in the guide ... please fork, branch, push and send a pull request. Thank you! :-)

Changes to Connector to make live

Oliver Welter — 2012-04-02T08:24:38

Hi,

I want to suggest some things that would make the whole connector 
monster easier to handle (at least at experienced while moving the 
Smartcard API).

1) Return of a hash containing all leaves under one node.
It is very often the case, that you expect a set of config values at a 
node. Retrieving them requires a loop construct each time on the 
programmers side. Besides, at least in most cases, the Connector has 
access to the hash which will make the access a lot faster then first 
exporting the keys and then getting them one by one.

2) Return of an array of values instead keys, when the node is an 
ordered list
Same problem as above.

3) Fixed naming convention and supporting method for the "multiple 
resolver problem".
In Smartcard Personalisation there are mulitple places that require the 
following:
* have a list of alternative "resolvers" (Proxy-Connectors)
* walk them in a defined order
* stop looping as soon as a resolver returns a result
* store the id of the used resolver to reuse it in the next request

4) Invention of a generic "primary key attribute" to support get/set 
operations:
When I fetch a value from an arbitrary connector and need to update the 
value later on, it would add extra robustness when we use the primary 
key of the backend (there might also be cases where it is impossible 
without a pkey). As the kind and name of the primary key differs between 
the backend implementations, it should be a generic mechanism (string 
representation in a known attribute name, suggestion: pkey)

Basically, there are two different ways to make this working:
1) Invent new method names to represent the different return types (or 
pass as parameter) and try to handle an "illegal" request with an exception.
2) Rely on the assumption that the programmer knows what he is asking 
for and choose the return value based on what we find on the data side.

I would go for 2, and as an addition, I would vote for a new method 
"meta" that returns meta-information on a node, so if a connector path 
should really be used with different "personalities", the code can check 
the node and behave accordingly.

One more issue, unrelated to the above: We need to find a way to handle 
exceptions in the Connector. Suggestion (very rough): Let the connector  
"die" on problems, add a wrapper around "get/set" in OXI:Config and 
transform it into an OXI:Exception.

Oli

Contexts (CTX) and OpenXPKI::Config

Oliver Welter — 2012-03-30T14:50:11

Hi,

I hit a strange problem that I cannot explain...

I use CTX('config') from inside a Workflow Action to access the new 
config layer (Instance of OpenXPKI::Config). I added a cache property to 
the config class which works fine as long as I am processing the same 
"workflow execution". If I now come back with another workflow action, 
the cache is empty again.
I added a "create" and "lastaccess" timestamp to the class -  the create 
time is always the same, the last access date is empty on every new 
workflow, so it looks like I get a new copy of the object on each new 
connect to the server.

As far as I understand the Context class, its just a singleton holding a 
reference to the instance which should not get copied on access.

Anybody has an idea whats going on?

Oliver

License

Sergei Vyshenski — 2012-03-30T12:51:33

Hi,

The FreeBSD project became anxious about licensing of 3 party products
associated with FreeBSD.
An automated system for license control of each bit of software is being
introduced.
They want me to state formally, which type of license does the openxpki
have.
This info will be checked by the FreeBSD officer through the internet
search.

At the moment pages

https://sourceforge.net/projects/openxpki/reviews/

and

http://www.openxpki.org/foundation/sw_license.txt

both say it is "AL v2.0".

But it seems to me that somewhere on SF or github I have recently seen
some words about dual license, like the license of perl.

Is it ok to use "AL v2.0" as a license for openxpki?
Before you answer, beware: page http://www.openxpki.org/ is in the
read-only mode (together with SVN), so nobody can change the content there.

Regards, Sergei


------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure

exploding load after search queries

Joachim Astel — 2012-03-29T15:21:42

I have recognized a load issue when doing parallel searches within the
openxpki database.

When I try ~ 20 parallel queries to the OpenXPKI database by dowing
a database search after certificates - each witch an increased number
of entries to be shown per page to 100, the load of the machine is
going up very fast because the machine is waiting for I/O.

Is there some way to finetune this (restrict maximum number of parallel
API connections, optimize I/O with search queries et al.)?

I'm using openxpki.1520 and MySQL as a backend database.

Greetings
-Achim

------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here 
http://p.sf.net/sfu/sfd2d-msazure

Develop Head now working / New Debian Packages

Oliver Welter — 2012-03-26T09:20:11

Hello Developers,

the current development head on github was fixed and should now work 
with the new config layer inside (but its not used in any place at the 
moment).

You can grab the current version including the new NICE workflows from 
the package repository (http://packages.openxpki.org/debian/).

If you want to build from source, you need to fetch the Connector Module 
from https://github.com/mrscotty/connector (use "develop" branch!).

For further information read the perldoc of OpenXPKI::Config ad the 
commit log of 39390c92

Oliver

Log4perl

Michael Bell — 2012-03-23T16:05:57

Hi,

I see the following message really often during "make test":

Log4perl: Seems like no initialization happened. Forgot to call init()?

The init CA test fails because "log" is empty in the server context. Any
ideas what I am missing?

Best regards

Michael

P.S. the SQL SELECT code is complete but I don't want to commit it if
the basic tests are failing.

Missing perl module

Michael Bell — 2012-03-23T11:17:19

Hi,

I am missing the perl module Connector::Proxy::Config::Versioned. Do you
know where I can find it?

I only discovered Config::Versioned 0.02 on CPAN.

Best regards

Michael

Project member status/permissions update

Martin Bartosch — 2012-03-07T16:56:51

Hi,

I just wanted to let you know that I have updated the project member permissions. I have removed SVN permissions for all members (see my earlier mail) and generally cleaned up permissions and roles to reflect *current* involvement in the project as known to us.

If anybody is missing anything (permission or role wise), please let us know.

Regards,

Martin


------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Repositories and development: git vs svn

Martin Bartosch — 2012-03-07T16:41:01

Dear colleagues,

We have reached the crossroads regarding svn and git. The histories have diverged due to recent commits on both. For technical reasons, there is no straightforward method of merging them in a clean manner.

Therefore, we suggest that the svn repository be officially switched to read-only and all further work be committed to git on SF.
If there are no objections we will completely disable the SF Subversion repository in a few weeks, eventually leaving only the git repositories.

We understand that there are still open issues regarding the nightly builds for freebsd and the replication of the www mirrors. Future efforts should be put on resolving these issues rather than coming up with a workaround to maintain both svn and git simultaneously.

***

Official repositories:

Main repository on SourceForge:
git://openxpki.git.sourceforge.net/gitroot/openxpki/openxpki

On GitHub:
git://github.com/openxpki/openxpki.git

Secondary (developer) repositories exist on GitHub.

***

Some notes on branch usage and development policy:

In the past, there was no formal release process. Commits have been made directly to the head of SVN, partly because of the way SVN worked, perhaps partly because of the way the developers worked. 

Here in Frankfurt, we have a few customer modifications that we would like to push upstream to the project. Since our commits involve more than just a couple of lines of code, we would like to leverage Git to assist in the release process. Specifically, we would like to suggest the following branch layout for using the Git repository:

We would like to utilize the following branch layout in the official repos:

 master    The stable, production branch containing officially-released versions
 develop   The unstable, development branch that may or may not work

In addition, active work in the Frankfurt team(s) will be done in the following branches that may or may not be published on SF and/or GitHub. (They are more likely to show up in the individual developer repositories.)

 feature/<name>     Development commits regarding a specific commit -- branch gets merged to develop
 release/<version>  Last-minute fixes for a pending official release -- branch gets merged to master
 hotfix/<name>      Bug fix needed outside of regular development workflow -- merged to master

For some nice insight into the branching and naming philosophy used here please see git-flow (https://github.com/nvie/gitflow)


With best regards,

Scott and Martin


------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

utf-8 Ansatz

Sergei Vyshenski — 2012-02-24T14:07:13

Hi,

We have finished testing of
(1) a "trivial" utf-8 patch and
(2) a perl 5.14 related patch

Two of them together are tested with perl 5.8, 5.10, 5.12, and 5.14.
That is 5.14 related patch is backward compatible.

(
Please consider: versions 5.8 and 5.10 are already in the "End of life"
status, and we are expecting a new stable 5.16 this year.
http://www.cpan.org/src/README.html
)

The utf-8 patch was invented after major reading about perl deltas:

http://perldoc.perl.org/index-history.html

It works with FF, and Chrome at the client side, and does not depend
on language settings in these browsers.
(Remember that IE8 does not work with or without these patches.)
Hope that Julia will commit both patches later today.

1. "Trivial" utf-8 hack for perls 5.8, 5.10, and 5.12.

Main (but not the only) idea here is to explicitly call an operator
Encode::_utf8_on() before each call of the Serialization module.

This operator has a very useful property. If applied twice it has the
same effect as if applied once: A * A = A.
It seems, it is this property which makes the hack portable trough all
versions of perl.

2. Perl 5.14 needs a special attention.

The same "trivial" utf-8 hack is valid here too.

But here we need another patch (unrelated to utf-8) specific to perl
5.14 new syntax requirements.
The problem is that oxi itself (with OR without utf-8 hack) does not
work with perl 5.14.

2a. The reason is that 5.14 is MORE strict about perl syntax compared to
any older versions of perl.

Say expression

foreach my $lang qw( en ru de ) { ... }

is an ERROR in perl 5.14 and should be replaced by more parenthesized
expression

foreach my $lang (qw( en ru de )) { ... }

which works in all versions of perl (this latter style is backward
compatible).

It happens that the code of oxi has multiple instances of both types of
programming style.

At the moment we see that (with both utf-8 patch and 5.14 patch applied)
a CSR with foreign letters is presented correctly. But we can not
guarantee that we have fixed all violations of new syntax requirements
in the code of oxi by means of this 5.14 patch.

And each committer should swear to comply with new syntax requirements
from now on?

2b. Another reason is that some functionality of perl 5.14 has moved
into external modules. One example which has hit us is a new external
dependency for oxi when perl 5.14 is used:

Switch.pm

This is fixed easily. If we install this module from CPAN, then oxi
works with all versions of perl. Now again we can not guarantee that all
new external dependencies were noticed.


Let us consider these patches as just a "demonstration of the method"
patches. They fix a problem reported by Oliver, and some other utf-8
related problems which we were aware of. Most probably both patches miss
many other places, that should be patched too. Here we need a collective
efforts and feedback.

But fixing the code is the easiest part of the story. It triggers a much
more complex problem.
Maybe we should solve this problem first, and only then go towards
ultimate utf-8 and 5.14 related fix.
It seems that this "trivial" utf-8 hack and 5.14 related patch should be
augmented by

- a new social agreement about future programming style, or by
- inventing another "clever" hack (instead of a "trivial" one) which
would make it very hard to ruin utf-8 support by future careless commits.

(You remember how in the past we several times tried to fix big commits
which again and again ruined crypto abstraction in oxi, and then finally
had to surrender, because other committers had no concern about this
crypto abstraction. This time we are not willing even to begin to act as
"code protectors".)

---

Now about the alternative for the "trivial" patch, a "clever" utf-8 patch.
Maybe somebody with system wide understanding about oxi suite could
invent a "clever" solution. That is find out, how this method of fixing
utf-8 could be moved into the Serialization module (?). So that it will
be called "automatically" when needed, and applied universally to all
places where we need a special utf-8 attention. Or it could be hidden in
some other place where it will be protected against future careless commits.

Otherwise each committer should swear to call Encode::_utf8_on() each
time when she thinks about utf-8 and each time he forgets to think about it?

All the best, Sergei

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Idea for implementation of paused workflows

Oliver Welter — 2012-02-13T23:11:14

Hi All,

as a follow up to the proposal on enhanced Workflow managment from last 
october, here are some ideas on implementing it.

Current situation: The actions in the workflows are marked either as 
autorun or end up in a state that expects a user interaction and are 
routed to frontend tasks (e.g. approval steps). There is no method to 
pause/resume a workflow o recover from a temporary failure.

Proposed Enhancements: All Activities should be able to catch 
foreseeable failures (e.g. availability of tokens or network connected 
resources) and provide a retry logic if the failure is expected to be 
temporary. Instead of making an obscure internal sleep/loop construct, 
the activity should be able to use the workflow engine to say "please 
execute me again in 5 Minutes".

The baseclass OpenXPKI::Server::Workflow::Activity is extended by a 
helper method "pause", that can be called from within the execute method 
of each activity. By default, the parameters are provided by the XML 
configuration as parameters in the activity definition:
* retry_count - max. number of retries before we fail finally
* retry_interval time to wait between each retry (OpenXPKI DateTime String)
The method should accept a single parameter which overrides the pause 
interval. There should be setters to manipulate both values from within 
the activity.

To wakeup the paused workflows, we need a new process inside the 
OpenXPKI Server daemon, I will call it "the watcher". The watcher is 
forked away once at the beginning of the server process as a standalone 
thread, which looks for resumable workflows once a minute.
Paused workflows are determined using a new field "resume_at" in the 
"workflow" table. The field is a timestamp which is non zero for paused 
workflows. In addition, we introduce another field "processing status", 
which gives a verbose indication about what the workflow engine is doing 
with this workflow:
* running - real perl process/activity is executed
* paused - waiting to be resumed
* manual - regular stop in a non-autorun state
* finished - reached the last state (usually SUCCESS)

The watcher can also take care of aborted/crashed workflows. A first, 
simple approach would be: Add another timestamp field to the workflow 
database "reap_at" and set it to "now + 5 Minutes" on every start of an 
activity. If an activity knows that it will regularly take longer than 5 
Minutes, this value must be increased or adjusted during runtime. If the 
time exceeds the time in "reap_at", the process can be considered dead 
and the watcher tries to run the last action again.

Improvements to support the above ideas: I consider it a good idea to 
introduce new methods "wakeup" and "restore" in the activity base class, 
which are called before the "execute" method, if a workflow is resumed 
from pause or a crash. Obviously there is also a need for setting the 
"reap_at" timestamps.

In an earlier discussion, Andreas Leibl suggested to set two timeouts 
(soft/hard) for different aspects of resuming a non-responsing workflow 
and provide a multi-node failover strategie. This will fit into this 
model by adding some more information to the workflow table.

I had a quick look at the Workwflow code base and think this ideas can 
be coded in a suitable manner, if anybody as objections or a better 
idea, comments are welcome.

regards

Oliver

git and svn

Sergei Vyshenski — 2012-02-10T09:20:02

Hi Martin,

Sorry for hitting this point again, but lastmidnight generator is stuck:

http://www7.openxpki.org/lastmidnight/index.html

For use with lastmidnight generator
I have a working copy of a "git snapshot" which was created with a command:

git clone git://openxpki.git.sourceforge.net/gitroot/openxpki/openxpki

To update this snapshot nightly from a SF repository I used to issue
command:

git svn rebase

which now says:

Unable to determine upstream SVN information from working tree history

What is a methodical way to update my git snapshot?

All the best, Sergei

------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing 
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/

Problems with Umlaut / UTF8

Oliver Welter — 2012-02-08T10:56:53

Hi All,

I was trying to issue certificates using german Umlaute and run into 
several problems...

The main issue is, that the Serialization system crashes with 
I18N_OPENXPKI_SERIALIZATION_SIMPLE_READ_HASH_KEY_LENGTH_FORMAT_CORRUPTED 
when you try to transmit umlauts. Remarkably, the message with the 
umlauts is received and properly decoded (as war as I see) , but the 
next following one throws the error.

However, in the database the umlaut chars show up as two characters and 
when you view such information on the frontend again, it gets decoded to 
clumsy boxes. Besides, the database (mysql 5.1) is by default 
initialised in  latin1, but changing it to utf8 does not help either.

I use a debian squeeze with de/en utf8 locale, the webpages are also 
delivered using utf8 as charset, so I dont see anything "outside" to 
create this problems.

Anybody here has positive experience with Umlauts and can help out?

Oliver

Issues with DBI/SQL abstraction layer

Oliver Welter — 2012-02-03T16:49:58

Hi,

I have a (severe?) issue with the SQL abstraction layer. I am using 
mysql as backend and unfortunatly no clue about other rdbms - therefore 
I need some feedback if the issue is mysql only or not.

Problem:
I am searching a certificate by its subject, the subject I am looking 
for contains a backshlash character.

my $issuer_dn = "OU=Trustcenter,O=Snakeoil\, Inc.,C=US";
CTX('dbi_backend')->first(
   TABLE   => 'CERTIFICATE',
   COLUMNS => [ 'IDENTIFIER' ],
   DYNAMIC => {
     'SUBJECT' => $issuer_dn,
     'PKI_REALM' => $pki_realm
});

I get an empty result, which I tracked down to:

According OpenXPKI::Server::DBI::SQL, Line 979 the query uses a "like" 
statement on the condition. In mysql the backslash is an escape 
character when used in like statements:
http://dev.mysql.com/doc/refman/5.0/en/string-comparison-functions.html

Workaround: I ended up now in adding slashes to this special query, 
which does the job, but I expect that can make some headache one day in 
any other query, too. Besides I do not now what happens on other RDBMS.

I guess that switching to a simple equotation will break the code as it 
prevents wildcard queries so we need some kind of escaping mechanism.

Any ideas on that ?

Oliver