sandesha-dev@ws.apache.org
[Top] [All Lists]

Re: [Mercury] Mercury restarting (was persistence policy)

Subject: Re: [Mercury] Mercury restarting was persistence policy
From: "Amila Suriarachchi"
Date: Thu, 19 Jun 2008 20:02:27 +0530


On Thu, Jun 19, 2008 at 7:23 PM, Jaliya Ekanayake <jnekanayake@xxxxxxxxx> wrote:
Hi Amila,
 
>you mean if it is a resumed sequence RM has to drop messages upto the number it has gone? 
>I think the better option is to give some API to Client app to get the number of messages has send. if crashed sequence has send 99 messages then this is a huge duplicate.
 
Yes, this is something that we can do to improve.
But, what is the requirement for the crashed client application to send the sequence from the beginning always?
Again it is upto the client application to decide its crash-recovery behavior. (I am sorry if I mentioned this usecase earlier. I did not mean to use it as the only option)
If it can recover and start sending the next messages we don't need to worry about it.
If it decide to use the very basic approach of starting the sequence again, then still the it will work.
 
I like the simple approach. Otherwise, we are injecting lot of RM related logic to the application.

I am not saying what you told was impossible. yes certainly we can do this.
so the best option would be to allow two ways and let users to choose when they use.
1. send a resume message and send all messages
2. examine the send messages and send messages from that point.

thanks,
Amila.
 

Thanks,
Jaliya
 
 
 
 
Sent: Thursday, June 19, 2008 9:20 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)



On Wed, Jun 18, 2008 at 7:00 PM, Jaliya Ekanayake <jnekanayake@xxxxxxxxx> wrote:
Hi Amila,
 
I think the discussion is mixing up the application recovery and the persistance requirements of RM.
We need to separate the two cases.
 
In your example, how the application client determines whether it has crashed or not is a separate problem to RM.
Agreed.
 
From RM point of view, this is what should happen.
 
The application client should be given a simple API to let RM know that this is the same sequance that it needs to continue before crashing.
Something like "resumeSequance(seqID)"

yes. In Mercury it uses to address and internal key to uniquely identify a RM sequence. I think sandesha2 also do the same.
 
We need this information from the application client simply because RM does not have a way to differentiate the different invocations of the application client.
 
In RM database, we only need to record the messages and the propety to know whether the sequance is completed or not.
yes. What about the acknowledged messages. I think we need to keep all the details to restart the sequence. In Mercury terms to populate the last state.
 
When the application client recovers, it can simply call "resumeSequance(seqID) and probably start sending the messages again from the begining.
However, in the RM database our records shows that we have successfully sent up to 25h message. So RM will continue only after 26th message.

you mean if it is a resumed sequence RM has to drop messages upto the number it has gone? 
I think the better option is to give some API to Client app to get the number of messages has send. if crashed sequence has send 99 messages then this is a huge duplicate.

thanks,
Amila.
 
Thanks,
Jaliya
 
 
----- Original Message -----
Sent: Wednesday, June 18, 2008 1:41 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)

Here is what I thought when writing the persistence.

Lets take this scenario,

There is an application client which process transaction logs and send messages to server. From another application it gets log files to a folder called *input* folder and then this application  moves this file to  *processing* folder and start reading log entries and send messages. After finish it move the file to *finish* folder.
Then say this Application client crashes when it processing the file. When at the start up the application can understand it has crashed by looking at the files in the *processing* folder.
Then it has to determine how many messages it has send. This can be done by checking the RM persistence database. Then it can restart the sequence and start sending messages from the stop point.
The main point here is that this application should have a recovery process and it should be able to understand something has gone wrong. (in this case looking at the files in the *processing* folder).

This logic can only be written in the Application level recovery process. That also can be written in a demon but I don't see any advantage.

thanks,
Amila.



On Wed, Jun 18, 2008 at 10:26 AM, Amila Suriarachchi <amilasuriarachchi@xxxxxxxxx> wrote:


On Tue, Jun 17, 2008 at 9:53 PM, Jaliya Ekanayake <jnekanayake@xxxxxxxxx> wrote:
Don't know if somebody has done this earlier, but as Chamikara mentioned, we can have a daemon process in the client side which acts as the:
 
1. Endpoint for the dual channel communications originating from that machine.
2. manager for the reliable communications.
 
The communications between the RMHandler (not the client application) and the daemon process should happen via a persistence storage mechanism.
isn't this mean the user has to write this logic in the deamon. So I think writing this straight away in the Application client would make it easier. 
The daemon will keep trying to send the messages, and if something fail, once in a while bug the user with a pop up window :)

Then anyway application client has to handle this isn't it? This means some recovery part should goes to application client.

thanks,
Amila.

Thanks,
Jaliya
 
----- Original Message -----
Sent: Tuesday, June 17, 2008 11:05 AM
Subject: Re: [Mercury] Mercury restarting (was persistence policy)

Amila, Paul,

I don't think I said that application client should have any logic to manage recovery. I simply said that one of following two has to be done to manage recovery in the client side.

1. There should be a permanent RM agent running in the client side, which manages recovery.
2. RM client should check for crashed sequences whenever it is invoked in the client side, if it find any they should be resumed.

In both cases work should be done by the RM client transparently, not by the application client.

Chamikara


On Tue, Jun 17, 2008 at 10:42 AM, Amila Suriarachchi <amilasuriarachchi@xxxxxxxxx> wrote:


On Tue, Jun 17, 2008 at 7:09 PM, Paul Fremantle <pzfreo@xxxxxxxxx> wrote:
Amila

The Mercury design is wrong.

The application client should not be responsible for understanding if
the system needs to restart. There isn't a single successful messaging
or transaction system in the world that requires the application to
incorporate logic to handle recovery. Sandesha2 manages automatic
recovery and so should Mercury.

Here the question is that when a client dies. (i.e. system crashed) how it automatically restart?
Actually I had a chat with Chamikara regarding this and he told that sandesha2 also can not
automatically restart.
Chamikara could you please comment on this?

thanks,
Amila.


Paul

On Tue, Jun 17, 2008 at 2:36 PM, Amila Suriarachchi
<amilasuriarachchi@xxxxxxxxx> wrote:
> +1 to add a policy to declare whether a service supports Persistence or not.
>
> IMO if RM provides persistence then it should happen in both client and
> server side.
> But at the client side Application layer has to do some part as well.
> In Mercury if a client dies when sending a sequence it can be restarted by
> sending a message by setting a property called MercuryResumeSequence to
> true.
> But in this case Application client have to check how many messages it had
> send to RM either by keeping a track of messages or by examining the Mercury
> data base store.
>
> thanks,
> Amila.
>
> On Tue, Jun 17, 2008 at 6:50 AM, Jaliya Ekanayake <jnekanayake@xxxxxxxxx>
> wrote:
>>
>> Hi Paul,
>>
>> I agree with the two scenarios you mentioned.
>> So, if the majority of the use cases for RM are of the above type with
>> long running communications, then we should stick to the persisted
>> reliability in every usecase.
>> Otherwise, I think we'd better keep the client side free from the
>> mandatory database integration.
>>
>> Thanks,
>> Jaliya
>>
>>
>> ----- Original Message ----- From: "Paul Fremantle" <pzfreo@xxxxxxxxx>
>> To: "Jaliya Ekanayake" <jaliya@xxxxxxxxxx>
>> Cc: <sandesha-dev@xxxxxxxxxxxxx>
>> Sent: Monday, June 16, 2008 1:05 PM
>> Subject: Re: Policy or other extensions to indicate persistent messaging
>>
>>
>>> Jaliya
>>>
>>> I agree that it is the responsibility of the client. But I'm worried
>>> about two scenarios.
>>>
>>> 1) The client crashes and restarts - the server has missing messages
>>> and without them it cannot deliver the later messages that it has.
>>> 2) The client crashes and restarts - the server has responses stored,
>>> ready to send, but the client has lost the sequence state and cannot
>>> accept the stored messages.
>>>
>>> I still think that there is a requirement to ENSURE that a particular
>>> communication is completely reliable.
>>>
>>> Paul
>>>
>>> On Mon, Jun 16, 2008 at 5:54 PM, Jaliya Ekanayake <jnekanayake@xxxxxxxxx>
>>> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> Please see my comments below.
>>>>
>>>> Thanks,
>>>> Jaliya
>>>> ----- Original Message ----- From: "Paul Fremantle" <pzfreo@xxxxxxxxx>
>>>> To: "Jaliya Ekanayake" <jaliya@xxxxxxxxxx>
>>>> Sent: Monday, June 16, 2008 11:40 AM
>>>> Subject: Re: Policy or other extensions to indicate persistent messaging
>>>>
>>>>
>>>>> Jaliya
>>>>>
>>>>>> My idea is that the persistence is a feature that the admin can turn
>>>>>> on
>>>>>> or
>>>>>> off per service.
>>>>>
>>>>> So I agree that it is a good feature to turn it on and off per
>>>>> service, and that we support today. So for this model, what I am
>>>>> proposing adding is the ability for the server to advertize that it
>>>>> supports that.
>>>>>
>>>>>> If it is to be per sequence, I think it will over
>>>>>> complicate the handshakes.
>>>>>
>>>>> It will *complicate* the handshakes :) I agree. I think to say it
>>>>> over-complicates the handshake is a subjective idea.
>>>>>
>>>> I agree.
>>>>
>>>>>> Also, Sandesha should not impose restrictions to the clients based on
>>>>>> their
>>>>>> persistence settings.
>>>>>
>>>>> I don't agree. WSRM imposes plenty of restrictions on the client and
>>>>> server. It is perfectly possible for a server to refuse to communicate
>>>>> with a client that does not support RM. With WSRM 1.1 It is possible
>>>>> for clients to demand that the server creates an association between
>>>>> the sequence and a security session.
>>>>>
>>>>> If the server supports persistence but the client doesn't, then there
>>>>> is no overall guarantee of reliability. So I believe that it ought to
>>>>> be *possible* for a server to send CreateSequenceRefused if it
>>>>> *requires* persistence and the client cannot provide it. Of course
>>>>> that should be configured by the server. Similarly the client should
>>>>> be able to demand (like a mustUnderstand) that the server provides a
>>>>> persistent capability.
>>>>>
>>>>> Why do I want this? Almost every customer I talk to says that WSRM
>>>>> without persistence is basically pointless.
>>>>
>>>> I totally agree with this. Without persistance, I can almost trust TCP
>>>> for
>>>> the reliability. (not for the multi-transport multi-hop cases)
>>>> So we need persistance for *real* use cases.
>>>>
>>>> The blogging from people
>>>>>
>>>>> who believe that WSRM should not support persistence has - in my view
>>>>> - been very harmful to the adoption of the spec. Now the normal
>>>>> argument is that "you can support persistence if you need it". But
>>>>> frankly, that is a weak argument, because in a real distributed SOA
>>>>> you cannot (at the moment) know if there is persistence involved,
>>>>> except maybe by phoning up the sysadmin at the other end. So if you
>>>>> require proper persistent reliability then you need a way of agreeing
>>>>> between both parties that it exists. Now WSRM has the perfect model
>>>>> for this negotiation - the CreateSequence/CSResponse. This ability to
>>>>> negotiate details is perfect for this, and I don't see any problem
>>>>> extending it to support this.
>>>>>
>>>>
>>>> I agree that the Client should be able to request persistence from a
>>>> Service, but I still don' t understand why we need the server to reject
>>>> a
>>>> client if it does not support persistence.
>>>> Here is a scenario.
>>>>
>>>> Say we have two companies A and B that perform some communications over
>>>> snail mail.
>>>> All the mails received by the company B first go through its mail
>>>> processing
>>>> system (MPS) which keeps information on all the mails received, and also
>>>> keeps a copy of them.
>>>> All the mails sent by the company B also go through the same MPS.
>>>>
>>>> Company A is not that sophisticated in its operations. They don't simply
>>>> have a system as above.
>>>>
>>>> Now consider the communications that could happen.
>>>>
>>>> 1.  A sends a mail to B
>>>> If this reach B, things are fine.
>>>> If this does not reach B then it is a problem of A. A should keep a copy
>>>> so
>>>> that it can send it again.
>>>>
>>>>
>>>> 2. B sends a mail to A as a response to A's request
>>>> If this reach A and processed by A things are fine
>>>> If this reach A but not processed by A, then B can send it again. (up to
>>>> some number of times)
>>>> If this does not reach A, B can still send it again.
>>>>
>>>> 3. B needs to send a mail to A requesting something. (Now A is the
>>>> service
>>>> provider)
>>>> In this case B can request that A provides the necessary reliability to
>>>> its
>>>> requests.
>>>> If A does not support it, B should not proceed.
>>>>
>>>> Therefore, IMO the client does not need to have a real persistence to
>>>> use a
>>>> service offered with persisted reliability, but it can request this
>>>> feature
>>>> from the service.
>>>>
>>>> Let me know your thoughts. probably I have missed some use case.
>>>>
>>>> Thanks,
>>>> Jaliya
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Say, the server supports persistance. In that case, any request sent by
>>>> the
>>>> client is guranteed to be served.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Paul
>>>>>
>>>>> , but we can advertise that we have persistence for this
>>>>>>
>>>>>> service.
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Jaliya
>>>>>>
>>>>>> ----- Original Message ----- From: "Paul Fremantle" <pzfreo@xxxxxxxxx>
>>>>>> To: "Danushka Menikkumbura" <danushka@xxxxxxxx>
>>>>>> Cc: <sandesha-dev@xxxxxxxxxxxxx>
>>>>>> Sent: Monday, June 16, 2008 8:12 AM
>>>>>> Subject: Re: Policy or other extensions to indicate persistent
>>>>>> messaging
>>>>>>
>>>>>>
>>>>>>> Danushka
>>>>>>>
>>>>>>> I'm not clear I understand your point.
>>>>>>>
>>>>>>> Firstly, I was just using AMQP as an example - I didn't mean that we
>>>>>>> wanted to be exactly the same as AMQP or do anything with AMQP.
>>>>>>>
>>>>>>> So firstly, in general, I don't believe that Sandesha2 can modify the
>>>>>>> persistence on a sequence by sequence basis - either it is on for a
>>>>>>> service or off. However, logically the sequence *is* the level at
>>>>>>> which persistence can be defined. So these are the options I see:
>>>>>>>
>>>>>>> * The server has persistence set permanently on for a service. It
>>>>>>> does
>>>>>>> not demand persistence from the client. It publishes a policy saying
>>>>>>> persistence is optional for the client. It accepts any sequence
>>>>>>> creation, and persistently stores messages. If the client "asks for
>>>>>>> persistence" during the create sequence, then it says "yes I'm
>>>>>>> persistent in reply" (by some yet to be determined mechanisms). This
>>>>>>> is how we operate today, except with the addition of the policy and
>>>>>>> the optional information passing during create sequence.
>>>>>>>
>>>>>>> * The server has persistence set permanently on for a service.  It
>>>>>>> demands persistence from the client. Therefore it publishes a policy
>>>>>>> saying that client's must be persistent. During the CreateSequence
>>>>>>> exchange, it can refuse any client that doesn't agree (by some
>>>>>>> yet-to-be-defined mechanism) to be persistent. This would be a new
>>>>>>> capability.
>>>>>>>
>>>>>>> * The server has some clever ability to turn persistence on or off
>>>>>>> per-sequence based on the create sequence. In this model, the server
>>>>>>> picks up the preference from the client. So the same service might
>>>>>>> have some persistent and some non-persistent sequences. Maybe this is
>>>>>>> overkill and beyond the basic requirements. I'm not clear. However,
>>>>>>> this seems to be a model that other systems allow. Of course, a
>>>>>>> server
>>>>>>> could respond that it doesn't support this capability.
>>>>>>>
>>>>>>> Ideally, all of this would be designed to allow backwards
>>>>>>> compatibility. So even in the cases where the server refuses a
>>>>>>> sequence because it requires persistence and the client doesn't
>>>>>>> support this extension, the failure is explained in the error and the
>>>>>>> administrator can see why.
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>> On Mon, Jun 16, 2008 at 12:20 PM, Danushka Menikkumbura
>>>>>>> <danushka@xxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>>>> 1) A policy element to indicate whether this endpoint supports
>>>>>>>>>>> and/or
>>>>>>>>>>> requires persistence
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> (b) Even if we go for transport level persistence, the endpoint
>>>>>>>>>> does
>>>>>>>>>> not
>>>>>>>>>> have a say in it, because in AMQP we can have either queue level
>>>>>>>>>> persistence
>>>>>>>>>> (i.e. transport receiver level abstraction) or message level
>>>>>>>>>> persistence
>>>>>>>>>> (i.e. message sender level abstraction).
>>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Paul,
>>>>>>>>  But still this is not doable with AMQP.
>>>>>>>>
>>>>>>>> Danushka
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Paul Fremantle
>>>>>>> Co-Founder and CTO, WSO2
>>>>>>> Apache Synapse PMC Chair
>>>>>>> OASIS WS-RX TC Co-chair
>>>>>>>
>>>>>>> blog: http://pzf.fremantle.org
>>>>>>> paul@xxxxxxxx
>>>>>>>
>>>>>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: sandesha-dev-unsubscribe@xxxxxxxxxxxxx
>>>>>>> For additional commands, e-mail: sandesha-dev-help@xxxxxxxxxxxxx
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Paul Fremantle
>>>>> Co-Founder and CTO, WSO2
>>>>> Apache Synapse PMC Chair
>>>>> OASIS WS-RX TC Co-chair
>>>>>
>>>>> blog: http://pzf.fremantle.org
>>>>> paul@xxxxxxxx
>>>>>
>>>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Paul Fremantle
>>> Co-Founder and CTO, WSO2
>>> Apache Synapse PMC Chair
>>> OASIS WS-RX TC Co-chair
>>>
>>> blog: http://pzf.fremantle.org
>>> paul@xxxxxxxx
>>>
>>> "Oxygenating the Web Service Platform", www.wso2.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: sandesha-dev-unsubscribe@xxxxxxxxxxxxx
>>> For additional commands, e-mail: sandesha-dev-help@xxxxxxxxxxxxx
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: sandesha-dev-unsubscribe@xxxxxxxxxxxxx
>> For additional commands, e-mail: sandesha-dev-help@xxxxxxxxxxxxx
>>
>
>
>
> --
> Amila Suriarachchi,
> WSO2 Inc.



--
Paul Fremantle
Co-Founder and CTO, WSO2
Apache Synapse PMC Chair
OASIS WS-RX TC Co-chair

blog: http://pzf.fremantle.org
paul@xxxxxxxx

"Oxygenating the Web Service Platform", www.wso2.com



--
Amila Suriarachchi,
WSO2 Inc.




--
Amila Suriarachchi,
WSO2 Inc.



--
Amila Suriarachchi,
WSO2 Inc.



--
Amila Suriarachchi,
WSO2 Inc.



--
Amila Suriarachchi,
WSO2 Inc.
<Prev in Thread] Current Thread [Next in Thread>