http://gaffneykev.no-ip.info/ca400blog/Docs/OnlineBrokerageTest.doc
Testing & HD18
Resloution
Broken ftrenq link
CM02 Customer Documents & Sessions
CM02 customer document management is complete and ready for UAT.
The biggest problems faced during development of this requirements were management of sessions & the lazy loading of customer documents in the Client POJO.
Uploading on the flex side is developed from the file upload & php article from coding cowboys:
http://weblog.cahlan.com/2006/09/uploading-files-with-flex-and-php.html
I then got the apache commons file upload libraries to lend a hand on the backend.
The actionscript from the above link handles the getting of the file from the file system and posts it off to our usual service, tagged with add document.
The request is then parsed using the apache commons library into a FileItem where we can extract the content & file name etc.
The request comes with the client key, we fetch the appropriate client and its file set adding the new document and saving the client.
One of the biggest problems we had with this as i mentioned earlier was a session problem. When logged in there is a session id associated with the authenticated user. However, the file upload implementation in flex begins a new session. There are numerous links and posts about these flex session id tomcat problems on the web. Our solution was to setup another request handler which can pass back the valid jsession id to flex itself. We then append this session id to the upload url which ensures the upload request is authenticated. The same issue occurs for file download.
The other issue i noted earlier was lazy loading and while my post on the hibernate forum was a start it wasn’t the sliver bullet to all my problems.
In the end I wrote a session interceptor in Spring which i mentioned as on wish list in a previous post – this however became essential to make the document lazy loading work.
I also had to the spilt the file POJO into two separate POJOs one for the file details and one for the file content itself. This is because as per the hibernate post all hibernate properties default to eager fetching and changing this through writing an ant task to change the byte code is not a very “done” thing. So as per the hibernate post i have setup an @lazyToOne relation between the File & FileContent POJOs. This now means when we do a fetch on a customer for populating the various customer tabs we are not fetching there documents as well, just a proxy list containing their file details. Before this was implemented even with small files there was a visible drop in performance of fetching customer details. This has now thankfully been resolved.
Also becuase of the spilt the adding of a file is a transaction and as such has been wrapped by a spring transaction interceptor (see my previous post on transactions for details).
Further I have implemenated a many to many relationship on the data model side to open the scope for one document to be linked to more than one customer. This however, will be beyond scope for this project iteration.
Now my task is to go back over all of the developed CMS components and do some ironing and polishing.
David is already writing JUnits for some of his code so I had better not let the side down. For GUI testing we are going to rely on manual use cases and checks as opposed to some of the automated GUI testing software out there – due simply to the ever tightening time constraints placed on the project.
Lazy loading issues Customers & Documents
Contact Log
Searching, Contact Log, Annontations, Deprecation, HD16 etc
Since my last update where I discussed an implementation of free form search, I now have this requirement fufilled with a few caveats.
The current implementation is a free from search which is implemented for all but two of Dermot’s requirements (see previous post).
That is for example you can type a policy number or a client name etc into the one search box and without specifying further options on type it will pull back the pertinent client results.
It is a free from search for Dermot’s search specification only.
I have implemented this in the following fashion:
The retreiveClientsRequestHandler bean which handles requests for client search has a list of query fields
<bean id=“retrieveAllCustomersRequestHandler” class=“com.zeninvent.dmib.web.handler.retrieveAllCustomersRequestHandler”>
<property name=“climasListProxyDAO”><ref bean=“climasListProxyDAO”/></property>
<property name=“queryFields”>
<list>
<!– Any value here must be specifed its table implmentation list in the climasListProxyDAO –>
<value>taxStatus</value>
<value>surname</value>
<value>phone1</value>
<value>address1</value>
<value>address2</value>
<value>occupationS</value>
<value>employerS</value>
<value>policyNum</value>
<value>maritalStatus</value>
</list>
</property>
</bean>
Any field\column which is signed up for search must first be entered here.
Then the clientMasterListProxy bean has a list of columns for each table to be queried.
<bean id=“climasListProxyDAO” class=“com.zeninvent.dmib.db.hibernate.HibernateClimasListProxyDAO”>
<property name=“sessionFactory”><ref bean=“sessionFactory”/></property>
<property name=“ftrenqFields”>
<list>
<value>occupationS</value>
<value>employerS</value>
</list>
</property>
<property name=“policyFields”>
<list>
<value>policyNum</value>
</list>
</property>
<property name=“climasFields”>
<list>
<value>taxStatus</value>
<value>surname</value>
<value>phone1</value>
<value>address1</value>
<value>address2</value>
<value>maritalStatus</value>
</list>
</property>
While this list currently corresponds only to match the values in the original list in the handler it may be nice in time to have some redundancy here where the search list in the requestHandler may not utilise all of the options in the lists for performance reasons etc.
Then in the ClientMasterListProxyDAO it is simply a case of iterating through the lists and adding the appropriate criteria to the right Criteria objects.
Example:
// builds a list for the ftrenqFiels set in the servlet context these must match fields in the ftrenq table
if(ftrenqFields.contains(queryCriteria.get(i).getPropertyName())){
log.debug(“Non climas value being added to query:”+queryCriteria.get(i).getPropertyName());
critFtrenq.add((Criterion) queryCriteria.get(i).getQueryCriterion());
isAdded = true;
}
Once the criteria list is then built we return the results to the calling requestHandler which places the results back into the model.
While one can argue that there are could be performance issues against checking for example a telephone number against all the search list options in our live database with two and a half thousand customers there are no such issues.
If performance issues were to arise in the future or the days were suddenly to get longer I would consider implementing some regular expression pattern matching on the query field to narrow the list of possible search fields.
That took me until late Saturday afternoon to implement after which I spent the rest of the evening knocking off requirements :
CM09(Mapping) & CM11(For quotation – re-using David’s already developed quotation & front end purchase component).
Today I focused on CM06 the customer contact log.
While reading more about relationship mapping in hibernate there is an improved implementation for mapping, which up to now we have not been using, through annotations. Today that changed.
The beauty about using hibernate annotations is that they provide a further abstraction away from the database and also remove the need for the rather awakard mapping.hbm.xml file.
The other added advantage is if we do need to recfactor the table itself, for example adding a column, under the old regieme it would require updating the mapping.hbm.xml and then updating the POJO. However, now we just add the field to the POJO and hey presto!
The hibernate annotations are placed in the POJO
Example:
@Entity
@Table(name = “climas”)
@Proxy(lazy=false)
public class Climas implements java.io.Serializable{
@Id
@GeneratedValue(strategy = GenerationType.AUTO)
@Column(name=“clientKey”)
private int clientKey;
@OneToOne
@PrimaryKeyJoinColumn
private Ftrenq ftrenq;
@OneToMany(fetch=FetchType.EAGER)
@JoinColumn(name=“clientKey”)
private Set<ContactLog> contactLog ;
}
Most of the above annotations are self explanatory.
@Proxy is the fix for the hibernate org.hibernate.LazyInitializationException which translates to:
“Indicates access to unfetched data outside of a session context. For example, when an uninitialized proxy or collection is accessed after the session was closed”
A way to fix this is to write a sessionIntercoter which has a similar theme to our transaction interceptor from a previous post.
I’ll do this if I get time near the end of the project.
However, we need to hydrate the Climas pojo with a contactLog Set eagerly which is hibernates default to avoid another dreaded null pointer!
The @OneToMany @OneToOne do as they say on the tin.
Now because our mappings are being phased out I have had to re-factor the ftrenq table column names to a nicer format as before the xml mapping swept the column names under the carpet.
The ftrenq’s tables has 156 columns which required manual renaming. I could have used the @Column annotation to override the defaults in the Ftrenq POJO but I felt the column renaming was quicker.
This change in moving away from hibernate mappings to annotations also requires a change in our servlet sessionFactory where we now have an annotated classes list as well. The nice thing about it is we can use mappings and annotations side by side so there doesn’t have to be a complete re-factoring of every POJO and table but more a gentle migration as we code new tables and POJO’s with annotations and re-factoring old POJOs as required like the case today.
I also made the bold decision to truncate all DATETIME types in the migrated data to DATE to remove the absurdity of know the exact second of when a client signed for a policy. Don’t worry all the times were 00:00:00 anyway! J
The Client POJO has also now been officially deprecated and we hope to remove it completely in time.
David has the purchase of mortgage protection nicely tied up. And has been liaising with Dermot over a couple of options regarding the quotation service and purchase options.
anon…
HD16 Tomorrow agenda here:
http://gaffneykev.no-ip.info/ca400blog/Docs/Meeting%20HD16%20Agenda.doc
Screenshot of the contact log UI:
Search
More Messy Data
Weekly Update
HD14 lasted just under an hour. As per the agenda ( http://gaffneykev.no-ip.info/ca400blog/Docs/Meeting%20HD14%20Agenda.doc ) we discussed our first release of the software to Dermot.
A number of topics were raised in relation to this area, in particular our failure to see the new requirements which showed up in Dermot’s eyes as being essential during the release.
These new requirements while a somewhat ugly spot in the project need to be addressed and it is also important to discuss why they have only come to light now & to address this problem going forward.
From early in the project it was agreed there would be a prototype development approach taken. The reasoning for this was two fold, while it brought David and I up to speed with the presentation technologies in particular, it would also be a good approach for eliciting requirements from Dermot. This was a corner stone we felt for building a successful application and meeting Dermot’s business needs.
From our first prototype ( http://gaffneykev.no-ip.info/ca400blog/Lists/Photos/cms01.JPG ) which we demonstrated to Dermot we discussed with him repeatedly what customer information & filter criteria he would like defined for his new system.
When we delivered our final “prototype” at the end of November ( http://www.redbrick.dcu.ie/~napalm/191207/bin/ms.html ) we again retried the need for him to use the system\test run through the system at some stage between then and February and inform us as to anything missing or that he wanted changed.
Again this feedback despite our best efforts never arrived. However, when we delivered the system he appeared to strike a eureka moment and finally realised he actually had use the software we delivered & the rest as they say is history.. & we finally got the requirements we had been badgering for all along.
While discussing this with Howard in HD14 he told us that this is a relatively common problem in software projects where the actors involved on the business side are not familiar with developing software systems (See our original risk assessment in the project proposal http://gaffneykev.no-ip.info/ca400blog/Docs/CA400InitialProposal%20-%20detailedVersion1Zero.doc )
I have been giving some considerable thought as to why the prototyping phase of the project was not as successful as we had originally hoped and have concluded and while not coming to any firm conclusions I still am unsure as to what we should have done differently to achieve a better result.
It is also important to note that the new requirements – policy information, client input information, client linking & filters are on the essential list in terms of critically to the project and have also required a significant redesign\update of our pervious data model, UI & backend processing.
In HD14 there was also a walkthrough of our system architecture which is as per out functional specification ( http://gaffneykev.no-ip.info/ca400blog/Docs/Functional%20Specification%201.0.doc – see 5.2 High Level Design Overview Diagram page 40)
This week I continued working primarily on the DM05 requirements, while talking to Sean @ best advice and getting him and Geoff to liaise with David on the quotation front. David demonstrated a returned and parsed quotation yesterday & it felt really good to see it!
We always knew Dermot’s existing data model was bad but these new requirements have highlighted jsut how atrocious it actually is.
Example:
Policy Holders have dependents. Now in database school the first lesson is a if you have a column in the policyholder table which is dependent, what happens if you have more than one? – Answers on a postcard.
In Dermot’s existing data model which he subscribed to for a monthly fee! – There are six dependent columns and no sign of a link table anywhere! – At the moment I am focusing on getting the system working with this data. However, new data will not be entered in the same awful way as previous & should time allow ill export the old data into the new model as well.
We have also managed to cross of some of the essential requirements which are decorating the spaces in front of our desks.
Theses are SF02, SF03, CM01 & CM14 – crossing them off felt very good!
Going forward, it is more of the same, finish up the new requirements by the end of next week if things go well and probably the week after if there are some unseen problems lying ahead.
