\section{Evaluation of Techniques for Workflow-Internal Data Processing}
\label{sec:Evaluation}


Now, we evaluate the effectiveness of the techniques introduced in Section \ref{sec:OptDP:Techniques} via experiments, in particular via the protein modeling workflow of \mbox{Section \ref{sec:prel:SampleScenario}}. In doing so, we evaluate some of the \mbox{arguments} discussed in the previous section regarding improved efficiency and reliability for \mbox{data-intensive} workflow execution.



\subsection{Prototype and Experimental Setup}
\label{sec:Evaluation:Setup}

\begin{figure}[tb]
	\centering
		\includegraphics[width=0.78\textwidth]{Evaluation/Prototype.pdf}
	\caption{Prototypes original ODE (a) and ODE-TI (b), corresponding to architectures shown in Figure \ref{fig:Architecture}(a) and \ref{fig:Architecture}(b).}
	\label{fig:Prototype}
\end{figure}


For our evaluation, we have developed two BPEL-based prototypes of a workflow execution environment. The first one reflects the current architecture of such environments as shown in \mbox{Figure \ref{fig:Architecture}(a)}. We use the open source workflow engine Apache Orchestration Director Engine\footnote{Apache ODE: http://ode.apache.org/} (ODE) V1.3.4 as workflow runtime, the IBM DB2\footnote{IBM DB2: http://www-01.ibm.com/software/data/db2/} V9.7 as DBS, and Hibernate\footnote{Hibernate: http://docs.jboss.org/hibernate/core/3.5/reference/en/html/} V3.2.5 as persistence manager (see \mbox{Figure \ref{fig:Prototype}(a))}. ODE represents variable contents as XML documents. These documents are managed as Java objects of the type \mbox{W3C Node}\footnote{W3C Node: http://download.oracle.com/javase/1.4.2/docs/api/org/w3c/dom/Node.html}. When storing the data in the DBS, Hibernate maps small \mbox{XML documents} to variable character fields (varchar) and larger ones to binary large objects (BLOBs). These data structures are appropriate for the original usage pattern of the local DBS, \ie it only serves as data store for variable contents. In the following, we call this prototype \h{original ODE}.

%Hibernate maps XML documents with a length of less or equal than 256 characters to variable character fields (varchar) and larger documents to binary large objects (BLOBs).

To implement the extended architecture presented in \mbox{Figure \ref{fig:Architecture}(b)}, we have changed the DBS-internal data structure and the corresponding mapping of Hibernate to a native XML data type in order to employ the XML-enabled query processing functionality of the DB2 pureXML technology\footnote{IBM DB2 pureXML: http://www-01.ibm.com/software/data/db2/xml/} \mbox{(see Figure \ref{fig:Prototype}(b))}. We abstained from the data processing optimizer. Instead, we have extended \mbox{Hibernate} to enforce the techniques \ap, \eep, and \wsp\ introduced in Section \ref{sec:OptDP:Techniques}. This leads to a tight integration of the database and the workflow engine. We call the resulting prototype \h{ODE-TI} (Tight Integration).

All system components ran on a \mbox{32-bit} Windows Server 2003 Enterprise Edition (Service Pack 2) operating system with two Intel Xeon PowerEdge 2850 \mbox{3.2 GHz} processors and \mbox{8 GB} main memory. With this system environment and the above-mentioned prototypes, we have investigated whether there exist break-even points when ODE-TI is more efficient and reliable than original ODE. Criteria for such break-even points are the data size, complexity of involved expressions, and complexity of workflows. In real scenarios, concrete break-even points for decisions of the data processing optimizer are system- and tool-dependent. 
% we investigate whether there exist break-even points when our extended version of Apache ODE is more efficient and reliable than the original one. Criteria for such break-even points are the data size, complexity of involved expressions, and complexity of workflows. Concrete break-even points for decisions of the data processing optimizer are system- and tool-dependent.
We have analyzed the effectiveness of the techniques introduced in Section \ref{sec:OptDP:Techniques} in two scenarios. The first one comprises small \mbox{BPEL processes} that facilitated the test of individual workflow activities, \ie a \mbox{BPEL \h{assign} activity} for the \ap, an \h{if} activity for the \eep, and an \mbox{\h{invoke} activity} for the \wsp. This allows us to evaluate the techniques in isolation. The second test scenario is the sample scenario of a data-intensive protein modeling workflow described in Section \ref{sec:prel:SampleScenario} that contains all these activities and runs most of them in a loop. Activity \h{Get List of Protein Sequences} is defined as \mbox{\h{invoke} activity,} the pattern matching step as \mbox{\h{if} activity}, and activities \h{Add Sequence Header} and \h{Count Negative} as \mbox{\h{assign}} activities. This allows us to evaluate all \mbox{techniques} \mbox{in combination.} %The Web Service used by activity \h{Get List of Protein Sequences} only delivers data sets up to approximately 4 MB\footnote{http://www.cyped.uni-stuttgart.de/}. In order to test bigger data sets, we simulated this activity as \h{assign} activity that stores the corresponding data set as literal value to the target variable. Comparative measurements with smaller data sizes have shown that the performance ratio between original ODE and ODE-TI is not substantially influenced by this simulated Web Service call. %since the loop is the most expensive part of the workflow.
\mbox{The underlying} \mbox{data set} of both test scenarios is the list of protein sequences of the protein modeling workflow. Each protein sequence is organized as \mbox{XML element} contained in an XML sequence of such elements. The tests have been carried out for five different data sizes: \mbox{100 KB,} \mbox{500 KB,} \mbox{4 MB,} \mbox{9 MB,} and \mbox{50 MB.} This corresponds to 40, 199, 697, 1394, and 7695 protein sequences, respectively, as well as the same number of iterations of the inner loop. Larger XML documents could not be tested since Apache ODE either causes main memory overloads for such data sets or generally stops workflow execution after 2000 seconds without result or failure notification. Remark that this is no severe limitation of our experimental setting as for performance reasons, processing such large amounts of data should be assigned to external resources anyway (see Figure \ref{fig:DataProcessingInWorkflows}).

%According to Figure \ref{fig:DataProcessingInWorkflows}, processing such amounts of data should be outsourced to external resources.


%%% Number of and protein sequences and iterations for different data sizes %%%


% 100 KB: 40

% 500 KB: 199

% 4 MB: 697

% 9 MB: 1394

% 15 MB: 2317

% 25 MB: 3856

% 50 MB: 7695

% 125 MB: 19184

% 200 MB: 30763



\subsection{Test Results}
\label{sec:Evaluation:Results}



\begin{figure}[tb]
	\centering
		\includegraphics[width=\textwidth]{Evaluation/TestResults1.pdf}
	\caption{Effectiveness of techniques for workflow-internal data processing.}
	\label{fig:TestResults1}
\end{figure}


Up to 9 MB, we have carried out each BPEL process of the first test scenario 100 times, and 50 times for 50 MB. Afterwards, we have calculated the mean average of the relevant activity durations. The durations of original ODE are taken as 100\%, wheras those of ODE-TI are reported in \mbox{Figure \ref{fig:TestResults1}} in relation to those of original ODE. For the \ap, we have used three different complexities of the involved XPath expressions. The underlying \h{assign} activity first carries out a read operation to evaluate this expression on the input list of protein sequences, followed by a write operation that stores the expression result to a variable. The first expression complexity involves \h{no expression}, \ie it stores the whole input data. The second one contains an expression to select exactly one protein sequence (\h{simple expression}), and the third one concatenates two sequences (\h{complex expression}). So, the write operation in these two cases stores a data set with constant size, \ie its size does not depend on the size of the input data. Regarding no or a simple expression, the \ap\ shows an average performance degradation for every data size that ranges between 358\% and 11\%. For the complex expression, it shows degradations by 161\% and 38\% for \mbox{100 KB} and 500 KB. When increasing the data size to more than 4 MB, we reach a break-even point in the used system environment where ODE-TI gets slightly better than original ODE by up to 18\%.

%nearly the same duration for 4 MB, and improvements up to 18\% for 9 MB and 50 MB. So, we reach a break-even point where ODE-TI gets slightly better than original ODE in the employed system environment.

In the same way, we tested the \eep\ with a simple expression that accesses one protein sequence and a complex expression that accesses two of them. The underlying \mbox{\h{if} activity} only performs a read operation, and no subsequent write operation. For the simple expression, ODE-TI shows performance degradations from 17\% to 127\% when reading data up to 9 MB. However, the break-even point is reached for 50 MB, where we get an improvement of 5\%. When testing the complex expression, we get the same performance for 100 KB and improvements between 74\% and 85\% for the other data sizes. The results of the \wsp\ get better with increasing data size and range from slight degradations of 12\% to slight improvements up to 14\%. \mbox{In summary} and as described in Section \ref{sec:Discussion:Efficiency}, these experiments show that our techniques considered in isolation result in performance improvements mainly for complex expressions that are evaluated on large data sets.

% The \wsp\ does not substantially influence activity duration. Performance improvements and degradations are both within a 3\% range. \todo{Zeiten beim Web Service Pushdown ändern.}

%As stated in Section \ref{sec:Discussion:Efficiency}, the results become better the more complex the involved expressions are.
Comparing the results of the \eep\ and the \ap\ and considering that the only relevant difference between the underlying \h{if} and \h{assign} activities is the additional write operation of the \h{assign}, we see that the workflow runtime in original ODE deals more efficiently with write operations than the database system in ODE-TI. The main reason for the additional overhead in the database system is that it needs to store log information on disk for the involved data manipulation queries. Furthermore, it needs to adapt indexes when new data is inserted or when existing data is updated. In case of the \ap\ without expression, the results first get better with an increasing data size. However, this turns into the opposite when crossing the 4 MB size. This is because for bigger data sets the effects of the write operation in the database system superimpose those of the read operation. All other test cases involving an expression either do not perform a write operation or the write operation stores data of constant size. Hence, the effects of write operations get less important with growing data sets and ODE-TI achieves better results the bigger the input data sets are. In original ODE, the \h{invoke} activity regarding the \wsp\ stores the result of the service in the local DBS for persistence and recovery purposes. This constitutes additional costs for transferring data from the workflow runtime to the local DBS, which are circumvented in ODE-TI by the \wsp\ (see Section \ref{sec:Discussion:Efficiency}). These transfer costs in original ODE largely compensate the above-mentioned overhead for writing the service result in the database system of ODE-TI, but they cross this overhead when the data size increases.


\begin{figure}[tb]
	\centering
		\includegraphics[width=\textwidth]{Evaluation/TestResults2.pdf}
	\caption{Performance improvements for the protein modeling workflow.}
	\label{fig:TestResults2}
\end{figure}


For the second test scenario, we have %investigated whether the optimization techniques are better suited for more complex data-intensive workflows such as the protein modeling workflow. We have 
executed the protein modeling workflow either in sequence or in parallel. For the sequential execution, we have carried out 100 instances for both \mbox{100 KB} and 500 KB. The 4 MB and 9 MB cases have been executed \mbox{50 times}. The 50 MB and larger documents could not be tested due to a main memory overload in original ODE and workflow execution times above 2 million ms in ODE-TI as described in \mbox{Section \ref{sec:Evaluation:Setup}}. \mbox{Figure \ref{fig:TestResults2}(a)} shows the absolute values of the average duration of one workflow instance compared between original ODE and ODE-TI. These results indicate the potential of our approach: \mbox{For 100 KB,} we can reduce the duration by nearly a factor of 3, and even by a factor of more than 8 using \mbox{500 KB}. For 4 and 9 MB, original ODE was not able to execute the entire workflow at least once. Due to a main memory overload  in the workflow runtime, it crashed after approximately 200 or 100 iterations, respectively. However, \mbox{ODE-TI} successfully executed all 50 instances of the workflow for both data sizes. Altogether, these results correspond to the argument of Section \ref{sec:Discussion:Efficiency} that the database technology can more efficiently and reliably deal with high volumes of data and complex as well as frequent data processing operations involved in data-intensive workflows. The standard version of Apache ODE comes with a Java-based and embedded Apache Derby\footnote{Apache Derby: http://db.apache.org/derby/} database system. When using Derby instead of DB2 in original ODE, the problem with main memory overloads would even be worsened since Derby would be executed in the same Java process as ODE and would thus share its main memory.

Figure \ref{fig:TestResults2}(b) shows the results for parallel workflow execution where all instances are started at the same time. Here, we have executed the 100 KB and 500 KB cases ten times, the 4 MB case five times, and three instances for 9 MB. For each data size, the figure reports the total durations of all executed workflow instances. ODE-TI outperforms original ODE in any of these cases. It reduces the total duration by a factor of more than 2 already for 100 KB. When the data size grows, original ODE gets problems to execute at least one of the workflow instances. After ten to 17 minutes, it aborts with a main memory overload again. ODE-TI was able to execute all instances in 145 seconds for 500 KB, 736 seconds for 4 MB, and 1620 seconds in the 9 MB case. In summary and as described in \mbox{Section \ref{sec:Discussion:Efficiency}}, the evaluated techniques lead to improved efficiency and reliability for both sequential and parallel execution of data-intensive workflows such as the protein modeling workflow.

%The comparison of the results for the sequential and parallel workflow execution indicates a significant increase in throughput of workflow instances for ODE-TI compared to original ODE. The factor by which ODE-TI reduces the average duration of one single workflow instance in the sequential execution increases from nearly 2 to 5.5 in case of the total duration of ten parallel workflow instances. %ODE-TI was able to execute ten parallel instances in 20 seconds, while executing one sequential instance in 12.1 seconds.
%For 500 KB, we get a similar increase in throughput. Furthermore, ODE-TI could execute two parallel workflow instances with a data size of 4 MB in approximately the same time as a single workflow instance in the sequential execution. This constitutes a significant increase in throughput of approximately 100\%. So, we see that the database system exhibits a more mature support for the concurrent execution of processes than the workflow runtime as stated in Section \ref{sec:Discussion:Efficiency}.

%For the protein modeling workflow, the evaluated optimization techniques lead to significant improvements in efficiency and reliability, an outstanding scalability with respect to the amount of data, and an increased throughput for parallel workflow execution. Altogether, we can recommend all optimization techniques when dealing with more complex workflows, in particular data-intensive workflows. For such workflows, the data processing optimizer should decide to carry out the workflow-local data processing completely in the local DBS.