PARAFAC Model

DESCRIPTION OF PARAFAC METHOD

History:
The model in its three-way form has been independently proposed in 1970 by R. Harshman [1] and by Carrol and Chang [2]. They also proposed the most part of the ALS algorithm used to fit this model and some of the techniques (e.g. line-search) used for accelerating it.

Since then, this model has been successfully employed in various fields (psychometrics, analytical chemistry, batch monitoring, etc) and many algorithms have been proposed (ALS, dGN, SWATLD...) [3,4]

One of the aspects that makes this model particularly appealing is the uniqueness of the solution under relatively mild conditions; this means that the solution cannot be rotated without loss of fit as it is possible with many bilinear model such as PCA. One of the main advantages is of course the consequential interpretability, which also stems from the simpler model structure (no interactions are allowed between factors as it is possible in Tucker) and the reduced number of parameters.

Aim:
PARAFAC (PARAllel Factor analysis) is an n-linear model fit on an n-way array.
I.e., given the n-way array X having elements x_ijkl..., the model can be written as:
,
where F is the number of extracted factors/components and e the residuals.
In the three-way case the model parameters a_if, b_jf and c_kf can be gathered in three loading matrices A, B and C, and the model expressed as:
,
where is the columnwise Kathri-Rao product, E is the residual matrix and (IxJK) refers to the way the array is matricised [5]; namely as a (two-way) matrix with I rows and JK columns.

When used for curve resolution, the aim is to recover the responses of every component in the different dimensions of the original array (e.g. elution profiles, kinetic profiles, etc) allowing (but it is not currently implemented in this version of CuBatch) also for quantitative determination of the components of a mixture.
The PARAFAC model can also be used for exploratory purposes given its relative simplicity of interpretation.

Criterion:
Depending on the different proposed algorithms the optimisation criterion may vary, and may be even "unknown" [3,4]. The criterion which appears as the most reliable (for both theoretical and practical purposes) is the least squares criterion. The problem that is solved is then the following (for the three-way case)
(1)
in other words, the square of the Euclidean distance between the model and the data is minimised.
When the ALS algorithm is used, it is also possible to apply some constraints and problem (1) may for example be solved subject to non-negativity for some of the parameters or orthogonality of some of the matrices[5].

Algorithm:
Currently, four different algorithms applicable to fitting a PARAFAC model are implemented in CuBatch:

PARAFAC-ALS
PARAFAC-LM (named 'dGN' within the program)
PARAFAC-ALS with compression
PARAFAC-LM with compression.

Thorough description of these algorithms can be found in the literature [1 3 4 5] and are here only outlined.

1. PARAFAC-ALS (Alternating Least Squares)

It is the algorithm originally proposed by Harshman [1] and later on expanded with the inclusion of several type of constraints and is Alternating Least Squares algorithm: the idea is to split the non-linear least squares problem (1) into linear subproblems by fixing all but one of the loading matrices and to solve them iteratively until convergence.
For the three-way case

i.	Set initial values for and
ii.
iii.
iv.
v.	Check convergence: if not converged repeat from step ii.

The subproblems can also be solved so that the new estimates of the loading matrices are found subject to the desired constraints [4].

2. PARAFAC-LM

The algorithm is well spread in the optimisation field and is described in [4].
Differently from ALS, problem (1) is solved as is, under the assumption that the behaviour of the residuals for the single element x_ijk is well approximated by a linear function with respect to the model parameters and that the residuals are not "too" large.

i.	Set initial values for , and
ii.	set e as the vectorised residuals and set as the vector of the parameters
iii.	Fill in the Jacobian matrix with the derivatives of the residuals with respect to the various elements
iv.	The update for p is computed as the solution to: where "lambda" is called damping parameter
v.	If the decrease of the residuals is insufficient reject the update and increase , otherwise update the parameters and reduce (or leave it the same depending on how good is the decrease)
vi.	Check convergence: if not converged repeat from step ii.

Constraints have not been implemented in this algorithm yet.

3. and 4. Compression

Compression is a technique that allows to reduce the size of the problem of several order of magnitude and is based on a Tucker model [4-6].

i.	Fit a Tucker model on X with F' = F+f' (F is the number of components in the PARAFAC model and f' = 2, normally) components in each mode
ii.	Fit a PARAFAC model with F components on G using either PARAFAC-ALS (1.) or PARAFAC (2.)
iii.	Expand the solution on the compressed space using R, S and T. E.g.
iv.	Fit the F components PARAFAC model on X using A, B and C from step iii. and PARAFAC-ALS

Constraints are not available when compression is used.

5. Initialisation

Several possibilities are available to set the initial estimates for the main algorithm: column-orthogonal random matrices, random matrices, NIPALS, DTLD [7], SWATLD [8] or the best among all these.
DTLD and SWATLD cannot deal with missing values and thus cannot be used in such case

Code:
The code for PARAFAC-ALS is the one from the n-way toolbox a version (2.1) of which is included in the software.
The same holds for NIPALS and DTLD. Most of the files for the n-way data treatment are part of the n-way toolbox [9]. Updates (but CuBatch is not guaranteed to work with them) can be downloaded at www.models.kvl.dk

The algorithms for PARAFAC-LM and SWATLD have been implemented by Tomasi & Bro and will soon be independently available for download.

Applications:
Curve resolution, batch process monitoring, exploratory analysis, multivariate calibration...

Dataset reference:
Fluorescence.mat

References:
[1] Harshman RA, UCLA working papers in phonetics, 16 (1970), 1-84
[2] Carrol JD, Chang JJ, Psychometrika, 35 (1970), 283
[3] Faber N, Bro R, Hopke PK, Chemometrics and Intelligent Laboratory Systems, 65 (2003), 119-137
[4] Tomasi G, Bro R, submitted
[5] Bro R, PhD dissertation, University of Amsterdam (1998)
[6] Bro R, Andersson CA, Chemometrics and Intelligent Laboratory Systems, 42 (1998), 105-113
[7] Sanchez E, Kowalski BR, Journal of Chemometrics, 4 (1990), 29-45
[8] Chen et al., Chemometrics and Intelligent Laboratory Systems, 52 (2000) 75-86.
[9] Andersson CA, Bro R, Chemometrics and Intelligent Laboratory Systems, 52 (2000), 1-4
[10] Ladefoged P et al, Journal of the Acoustical Society of America, 64 (1978), 1027-1035.
[11] Ossenkopp KP et al, Behavioural Processes, 31 (1994), 129-144