1. Introduce discrimination into GPDM (D-GPDM), the classification is based on CRF

2. Introduce discrimination into GPDM (D-GPDM), the classification is take care of by based on MIL. The tem In MIL, we use the learned latent point as the feature,
   and we learn a set of instance prototypes. The MIL learning is based on Diverse Density.

3. Using GPDM to learn the latent space trajectory, then classification is done by matching trajectory in the latent space via method in [1]

4. Introduce discrination into GPDM, the classification is taken care of by Gaussian Process classification. But it is hard to really innovate on the
Gaussian Process Classification part. The GPC inference involves approximation approaches like (loopy) belief propagation, expectation propagation,
variational approximations, or Monte Carlo Sampling.

[1] M. Black and A. Jepson. A probabilistic framework for matching temporal trajectories:Condensation-based recognition of
gestures and expressions. In ECCV, 1998.

I am reading dozens of papers in order to do my surgery video project. Now I have the ideas, and I only have two months before my graduation to fully develop those ideas including coding, running experiment, analyze the data and publish paper if possible. Too short to work on, so I really need to manage my time efficiently.

The papers that I have been reading

1. Conditional random field

1.1 Lafferty, J., Zhu, X. and Liu, Y. 2004. Kernel conditional random fields: Representation and clique selection. In Proc. Twenty-First International Conference on Machine Learning (ICML). 

1.2 Sanjiv Kumar and Martial Hebert, Discriminative random field, Volume 68, Number 2 / June, 2006, pp. 179-201.

1.3 C. Sutton, A. McCallum, An introduction to Conditional random fields for relational learning, Introduction to Statistical Relational Learning, Ed. Lise Getoor and Ben Taskar, the MIT press.

1.4 C. Sminchisescu, A. Kanaujia, Z. Li, D. Metaxas, Conditional models for contextual human motion recognition, in: IEEE International Conference on Computer Vision, Vol. 2, 2005, pp. 1808--1815.

1.5 B. Taskar, C. Guestrin, and D. Koller. Max-margin markov networks. In Sebastian Thrun, Lawrence Saul, and Bernhard Sch olkopf, editors, Advances in Neural Information Processing Systems 16, 2003.

1.6 Ariadna Quattoni, Michael Collins and Trevor Darrel. Conditional Random Fields for Object Recognition. In Advances in Neural Information Processing Systems 17 (NIPS 2004), 2004.

2 Gaussian Process

2.1 Gaussian Process Regression, Chapter 2, Machine Learning for Pattern Recognition, M. Bishop, MIT Press.

2.2 Gaussian Process Classification, Chapter 3, Machine Learning for Pattern Recognition, M. Bishop, MIT Press.

2.3 N. Lawrence, Probabilistic Non-linear Principle Component Analysis with Gaussian Process Latent Variable Models, Journal of Machine Learning Research 6, pp. 1783-1816, 2005.

2.4 Wang, J. M., Fleet, D. J., Hertzmann, A. Gaussian Process Dynamical Models for Human Motion. In IEEE Trans. PAMI. February, 2008. pp. 283-298.

2.5 Yasemin Altun, Thomas Hofmann and Alexander J. Smola. Gaussian process classification for segmenting and annotating sequences. In Proceedings of the Twenty-First International Conference on Machine Learning (ICML 2004), 2004.

2.6 Y. Altun, T. Hofmann, and M. Johnson. Discriminative learning for label sequences via boosting. In Advances in Neural Information Processing Systems (NIPS*15), 2003.

2.7  Urtasun and T. Darrell. Discriminative Gaussian process latent variable model for classification. In 24th International Conference on Machine Learning, 2007.

3, Combine multiple data source

3.1 M. Girolami, M. Zhong, Data Integration for classification problems employing gaussian process priors, NIPS 2006.

1. Compare with PCA in terms of latent space mapping

2. What is the advantage of our method? Using examples to illustrate 'smart sampling'

3. Partition the sample space in upper and lower body to see the performance.

4. Try a more sophisticated dynamical model, at least a second order markov model. "Your dynamical model is very simple, just a 1st order markov model. Why don't you use something more complicated, or at least a second order markov model?"

5. Using kernel to capture nonlinearties in the motion space. "Why are you using a linear relationship between left and right side of the body? "

 6. Analyze the error cases. "I also concern about the error cases shown in the vidoes. They should have been analyzed and discussed. "

The mocap data from HumanEva records the position, I need to compute the global and local transform from the marker data and then compute the body angles from the transform.

First try to project the 3D poses to 2D and see how the pose fit the image data using trial one of any subject (S1, S2, S3 or S4);

If it fits well, then use three sets of data(S1_walking_trial3,S2_walking_trial3,S3_walking_trial3) to learn motion prior.

After that, track the moving person in sequece S1_walking_trial1,S2_walking_trial1 and S3_walking_trial1), and do evaluation by comparing the estimated poses with the ground truth MoCap data. I can also track the moving person in sequences S1_walking_trial2,S2_walking_trial2 and S2_walking_trial2, but since no motion capture data is given, I can not do any further evaluation, and I can only do visual inspection to see the performance of the algorithm.

What have been done:

1. Given an image, using the little CMS C++ API, I have been able to apply differnt output color profiles to the same image and show the appearance of the same image side by side after different ouput color profile transformation.

What is the next step:

2. After a discussion with my advisor, I am going to achieve the following objective:

Given an image I shown on LCD A, try to find the image J so that the appearance of J in a desktop monitor looks the same with the appearance of I on A. Basically I would like to find an approapriate output color profile (xxx.icc)so that after I apply the ICC transfom (J=icctrans(I,'-t0 -o xxxx.icc'), image J looks the same on monitor with the appearance of I on a LCD.

But then fundamental questions arise:

 the purpose of color management is to make the same image look consistently the same under different device, but why when I display the same image on one LCD and one CRT monitor, the appearance of the same image is different? And what is the color profile used in the working color space, namely the PCS space, and also what is the color profile for the LCD (or a CRT). Is the profile used for the output device (like LCD or CRT) the same with the one used in the working color space?

The steps in using matlab compiler to call matlab code from C++ enviornment.

1. in Matlab prompt, type

>>mbuild -setup to configure the C++ compiler

2 in Matlab prompt, type

>>mcc -W cpplib:libIMclassify -T link:lib addshape.m colorf.m fea4blocks.m

This will create a shared library from an arbitrary set of M-files (or mex-file). The MATLAB Compiler generates a wrapper file, a libIMclassify.h, a libIMclassify.dll file, a libIMclassify.lib file, a libIMclassify.ctf file. What will used in the C++ application include the .dll, .lib, .h and .ctf file and other matlab files (e.g. .mat or .dll or that the matlab .m files depend on).

3 Put the the .dll, .lib, .h and .ctf file in the folder of the C++ application

4 Write a C++ wrapper that calls the matlab compiler generated shared library

In my image categorization project, I wrote the following C++ wrapper named main.cpp

#ifdef __APPLE_CC__
#include
#include
#endif

#include "libIMclassify.h"
using namespace std;

#ifdef _MSC_VER
extern "C"
#endif

void* ClassifyIM(const char** inIMfn, const int &nIM, void *x)
{
    int *err = (int *)x;
    if (err == NULL) return 0;

    // Call application and library initialization. Perform this
    // initialization before calling any API functions or
    // Compiler-generated libraries.
    if (!mclInitializeApplication(NULL,0))
    {
        std::cerr << "could not initialize the application properly" << std::endl;
        *err = -1;
        return x;
    }
    if( !libIMclassifyInitialize() )
    {
        std::cerr << "could not initialize the library properly"<< std::endl;                  
        *err = -1;
    }
    else
    {     
        try
        {
   // Create the input data
   // Input parameters:
   // B:   test image file name cell array and
   // nIM:  number of test images to be classifile   
   int arr[1] = {nIM};
   mwArray N(1,1,mxINT8_CLASS,mxREAL);
   N.SetData(arr, 1);   
   mwArray B(1, nIM, mxCELL_CLASS);//the image name in cell array format
   for(int i = 1; i<= nIM; i++)
   {
    mwArray fn(inIMfn[i-1]);
    B.Get(1,i).Set(fn);
   }   
   //Create output parameters
   //lbl: the predicted label, an array of size nIM
   mwArray lbl(nIM,1,mxUINT8_CLASS,mxREAL);

   //Call the library function
   ImageCategorization(1, lbl, B, N);

   //convert the predicted label from mwArray in matlab to unsigned int array in C++
   unsigned int *lblp = new unsigned int[nIM];
   lbl.GetData(lblp,nIM);

   //Display the returned predicted label for each image
   for(int i = 0; i < nIM; i++)
    std::cout << "Label of "  << inIMfn[i] << " is: " << lblp[i]<  }
  catch (const mwException& e)
        {
          std::cerr << e.what() << std::endl;
          *err = -2;
        }
        catch (...)
        {
          std::cerr << "Unexpected error thrown" << std::endl;
          *err = -3;
        }   
  // Call the library termination routine
  libIMclassifyTerminate();
    }
/* On MAC, you need to call mclSetExitCode with the appropriate exit status
 * Also, note that you should call mclTerminate application in the end of
 * your application. mclTerminateApplication terminates the entire
 * application and exits with the exit code set using mclSetExitCode. Note
 * that this behavior is only on MAC platform.
 */
#ifdef __APPLE_CC__
    mclSetExitCode(*err);
#endif
    mclTerminateApplication();
    return 0;
}

int main()
{
 int err = 0;
 int nIM = 3;
 const char* imfn[] = {"test001.jpg", "test2.jpg","test0003.jpg"};
 
#ifdef __APPLE_CC__
    pthread_t id;
    pthread_create(&id, NULL, run_main, &err);

    CFRunLoopSourceContext sourceContext;
    sourceContext.version         = 0;
    sourceContext.info            = NULL;
    sourceContext.retain          = NULL;
    sourceContext.release         = NULL;
    sourceContext.copyDescription = NULL;
    sourceContext.equal           = NULL;
    sourceContext.hash            = NULL;
    sourceContext.schedule        = NULL;
    sourceContext.cancel          = NULL;
    sourceContext.perform         = NULL;

    CFRunLoopSourceRef sourceRef = CFRunLoopSourceCreate(NULL, 0, &sourceContext);
    CFRunLoopAddSource(CFRunLoopGetCurrent(), sourceRef, kCFRunLoopCommonModes);
    CFRunLoopRun();
#else 
    ClassifyIM(imfn, nIM, &err);
#endif
    return err;
}

5. Create a win32 console project using Visual Studio C++ 2003, add main.cpp and libIMclassify.h to the project.

6 Set C++ project settings in Visual Studio .NET 2003.

 6.1

Click menu Project->Project settings->Linker->General, add C:\Program Files\MATLAB\R2006b\extern\lib\win32\microsoft;F:\RA\TactileOurData\NET\ImageCategorization\ImageCategorization to the "Additional Library Directories. The second directory is the one that holds your C++ project.

6.2 Click Project->Project settings->Linker->Input, add "libdflapack.lib libeng.lib libfixedpoint.lib libmat.lib libmex.lib libmwlapack.lib libmwservices.lib libmx.lib libut.lib mclcom.lib mclcommain.lib mclmcr.lib mclmcrrt.lib mclxlmain.lib libIMclassify.lib" to the "Additional dependencies" entry. The last libIMclassify.lib is the lib file generated by the matlab compiler and is located in the current project directory (F:\RA\TactileOurData\NET\ImageCategorization\ImageCategorization )

7. Build and run.

Standalone matlab application may also be built by using mcc and mbuild command. For example, in matlab prompt or OS prompt, type

mbuild main.cpp libIMclassify.lib

will create an executable named main.exe