The need of optical fiber network operators to cope with existing dispersion and to increase spectral efficiency has prompted us to propose and implement a new kind of quadrature amplitude modulation, where the spectrally efficient duobinary modulation is applied to two quadratures of the signal with unequal amplitudes. Together they form a 4-level intensity-modulated signal with high spectral efficiency and improved dispersion tolerance. The two quadratures may either be the in-phase and the quadrature part of the field in one polarization, or they may be signals in two orthogonal polarizations. 

Publications about the two schemes are here:

• Selwan. K. Ibrahim, Suhas Bhandare, and Reinhold Noé, Performance of 20 Gbit/s Quaternary Intensity Modulation Based on Binary or Duobinary Modulation in Two Quadratures with Unequal Amplitudes , IEEE J. Selected Topics in Quantum Electronics, vol. 12, 2006, No. 4, pp. 596-602. 
• Selwan. K. Ibrahim, Suhas Bhandare, and Reinhold Noé, 20 Gbit/s Quaternary Intensity Modulation Based on Duobinary Modulation with Unequal Amplitude in Two Polarizations, IEEE Photonics Technology Letters, vol. 18, 2006, pp. 1482-1484. 
• Selwan. K. Ibrahim, Suhas Bhandare, Reinhold Noé, Narrowband 2x10 Gbit/s Quaternary Intensity Modulation Based on Duobinary Modulation in Two Polarizations with Unequal Amplitudes, Optical Fiber Communication Conference (OFC 2006), 5-10 March 2006, Anaheim, CA, USA, paper OThI2 
• Selwan K. Ibrahim, Suhas Bhandare, Hongbin Zhang, and Reinhold Noé, 2×10 Gbit/s Quaternary Intensity Modulation Generation using an Optical QPSK modulator, Proc. SPIE, Vol. 6021, paper 602119, Dec. 8, 2005. 
• S.K. Ibrahim, S. Bhandare, H. Zhang, R. Noé, 2×10 Gbit/s Quaternary Intensity Modulation Generation using an Optical QPSK modulator, Asia-Pacific Optical Communications Conference (APOC 2005), 6–10 November 2005, Shanghai, China, Session 5b, paper 6021-44 
• S.K. Ibrahim, S. Bhandare, R. Noé, Narrowband 20 Gbit/s Quaternary Intensity Modulation Generated by Duobinary 10 Gbit/s Modulation in 2 Quadratures , Proc. 31st European Conference on Optical Communication ECOC 2005, 25-29 September 2005, Glasgow, Scotland, Th2.6.5, Vol. 4, pp. 909-910 

As an example, the NRZ rather than duobinary version of modulating quadratures in orthogonal polarization is shown in the following pictures. A clear quaternary eye diagram is seen, and the bit error ratio counter displays no errors. The pictures were taken in our new spacious optics laboratory in building P8.