The biggest problem is that people are using OGM primarily to encode DVD's with DivX video and Vorbis audio. I fear that people might be using OGM, and convince themselves that they're using completely free software, when they're not. After all, 'Ogg' is from Xiph, and Xiph only produces free stuff. DivX and XviD are MPEG-4 variants, and subject to any holds that the MPEG patents have on those technologies.
Again; Nothing wrong with DivX or XviD, they're just not completely open standards. Patents apply. It doesn't mean they're not useful, it just means that you might be getting more than you bargained for on the legal side of things.