SGML means Standard Generalized Markup Language. It is an early standard for defining Markup to use in documents. It is very generalized and can be used to define all kinds of markups. XML is a specific implementation of SGML. Framemaker also has a markup language called MIF.
Files conforming to SGML can have any extension. Generically the extension could be .SGM or .SGML