SGML means Standard Generalized Markup Language.
It is an early standard for defining Markup to use in documents. It is very generalized and can be used to define all kinds of markups. XML is a specific implementation of SGML. Framemaker also has a markup language called MIF. Technically an XHTML document is an SGML.
Files conforming to SGML can have any extension. Generically the extension could be .SGM or .SGML
A conforming SGML document must be either a type-valid SGML document, a tag-valid SGML document, or both.
 A type-valid SGML document
An SGML document in which, for each document instance, there is an associated document type declaration (DTD) to whose DTD that instance conforms.
 A tag-valid SGML document
An SGML document, all of whose document instances are fully tagged. There need not be a document type declaration associated with any of the instances. If there is a document type declaration, the instance can be parsed with or without reference to it.