1   
  2   
  3   
  4   
  5   
  6   
  7   
  8   
  9   
 10   
 11   
 12   
 13   
 14   
 15   
 16   
 17   
 18   
 19   
 20   
 21   
 22   
 23   
 24   
 25   
 26   
 27   
 28   
 29   
 30   
 31   
 32   
 33   
 34   
 35   
 36   
 37   
 38  """ 
 39  Provides an extension to back up MySQL databases. 
 40   
 41  This is a Cedar Backup extension used to back up MySQL databases via the Cedar 
 42  Backup command line.  It requires a new configuration section <mysql> and is 
 43  intended to be run either immediately before or immediately after the standard 
 44  collect action.  Aside from its own configuration, it requires the options and 
 45  collect configuration sections in the standard Cedar Backup configuration file. 
 46   
 47  The backup is done via the C{mysqldump} command included with the MySQL 
 48  product.  Output can be compressed using C{gzip} or C{bzip2}.  Administrators 
 49  can configure the extension either to back up all databases or to back up only 
 50  specific databases.  Note that this code always produces a full backup.  There 
 51  is currently no facility for making incremental backups.  If/when someone has a 
 52  need for this and can describe how to do it, I'll update this extension or 
 53  provide another. 
 54   
 55  The extension assumes that all configured databases can be backed up by a 
 56  single user.  Often, the "root" database user will be used.  An alternative is 
 57  to create a separate MySQL "backup" user and grant that user rights to read 
 58  (but not write) various databases as needed.  This second option is probably 
 59  the best choice. 
 60   
 61  The extension accepts a username and password in configuration.  However, you 
 62  probably do not want to provide those values in Cedar Backup configuration. 
 63  This is because Cedar Backup will provide these values to C{mysqldump} via the 
 64  command-line C{--user} and C{--password} switches, which will be visible to 
 65  other users in the process listing. 
 66   
 67  Instead, you should configure the username and password in one of MySQL's 
 68  configuration files.  Typically, that would be done by putting a stanza like 
 69  this in C{/root/.my.cnf}:: 
 70   
 71     [mysqldump] 
 72     user     = root 
 73     password = <secret> 
 74   
 75  Regardless of whether you are using C{~/.my.cnf} or C{/etc/cback.conf} to store 
 76  database login and password information, you should be careful about who is 
 77  allowed to view that information.  Typically, this means locking down 
 78  permissions so that only the file owner can read the file contents (i.e. use 
 79  mode C{0600}). 
 80   
 81  @author: Kenneth J. Pronovici <pronovic@ieee.org> 
 82  """ 
 83   
 84   
 85   
 86   
 87   
 88   
 89  import os 
 90  import logging 
 91  from gzip import GzipFile 
 92  from bz2 import BZ2File 
 93   
 94   
 95  from CedarBackup2.xmlutil import createInputDom, addContainerNode, addStringNode, addBooleanNode 
 96  from CedarBackup2.xmlutil import readFirstChild, readString, readStringList, readBoolean 
 97  from CedarBackup2.config import VALID_COMPRESS_MODES 
 98  from CedarBackup2.util import resolveCommand, executeCommand 
 99  from CedarBackup2.util import ObjectTypeList, changeOwnership 
100   
101   
102   
103   
104   
105   
106  logger = logging.getLogger("CedarBackup2.log.extend.mysql") 
107  MYSQLDUMP_COMMAND = [ "mysqldump", ] 
115   
116     """ 
117     Class representing MySQL configuration. 
118   
119     The MySQL configuration information is used for backing up MySQL databases. 
120   
121     The following restrictions exist on data in this class: 
122   
123        - The compress mode must be one of the values in L{VALID_COMPRESS_MODES}. 
124        - The 'all' flag must be 'Y' if no databases are defined. 
125        - The 'all' flag must be 'N' if any databases are defined. 
126        - Any values in the databases list must be strings. 
127   
128     @sort: __init__, __repr__, __str__, __cmp__, user, password, all, databases 
129     """ 
130   
131 -   def __init__(self, user=None, password=None, compressMode=None, all=None, databases=None):   
 132        """ 
133        Constructor for the C{MysqlConfig} class. 
134   
135        @param user: User to execute backup as. 
136        @param password: Password associated with user. 
137        @param compressMode: Compress mode for backed-up files. 
138        @param all: Indicates whether to back up all databases. 
139        @param databases: List of databases to back up. 
140        """ 
141        self._user = None 
142        self._password = None 
143        self._compressMode = None 
144        self._all = None 
145        self._databases = None 
146        self.user = user 
147        self.password = password 
148        self.compressMode = compressMode 
149        self.all = all 
150        self.databases = databases 
 151   
153        """ 
154        Official string representation for class instance. 
155        """ 
156        return "MysqlConfig(%s, %s, %s, %s)" % (self.user, self.password, self.all, self.databases) 
 157   
159        """ 
160        Informal string representation for class instance. 
161        """ 
162        return self.__repr__() 
 163   
165        """ 
166        Definition of equals operator for this class. 
167        @param other: Other object to compare to. 
168        @return: -1/0/1 depending on whether self is C{<}, C{=} or C{>} other. 
169        """ 
170        if other is None: 
171           return 1 
172        if self.user != other.user: 
173           if self.user < other.user: 
174              return -1 
175           else: 
176              return 1 
177        if self.password != other.password: 
178           if self.password < other.password: 
179              return -1 
180           else: 
181              return 1 
182        if self.compressMode != other.compressMode: 
183           if self.compressMode < other.compressMode: 
184              return -1 
185           else: 
186              return 1 
187        if self.all != other.all: 
188           if self.all < other.all: 
189              return -1 
190           else: 
191              return 1 
192        if self.databases != other.databases: 
193           if self.databases < other.databases: 
194              return -1 
195           else: 
196              return 1 
197        return 0 
 198   
200        """ 
201        Property target used to set the user value. 
202        """ 
203        if value is not None: 
204           if len(value) < 1: 
205              raise ValueError("User must be non-empty string.") 
206        self._user = value 
 207   
209        """ 
210        Property target used to get the user value. 
211        """ 
212        return self._user 
 213   
215        """ 
216        Property target used to set the password value. 
217        """ 
218        if value is not None: 
219           if len(value) < 1: 
220              raise ValueError("Password must be non-empty string.") 
221        self._password = value 
 222   
224        """ 
225        Property target used to get the password value. 
226        """ 
227        return self._password 
 228   
230        """ 
231        Property target used to set the compress mode. 
232        If not C{None}, the mode must be one of the values in L{VALID_COMPRESS_MODES}. 
233        @raise ValueError: If the value is not valid. 
234        """ 
235        if value is not None: 
236           if value not in VALID_COMPRESS_MODES: 
237              raise ValueError("Compress mode must be one of %s." % VALID_COMPRESS_MODES) 
238        self._compressMode = value 
 239   
241        """ 
242        Property target used to get the compress mode. 
243        """ 
244        return self._compressMode 
 245   
247        """ 
248        Property target used to set the 'all' flag. 
249        No validations, but we normalize the value to C{True} or C{False}. 
250        """ 
251        if value: 
252           self._all = True 
253        else: 
254           self._all = False 
 255   
257        """ 
258        Property target used to get the 'all' flag. 
259        """ 
260        return self._all 
 261   
263        """ 
264        Property target used to set the databases list. 
265        Either the value must be C{None} or each element must be a string. 
266        @raise ValueError: If the value is not a string. 
267        """ 
268        if value is None: 
269           self._databases = None 
270        else: 
271           for database in value: 
272              if len(database) < 1: 
273                 raise ValueError("Each database must be a non-empty string.") 
274           try: 
275              saved = self._databases 
276              self._databases = ObjectTypeList(basestring, "string") 
277              self._databases.extend(value) 
278           except Exception, e: 
279              self._databases = saved 
280              raise e 
 281   
283        """ 
284        Property target used to get the databases list. 
285        """ 
286        return self._databases 
 287   
288     user = property(_getUser, _setUser, None, "User to execute backup as.") 
289     password = property(_getPassword, _setPassword, None, "Password associated with user.") 
290     compressMode = property(_getCompressMode, _setCompressMode, None, "Compress mode to be used for backed-up files.") 
291     all = property(_getAll, _setAll, None, "Indicates whether to back up all databases.") 
292     databases = property(_getDatabases, _setDatabases, None, "List of databases to back up.") 
293   
300   
301     """ 
302     Class representing this extension's configuration document. 
303   
304     This is not a general-purpose configuration object like the main Cedar 
305     Backup configuration object.  Instead, it just knows how to parse and emit 
306     MySQL-specific configuration values.  Third parties who need to read and 
307     write configuration related to this extension should access it through the 
308     constructor, C{validate} and C{addConfig} methods. 
309   
310     @note: Lists within this class are "unordered" for equality comparisons. 
311   
312     @sort: __init__, __repr__, __str__, __cmp__, mysql, validate, addConfig 
313     """ 
314   
315 -   def __init__(self, xmlData=None, xmlPath=None, validate=True): 
 316        """ 
317        Initializes a configuration object. 
318   
319        If you initialize the object without passing either C{xmlData} or 
320        C{xmlPath} then configuration will be empty and will be invalid until it 
321        is filled in properly. 
322   
323        No reference to the original XML data or original path is saved off by 
324        this class.  Once the data has been parsed (successfully or not) this 
325        original information is discarded. 
326   
327        Unless the C{validate} argument is C{False}, the L{LocalConfig.validate} 
328        method will be called (with its default arguments) against configuration 
329        after successfully parsing any passed-in XML.  Keep in mind that even if 
330        C{validate} is C{False}, it might not be possible to parse the passed-in 
331        XML document if lower-level validations fail. 
332   
333        @note: It is strongly suggested that the C{validate} option always be set 
334        to C{True} (the default) unless there is a specific need to read in 
335        invalid configuration from disk. 
336   
337        @param xmlData: XML data representing configuration. 
338        @type xmlData: String data. 
339   
340        @param xmlPath: Path to an XML file on disk. 
341        @type xmlPath: Absolute path to a file on disk. 
342   
343        @param validate: Validate the document after parsing it. 
344        @type validate: Boolean true/false. 
345   
346        @raise ValueError: If both C{xmlData} and C{xmlPath} are passed-in. 
347        @raise ValueError: If the XML data in C{xmlData} or C{xmlPath} cannot be parsed. 
348        @raise ValueError: If the parsed configuration document is not valid. 
349        """ 
350        self._mysql = None 
351        self.mysql = None 
352        if xmlData is not None and xmlPath is not None: 
353           raise ValueError("Use either xmlData or xmlPath, but not both.") 
354        if xmlData is not None: 
355           self._parseXmlData(xmlData) 
356           if validate: 
357              self.validate() 
358        elif xmlPath is not None: 
359           xmlData = open(xmlPath).read() 
360           self._parseXmlData(xmlData) 
361           if validate: 
362              self.validate() 
 363   
365        """ 
366        Official string representation for class instance. 
367        """ 
368        return "LocalConfig(%s)" % (self.mysql) 
 369   
371        """ 
372        Informal string representation for class instance. 
373        """ 
374        return self.__repr__() 
 375   
377        """ 
378        Definition of equals operator for this class. 
379        Lists within this class are "unordered" for equality comparisons. 
380        @param other: Other object to compare to. 
381        @return: -1/0/1 depending on whether self is C{<}, C{=} or C{>} other. 
382        """ 
383        if other is None: 
384           return 1 
385        if self.mysql != other.mysql: 
386           if self.mysql < other.mysql: 
387              return -1 
388           else: 
389              return 1 
390        return 0 
 391   
393        """ 
394        Property target used to set the mysql configuration value. 
395        If not C{None}, the value must be a C{MysqlConfig} object. 
396        @raise ValueError: If the value is not a C{MysqlConfig} 
397        """ 
398        if value is None: 
399           self._mysql = None 
400        else: 
401           if not isinstance(value, MysqlConfig): 
402              raise ValueError("Value must be a C{MysqlConfig} object.") 
403           self._mysql = value 
 404   
406        """ 
407        Property target used to get the mysql configuration value. 
408        """ 
409        return self._mysql 
 410   
411     mysql = property(_getMysql, _setMysql, None, "Mysql configuration in terms of a C{MysqlConfig} object.") 
412   
414        """ 
415        Validates configuration represented by the object. 
416   
417        The compress mode must be filled in.  Then, if the 'all' flag I{is} set, 
418        no databases are allowed, and if the 'all' flag is I{not} set, at least 
419        one database is required. 
420   
421        @raise ValueError: If one of the validations fails. 
422        """ 
423        if self.mysql is None: 
424           raise ValueError("Mysql section is required.") 
425        if self.mysql.compressMode is None: 
426           raise ValueError("Compress mode value is required.") 
427        if self.mysql.all: 
428           if self.mysql.databases is not None and self.mysql.databases != []: 
429              raise ValueError("Databases cannot be specified if 'all' flag is set.") 
430        else: 
431           if self.mysql.databases is None or len(self.mysql.databases) < 1: 
432              raise ValueError("At least one MySQL database must be indicated if 'all' flag is not set.") 
 433   
435        """ 
436        Adds a <mysql> configuration section as the next child of a parent. 
437   
438        Third parties should use this function to write configuration related to 
439        this extension. 
440   
441        We add the following fields to the document:: 
442   
443           user           //cb_config/mysql/user 
444           password       //cb_config/mysql/password 
445           compressMode   //cb_config/mysql/compress_mode 
446           all            //cb_config/mysql/all 
447   
448        We also add groups of the following items, one list element per 
449        item:: 
450   
451           database       //cb_config/mysql/database 
452   
453        @param xmlDom: DOM tree as from C{impl.createDocument()}. 
454        @param parentNode: Parent that the section should be appended to. 
455        """ 
456        if self.mysql is not None: 
457           sectionNode = addContainerNode(xmlDom, parentNode, "mysql") 
458           addStringNode(xmlDom, sectionNode, "user", self.mysql.user) 
459           addStringNode(xmlDom, sectionNode, "password", self.mysql.password) 
460           addStringNode(xmlDom, sectionNode, "compress_mode", self.mysql.compressMode) 
461           addBooleanNode(xmlDom, sectionNode, "all", self.mysql.all) 
462           if self.mysql.databases is not None: 
463              for database in self.mysql.databases: 
464                 addStringNode(xmlDom, sectionNode, "database", database) 
 465   
467        """ 
468        Internal method to parse an XML string into the object. 
469   
470        This method parses the XML document into a DOM tree (C{xmlDom}) and then 
471        calls a static method to parse the mysql configuration section. 
472   
473        @param xmlData: XML data to be parsed 
474        @type xmlData: String data 
475   
476        @raise ValueError: If the XML cannot be successfully parsed. 
477        """ 
478        (xmlDom, parentNode) = createInputDom(xmlData) 
479        self._mysql = LocalConfig._parseMysql(parentNode) 
 480   
481     @staticmethod 
483        """ 
484        Parses a mysql configuration section. 
485   
486        We read the following fields:: 
487   
488           user           //cb_config/mysql/user 
489           password       //cb_config/mysql/password 
490           compressMode   //cb_config/mysql/compress_mode 
491           all            //cb_config/mysql/all 
492   
493        We also read groups of the following item, one list element per 
494        item:: 
495   
496           databases      //cb_config/mysql/database 
497   
498        @param parentNode: Parent node to search beneath. 
499   
500        @return: C{MysqlConfig} object or C{None} if the section does not exist. 
501        @raise ValueError: If some filled-in value is invalid. 
502        """ 
503        mysql = None 
504        section = readFirstChild(parentNode, "mysql") 
505        if section is not None: 
506           mysql = MysqlConfig() 
507           mysql.user = readString(section, "user") 
508           mysql.password = readString(section, "password") 
509           mysql.compressMode = readString(section, "compress_mode") 
510           mysql.all = readBoolean(section, "all") 
511           mysql.databases = readStringList(section, "database") 
512        return mysql 
  513   
514   
515   
516   
517   
518   
519   
520   
521   
522   
523   
524 -def executeAction(configPath, options, config): 
 525     """ 
526     Executes the MySQL backup action. 
527   
528     @param configPath: Path to configuration file on disk. 
529     @type configPath: String representing a path on disk. 
530   
531     @param options: Program command-line options. 
532     @type options: Options object. 
533   
534     @param config: Program configuration. 
535     @type config: Config object. 
536   
537     @raise ValueError: Under many generic error conditions 
538     @raise IOError: If a backup could not be written for some reason. 
539     """ 
540     logger.debug("Executing MySQL extended action.") 
541     if config.options is None or config.collect is None: 
542        raise ValueError("Cedar Backup configuration is not properly filled in.") 
543     local = LocalConfig(xmlPath=configPath) 
544     if local.mysql.all: 
545        logger.info("Backing up all databases.") 
546        _backupDatabase(config.collect.targetDir, local.mysql.compressMode, local.mysql.user, local.mysql.password, 
547                        config.options.backupUser, config.options.backupGroup, None) 
548     else: 
549        logger.debug("Backing up %d individual databases.", len(local.mysql.databases)) 
550        for database in local.mysql.databases: 
551           logger.info("Backing up database [%s].", database) 
552           _backupDatabase(config.collect.targetDir, local.mysql.compressMode, local.mysql.user, local.mysql.password, 
553                           config.options.backupUser, config.options.backupGroup, database) 
554     logger.info("Executed the MySQL extended action successfully.") 
 555   
556 -def _backupDatabase(targetDir, compressMode, user, password, backupUser, backupGroup, database=None): 
 557     """ 
558     Backs up an individual MySQL database, or all databases. 
559   
560     This internal method wraps the public method and adds some functionality, 
561     like figuring out a filename, etc. 
562   
563     @param targetDir:  Directory into which backups should be written. 
564     @param compressMode: Compress mode to be used for backed-up files. 
565     @param user: User to use for connecting to the database (if any). 
566     @param password: Password associated with user (if any). 
567     @param backupUser: User to own resulting file. 
568     @param backupGroup: Group to own resulting file. 
569     @param database: Name of database, or C{None} for all databases. 
570   
571     @return: Name of the generated backup file. 
572   
573     @raise ValueError: If some value is missing or invalid. 
574     @raise IOError: If there is a problem executing the MySQL dump. 
575     """ 
576     (outputFile, filename) = _getOutputFile(targetDir, database, compressMode) 
577     try: 
578        backupDatabase(user, password, outputFile, database) 
579     finally: 
580        outputFile.close() 
581     if not os.path.exists(filename): 
582        raise IOError("Dump file [%s] does not seem to exist after backup completed." % filename) 
583     changeOwnership(filename, backupUser, backupGroup) 
 584   
586     """ 
587     Opens the output file used for saving the MySQL dump. 
588   
589     The filename is either C{"mysqldump.txt"} or C{"mysqldump-<database>.txt"}.  The 
590     C{".bz2"} extension is added if C{compress} is C{True}. 
591   
592     @param targetDir: Target directory to write file in. 
593     @param database: Name of the database (if any) 
594     @param compressMode: Compress mode to be used for backed-up files. 
595   
596     @return: Tuple of (Output file object, filename) 
597     """ 
598     if database is None: 
599        filename = os.path.join(targetDir, "mysqldump.txt") 
600     else: 
601        filename = os.path.join(targetDir, "mysqldump-%s.txt" % database) 
602     if compressMode == "gzip": 
603        filename = "%s.gz" % filename 
604        outputFile = GzipFile(filename, "w") 
605     elif compressMode == "bzip2": 
606        filename = "%s.bz2" % filename 
607        outputFile = BZ2File(filename, "w") 
608     else: 
609        outputFile = open(filename, "w") 
610     logger.debug("MySQL dump file will be [%s].", filename) 
611     return (outputFile, filename) 
 612   
613   
614   
615   
616   
617   
618 -def backupDatabase(user, password, backupFile, database=None): 
 619     """ 
620     Backs up an individual MySQL database, or all databases. 
621   
622     This function backs up either a named local MySQL database or all local 
623     MySQL databases, using the passed-in user and password (if provided) for 
624     connectivity.  This function call I{always} results a full backup.  There is 
625     no facility for incremental backups. 
626   
627     The backup data will be written into the passed-in backup file.  Normally, 
628     this would be an object as returned from C{open()}, but it is possible to 
629     use something like a C{GzipFile} to write compressed output.  The caller is 
630     responsible for closing the passed-in backup file. 
631   
632     Often, the "root" database user will be used when backing up all databases. 
633     An alternative is to create a separate MySQL "backup" user and grant that 
634     user rights to read (but not write) all of the databases that will be backed 
635     up. 
636   
637     This function accepts a username and password.  However, you probably do not 
638     want to pass those values in.  This is because they will be provided to 
639     C{mysqldump} via the command-line C{--user} and C{--password} switches, 
640     which will be visible to other users in the process listing. 
641   
642     Instead, you should configure the username and password in one of MySQL's 
643     configuration files.  Typically, this would be done by putting a stanza like 
644     this in C{/root/.my.cnf}, to provide C{mysqldump} with the root database 
645     username and its password:: 
646   
647        [mysqldump] 
648        user     = root 
649        password = <secret> 
650   
651     If you are executing this function as some system user other than root, then 
652     the C{.my.cnf} file would be placed in the home directory of that user.  In 
653     either case, make sure to set restrictive permissions (typically, mode 
654     C{0600}) on C{.my.cnf} to make sure that other users cannot read the file. 
655   
656     @param user: User to use for connecting to the database (if any) 
657     @type user: String representing MySQL username, or C{None} 
658   
659     @param password: Password associated with user (if any) 
660     @type password: String representing MySQL password, or C{None} 
661   
662     @param backupFile: File use for writing backup. 
663     @type backupFile: Python file object as from C{open()} or C{file()}. 
664   
665     @param database: Name of the database to be backed up. 
666     @type database: String representing database name, or C{None} for all databases. 
667   
668     @raise ValueError: If some value is missing or invalid. 
669     @raise IOError: If there is a problem executing the MySQL dump. 
670     """ 
671     args = [ "-all", "--flush-logs", "--opt", ] 
672     if user is not None: 
673        logger.warn("Warning: MySQL username will be visible in process listing (consider using ~/.my.cnf).") 
674        args.append("--user=%s" % user) 
675     if password is not None: 
676        logger.warn("Warning: MySQL password will be visible in process listing (consider using ~/.my.cnf).") 
677        args.append("--password=%s" % password) 
678     if database is None: 
679        args.insert(0, "--all-databases") 
680     else: 
681        args.insert(0, "--databases") 
682        args.append(database) 
683     command = resolveCommand(MYSQLDUMP_COMMAND) 
684     result = executeCommand(command, args, returnOutput=False, ignoreStderr=True, doNotLog=True, outputFile=backupFile)[0] 
685     if result != 0: 
686        if database is None: 
687           raise IOError("Error [%d] executing MySQL database dump for all databases." % result) 
688        else: 
689           raise IOError("Error [%d] executing MySQL database dump for database [%s]." % (result, database)) 
 690