So I had a customer come to me asking me to check his domain.
I see the error establishing connection message. SSH in and I see this:
Now, the log itself says:
Obviously, something is very wrong here... and it may be because I just barely woke up, but I'm not seeing anything immediately obvious besides this bit which doesn't tell me anything useful:
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2 3610099732
150906 12:14:05 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54$
InnoDB: Apply batch completed
What exactly caused the SQL server to die?
Edit: Also, already optimized and repaired tables, but I'm still working on investigating exactly what caused the server to die since there doesn't seem to be anything obviously wrong with the server.
I see the error establishing connection message. SSH in and I see this:
Quote:
Originally Posted by server
[root@daedalus ~]# service mysqld statusmysqld dead but subsys locked
[root@daedalus ~]# service mysqld stop
Stopping mysqld: [ OK ]
[root@daedalus ~]# service mysqld start
Starting mysqld: [ OK ]
[root@daedalus ~]# service mysqld stop
Stopping mysqld: [ OK ]
[root@daedalus ~]# service mysqld start
Starting mysqld: [ OK ]
Quote:
Originally Posted by mysqld.log
150906 03:46:48 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150906 12:14:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150906 12:14:05 InnoDB: Initializing buffer pool, size = 8.0M
150906 12:14:05 InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 2 3610098884
150906 12:14:05 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2 3610099732
150906 12:14:05 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54$
InnoDB: Apply batch completed
150906 12:14:06 InnoDB: Started; log sequence number 2 3610099732
150906 12:14:06 [Note] Event Scheduler: Loaded 0 events
150906 12:14:06 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150906 12:25:02 [Note] /usr/libexec/mysqld: Normal shutdown
150906 12:25:02 [Note] Event Scheduler: Purging the queue. 0 events
150906 12:25:02 InnoDB: Starting shutdown...
150906 12:25:02 InnoDB: Shutdown completed; log sequence number 2 3610664100
150906 12:25:02 [Note] /usr/libexec/mysqld: Shutdown complete
150906 12:25:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150906 12:25:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150906 12:25:03 InnoDB: Initializing buffer pool, size = 8.0M
150906 12:25:03 InnoDB: Completed initialization of buffer pool
150906 12:25:03 InnoDB: Started; log sequence number 2 3610664100
150906 12:25:03 [Note] Event Scheduler: Loaded 0 events
150906 12:25:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150906 12:14:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150906 12:14:05 InnoDB: Initializing buffer pool, size = 8.0M
150906 12:14:05 InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 2 3610098884
150906 12:14:05 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2 3610099732
150906 12:14:05 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54$
InnoDB: Apply batch completed
150906 12:14:06 InnoDB: Started; log sequence number 2 3610099732
150906 12:14:06 [Note] Event Scheduler: Loaded 0 events
150906 12:14:06 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150906 12:25:02 [Note] /usr/libexec/mysqld: Normal shutdown
150906 12:25:02 [Note] Event Scheduler: Purging the queue. 0 events
150906 12:25:02 InnoDB: Starting shutdown...
150906 12:25:02 InnoDB: Shutdown completed; log sequence number 2 3610664100
150906 12:25:02 [Note] /usr/libexec/mysqld: Shutdown complete
150906 12:25:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150906 12:25:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150906 12:25:03 InnoDB: Initializing buffer pool, size = 8.0M
150906 12:25:03 InnoDB: Completed initialization of buffer pool
150906 12:25:03 InnoDB: Started; log sequence number 2 3610664100
150906 12:25:03 [Note] Event Scheduler: Loaded 0 events
150906 12:25:03 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
Quote:
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 2 3610099732
150906 12:14:05 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54$
InnoDB: Apply batch completed
Edit: Also, already optimized and repaired tables, but I'm still working on investigating exactly what caused the server to die since there doesn't seem to be anything obviously wrong with the server.