Sep  8 14:11:27 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: fatal: invalid metadata block
Sep  8 14:11:27 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1:   bh = 312738524 (magic number)
Sep  8 14:11:27 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line
 = 334
Sep  8 14:11:27 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: about to withdraw this file system
Sep  8 14:11:27 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: telling LM to withdraw
Sep  8 14:11:28 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: jid=1: Trying to acquire journal lock...
Sep  8 14:11:28 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: jid=1: Trying to acquire journal lock...
Sep  8 14:11:28 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: jid=1: Trying to acquire journal lock...
Sep  8 14:11:28 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: jid=1: Busy
Sep  8 14:11:28 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: jid=1: Busy
Sep  8 14:11:28 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: jid=1: Busy
Sep  8 14:11:28 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: withdrawn
Sep  8 14:11:28 192.168.100.105 kernel: 
Sep  8 14:11:28 192.168.100.105 kernel: Call Trace:
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff885084ce>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80262907>] __wait_on_bit+0x60/0x6e
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80215788>] sync_buffer+0x0/0x3f
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80262981>] out_of_line_wait_on_bit+0x6c/0x78
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8029a01a>] wake_bit_function+0x0/0x23
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8021a7f1>] submit_bh+0x10a/0x111
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8851b4a7>] :gfs2:gfs2_meta_check_ii+0x2c/0x38
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850bd30>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x160
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff884fcfc3>] :gfs2:gfs2_block_map+0x1dc/0x33e
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8021a821>] poll_freewait+0x29/0x6a
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff884fd199>] :gfs2:gfs2_extent_map+0x74/0xac
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff884fd2ce>] :gfs2:gfs2_write_alloc_required+0xfd/0x122
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff885058d5>] :gfs2:gfs2_glock_nq+0x248/0x273
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850d27c>] :gfs2:gfs2_write_begin+0x99/0x36a
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850ed1b>] :gfs2:gfs2_file_buffered_write+0x14b/0x2e5
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8020d3a5>] file_read_actor+0x0/0xfc
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850f151>] :gfs2:__gfs2_file_aio_write_nolock+0x29c/0x2d4
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850f2f4>] :gfs2:gfs2_file_write_nolock+0xaa/0x10f
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8022eca0>] __wake_up+0x38/0x4f
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80299fec>] autoremove_wake_function+0x0/0x2e
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8022fbe4>] pipe_readv+0x38e/0x3a2
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80263bce>] lock_kernel+0x1b/0x32
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8850f444>] :gfs2:gfs2_file_write+0x49/0xa7
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff80216da9>] vfs_write+0xce/0x174
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff802175e1>] sys_write+0x45/0x6e
Sep  8 14:11:28 192.168.100.105 kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6
Sep  8 14:11:28 192.168.100.105 kernel: 
.......
Sep  8 14:11:35 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: fatal: filesystem consistency error
Sep  8 14:11:35 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0:   function = trans_go_xmote_bh, file = fs/gfs2/glops.c, line = 401
Sep  8 14:11:35 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: about to withdraw this file system
Sep  8 14:11:35 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: telling LM to withdraw
......
Sep  8 15:54:46 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: fatal: invalid metadata block
Sep  8 15:54:46 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1:   bh = 567447924 (magic number)
Sep  8 15:54:46 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line
 = 334
Sep  8 15:54:46 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: about to withdraw this file system
Sep  8 15:54:46 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: telling LM to withdraw
Sep  8 15:54:47 192.168.100.103 kernel: GFS2: fsid=xencluster1:xenclusterfs1.1: withdrawn
Sep  8 15:54:47 192.168.100.103 kernel: 
Sep  8 15:54:47 192.168.100.103 kernel: Call Trace:
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff884da4ce>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff80262907>] __wait_on_bit+0x60/0x6e
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff80215788>] sync_buffer+0x0/0x3f
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff884d7b07>] :gfs2:glock_work_func+0x0/0xb8
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff80262981>] out_of_line_wait_on_bit+0x6c/0x78
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff8029a01a>] wake_bit_function+0x0/0x23
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff8021a7f1>] submit_bh+0x10a/0x111
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff884ed4a7>] :gfs2:gfs2_meta_check_ii+0x2c/0x38
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff884ddd30>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x160
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff802855c8>] __wake_up_common+0x3e/0x68
Sep  8 15:54:47 192.168.100.103 kernel:  [<ffffffff884d8ca7>] :gfs2:gfs2_inode_refresh+0x22/0x2cf
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff884d827d>] :gfs2:inode_go_lock+0x44/0xbe
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff884d5f75>] :gfs2:do_promote+0xad/0x137
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff884d72ec>] :gfs2:finish_xmote+0x28c/0x2b7
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff884d7b24>] :gfs2:glock_work_func+0x1d/0xb8
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff8024ee1e>] run_workqueue+0x94/0xe4
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff8024b727>] worker_thread+0x0/0x122
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff8024b817>] worker_thread+0xf0/0x122
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80286daf>] default_wake_function+0x0/0xe
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80233483>] kthread+0xfe/0x132
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff8025fb2c>] child_rip+0xa/0x12
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff80233385>] kthread+0x0/0x132
Sep  8 15:54:48 192.168.100.103 kernel:  [<ffffffff8025fb22>] child_rip+0x0/0x12
Sep  8 15:54:48 192.168.100.103 kernel:
.....
Sep  8 16:18:15 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: fatal: invalid metadata block 
Sep  8 16:18:15 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3:   bh = 567447924 (magic number) 
Sep  8 16:18:15 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line
 = 334 
Sep  8 16:18:15 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: about to withdraw this file system 
Sep  8 16:18:15 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: telling LM to withdraw 
Sep  8 16:18:17 192.168.100.102 kernel: GFS2: fsid=xencluster1:xenclusterfs1.3: withdrawn 
Sep  8 16:18:17 192.168.100.102 kernel:  
Sep  8 16:18:17 192.168.100.102 kernel: Call Trace: 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8850b4ce>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff80262907>] __wait_on_bit+0x60/0x6e 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff80215788>] sync_buffer+0x0/0x3f 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff80262981>] out_of_line_wait_on_bit+0x6c/0x78 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8029a01a>] wake_bit_function+0x0/0x23 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8021a7f1>] submit_bh+0x10a/0x111 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8851e4a7>] :gfs2:gfs2_meta_check_ii+0x2c/0x38 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8850ed30>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x160 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff88516488>] :gfs2:gfs2_permission+0xaf/0xd3 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff88502f35>] :gfs2:gfs2_dir_read+0x403/0x535 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8022590c>] filldir+0x0/0xb7 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8022d2e9>] wake_up_bit+0x11/0x22 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8022d2e9>] wake_up_bit+0x11/0x22 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff88506fbd>] :gfs2:do_promote+0xf5/0x137 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff88508561>] :gfs2:run_queue+0xad/0x17c 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8022590c>] filldir+0x0/0xb7 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff88512977>] :gfs2:gfs2_readdir+0x6b/0x8f 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8851294c>] :gfs2:gfs2_readdir+0x40/0x8f 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8022590c>] filldir+0x0/0xb7 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff80235fc7>] vfs_readdir+0x77/0xa9 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff802397ee>] sys_getdents+0x75/0xbd 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8025f295>] tracesys+0x47/0xb6 
Sep  8 16:18:17 192.168.100.102 kernel:  [<ffffffff8025f2f9>] tracesys+0xab/0xb6 
Sep  8 16:18:17 192.168.100.102 kernel:  
.....
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4: fatal: invalid metadata block 
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4:   bh = 567447924 (magic number) 
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4:   function = gfs2_meta_indirect_buffer, file = fs/gfs2/meta_io.c, line
 = 334 
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4: about to withdraw this file system 
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4: telling LM to withdraw 
Sep  8 16:42:12 192.168.100.105 kernel: GFS2: fsid=xencluster1:xenclusterfs1.4: withdrawn 
Sep  8 16:42:12 192.168.100.105 kernel:  
Sep  8 16:42:12 192.168.100.105 kernel: Call Trace: 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff885044ce>] :gfs2:gfs2_lm_withdraw+0xc1/0xd0 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80262907>] __wait_on_bit+0x60/0x6e 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80215788>] sync_buffer+0x0/0x3f 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff88501b07>] :gfs2:glock_work_func+0x0/0xb8 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80262981>] out_of_line_wait_on_bit+0x6c/0x78 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8029a01a>] wake_bit_function+0x0/0x23 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8021a7f1>] submit_bh+0x10a/0x111 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff885174a7>] :gfs2:gfs2_meta_check_ii+0x2c/0x38 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff88507d30>] :gfs2:gfs2_meta_indirect_buffer+0x104/0x160 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff802855c8>] __wake_up_common+0x3e/0x68 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff88502ca7>] :gfs2:gfs2_inode_refresh+0x22/0x2cf 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8850227d>] :gfs2:inode_go_lock+0x44/0xbe 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff884fff75>] :gfs2:do_promote+0xad/0x137 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff885012ec>] :gfs2:finish_xmote+0x28c/0x2b7 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff88501b24>] :gfs2:glock_work_func+0x1d/0xb8 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8024ee1e>] run_workqueue+0x94/0xe4 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8024b727>] worker_thread+0x0/0x122 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8024b817>] worker_thread+0xf0/0x122 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80286daf>] default_wake_function+0x0/0xe 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80233483>] kthread+0xfe/0x132 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8025fb2c>] child_rip+0xa/0x12 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80299dd4>] keventd_create_kthread+0x0/0xc4 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8026df02>] monotonic_clock+0x35/0x7b 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff80233385>] kthread+0x0/0x132 
Sep  8 16:42:12 192.168.100.105 kernel:  [<ffffffff8025fb22>] child_rip+0x0/0x12 
Sep  8 16:42:12 192.168.100.105 kernel:  
......
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: jid=4: Replaying journal... 
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: fatal: invalid metadata block 
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2:   bh = 143515 (type: exp=9, found=4) 
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2:   function = foreach_descriptor, file = fs/gfs2/recovery.c, line = 350
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: about to withdraw this file system 
Sep  8 16:42:13 192.168.100.101 kernel: GFS2: fsid=xencluster1:xenclusterfs1.2: telling LM to withdraw 
.....
Sep  8 16:54:22 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: jid=4: Trying to acquire journal lock... 
Sep  8 16:54:22 192.168.100.104 kernel: GFS2: fsid=xencluster1:xenclusterfs1.0: jid=4: Busy