#ceph IRC Log

Index

IRC Log for 2013-03-25

Timestamps are in GMT/BST.

[0:09] * dxd828 (~dxd828@host-80-41-47-195.as13285.net) Quit (Quit: Textual IRC Client: www.textualapp.com)
[0:09] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[0:16] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Quit: Leaving.)
[0:16] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) has joined #ceph
[0:22] * The_Bishop (~bishop@2001:470:50b6:0:145d:b04b:12e9:8807) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[0:23] * tnt_ (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[0:23] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Remote host closed the connection)
[0:24] * tnt (~tnt@82.195-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[0:26] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[0:27] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[0:28] * ssedov (stas@ssh.deglitch.com) has joined #ceph
[0:30] * stass (stas@ssh.deglitch.com) Quit (Read error: Connection reset by peer)
[0:34] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Ping timeout: 480 seconds)
[0:36] * leseb (~leseb@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[0:52] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (Read error: No route to host)
[0:56] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[0:56] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[0:58] * infernix (nix@cl-1404.ams-04.nl.sixxs.net) Quit (Ping timeout: 480 seconds)
[1:03] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[1:03] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[1:07] * tnt_ (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[1:12] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) has joined #ceph
[1:16] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[1:19] * LeaChim (~LeaChim@b0fae63d.bb.sky.com) Quit (Read error: Operation timed out)
[1:24] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[1:25] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:27] * esammy (~esamuels@host-2-103-103-175.as13285.net) Quit (Quit: esammy)
[1:28] * leseb (~leseb@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Remote host closed the connection)
[1:58] * ssedov (stas@ssh.deglitch.com) Quit (Read error: Connection reset by peer)
[1:59] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[2:02] * stass (stas@ssh.deglitch.com) has joined #ceph
[2:10] * The_Bishop (~bishop@e179007144.adsl.alicedsl.de) has joined #ceph
[2:25] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[3:00] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Quit: themgt)
[3:04] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[3:22] * themgt (~themgt@97-95-235-55.dhcp.sffl.va.charter.com) has joined #ceph
[3:29] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[3:31] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[3:38] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[3:41] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[3:42] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[3:51] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[3:52] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[4:02] * themgt (~themgt@97-95-235-55.dhcp.sffl.va.charter.com) Quit (Quit: themgt)
[4:03] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[4:04] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[4:07] * xiaoxi (~xiaoxiche@134.134.137.71) has joined #ceph
[4:12] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) has joined #ceph
[4:50] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Quit: themgt)
[5:10] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[5:11] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[5:12] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) has joined #ceph
[5:25] <xiaoxi> Hi, I am hitting the same problem for a lot of times.that is "sync entry timeout after 600s".
[5:26] <xiaoxi> Bacically I am doing random write test on top of the RBD, I could see extremly high IOPS (1W+) for 80 spindles.From the ceph code, I assume all the writes are just go to pagecache,than the sync will take extremly long time
[5:26] <xiaoxi> which seems to cause this issue
[5:43] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[5:44] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[5:49] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[5:58] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[6:22] <iggy> xiaoxi: it depends, most benchmarks use o_direct or sync frequently to avoid a multi gig sync
[6:23] <iggy> but you could be hitting the limits of your journal size while syncing
[6:26] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[6:28] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[6:43] * Ludo_ (~Ludo@falbala.zoxx.net) has joined #ceph
[6:43] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * BillK (~BillK@58-7-123-79.dyn.iinet.net.au) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * mauilion (~root@75.98.92.113) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * ivoks (~ivoks@jupiter.init.hr) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * TMM (~hp@535240C7.cm-6-3b.dynamic.ziggo.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * scalability-junk (uid6422@id-6422.tooting.irccloud.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * Svedrin (svedrin@2a01:4f8:100:3061::10) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * jantje (~jan@paranoid.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * fred1 (~fredl@2a00:1a48:7803:107:8532:c238:ff08:354) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * frank9999 (~frank@kantoor.transip.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * hijacker (~hijacker@213.91.163.5) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * mistur (~yoann@kewl.mistur.org) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * dignus (~dignus@bastion.jkit.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * liiwi (liiwi@idle.fi) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * Ludo (~Ludo@falbala.zoxx.net) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * sbadia (~sbadia@yasaw.net) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * thelan (~thelan@paris.servme.fr) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * tjikkun_ (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * wonko_be (bernard@november.openminds.be) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * Ormod (~valtha@ohmu.fi) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * Zethrok (~martin@95.154.26.34) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * HauM1 (~HauM1@login.univie.ac.at) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:43] * Meyer__ (meyer@c64.org) Quit (reticulum.oftc.net solenoid.oftc.net)
[6:45] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[6:45] * BillK (~BillK@58-7-123-79.dyn.iinet.net.au) has joined #ceph
[6:45] * mauilion (~root@75.98.92.113) has joined #ceph
[6:45] * ivoks (~ivoks@jupiter.init.hr) has joined #ceph
[6:45] * TMM (~hp@535240C7.cm-6-3b.dynamic.ziggo.nl) has joined #ceph
[6:45] * scalability-junk (uid6422@id-6422.tooting.irccloud.com) has joined #ceph
[6:45] * Svedrin (svedrin@2a01:4f8:100:3061::10) has joined #ceph
[6:45] * jantje (~jan@paranoid.nl) has joined #ceph
[6:45] * fred1 (~fredl@2a00:1a48:7803:107:8532:c238:ff08:354) has joined #ceph
[6:45] * frank9999 (~frank@kantoor.transip.nl) has joined #ceph
[6:45] * thelan (~thelan@paris.servme.fr) has joined #ceph
[6:45] * hijacker (~hijacker@213.91.163.5) has joined #ceph
[6:45] * mistur (~yoann@kewl.mistur.org) has joined #ceph
[6:45] * dignus (~dignus@bastion.jkit.nl) has joined #ceph
[6:45] * liiwi (liiwi@idle.fi) has joined #ceph
[6:45] * sbadia (~sbadia@yasaw.net) has joined #ceph
[6:45] * tjikkun_ (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[6:45] * wonko_be (bernard@november.openminds.be) has joined #ceph
[6:45] * Ormod (~valtha@ohmu.fi) has joined #ceph
[6:45] * Zethrok (~martin@95.154.26.34) has joined #ceph
[6:45] * HauM1 (~HauM1@login.univie.ac.at) has joined #ceph
[6:45] * Meyer__ (meyer@c64.org) has joined #ceph
[6:47] <- *Svedrin* "I am currently away. Please leave a message, I will see it as soon as I get back."
[6:58] <xiaoxi> iggy:I really have o_direct for workload on top of RBD
[7:00] <xiaoxi> But ceph is a black box, I cannot (even can, is it really a good choice ?) to force ceph use ODirect to write file
[7:01] <xiaoxi> What's more, for random write (4K), you can hardly hit any capability throttle, even a 64M journal can contains 16K IOs..
[7:11] <iggy> for rbd, it's going to respect whatever is using the rbd asks for
[7:16] <xiaoxi> iggy:sorry,not following...
[7:17] <xiaoxi> but I will try if I can reproduce this by using rados bench...seems promising.
[7:31] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:52] * tnt (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:07] * Cube1 (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[8:10] * ScOut3R (~ScOut3R@5401D8E4.dsl.pool.telekom.hu) has joined #ceph
[8:13] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit (Read error: Operation timed out)
[8:16] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[8:16] * Cube1 (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[8:18] * Cube1 (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[8:24] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[8:30] * The_Bishop_ (~bishop@e179004103.adsl.alicedsl.de) has joined #ceph
[8:34] * The_Bishop (~bishop@e179007144.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[8:35] * barryo (~borourke@cumberdale.ph.ed.ac.uk) has joined #ceph
[8:35] * ScOut3R (~ScOut3R@5401D8E4.dsl.pool.telekom.hu) Quit (Ping timeout: 480 seconds)
[8:39] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[8:44] * sleinen (~Adium@2001:620:0:46:c173:dbc8:1745:5052) has joined #ceph
[8:46] * Cube1 (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[8:55] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: IceChat - Keeping PC's cool since 2000)
[8:57] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[8:58] * SvenHome (~scarter@71-209-155-46.phnx.qwest.net) has joined #ceph
[8:58] * SvenHome (~scarter@71-209-155-46.phnx.qwest.net) has left #ceph
[9:08] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit (Quit: Leaving.)
[9:16] * The_Bishop (~bishop@f052097050.adsl.alicedsl.de) has joined #ceph
[9:17] * The_Bishop_ (~bishop@e179004103.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[9:19] * vipr_ (~root@78-23-119-42.access.telenet.be) Quit (Quit: leaving)
[9:20] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:21] * leseb (~leseb@83.167.43.235) has joined #ceph
[9:21] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) has joined #ceph
[9:23] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) has joined #ceph
[9:24] * dosaboy (~gizmo@faun.canonical.com) has joined #ceph
[9:25] * vipr (~root@78-23-119-42.access.telenet.be) Quit (Quit: leaving)
[9:26] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:26] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:27] * Cube (~Cube@cpe-76-95-217-215.socal.res.rr.com) Quit ()
[9:28] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:31] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:31] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:32] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:33] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:33] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:33] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:33] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:34] * tnt (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:35] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:36] * themgt (~themgt@24-177-232-181.dhcp.gnvl.sc.charter.com) Quit (Quit: Pogoapp - http://www.pogoapp.com)
[9:37] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:37] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[9:37] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:37] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) has joined #ceph
[9:39] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:40] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[9:41] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:41] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[9:42] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:43] * vipr (~root@78-23-119-42.access.telenet.be) Quit ()
[9:44] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:45] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[9:53] * sivanov (~sivanov@gw2.maxtelecom.bg) has joined #ceph
[10:00] * jamespage (~jamespage@culvain.gromper.net) Quit (Quit: Coyote finally caught me)
[10:00] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[10:02] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:04] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:04] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:07] * xiaoxi (~xiaoxiche@134.134.137.71) Quit (Ping timeout: 480 seconds)
[10:08] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:12] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:13] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[10:14] * LeaChim (~LeaChim@b0fae63d.bb.sky.com) has joined #ceph
[10:16] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[10:25] * stacker666 (~stacker66@206.pool85-61-191.dynamic.orange.es) Quit (Quit: Saliendo)
[10:34] * vipr (~root@78-23-119-42.access.telenet.be) Quit (Quit: leaving)
[10:35] * vipr (~root@78-23-119-42.access.telenet.be) has joined #ceph
[10:38] * MK_FG (~MK_FG@00018720.user.oftc.net) Quit (Remote host closed the connection)
[10:40] * leseb (~leseb@83.167.43.235) Quit (Remote host closed the connection)
[10:41] * MK_FG (~MK_FG@00018720.user.oftc.net) has joined #ceph
[10:42] * leseb (~leseb@83.167.43.235) has joined #ceph
[10:43] * The_Bishop_ (~bishop@e179011102.adsl.alicedsl.de) has joined #ceph
[10:44] * l0nk (~alex@83.167.43.235) Quit (Read error: Operation timed out)
[10:46] * jtangwk (~Adium@2001:770:10:500:499c:2ad8:78dd:2edb) has joined #ceph
[10:49] * The_Bishop (~bishop@f052097050.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[10:49] * The_Bishop__ (~bishop@e179011085.adsl.alicedsl.de) has joined #ceph
[10:55] * The_Bishop_ (~bishop@e179011102.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[11:00] * l0nk (~alex@83.167.43.235) has joined #ceph
[11:10] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[11:11] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[11:11] * esammy (~esamuels@host-2-103-103-175.as13285.net) has joined #ceph
[11:11] * mcclurmc (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[11:37] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Quit: No Ping reply in 180 seconds.)
[11:38] * JohansGlock (~quassel@kantoor.transip.nl) has joined #ceph
[12:02] * mcclurmc (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[12:04] * jtangwk (~Adium@2001:770:10:500:499c:2ad8:78dd:2edb) Quit (Quit: Leaving.)
[12:05] * leseb (~leseb@83.167.43.235) Quit (Remote host closed the connection)
[12:11] * l0nk (~alex@83.167.43.235) Quit (Ping timeout: 480 seconds)
[12:13] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[12:36] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[12:39] * leseb (~leseb@83.167.43.235) has joined #ceph
[12:43] * The_Bishop_ (~bishop@e179014156.adsl.alicedsl.de) has joined #ceph
[12:44] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[12:48] * The_Bishop__ (~bishop@e179011085.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[12:49] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[12:51] * markbby (~Adium@168.94.245.6) has joined #ceph
[12:51] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[13:19] <sivanov> Hello all
[13:19] <sivanov> I have a problem with "rbd"
[13:20] <sivanov> When i try to map image someting like "rbd map testimage --pool test"
[13:21] <sivanov> server is going down
[13:21] <sivanov> any ideas?
[13:21] <sivanov> thanks in advance
[13:27] * l0nk (~alex@83.167.43.235) has joined #ceph
[13:41] * jtangwk (~Adium@2001:770:10:500:499c:2ad8:78dd:2edb) has joined #ceph
[13:49] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) has joined #ceph
[13:55] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[13:55] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[13:57] <Gugge-47527> sivanov: "server is going down"
[13:57] <Gugge-47527> sivanov: can you explain that?
[13:57] * Kioob`Taff (~plug-oliv@local.plusdinfo.com) has joined #ceph
[13:58] <sivanov> I mean : server is with kernel panic and going to shut down
[13:58] <Kioob`Taff> Hi
[13:59] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[13:59] <Kioob`Taff> is there a way do *NOT* delete data asynchronously ? If in a script I delete 100 snapshot of different RBD images (backup rotation), all deletes are run simultaneously, which terribly slow down the cluster
[13:59] <Gugge-47527> sivanov: what distro are you using, and what kernel version?
[14:00] * wschulze (~wschulze@cpe-69-203-80-81.nyc.res.rr.com) has joined #ceph
[14:01] <sivanov> 3.5.0-23-generic Ubuntu
[14:01] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[14:02] * mcclurmc (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[14:03] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:04] * allsystemsarego (~allsystem@188.27.165.190) has joined #ceph
[14:12] * yehuda_hm (~yehuda@2602:306:330b:1410:5cf:6274:b951:9c4b) Quit (Ping timeout: 480 seconds)
[14:12] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Read error: Connection reset by peer)
[14:16] * leseb (~leseb@83.167.43.235) Quit (Remote host closed the connection)
[14:16] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[14:17] <Gugge-47527> sivanov: do you run any of the ceph daemons on the server you are trying to map the rbd on?
[14:18] * xiaoxi (~xiaoxiche@134.134.139.74) has joined #ceph
[14:19] <xiaoxi> hi
[14:20] <joao> howdy
[14:21] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[14:21] * yehuda_hm (~yehuda@2602:306:330b:1410:7cf2:d464:7dd6:4b6) has joined #ceph
[14:21] <sivanov> yes deamons are up
[14:21] <sivanov> without any errors
[14:22] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[14:22] <iggy> that wasn't the question
[14:25] <sivanov> ohh sorry
[14:25] <sivanov> yes daemons are on the same machine
[14:26] <iggy> bad
[14:27] <sivanov> why?
[14:27] <iggy> because?
[14:29] * The_Bishop__ (~bishop@e177088108.adsl.alicedsl.de) has joined #ceph
[14:29] * The_Bishop_ (~bishop@e179014156.adsl.alicedsl.de) Quit (Read error: Connection reset by peer)
[14:31] <iggy> it's an age old problem... kernel modules and user space allocating memory, etc
[14:32] <absynth> i think the issue is the kernel
[14:32] <absynth> you will want to upgrade to 3.7+
[14:32] <absynth> sivanov: you, in that case
[14:33] <xiaoxi> joao:have you seen my mail in the ML? what's your idea about it ? is it a known issue?
[14:33] <joao> haven't seen it; thunderbird must be acting out again
[14:33] * wer_ (~wer@206-248-239-142.unassigned.ntelos.net) Quit (Read error: No route to host)
[14:33] <joao> let me restart it
[14:34] * wer_ (~wer@206-248-239-142.unassigned.ntelos.net) has joined #ceph
[14:34] <absynth> [ceph-users] Ceph Crach at sync_thread_timeout after heavy random writes.
[14:34] <absynth> that one?
[14:34] <xiaoxi> yes
[14:34] <absynth> just a clarification for that one
[14:34] <absynth> Bug #3737 (we reported it) happens with rbd_cache turned on
[14:34] * drokita (~drokita@199.255.228.128) has joined #ceph
[14:34] <absynth> you mentioned qemu cache
[14:34] <joao> does that happen on the osd or the monitor?
[14:35] <joao> osd I presume
[14:35] <xiaoxi> osd
[14:35] <xiaoxi> I think maybe I need to rewrite a mail... I wrote that one in hurry to catch up suttle for home :)
[14:35] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[14:36] <xiaoxi> absynth:I didn't turn on qemu cache and I am sure it's not 3737
[14:36] <absynth> xiaoxi: qemu cache != rbd cache
[14:36] <absynth> that is my point
[14:37] <absynth> from what you describe, it's pretty surely not 3737
[14:38] <joao> xiaoxi, I don't really know what might be happening there, but maybe someone else might when they come in later today
[14:38] <joao> I would look further into it, but am caught up in-between two mon bugs that I should sort out asap
[14:39] <xiaoxi> joao:OK, thanks a lot :) Monitor refactor is a big stuff :)
[14:40] <joao> yeah, some corner case bugs have been popping up
[14:41] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[14:41] * wer_ (~wer@206-248-239-142.unassigned.ntelos.net) Quit (Read error: Connection reset by peer)
[14:41] * wer_ (~wer@206-248-239-142.unassigned.ntelos.net) has joined #ceph
[14:45] * leseb (~leseb@83.167.43.235) has joined #ceph
[14:45] <sivanov> okay many thanks
[14:46] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[14:47] * esammy (~esamuels@host-2-103-103-175.as13285.net) has left #ceph
[14:49] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) has joined #ceph
[14:51] * esammy (~esamuels@host-2-103-103-175.as13285.net) has joined #ceph
[14:54] * portante (~user@66.187.233.206) Quit (Quit: updates)
[15:01] * BillK (~BillK@58-7-123-79.dyn.iinet.net.au) Quit (Read error: Connection reset by peer)
[15:03] * calebamiles (~caleb@c-107-3-1-145.hsd1.vt.comcast.net) has joined #ceph
[15:07] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[15:09] * portante (~user@66.187.233.206) has joined #ceph
[15:22] <joao> Karcaw, around?
[15:27] * leseb (~leseb@83.167.43.235) Quit (Remote host closed the connection)
[15:33] * loicd (~loic@c-76-119-91-36.hsd1.ma.comcast.net) Quit (Quit: Leaving.)
[15:37] * mikedawson_ (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:41] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[15:41] * mikedawson_ is now known as mikedawson
[15:48] * BManojlovic (~steki@91.195.39.5) Quit (Read error: Operation timed out)
[15:49] * BManojlovic (~steki@bojanka.net) has joined #ceph
[15:53] * liiwi (liiwi@idle.fi) Quit (Ping timeout: 480 seconds)
[15:53] * liiwi (liiwi@77.86.213.88) has joined #ceph
[15:53] * leseb (~leseb@83.167.43.235) has joined #ceph
[15:54] * loicd (~loic@74-94-156-210-NewEngland.hfc.comcastbusiness.net) has joined #ceph
[16:05] * xiaoxi (~xiaoxiche@134.134.139.74) Quit (Remote host closed the connection)
[16:07] * sagelap (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[16:13] * sivanov (~sivanov@gw2.maxtelecom.bg) Quit (Ping timeout: 480 seconds)
[16:16] * jlogan1 (~Thunderbi@2600:c00:3010:1:51a1:82c7:4e19:4fa) has joined #ceph
[16:19] * sivanov (~sivanov@gw2.maxtelecom.bg) has joined #ceph
[16:21] * sagelap (~sage@2600:1012:b01e:d2b3:5197:11d5:e4d1:8b0a) has joined #ceph
[16:30] * sagelap (~sage@2600:1012:b01e:d2b3:5197:11d5:e4d1:8b0a) Quit (Ping timeout: 480 seconds)
[16:32] * gregaf1 (~Adium@2607:f298:a:607:dd7f:fab2:e671:5e68) Quit (Quit: Leaving.)
[16:34] * gregaf (~Adium@2607:f298:a:607:121:6248:db58:9b05) has joined #ceph
[16:35] * josef (~seven@li70-116.members.linode.com) has joined #ceph
[16:36] <josef> sagewk: ok just pushed 0.56.3 to epel
[16:36] <josef> i suppose you have a new stable version out by now
[16:39] <absynth> i think .4 or .5 is in the works
[16:39] <absynth> not sure which
[16:42] * vata (~vata@207.34.230.20) has joined #ceph
[16:42] * sagelap (~sage@2600:1012:b015:c29:414a:7a9c:4bc5:ec54) has joined #ceph
[16:43] <yehuda_hm> josef, absynth: 0.56.3 is still the latest
[16:46] <josef> sweet
[16:55] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) Quit (Quit: gerard_dethier)
[16:56] * sagelap (~sage@2600:1012:b015:c29:414a:7a9c:4bc5:ec54) Quit (Quit: Leaving.)
[17:00] * sagelap (~sage@2607:f298:a:607:f10e:c0e:3078:e1c6) has joined #ceph
[17:06] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[17:09] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:13] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:16] * rzerres (~ralf.zerr@pd95b5253.dip0.t-ipconnect.de) has joined #ceph
[17:16] * jskinner (~jskinner@69.170.148.179) Quit (Ping timeout: 481 seconds)
[17:20] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[17:21] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[17:22] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[17:24] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) Quit ()
[17:26] * glowell (~glowell@c-98-210-224-250.hsd1.ca.comcast.net) has joined #ceph
[17:28] <sagewk> josef: haha, its a day or two away. that's awesome, though, thanks!
[17:29] <josef> perfect, gives me a few weeks to forget about it, a few days after taht to fight with the fedora stuff, then two weeks to push it to stable just in time for a new release again
[17:30] * jskinner_ (~jskinner@69.170.148.179) has joined #ceph
[17:30] * jskinner (~jskinner@69.170.148.179) Quit (Read error: Connection reset by peer)
[17:33] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[17:33] * jskinner_ (~jskinner@69.170.148.179) Quit (Read error: Connection reset by peer)
[17:35] * jskinner_ (~jskinner@69.170.148.179) has joined #ceph
[17:35] * jskinner (~jskinner@69.170.148.179) Quit (Read error: Connection reset by peer)
[17:37] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[17:38] * infernix (nix@cl-1404.ams-04.nl.sixxs.net) has joined #ceph
[17:39] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[17:40] * alram (~alram@38.122.20.226) has joined #ceph
[17:43] * jskinner_ (~jskinner@69.170.148.179) Quit (Ping timeout: 480 seconds)
[17:45] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) Quit (Remote host closed the connection)
[17:46] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[17:47] * mauilion_ (~nicira@75.98.92.113) has joined #ceph
[17:48] <mauilion_> anyone really know cephfs?
[17:49] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[17:51] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[17:51] * mjblw1 (~mbaysek@wsip-174-79-34-244.ph.ph.cox.net) Quit (Read error: Connection reset by peer)
[17:51] * mjblw (~mbaysek@wsip-174-79-34-244.ph.ph.cox.net) has joined #ceph
[17:52] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[17:58] <gregaf> yeah, what's up mauilion_?
[18:00] * rturk-away is now known as rturk
[18:04] * BManojlovic (~steki@bojanka.net) Quit (Quit: Ja odoh a vi sta 'ocete...)
[18:06] * Cube (~Cube@12.248.40.138) has joined #ceph
[18:06] * Cube (~Cube@12.248.40.138) Quit ()
[18:06] * tnt (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:07] * nolan (~nolan@2001:470:1:41:20c:29ff:fe9a:60be) Quit (Quit: ZNC - http://znc.sourceforge.net)
[18:07] * Cube (~Cube@12.248.40.138) has joined #ceph
[18:09] * nolan (~nolan@2001:470:1:41:20c:29ff:fe9a:60be) has joined #ceph
[18:12] <mauilion_> hey gregaf I was just reading the cephfs space accounting and quotas thread
[18:12] <mauilion_> small world
[18:12] <mauilion_> :)
[18:12] <gregaf> there aren't that many of us and we all tend to hang out here ;)
[18:13] <mauilion_> I am using cephfs to support the shared directory that openstack uses to host cached images.
[18:14] <mauilion_> I am seeing a problem that is hard to find. When openstack deploys an image from the cached image it makes a cow image on disk local to the HV.
[18:14] <mauilion_> the backing image is the one stored on cephs
[18:14] * dmick (~dmick@2607:f298:a:607:c8a7:5c47:7246:d823) has joined #ceph
[18:14] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[18:14] <mauilion_> Some image that are started this way take A long time to boot and when they do boot operate as though they are backed by a bad raid
[18:15] <mauilion_> slow disk io
[18:15] <mauilion_> etc
[18:15] <mauilion_> The trecherous part is that it's not consistent
[18:15] <mauilion_> When there is slowness on an image all HV's feel it
[18:16] <mauilion_> but when there isn't none of the HV's have a problem.
[18:16] <gregaf> okay…any reason you're not using rbd instead? it's both designed for storing disk images and supported in production by Inktank :p
[18:16] <gregaf> is this something you see per-image or is it time-based?
[18:16] <mauilion_> I am using that for volumes.
[18:16] <mauilion_> and for glance's image repo
[18:16] * yehudasa (~yehudasa@2607:f298:a:607:81fa:6d54:10a1:841) Quit (Ping timeout: 480 seconds)
[18:16] <mauilion_> This is a cache mechanism glance uses to store the image on the HV
[18:17] <mauilion_> I would actually like to use rbd for this.
[18:17] <mauilion_> but it would take some work on openstack to acheive that
[18:18] <mauilion_> it would look pretty similar to booting from volume
[18:18] <gregaf> okay; I'm not really familiar with the mechanisms you're describing although joshd might be when he comes back around
[18:18] <mauilion_> ok
[18:18] <mauilion_> another question that you can answer.
[18:19] <gregaf> given what you've described my guess is that maybe you're opening the file read-write somewhere, in which case CephFS is going to make it go into "synchronous" mode
[18:19] <gregaf> if it's only opened read then it won't require coordination between all the readers
[18:19] <mauilion_> it should be read only
[18:19] <mauilion_> only one server is making it read/write
[18:19] <mauilion_> and only when creating the file.
[18:19] <mauilion_> once it's there it's read only
[18:20] <gregaf> okay, well, without more detail that's my best guess
[18:20] <mauilion_> okay I will write something up and send to the mailing list.
[18:20] <mauilion_> Here is another question.
[18:20] <mauilion_> When I put a file in cephfs
[18:21] <mauilion_> this file is replicated in a way similar to an object right?
[18:21] <gregaf> exactly the same; CephFS files are striped across RADOS objects
[18:21] <mauilion_> okay
[18:22] <mauilion_> when the object is being replicated.
[18:22] <mauilion_> would the client experience slowness while the replication is happening?
[18:22] <mauilion_> is there some io contention in other words?
[18:23] <mauilion_> this might be happening at the osd level I suspect.
[18:23] <gregaf> I'm not sure what you're asking
[18:23] <mauilion_> When I create a file on cephfs. It's striped across the rados objects.
[18:23] <mauilion_> It's also replicated per my settings.
[18:24] <mauilion_> Currently I have it set to 2
[18:24] <mauilion_> so there are 2 copies of my object striped acress rados objects.
[18:24] <mauilion_> When does the replication take place?
[18:24] * yehudasa (~yehudasa@2607:f298:a:607:c0af:5857:7027:84a7) has joined #ceph
[18:25] <gregaf> it's synchronous during the client write
[18:25] <mauilion_> ok
[18:25] <gregaf> client sends write to "primary" OSD, then the primary OSD sends it to all the other replicas and the write returns to a client once it's on-disk on each OSD
[18:28] * rturk is now known as rturk-away
[18:28] <mauilion_> got it
[18:30] <mauilion_> alright off to scour logs and write this all up.
[18:30] <mauilion_> Thanks for your time.
[18:30] * rturk-away is now known as rturk
[18:31] * sagelap1 (~sage@38.122.20.226) has joined #ceph
[18:32] * sagelap (~sage@2607:f298:a:607:f10e:c0e:3078:e1c6) Quit (Ping timeout: 480 seconds)
[18:42] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:47] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has joined #ceph
[18:47] * leseb (~leseb@83.167.43.235) Quit (Remote host closed the connection)
[18:56] * sivanov (~sivanov@gw2.maxtelecom.bg) Quit (Ping timeout: 480 seconds)
[18:59] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[18:59] <buck> is anyone seeing gpg signature issues when teuthology installs packages from gitbuilder.ceph.com?
[19:00] * loicd (~loic@74-94-156-210-NewEngland.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[19:03] * sleinen (~Adium@2001:620:0:46:c173:dbc8:1745:5052) Quit (Ping timeout: 480 seconds)
[19:10] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:18] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[19:18] * loicd (~loic@74-94-156-210-NewEngland.hfc.comcastbusiness.net) has joined #ceph
[19:23] * rturk is now known as rturk-away
[19:24] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[19:26] * sleinen1 (~Adium@2001:620:0:26:4d5c:5623:bc5:67f) has joined #ceph
[19:32] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:35] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[19:36] * sleinen1 (~Adium@2001:620:0:26:4d5c:5623:bc5:67f) Quit (Quit: Leaving.)
[19:44] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[19:45] * sleinen1 (~Adium@2001:620:0:25:ade9:6f13:1dfd:d483) has joined #ceph
[19:49] <sstan> gregaf : actually, is it sufficient that it is on journal , or does it return when it's actually on disk ?
[19:52] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:53] * leseb (~leseb@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[19:57] * dosaboy (~gizmo@faun.canonical.com) Quit (Quit: Leaving.)
[19:58] * portante is now known as portante|afk
[20:04] * timmclau_ (~timmclaug@69.170.148.179) has joined #ceph
[20:04] * jskinner (~jskinner@69.170.148.179) Quit (Read error: Operation timed out)
[20:04] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[20:04] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) Quit (Quit: noahmehl)
[20:11] * timmclaughlin (~timmclaug@69.170.148.179) Quit (Ping timeout: 480 seconds)
[20:15] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[20:18] <drokita> So, if my ceph health is OK, is it okay to take an OSD down and replace it? By that I mean skip the marking out part? The reason I would do such a ridiculous thing is that the disk is unhealthy and not outright failing. By marking it out, the massive rebalancing is making the matters worse.
[20:19] <absynth> yep
[20:20] <absynth> you can issue a ceph osd noout to prevent the rebalancing to take place
[20:20] <absynth> then take the osd down (kill the process, do not mark it down or so)
[20:20] <absynth> and replace the failing disk
[20:20] <absynth> after it has been replaced and a new disk put in (with a filesystem and all), unset noout and the backfill should take place
[20:20] <drokita> That's what I thought. Thanks for the confirmation!
[20:21] <absynth> at least that's the theory
[20:34] <t0rn> When using qemu-rbd , would anyone be surprised to see higher memory usage from the qemu proc when writing to a rbd volume backed by a pool with fewer placement groups than one backed by a pool with more placement groups? That is the behavior i am seeing, and was expecting the opposite.
[20:35] <absynth> err, can you rephrase that?
[20:35] <absynth> what were you expecting and what did you measure?
[20:38] <t0rn> was testing if more placement groups in a pool impacts memory usage of the qemu process (qemu process == instance with rbd block device attached)
[20:38] <t0rn> i was expecting memory usage of the qemu process to be lower when using rbd volumes backed by pools with fewer placement groups
[20:39] * rturk-away is now known as rturk
[20:39] <absynth> i'm not sure if i would be expecting that
[20:39] <t0rn> well, thats the question. it did seem to be the case for format 1, but not format 2 volumes
[20:40] <absynth> we have no experience with f2 volumes at all, yet
[20:40] <absynth> all our vms are format1
[20:41] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[20:45] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[20:50] <gregaf> sstan: just the journal — it can service reads out of the page cache so that doesn't need to have hit the disk yet
[20:50] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[20:51] <gregaf> t0rn: it wouldn't be surprising if in your test case the use of lower PG numbers meant you were getting less parallelism in your writes so they're taking longer and the cache needs to hold more data
[20:54] <t0rn> gregaf: interesting thought, the initial testing is just a dd /dev/zero onto rbd volume constantly for 15 minutes. For cache, are you referring to the rbd cache?
[20:57] <sstan> gregaf: is it possible to double-check that behavior? I don't understand why small writes are slow to process (200KB/s).
[20:58] * markbby (~Adium@168.94.245.6) Quit (Quit: Leaving.)
[21:01] <gregaf> t0rn: yes, rbd cache; joshd can talk about it more intelligently than I can though
[21:02] <gregaf> sstan: the journal will prevent writes from getting too far ahead of what's synced to the disk, so if you're doing a lot of them that could be it
[21:02] <t0rn> ty gregaf , i have rbd cache off (i do not specify any rbd cache settings) since off is default from what i understand
[21:02] <gregaf> okay, but even if you're not caching the messenger needs to keep a copy of the data until it's been committed
[21:03] <gregaf> so a longer time to commit means keeping that message around for longer
[21:05] <sstan> gregaf : that would explain why write speed decreases... Is there a way to tell the journal to keep more writes in memory in order to coalesce writes more efficiently?
[21:06] <gregaf> umm, sjust, what are the tuning parameters there?
[21:08] <sjust> sstan: can you describe the workload?
[21:08] <sjust> just 4k random IO?
[21:09] <sjust> rbd?
[21:09] <sstan> iozone small writes, rbd with dd bs=4k , etc.
[21:09] <sjust> and you have caching off/
[21:09] <sjust> ?
[21:10] <sstan> yes, oflag=direct and such for dd. iozone has options against that too
[21:10] <sjust> no, rbd caching
[21:10] <sstan> ah
[21:10] <sjust> is that enabled?
[21:10] <sjust> otherwise, the writes are syncronous
[21:10] <sstan> hmm I'd have to check
[21:10] <sjust> which is cripplingly slow for small io
[21:10] <sjust> essentially, you are latency bound
[21:10] <sstan> is there a quick command for that ?
[21:11] <sjust> joshd: are there docs for rbd caching?
[21:12] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[21:13] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has joined #ceph
[21:17] * josef (~seven@li70-116.members.linode.com) has left #ceph
[21:23] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[21:24] <joshd> the settings are here: http://ceph.com/docs/master/rbd/rbd-config-ref/, but it's off by default
[21:25] <joshd> writes will only be coalesced by the guest block layer (if applicable) and the rbd cache in writeback mode. the osds don't coalesce them
[21:25] * jtang1 (~jtang@79.97.135.214) has joined #ceph
[21:25] <sjust> joshd: well, the OSD page cache certainly coalesces writes
[21:25] <sjust> but on a less useful basis
[21:26] <joshd> they're still serialized by the osd though
[21:26] <sjust> yep
[21:26] <sjust> true, two writes means two journal entries
[21:27] <absynth> i wonder when we will be able to turn rbd_cache back on
[21:27] <sstan> but after the first write, the OSD should acknowledge but NOT write to disk yet, right?
[21:27] <absynth> and what performance boost we will see :)
[21:27] <sjust> the journal is a disk
[21:27] <sjust> or on a disk
[21:27] <sjust> etc
[21:27] <sjust> but it's a streaming write, so you don't pay seek costs
[21:28] <sjust> that is, the journal does large sequential IO with whatever happens to be queued for the journal at the time
[21:28] <sjust> the ios are then written to the OSD filesystem which is periodically synced
[21:29] <sstan> does the journal take writes while it syncs its contents with the drive ?
[21:29] <sjust> yes
[21:29] <sjust> up to a point
[21:29] <sjust> but to take advantage of any of this with the benchmarks you mentioned, you need rbd writeback caching to allow more than one write in flight at a time
[21:29] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has left #ceph
[21:30] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[21:30] <sstan> why is absynth saying "i wonder when we will be able to turn rbd_cache back on"
[21:30] <sstan> does it not work right now?
[21:30] <sjust> it works IF the layer above it is sending the correct bariers
[21:30] <sjust> as with any disk
[21:31] <sjust> it works like a disk cache
[21:32] * timmclau_ (~timmclaug@69.170.148.179) Quit (Remote host closed the connection)
[21:32] <sstan> so that caching is done locally to the client
[21:32] * timmclaughlin (~timmclaug@69.170.148.179) has joined #ceph
[21:32] <sjust> correct, if you have two clients accessing the same rbd, it will be exactly as calamitous as you would expect
[21:32] <sjust> (with caching turned on)
[21:32] <nhm> sstan: some folks have had some issues with rbd caching causing mouse stuttering or other interactivity issues with VMs.
[21:33] <sjust> with caching turned off, that could be made to work probably
[21:33] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[21:33] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[21:34] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Quit: Leaving)
[21:35] <sstan> then one has to "cache" at the osd level... so is there a way to tell the journal to take all the writes even when data isn't synced ?
[21:35] <sstan> i.e. the OSD would sync files in its spare time
[21:36] <sjust> sstan: the OSD does cache in it's own page cache
[21:36] <sjust> it just uses a linux filesystem to back the OSD data
[21:36] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 19.0.2/20130307023931])
[21:37] <sjust> with a journal (as mentioned above) to provide transactions and sequential performance for short bursts of random IO
[21:37] <sjust> so it also caches
[21:37] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[21:37] <sjust> the journal always takes writes first
[21:37] <sjust> so you get a commit based on how long it takes to go to journal
[21:37] <sjust> BUT
[21:38] <sjust> if the journal is allowed to be unboundedly far ahead of the backing disk (which is easy...sequential io is faster than random io) you would get a situation where the OSD would consume unbounded memory queueing IO for the backing disk and reads might have to block for an unbounded amount of time for a write to clear the queue
[21:38] <sjust> so we don't do that
[21:38] <sjust> once the journal gets too far ahead, it blocks until some throttle opens up
[21:39] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has joined #ceph
[21:39] * noahmehl (~noahmehl@cpe-75-186-45-161.cinci.res.rr.com) has joined #ceph
[21:41] <sstan> dd if=/dev/zero of=/dev/rbd/rbd/testttt bs=4k count=1 oflag=direct
[21:41] <sstan> is way faster that when count=1000
[21:41] <sstan> your explanation explains that really well
[21:43] <sstan> where should I look if I wanted it to remain as fast for bs=1000 ?
[21:44] <sstan> is it the 5second journal flush ?
[21:49] * timmclau_ (~timmclaug@69.170.148.179) has joined #ceph
[21:51] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[21:55] <sjust> I don't quite understand, you want to be able to decrease the write size but retain the same speed?
[21:56] <sjust> you'd need rbd writeback caching for that
[21:56] * The_Bishop__ (~bishop@e177088108.adsl.alicedsl.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[21:56] * timmclaughlin (~timmclaug@69.170.148.179) Quit (Ping timeout: 480 seconds)
[22:02] <rzerres> hello joao, just wanted to give you a quick showup
[22:09] * drokita1 (~drokita@199.255.228.128) has joined #ceph
[22:12] * timmclau_ (~timmclaug@69.170.148.179) Quit (Remote host closed the connection)
[22:14] * drokita (~drokita@199.255.228.128) Quit (Ping timeout: 480 seconds)
[22:14] * diegows (~diegows@190.190.2.126) has joined #ceph
[22:16] * rzerres (~ralf.zerr@pd95b5253.dip0.t-ipconnect.de) has left #ceph
[22:22] * drokita1 (~drokita@199.255.228.128) Quit (Ping timeout: 480 seconds)
[22:22] * Rocky (~r.nap@188.205.52.204) Quit (Quit: **Poof**)
[22:38] <elder> http://www.businessinsider.com/a-dangerous-sign-for-vmware-paypal-chooses-rival-openstack-2013-3
[22:47] <gregaf> haha, tv just pushed some downburst work to his github account
[22:47] <gregaf> houkouonchi-home, might want to check that out? :)
[22:51] * KindTwo (KindOne@h198.235.22.98.dynamic.ip.windstream.net) has joined #ceph
[22:53] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Remote host closed the connection)
[22:53] * KindTwo is now known as KindOne
[23:01] * allsystemsarego (~allsystem@188.27.165.190) Quit (Quit: Leaving)
[23:01] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[23:12] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[23:15] <sstan> sjust: as you said, when writes are performed, the journal may be out of sync with the disk _up to a certain point_.
[23:15] <sstan> with bs=1 it's fast (journal out of sync) , bs=10 also fast . bs=1000 (not fast; journal WAITS for disk to be in sync)
[23:16] <sstan> *count
[23:16] <sstan> all that time I meant count instead of blocksize, sorry :/
[23:17] <sjust> yeah
[23:23] * pioto (~pioto@pool-96-235-30-25.pitbpa.fios.verizon.net) has joined #ceph
[23:24] <pioto> hi, i'm hitting an ENOMEM trying to mount cephfs w/ the kernel module; mounting with fuse doesn't seem to have this issue. any suggestions of where to dig? i don't think it should actually be needing > 500 M for a single, almost empty mount, should it?
[23:24] <sstan> tried rbd cache. Speed of small i/o writes increased 10x
[23:25] * mauilion (~root@75.98.92.113) Quit (Quit: leaving)
[23:26] <gregaf> pioto: that does sound pretty unusual; is that being returned by mount.ceph?
[23:27] <pioto> yes. it's from the mount(2) call specifically:
[23:27] <pioto> mount("192.168.11.50:6789:/", "/mnt/mycephfs", "ceph", 0, "name=admin,key=client.admin") = -1 ENOMEM (Cannot allocate memory)
[23:28] <pioto> wait, huh?
[23:28] <pioto> hm
[23:28] <pioto> well. no, i guess that's how the command rewrites my arguments
[23:28] <pioto> which were: sbin/mount.ceph 192.168.11.50:6789:/ /mnt/mycephfs -v -o rw,secretfile=/etc/ceph/secret,name=admin
[23:28] <pioto> in case it helps:
[23:28] <pioto> $ uname -a
[23:28] <pioto> Linux aether 3.2.0-39-generic #62-Ubuntu SMP Thu Feb 28 00:28:53 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[23:29] <pioto> $ ceph -v
[23:29] <pioto> ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5)
[23:30] <pioto> if i drop the name=, then i get: mount("192.168.11.50:6789:/", "/mnt/mycephfs", "ceph", 0, "key=client.guest") = -1 EPERM (Operation not permitted)
[23:30] <gregaf> that's strace output?
[23:30] <pioto> yes
[23:31] <pioto> so, it seem slike auth is working.
[23:31] <pioto> because... dmesg says: [50149.082429] libceph: client0 fsid e3195ed8-80a3-4638-a8b8-1339cfcba30f
[23:31] <pioto> [50149.092525] libceph: mon0 192.168.11.50:6789 session established
[23:31] <pioto> or, in that second case, [50416.886469] libceph: client0 fsid e3195ed8-80a3-4638-a8b8-1339cfcba30f
[23:31] <pioto> [50416.889711] libceph: auth method 'x' error -1
[23:32] <gregaf> hrm
[23:32] <pioto> i'm on ubuntu 12.04 LTS, and just trying to follow the 'quick start'
[23:32] <gregaf> yeah
[23:32] <pioto> with the client being my laptop, and the cluster server all running in virtualbox on my desktop
[23:33] <gregaf> does it print out anything following "session established"?
[23:33] <pioto> no
[23:33] <gregaf> this isn't an issue I've run into before
[23:33] <pioto> yeah, my googling came up with basically nothing like this
[23:33] * eschnou (~eschnou@227.159-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[23:34] <dmick> this might be a job for the magic kernel debug?
[23:34] <pioto> hm, wazzat?
[23:34] <pioto> i'm just using the stock ubuntu kernel, which seems to be what provides my kernel module
[23:34] <pioto> do i need to build a new one, then?
[23:34] <gregaf> it's conceivable that it really is just failing to allocate the MDS messages or something, but I'm…surprised
[23:34] <dmick> trying to remember....something you can set in /sys somewhere or something. but it may be debug only
[23:35] <pioto> hm
[23:35] <dmick> gregaf do you remember wth I'm talking about? :)
[23:35] <pioto> well. after closing chrome and thunderbird, i have > 1.5G free
[23:35] <pioto> so... i really hope it doesn't need that much ram just to mount
[23:35] <gregaf> ah, yeah, that's not enabled in the Ubuntu kernel sadly I don't think
[23:36] <pioto> k
[23:36] <pioto> well... lemme spin up a fresh vm to act as a client
[23:36] <pioto> if i can reproduce in that
[23:36] <pioto> i'll build a custom kernel
[23:36] <pioto> but, well, i don't wanna deal with that hassle on my lapotp
[23:37] <gregaf> if you want to check it you can see if /sys/kernel/debug exists
[23:37] <pioto> $ sudo ls /sys/kernel/debug/ceph
[23:37] <pioto> e3195ed8-80a3-4638-a8b8-1339cfcba30f.client0
[23:38] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[23:38] <pioto> i bet taht's my rbd map?
[23:38] <gregaf> ah, hey, it is turned on!
[23:38] <gregaf> umm, not sure, elder or joshd?
[23:38] <gregaf> pioto: what are the contents of that dir?
[23:39] <pioto> $ sudo ls -l /sys/kernel/debug/ceph/e3195ed8-80a3-4638-a8b8-1339cfcba30f.client0
[23:39] <pioto> total 0
[23:39] <pioto> -rw------- 1 root root 0 Mar 25 17:06 monc
[23:39] <pioto> -rw------- 1 root root 0 Mar 25 17:06 monmap
[23:39] <pioto> -rw------- 1 root root 0 Mar 25 17:06 osdc
[23:39] <pioto> -rw------- 1 root root 0 Mar 25 17:06 osdmap
[23:40] <joshd> if that's the only .client directory, then yeah, it's the one being used by rbd
[23:40] <pioto> no mdsok
[23:40] <pioto> err, yeah. no mds means that, right?
[23:41] <pioto> so, that seems like it would let me debug the mount when it succeeds
[23:41] <joshd> I think so, although I haven't looked at those debugfs entries for a while
[23:41] <dmick> I think dynamic-debug-howto.txt is what I'm thinking of
[23:41] <pioto> but.. what about this case, when it doesn/t
[23:41] <pioto> ok, lemme dig that up
[23:43] <gregaf> well, you could try
[23:43] <dmick> hm. my Quantal kernel says # CONFIG_DYNAMIC_DEBUG is not set
[23:43] <gregaf> $ cat /home/gregf/ceph_kclient_debug_on.sh
[23:43] <gregaf> #!/bin/sh -x
[23:43] <gregaf> p() {
[23:43] <gregaf> echo "$*" > /sys/kernel/debug/dynamic_debug/control
[23:43] <gregaf> }
[23:43] <gregaf> echo 9 > /proc/sysrq-trigger
[23:43] <gregaf> p 'module ceph +p'
[23:43] <gregaf> p 'module libceph +p'
[23:43] <gregaf> p 'module rbd +p'
[23:43] <gregaf> p 'file net/ceph/messenger.c -p'
[23:43] <gregaf> p 'file' `grep -- --- /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` '+p'
[23:43] <gregaf> p 'file' `grep -- === /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` '+p'
[23:43] <dmick> maybe /sys/kernel/debug is used for multiple things
[23:43] <gregaf> which I believe will print to dmesg all the messages exchanged (along with a bunch of other stuff) to give us some clues about where it's ENOMEMing
[23:44] <gregaf> oh, maybe you need to check the dynamic_debug folder then (in debug/)
[23:45] <pioto> $ sudo ls -ld /sys/kernel/debug/dynamic_debug/control
[23:45] <pioto> ls: cannot access /sys/kernel/debug/dynamic_debug/control: No such file or directory
[23:45] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[23:46] <dmick> yeah, you don't have the right kernel option.
[23:46] <dmick> those other files are just 'normal' sysfiles
[23:46] <pioto> ok
[23:46] <pioto> so... i'll see if i can reproduce in a 'vanilla' ubuntu-server image
[23:46] <dmick> this would be such a great thing for dtrace. sigh.
[23:46] <pioto> if not, then my laptop is weird
[23:47] <pioto> if so, then i'll build a debug kernel (and watch the bug magically go away, with my luck :)
[23:48] <pioto> thanks for the pointers
[23:48] <gregaf> I've gotta say that in a quick review I didn't see anything returning ENOMEM that shouldn't be…maybe the right kind of memory just isn't available?
[23:48] <dmick> it's also kind of an old kernel/module in the scheme of things.
[23:49] <dmick> you could try just a newer kernel
[23:50] <gregaf> I really doubt the kernel is the issue, although I guess with CephFS we're still allowed to use that as an excuse :p
[23:50] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[23:50] <dmick> well, I mean "the ceph module involved in the kernel" when I say that, of course
[23:54] * mauilion_ is now known as mauilion
[23:55] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[23:56] * sleinen1 (~Adium@2001:620:0:25:ade9:6f13:1dfd:d483) Quit (Quit: Leaving.)
[23:57] <pioto> yeah. ii was wondering if a module from the main linux upstream is "good enough" to use with the current ceph
[23:57] <pioto> but maybe that inferface doesn't change as much as other things
[23:57] * tnt (~tnt@206.212-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.