#ceph IRC Log

Index

IRC Log for 2013-07-14

Timestamps are in GMT/BST.

[0:00] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[0:19] * Arranca56 (~arrebatam@188.250.3.195) has joined #ceph
[0:39] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[0:43] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[0:51] * Arranca56 (~arrebatam@188.250.3.195) Quit (autokilled: Do not spam. mail support@oftc.net (2013-07-13 22:51:22))
[1:04] * scuttlemonkey_ (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[1:10] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Ping timeout: 480 seconds)
[1:45] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: sputnik13)
[1:47] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Remote host closed the connection)
[1:49] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:51] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[1:52] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[1:58] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[2:00] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[2:00] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[2:02] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit ()
[2:02] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Remote host closed the connection)
[2:02] * LeaChim (~LeaChim@2.216.167.255) Quit (Ping timeout: 480 seconds)
[2:17] * agaran (~agaran@00017ab1.user.oftc.net) has left #ceph
[2:21] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[2:26] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) has joined #ceph
[2:48] * Henson_D (~kvirc@69.166.23.191) has joined #ceph
[2:50] <Henson_D> hey everyone, I have a question about Ceph. I have a 2 OSD system with 85 MB/s SATA for data and 35 MB/s SATA drives for Journalling connected with gigabit ethernet. With RADOS benchmarks I can get about 35 MB/s write speed (expected) and 150 MB/s read speed (expected). However with an RBD benchmark I can only get about 5-8 MB/s write and read speeds. Does anyone have any suggestions as to
[2:50] <Henson_D> why my RBD devices are way slower than RADOS is able to go?
[2:58] <BillK> Henson_D: how many mons on the host system" - I dropped down to one mon and cepg sped up a lot as it was being throttled by lack of resources
[2:58] <Henson_D> BillK: I
[2:58] <Henson_D> BillK: I'm running with 3 mons, the minimum number for fault-tolerance.
[2:59] <Henson_D> BillK: so your system got faster when you disabled some monitors?
[3:01] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[3:02] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[3:02] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit ()
[3:06] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[3:28] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[3:32] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[3:33] * Tovinhas (~dg@bl7-99-168.dsl.telepac.pt) has joined #ceph
[3:41] * Tovinhas (~dg@bl7-99-168.dsl.telepac.pt) Quit (Ping timeout: 480 seconds)
[3:42] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Read error: Connection reset by peer)
[3:42] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[3:53] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: sputnik13)
[3:57] * yy (~michealyx@58.100.82.159) has joined #ceph
[4:12] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) Quit (Ping timeout: 480 seconds)
[4:18] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[4:32] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[4:39] * Henson_D (~kvirc@69.166.23.191) Quit (Quit: KVIrc 4.1.3 Equilibrium http://www.kvirc.net/)
[4:45] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) has joined #ceph
[4:53] * zhangjf_zz2 (~zjfhappy@222.128.1.105) has joined #ceph
[4:54] * lightspeed (~lightspee@fw-carp-wan.ext.lspeed.org) has joined #ceph
[5:00] * fireD (~fireD@93-142-243-73.adsl.net.t-com.hr) has joined #ceph
[5:03] * fireD1 (~fireD@93-139-187-204.adsl.net.t-com.hr) Quit (Read error: Operation timed out)
[5:08] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[5:20] * yy (~michealyx@58.100.82.159) has left #ceph
[5:31] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[6:06] * yy (~michealyx@58.100.82.159) has joined #ceph
[6:11] * Psi-Jack_ (~psi-jack@psi-jack.user.oftc.net) Quit (Quit: http://www.psi-jack.info/)
[6:17] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley)
[6:43] * yy (~michealyx@58.100.82.159) Quit (Ping timeout: 480 seconds)
[6:50] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Read error: Operation timed out)
[7:01] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[7:02] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[7:32] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[7:37] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) has joined #ceph
[8:25] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley)
[9:11] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: sputnik13)
[10:00] * iggy (~iggy@theiggy.com) Quit (Quit: No Ping reply in 180 seconds.)
[10:00] * iggy (~iggy@theiggy.com) has joined #ceph
[10:04] * haomaiwang (~haomaiwan@notes4.com) Quit (Remote host closed the connection)
[10:05] * haomaiwang (~haomaiwan@notes4.com) has joined #ceph
[10:36] * odyssey4me (~odyssey4m@165.233.71.2) has joined #ceph
[10:37] * LeaChim (~LeaChim@2.216.167.255) has joined #ceph
[11:51] * haomaiwang (~haomaiwan@notes4.com) Quit (Remote host closed the connection)
[11:53] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[12:38] * arrebatamos (~javais69@188.250.5.103) has joined #ceph
[12:51] * toMeloos (~tom@53545693.cm-6-5b.dynamic.ziggo.nl) has joined #ceph
[12:56] * toMeloos (~tom@53545693.cm-6-5b.dynamic.ziggo.nl) Quit (Remote host closed the connection)
[12:58] * ScOut3R (~ScOut3R@catv-80-98-239-160.catv.broadband.hu) has joined #ceph
[13:04] * grepory (~Adium@c-69-181-42-170.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[13:23] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[13:29] * zhangjf_zz2 (~zjfhappy@222.128.1.105) Quit (Remote host closed the connection)
[13:47] * ScOut3R (~ScOut3R@catv-80-98-239-160.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[13:57] * arrebatamos (~javais69@188.250.5.103) Quit (autokilled: Please do not spam on IRC. Contact support@oftc.net with questions. (2013-07-14 11:57:48))
[13:59] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[14:24] * ScOut3R (~ScOut3R@catv-80-98-239-160.catv.broadband.hu) has joined #ceph
[14:26] <loicd> Hi, where can I find the repository matching http://tracker.ceph.com/projects/calamari/ ?
[14:36] * odyssey4me (~odyssey4m@165.233.71.2) Quit (Ping timeout: 480 seconds)
[14:40] * odyssey4me (~odyssey4m@165.233.71.2) has joined #ceph
[14:42] * julian (~julianwa@125.69.105.128) has joined #ceph
[14:52] * Tovinhas (~dg@bl10-138-114.dsl.telepac.pt) has joined #ceph
[14:53] * KindTwo (KindOne@h158.19.131.174.dynamic.ip.windstream.net) has joined #ceph
[14:56] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[14:56] * KindTwo is now known as KindOne
[15:03] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[15:04] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[15:05] * haomaiwa_ (~haomaiwan@notes4.com) has joined #ceph
[15:05] * Tovinhas (~dg@bl10-138-114.dsl.telepac.pt) Quit (autokilled: Please do not spam on IRC. Contact support@oftc.net with questions. (2013-07-14 13:05:35))
[15:06] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Read error: Connection reset by peer)
[15:06] * DarkAce-Z (~BillyMays@50.107.55.36) has joined #ceph
[15:09] * ScOut3R (~ScOut3R@catv-80-98-239-160.catv.broadband.hu) Quit (Remote host closed the connection)
[15:10] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[15:10] * DarkAceZ (~BillyMays@50.107.55.36) Quit (Ping timeout: 480 seconds)
[15:21] * haomaiwang (~haomaiwan@notes4.com) has joined #ceph
[15:21] * haomaiwa_ (~haomaiwan@notes4.com) Quit (Read error: Connection reset by peer)
[15:23] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[15:35] * john_barbee (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) has joined #ceph
[15:38] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley)
[15:47] * diegows (~diegows@190.190.2.126) has joined #ceph
[15:52] * infinitytrapdoor (~infinityt@ip-109-84-234-234.web.vodafone.de) has joined #ceph
[15:52] * infinitytrapdoor (~infinityt@ip-109-84-234-234.web.vodafone.de) has left #ceph
[16:09] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[16:18] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[16:22] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit ()
[16:24] * allsystemsarego (~allsystem@188.27.166.68) has joined #ceph
[16:33] * KevinPerks1 (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[16:34] * Fetch__ (fetch@gimel.cepheid.org) has joined #ceph
[16:36] * Fetch_ (fetch@gimel.cepheid.org) Quit (Read error: Connection reset by peer)
[16:38] * partner (joonas@ajaton.net) Quit (Remote host closed the connection)
[16:38] * partner (joonas@ajaton.net) has joined #ceph
[16:39] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[16:45] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[16:46] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit ()
[16:58] * jeroenmoors (~quassel@193.104.8.40) Quit (Remote host closed the connection)
[16:58] * jeroenmoors (~quassel@193.104.8.40) has joined #ceph
[16:58] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[17:10] * smiley (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[17:10] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[17:27] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[17:31] * KevinPerks1 (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[17:40] * john_barbee (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[17:48] * odyssey4me (~odyssey4m@165.233.71.2) Quit (Ping timeout: 480 seconds)
[18:02] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[18:08] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[18:08] * ChanServ sets mode +o scuttlemonkey
[18:11] * scuttlemonkey_ (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Read error: Operation timed out)
[18:11] * john_barbee (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) has joined #ceph
[18:14] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[18:26] * john_barbee_ (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) has joined #ceph
[18:31] * john_barbee (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[18:39] * john_barbee_ (~jbarbee@c-50-165-106-164.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[18:40] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:42] * haomaiwa_ (~haomaiwan@106.120.176.116) has joined #ceph
[18:47] <paravoid> sage: the slow peering thread sounds exactly like the thing I've been debugging with sjust...
[18:48] * haomaiwang (~haomaiwan@notes4.com) Quit (Ping timeout: 480 seconds)
[18:48] * julian (~julianwa@125.69.105.128) Quit (Quit: afk)
[19:07] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[19:09] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit ()
[19:28] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:35] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[19:44] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:01] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[20:04] * Henson_D (~kvirc@69.166.23.191) has joined #ceph
[20:04] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:13] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[20:14] <Henson_D> would anyone be able to give me some suggestions on improving the performance of Ceph's RBDs? A RADOS benchmark shows my system has about 150 MB/s of read bandwidth, but using "dd" to do a sequential read on an RBD only gives me about 15-20 MB/s read. With a filesystem on the RBD the performance is even worse, about 8 MB/s.
[20:15] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[20:30] * fridudad (~oftc-webi@p5B09C393.dip0.t-ipconnect.de) has joined #ceph
[20:33] * danieagle (~Daniel@177.205.183.252.dynamic.adsl.gvt.net.br) has joined #ceph
[20:41] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[20:47] * xmltok_ (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph
[20:49] <Gugge-47527> Henson_D: what dd command do you use?
[20:51] <Gugge-47527> Henson_D: try using 4MB blocksize in the dd test :)
[21:06] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[21:10] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[21:11] * allsystemsarego (~allsystem@188.27.166.68) Quit (Quit: Leaving)
[21:14] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[21:19] * xmltok_ (~xmltok@cpe-76-170-26-114.socal.res.rr.com) Quit (Quit: Leaving...)
[21:21] * xmltok_ (~xmltok@relay.els4.ticketmaster.com) has joined #ceph
[21:26] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[21:34] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[21:35] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[21:39] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[22:00] <Henson_D> Gugge-47527: even with 4mb block sized i get 23 mb/s with a flushed cache and 43 mb/s without
[22:00] <Henson_D> Gugge-47527: read speed
[22:02] <Gugge-47527> and what do you get with rados bench with only one thread?
[22:07] <Henson_D> Gugge-47527: ahhh, very interesting. with only 1 thead I get about the same performance! I guess the default test uses more than 1, because I get 150 to 200 MB/s with the default number of threads.
[22:07] <Gugge-47527> yes
[22:07] <Henson_D> Gugge-47527: so I guess using multiple threads just has multiple workers extracting data from RADOS, and it just adds up all the throughput? Is there any way of getting RBD or filesystems to use multiple threads?
[22:07] <Gugge-47527> yes
[22:08] <Gugge-47527> run multiple concurrent ios .)
[22:08] <Gugge-47527> :)
[22:08] <Gugge-47527> if you _want_ to use dd, run more than one
[22:08] <Gugge-47527> or use a multithreaded tester
[22:08] <Gugge-47527> or, test with whatever workload you really are gonna put on the system
[22:09] <Henson_D> Gugge-47527: I'm more interested in trying to speed up a filesystem that I put on the RBD. I'll take a look and see what I can find, but thank you for clearing that up.
[22:09] <Henson_D> Gugge-47527: my workload would pretty much just be a single program loading and storing files on a filesystem on the RBD. I'm hoping to use Ceph in a high-availability power computer.
[22:10] <Gugge-47527> okay
[22:10] <Henson_D> high-availity power computer -> high-availability computer system
[22:11] <Henson_D> Gugge-47527: anyway, thank you for helping me figure out why it was going so slowly. I will continue my investigations in a multi-threaded direction.
[22:11] <Gugge-47527> one of my setups is 500-5000 concurrent webrequest -> nfs -> zfs -> 10xRBD images
[22:11] <Gugge-47527> im not concerned about the speed of a single thread :)
[22:12] <Henson_D> Gugge-47527: wow, that sounds like quite a setup!
[22:12] <Henson_D> Gugge-47527: what's a good filesystem to put on an RBD? I've tried ext4, xfs, and btrfs, and btrfs seems to outperform the others.
[22:13] * danieagle (~Daniel@177.205.183.252.dynamic.adsl.gvt.net.br) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[22:13] <Gugge-47527> i like zfs
[22:13] <Gugge-47527> i like btrfs .... when its ready in a few years :)
[22:14] <Henson_D> Gugge-47527: I know the ceph developers thing that btrfs is a good match, but recommend that people use xfs in production. I'm trying out btrfs on my non-production system, as I've read some of the BTRFS maintainers say it's pretty stable now, and if you're not doing anything wierd it should be pretty stable. Do you have any comments on that?
[22:15] <Henson_D> Gugge-47527: I noticed that ceph automatically created some snapshots in my ceph storage directory, so I imagine it leverages BTRFS's snapshotting abilities for the RADOS snapshots.
[22:15] <Gugge-47527> i tries btrfs, and hit some "5 minutes to unmount" issues and "btrfs-transacion taking 100% io for 5 hours after mount" issues
[22:16] <Gugge-47527> and i dropped it again :)
[22:17] <Henson_D> Gugge-47527: perhaps I should start over and use XFS with LVM.
[22:18] <Gugge-47527> my tests were with btrfs on top of rbd though, i never tried btrfs for the osd's
[22:20] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[22:23] <Henson_D> Gugge-47527: ahh, ok. Do you know why RBD is slow for single-threaded? I thought that if you have N OSDs, and your data is being "striped" across all of them, you should end up with N times the throughput of a single OSD.
[22:24] <guppy> Henson_D: I think you have to set up striping
[22:24] <Gugge-47527> a distribued system has a higher latency than a local disk
[22:25] <Gugge-47527> = you have longer pause between each io
[22:25] <Gugge-47527> = less performance
[22:26] <Henson_D> Gugge-47527: hmm, that makes sense. so using infiniband instead of ethernet should improve things :-)
[22:28] <ron-slc> Henson_D: Also, if using replication_size of 2 or 3, you are best to have a 10G network for the cluster network. 100MB into the public network means 200MB+ need to go out the cluster network on rep size = 3
[22:29] <ron-slc> so often times your back-end network is an unseen bottle neck.
[22:32] <Henson_D> ron-slc: thanks for the suggestion. My OSDs are on a gigabit back-end network, but they are also the clients loading data off of ceph. My network, drives, and CPUs don't seem to be the bottleneck for reading, so I guess it's the latency. For writing I only have 30 MB/s drives for the journal drives, so they are definitely the bottleneck.
[22:33] <Henson_D> it's a pretty low-performance system, but even then I was noticing performance quite a bit less than what I would expect. So I'll have to see how things scale with multiple parallel IOs.
[22:37] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[22:38] <ron-slc> Henson_D: Yea, correct, if your journal disk can only do 30Mb/s your writes will be constrained. Personally I use 1GB Cache on a Raid Card to help this situation, and also helps with having the Journal on same physical disk as OSD.
[22:40] <ron-slc> I used to have an SSD for Journals, but quickly did the math on the SSD's daily write cycle and MTBF. Based on Manufacturer's duty cycle, my SSD's would have lickely died in just 6 months. Thus my need for Flash-Backed RAID cards, in single-disk mode. (I mention this in-case you are considering SSD's for Journal.)
[22:41] * leseb1 (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:42] <ron-slc> Henson_d: Multiple IOs do help in most cases with RBD. Also, the huge performance boosts are seen when you have MANY OSD's, breaking the work into smaller chunks per-disk.
[22:47] <Henson_D> ron-slc: yeah, from some of the ceph performance analyses I've seen, they have a bunch of OSDs and are able to get really good performance, but I guess with single-threaded IO and 2 OSDs, I can't expect too much performance :-(
[22:48] <ron-slc> Henson_D: yea, Ceph and most distributed storage systems see performance upon horizontal scale. Us smaller guys, though get the benefit of much higher redundancy or reliability.
[22:48] * leseb1 (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[22:49] <ron-slc> Henson_D: If possible, a HUGE boost can be seen with larger "streaming" files/datasets by formatting the file system you have running on-top of RBD to 128k sector sizes. This will not help much if your I/O is not streaming, and relatively small like a SQL database.
[22:50] * fridudad (~oftc-webi@p5B09C393.dip0.t-ipconnect.de) Quit (Quit: Page closed)
[22:50] <ron-slc> This means less back and forth I/O per each large/streaming file you are reading / writing.
[22:52] * leseb1 (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:54] <Henson_D> ron-slc: cool, thanks for the advice. I was attracted to ceph for it's distributed and reliability properties. I guess if I had a larger system, I would also see great performance :-)
[22:57] <ron-slc> yea, give the 128k formatted file system a try, it is a BIG help in my experience with read performance. If your 2 OSD's are on 2 separate hosts, using just a cross-over cable, and setting your network MTU higher will help; same host would make no difference.
[22:58] <ron-slc> Network MTU on the cluster Network, mind you. I'd avoid MTU adjustments on the Public network.
[23:03] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[23:05] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[23:06] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[23:09] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit ()
[23:10] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[23:12] * mschiff (~mschiff@port-1321.pppoe.wtnet.de) has joined #ceph
[23:14] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[23:18] <Henson_D> ron-slc: ok, i'll give those a try
[23:20] * mschiff (~mschiff@port-1321.pppoe.wtnet.de) Quit (Ping timeout: 480 seconds)
[23:22] * mschiff (~mschiff@port-1321.pppoe.wtnet.de) has joined #ceph
[23:23] * diegows (~diegows@190.190.2.126) has joined #ceph
[23:24] <ron-slc> Henson_D: have fun!
[23:30] * mschiff (~mschiff@port-1321.pppoe.wtnet.de) Quit (Remote host closed the connection)
[23:32] * mschiff (~mschiff@port-1321.pppoe.wtnet.de) has joined #ceph
[23:32] * nhm (~nhm@184-97-193-106.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[23:34] <Henson_D> ron-slc: will do. Ceph has been a lot of fun to play with so far.
[23:35] <Azrael> sage: around?
[23:38] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[23:57] * tremendous (~xmltok@cpe-76-170-26-114.socal.res.rr.com) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.