#ceph IRC Log

Index

IRC Log for 2013-04-16

Timestamps are in GMT/BST.

[0:00] * mattm__ (~matt@45.b0.425e.cidr.airmail.net) has joined #ceph
[0:03] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[0:10] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[0:15] <nigly> dmick: awesome, new kernel and a rbd that failed to mount the fs works perfectly
[0:15] <dmick> cool
[0:15] * loicd (~loic@67.23.204.150) has joined #ceph
[0:18] * loicd (~loic@67.23.204.150) Quit ()
[0:20] * portante (~user@67.23.204.150) has joined #ceph
[0:21] * wogri (~wolf@nix.wogri.at) Quit (Remote host closed the connection)
[0:21] * loicd (~loic@67.23.204.150) has joined #ceph
[0:21] * wogri (~wolf@nix.wogri.at) has joined #ceph
[0:21] * loicd (~loic@67.23.204.150) Quit ()
[0:27] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) has joined #ceph
[0:28] * BillK (~BillK@124.149.139.60) has joined #ceph
[0:29] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) Quit ()
[0:29] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) has joined #ceph
[0:30] * smeven (~diffuse@101.170.97.238) Quit (Ping timeout: 480 seconds)
[0:31] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) Quit (Remote host closed the connection)
[0:32] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:39] * joshd1 (~joshd@67.23.204.150) has joined #ceph
[0:41] * fox_ (~fox@95.154.88.151) has joined #ceph
[0:44] * LeaChim (~LeaChim@90.215.24.238) has joined #ceph
[0:49] * loicd (~loic@67.23.204.150) has joined #ceph
[0:50] * loicd (~loic@67.23.204.150) Quit ()
[0:50] * diegows (~diegows@host28.190-30-144.telecom.net.ar) Quit (Ping timeout: 480 seconds)
[0:50] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[0:57] * danieagle (~Daniel@177.133.174.100) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[0:57] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[0:58] * imjustmatthew (~imjustmat@pool-173-53-54-223.rcmdva.fios.verizon.net) has joined #ceph
[0:58] * jamespage (~jamespage@culvain.gromper.net) Quit (Quit: Coyote finally caught me)
[1:24] * dosaboy_ (~dosaboy@67.23.204.150) Quit (Remote host closed the connection)
[1:24] * dosaboy (~dosaboy@67.23.204.150) has joined #ceph
[1:25] * joshd2 (~joshd@67.23.204.150) has joined #ceph
[1:25] * joshd1 (~joshd@67.23.204.150) Quit (Read error: Connection reset by peer)
[1:25] * thelan (~thelan@paris.servme.fr) Quit (Ping timeout: 480 seconds)
[1:26] * LeaChim (~LeaChim@90.215.24.238) Quit (Ping timeout: 480 seconds)
[1:28] * loicd (~loic@67.23.204.150) has joined #ceph
[1:31] * jlogan1 (~Thunderbi@2600:c00:3010:1:1::40) Quit (Read error: Connection reset by peer)
[1:31] * loicd (~loic@67.23.204.150) Quit ()
[1:31] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[1:32] * dosaboy (~dosaboy@67.23.204.150) Quit (Ping timeout: 480 seconds)
[1:32] * __jt__ (~james@rhyolite.bx.mathcs.emory.edu) Quit (Ping timeout: 480 seconds)
[1:35] * loicd (~loic@67.23.204.150) has joined #ceph
[1:36] <Elbandi_> is a way to clean a pool?
[1:37] <Elbandi_> i do a mds newfs, but there are lots of object in the pool
[1:38] <dmick> Elbandi_: yes, even an empty cephfs has objects in data/metadata
[1:38] <dmick> unless you know what you're doing, it's unsafe to remove objects from mds or rbd pools
[1:39] <dmick> if you really want to throw it all away and start over, you can remove and recreate the pool
[1:39] <dmick> or you can write a script using rados ls and rados rm
[1:41] <Elbandi_> yes, i thought rados ls+rm, but i wanted to ask is a easiest way..
[1:42] <dmick> rmpool/mkpool will definitely destroy every object in the pool
[1:43] <nigwil> if I have a storage node with 12 drives, I am thinking I would use two for OS (RAID1), one for OSD journals, and 9 for OSDs, is that a good choice?
[1:44] <Kioob> the one for journals will be a huge bottleneck for the 9 others
[1:45] * thelan (~thelan@paris.servme.fr) has joined #ceph
[1:45] <nigwil> is it practical though (for a 500TB deployment) to have one OSD journal for every OSD? that seems very expensive
[1:46] <Kioob> You can put journal directly on the OSD : it's slower than with SSD, you will be half the speed of a normal drive, but it works.
[1:46] <Kioob> (but I'm just a customer)
[1:46] * __jt__ (~james@rhyolite.bx.mathcs.emory.edu) has joined #ceph
[1:47] <dmick> nigwil: yes. think of the journal as a required part of every OSD
[1:47] <nigwil> ok, I thought that originally, but wondered how people feasibly separate OSD journals(perhaps they don't in practice?)
[1:48] <dmick> separate in what sense?
[1:48] <nigwil> separate drives for IO
[1:49] <dmick> I think the prevailing wisdom is that separating to SSD might make some sense
[1:49] * buck (~buck@bender.soe.ucsc.edu) Quit (Quit: Leaving.)
[1:49] <Kioob> You can use a *fast* SSD for journaling 1, 2, 3 or more OSD...
[1:49] <dmick> but otherwise there's not a lot of point
[1:49] <nigwil> ahh, ok that was the advice I was looking for :-)
[1:50] <nigwil> over the weekend someone suggestion that TRIM-ing an SSD so that the journal allocation is small will help with the wear-leveling so it lasts a reasonable time
[1:53] <nigwil> I know people like to keep the OS on its own disks, but has anyone tried having an OSD on the OS disk? does the OS really need to do much traffic to the OS drive once it has settled down, aside from /var/log activity?
[1:54] <nigwil> in other words, can I afford to have an OS/OSD combination that might run slower than the other OSDs due to sharing the drive with the OS, but how much slower? and does it impact the cluster overall. I would expect it would just be slow for the objects headed to that OSD
[1:55] <Kioob> I partitionned the drive, take a 4GB partition on each and created a (software) RAID10 for the OS. It works, but one of the problem is logging : enabling logging on OSD slow down my OSD since it throw a lot of writes on same disks.
[1:55] <nigwil> but the win is two more OSDs
[1:57] <nigwil> Kioob: right that makes sense, I suppose my question is how much slower is the "slow down"?
[1:58] <Kioob> Depending of the level of logging I suppose. With standard parameters, and only "ms = 1", that RAID10 of 8 drive throw between 10 and 20 writes per second, per drive
[1:59] <Kioob> but without "ms = 1", it's lower, of course
[2:02] * loicd (~loic@67.23.204.150) Quit (Quit: Leaving.)
[2:02] * loicd (~loic@67.23.204.150) has joined #ceph
[2:02] <nigwil> with an array like that you would expect 1600 IOPs, so are you "10 - 20 writes per second" perhaps 10% of the available IO bandwidth?
[2:03] <nigwil> 1600 if it was SAS I mean, lower for SATA
[2:04] * fox_ (~fox@95.154.88.151) Quit (Quit: This computer has gone to sleep)
[2:05] <dmick> nigwil: keep in mind that the cluster is generally as slow as the slowest OSD
[2:06] <nigwil> but only at the time that those slow OSDs are involved? to get write consistency they're waiting for those slow OSDs
[2:07] <nigwil> Kioob: your OSDs are not using RAID are they?
[2:07] <Kioob> no they don't ;)
[2:07] <nigwil> ok, just a sanity check on my part :-)
[2:08] <dmick> nigwil: yes, only when they're involved. Statistically, that's for every 1/NOSD object
[2:08] <dmick> or
[2:08] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[2:08] <dmick> NREP/NOSD, I should say
[2:09] <nigwil> but statistically it would also have to be the case that the OS IO collides with the OSD IO (to cause the slowness)
[2:09] <Kioob> 1600 IOPs... I don't know what is wrong in my setup, but... I really haven't that kind of performance
[2:10] <dmick> nigwil: sure.
[2:10] <nigwil> Kioob: that is probably an optimistic number
[2:10] <dmick> but what I mean is, if you make a mistake there, the rest of the cluster suffers, not just that machine
[2:10] * loicd (~loic@67.23.204.150) Quit (Ping timeout: 480 seconds)
[2:11] <Kioob> +1 dmick. I take a *lot* of time to fix problems on my cluster, and moving (production) data, is not easy
[2:11] <nigwil> dmick: yes, I see that, but I remain skeptical about the level of OS traffic and its potential impact. Or alternatively perhaps there is a way I can either quiet the OS down or perhaps gate the IO traffic somewhere else or quarantine it (perhaps get the OSD onto a SSD on a PCIe card)
[2:12] <nigwil> s/OSD/OS in that last line
[2:12] <dmick> I'm not saying OS traffic *will* be a problem; I'm just saying, be wary.
[2:12] <dmick> disks are much cheaper than time
[2:12] <nigwil> yes understood, I am all for being wary :-)
[2:12] <Kioob> nigwil : you can make ceph log to syslog, then syslog log thought network
[2:12] <nigwil> Kioob: exactly, that removes one (somewhat) heavy source of IO
[2:13] <Kioob> and... what about monitors ? ;)
[2:14] <nigwil> I was hoping they could share time/space with the storage nodes
[2:14] <Kioob> I choosed to put them on same host as OSD, which of course doesn't seem to be good idea.
[2:14] <nigwil> provided the storage nodes are well-resourced (CPUs and memory)
[2:14] <dmick> I...don't know that logging across the network decreases the load on the machine exactly
[2:14] <dmick> monitors are pretty cheap; the important part is to have them distributed well across failure domains
[2:15] <nigwil> dmick: it perhaps assumes network IO is lower cost than disk IO? and that syslogd avoids the drisk altogether with centralised logging (I don't know off-hand)
[2:15] <Kioob> well right now (during a recovery), the mon processes make 160 IO/s on my cluster
[2:16] <nigwil> how is your cluster arranged Kioob ?
[2:16] * loicd (~loic@67.23.204.150) has joined #ceph
[2:16] <Kioob> I have 5 hosts, with each 8 SAS drive + 4 (old) SSD
[2:16] * joshd2 (~joshd@67.23.204.150) Quit (Ping timeout: 480 seconds)
[2:17] <dmick> Kioob: I hope that's significantly less than the OSDs during recovery?
[2:17] <Kioob> But... I repeat : I made a lot of mistakes...
[2:17] <nigwil> how have you asssigned your drives to OSDs / OS etc?
[2:17] <Kioob> yes dmick, of course. But it doesn't help
[2:17] <dmick> well I mean the point is that recovery does a lot of IO
[2:17] <dmick> beats resilvering swraid
[2:18] <nigwil> do you have separate LANs (For clients and storage)?
[2:18] <Kioob> nigwil : a 4GB partition on each SAS drive, and a RAID10 with that 8 * 4 GB partitions
[2:18] <Kioob> nigwil: no, my provider doesn't allow that :(
[2:18] <nigwil> ahh
[2:19] <nigwil> bbl, off to a meeting. Kioob: if you around later I have more questions :-) thanks dmick for the insights, very useful
[2:20] <dmick> yw
[2:20] * joshd1 (~joshd@67.23.204.150) has joined #ceph
[2:21] <Kioob> no problem ;) I'm often here... to ask questions to dmick :D
[2:38] * joshd1 (~joshd@67.23.204.150) Quit (Ping timeout: 480 seconds)
[2:38] * john_barbee_ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) has joined #ceph
[2:39] * dosaboy (~dosaboy@67.23.204.20) has joined #ceph
[2:40] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[2:49] * loicd (~loic@67.23.204.150) Quit (Quit: Leaving.)
[2:55] * dosaboy (~dosaboy@67.23.204.20) Quit (Ping timeout: 480 seconds)
[2:59] * portante (~user@67.23.204.150) Quit (Ping timeout: 480 seconds)
[3:02] * capri (~capri@p54A5433C.dip0.t-ipconnect.de) has joined #ceph
[3:09] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[3:10] * capri (~capri@p54A5433C.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[3:10] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit ()
[3:10] * fox_ (~fox@gw.farpost.ru) has joined #ceph
[3:12] * mattm__ (~matt@45.b0.425e.cidr.airmail.net) Quit (Quit: Leaving.)
[3:12] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[3:16] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[3:18] * mattm__ (~matt@45.b0.425e.cidr.airmail.net) has joined #ceph
[3:18] * leseb1 (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[3:18] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Read error: Connection reset by peer)
[3:20] * calebamiles (~caleb@c-50-138-218-203.hsd1.vt.comcast.net) Quit (Ping timeout: 480 seconds)
[3:24] * fox_ (~fox@gw.farpost.ru) Quit (Ping timeout: 480 seconds)
[3:25] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[3:25] * leseb1 (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Read error: Connection reset by peer)
[3:27] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit ()
[3:49] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Quit: Leaving.)
[3:58] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[4:02] * fox_ (~fox@gw.farpost.ru) has joined #ceph
[4:16] * markl (~mark@tpsit.com) Quit (Ping timeout: 480 seconds)
[4:22] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[4:30] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[4:44] * __jt__ (~james@rhyolite.bx.mathcs.emory.edu) Quit (Ping timeout: 480 seconds)
[4:46] * Big (~textual@220-135-68-130.HINET-IP.hinet.net) has joined #ceph
[5:08] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[5:14] * mattm__ (~matt@45.b0.425e.cidr.airmail.net) Quit (Quit: Leaving.)
[5:16] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[5:16] * __jt__ (~james@rhyolite.bx.mathcs.emory.edu) has joined #ceph
[5:17] * gregaf (~Adium@cpe-76-174-249-52.socal.res.rr.com) Quit (Quit: Leaving.)
[5:23] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[5:24] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[5:37] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[5:53] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[5:54] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[5:58] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[6:05] * nhm (~nh@65-128-150-185.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[6:05] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[6:10] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[6:19] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[6:26] * wogri_risc (~wogri_ris@ro.risc.uni-linz.ac.at) Quit (Remote host closed the connection)
[6:26] * wogri (~wolf@nix.wogri.at) Quit (Quit: Lost terminal)
[6:35] * wogri (~wolf@nix.wogri.at) has joined #ceph
[6:36] * wogri (~wolf@nix.wogri.at) Quit (Remote host closed the connection)
[6:39] * wogri (~wolf@nix.wogri.at) has joined #ceph
[6:57] <athrift> The blueprints for dumpling are exciting
[7:05] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[7:06] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[7:13] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[7:15] * jjgalvez1 (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[7:15] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Read error: Connection reset by peer)
[7:34] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[7:45] * capri (~capri@pd95c3283.dip0.t-ipconnect.de) has joined #ceph
[7:52] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Quit: It's a dud! It's a dud! It's a du...)
[7:54] * norbi (~nonline@buerogw01.ispgateway.de) has joined #ceph
[7:59] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[8:00] * jjgalvez1 (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[8:02] * scuttlemonkey (~scuttlemo@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[8:02] * ChanServ sets mode +o scuttlemonkey
[8:06] * fox_ (~fox@gw.farpost.ru) Quit (Quit: This computer has gone to sleep)
[8:07] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[8:08] * madkiss (~madkiss@2001:6f8:12c3:f00f:189:b6d7:1c53:c946) Quit (Quit: Leaving.)
[8:08] * Havre (~Havre@2a01:e35:8a2c:b230:45c4:d437:e951:ce52) has joined #ceph
[8:15] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[8:16] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[8:30] * Havre (~Havre@2a01:e35:8a2c:b230:45c4:d437:e951:ce52) Quit (Ping timeout: 480 seconds)
[8:53] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[8:55] * joshd1 (~joshd@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[9:01] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[9:04] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:16] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[9:17] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit ()
[9:17] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) has joined #ceph
[9:18] * boges (~te220182k@193.93.19.66) has joined #ceph
[9:21] <boges> приветствую. на руском можно вопрос задать?
[9:25] * wogri (~wolf@nix.wogri.at) Quit (Remote host closed the connection)
[9:25] * wogri (~wolf@nix.wogri.at) has joined #ceph
[9:28] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:29] * fox_ (~fox@95.154.88.151) has joined #ceph
[9:32] * joshd1 (~joshd@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[9:33] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) has joined #ceph
[9:35] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:38] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[9:41] * madkiss1 (~madkiss@089144192204.atnat0001.highway.a1.net) has joined #ceph
[9:41] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) Quit (Read error: Connection reset by peer)
[9:41] * madkiss1 (~madkiss@089144192204.atnat0001.highway.a1.net) Quit ()
[9:41] * boges (~te220182k@193.93.19.66) Quit (Quit: Покидаю)
[9:43] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) has joined #ceph
[9:52] * fox_ (~fox@95.154.88.151) Quit (Ping timeout: 480 seconds)
[10:07] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[10:08] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[10:08] * Forced (~Forced@205.132.255.75) Quit (Remote host closed the connection)
[10:08] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[10:09] * Forced (~Forced@205.132.255.75) has joined #ceph
[10:11] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[10:12] * LeaChim (~LeaChim@90.215.24.238) has joined #ceph
[10:15] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Read error: Operation timed out)
[10:17] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[10:17] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[10:20] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[10:21] * rustam (~rustam@94.15.91.30) has joined #ceph
[10:21] * sleinen (~Adium@2001:620:0:25:7844:283d:751f:b1f9) has joined #ceph
[10:22] * fox_ (~fox@95.154.88.151) has joined #ceph
[10:25] * smeven (~diffuse@1.145.222.157) has joined #ceph
[10:32] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) Quit (Quit: Leaving.)
[10:37] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Remote host closed the connection)
[10:37] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[10:39] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[10:40] * rustam (~rustam@94.15.91.30) has joined #ceph
[10:41] * rustam (~rustam@94.15.91.30) Quit (Remote host closed the connection)
[10:42] * scuttlemonkey (~scuttlemo@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[10:48] * v0id (~v0@62-46-172-131.adsl.highway.telekom.at) has joined #ceph
[10:49] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[10:52] * fox_ (~fox@95.154.88.151) Quit (Quit: This computer has gone to sleep)
[10:55] * vo1d (~v0@91-115-224-180.adsl.highway.telekom.at) Quit (Ping timeout: 480 seconds)
[10:56] * fox_ (~fox@95.154.88.151) has joined #ceph
[10:59] * sleinen (~Adium@2001:620:0:25:7844:283d:751f:b1f9) Quit (Quit: Leaving.)
[10:59] * sleinen (~Adium@130.59.94.211) has joined #ceph
[11:01] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[11:04] * sleinen1 (~Adium@2001:620:0:26:9df9:8253:b8c4:ea0f) has joined #ceph
[11:07] * sleinen1 (~Adium@2001:620:0:26:9df9:8253:b8c4:ea0f) Quit ()
[11:10] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[11:11] * sleinen (~Adium@130.59.94.211) Quit (Ping timeout: 480 seconds)
[11:13] * fox_ (~fox@95.154.88.151) Quit (Ping timeout: 480 seconds)
[11:25] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) has joined #ceph
[11:26] * sleinen (~Adium@130.59.94.211) has joined #ceph
[11:28] * sleinen (~Adium@130.59.94.211) Quit ()
[11:36] * sleinen (~Adium@130.59.94.211) has joined #ceph
[11:37] * sleinen1 (~Adium@2001:620:0:26:798d:2d0e:11b7:8f5b) has joined #ceph
[11:44] * sleinen (~Adium@130.59.94.211) Quit (Ping timeout: 480 seconds)
[11:48] * capri_on (~capri@pd95c3283.dip0.t-ipconnect.de) has joined #ceph
[11:50] * fox_ (~fox@95.154.88.151) has joined #ceph
[11:53] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[11:53] * fox_ (~fox@95.154.88.151) Quit (Read error: Connection reset by peer)
[11:55] * capri (~capri@pd95c3283.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[11:56] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[12:04] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[12:04] * Big (~textual@220-135-68-130.HINET-IP.hinet.net) Quit (Quit: Textual IRC Client: www.textualapp.com)
[12:06] * fox_ (~fox@95.154.88.151) has joined #ceph
[12:13] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[12:21] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[12:30] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) Quit (Read error: Operation timed out)
[12:46] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) has joined #ceph
[12:50] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[12:50] * jakku (~jakku@ad046161.dynamic.ppp.asahi-net.or.jp) Quit (Remote host closed the connection)
[12:53] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[12:58] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[13:02] * madkiss (~madkiss@089144192204.atnat0001.highway.a1.net) Quit (Ping timeout: 480 seconds)
[13:08] * agh (~agh@www.nowhere-else.org) has joined #ceph
[13:08] <agh> hello ! Help me pleeeeaaase :)
[13:09] <agh> One of my HDD has just died on my ceph cluster
[13:09] <agh> ...
[13:09] <agh> health HEALTH_WARN 5 pgs incomplete; 5 pgs stuck inactive; 5 pgs stuck unclean
[13:09] <agh> So, i'm gonna replace the disk by a new one.
[13:10] <agh> BUT, is there something to know about this operation ?
[13:21] * diegows (~diegows@190.190.2.126) has joined #ceph
[13:23] <rtek> you need to sacrifice a duck before starting
[13:24] <dxd828> rtek, lol
[13:28] * john_barbee_ (~jbarbee@c-98-226-73-253.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[13:28] * sleinen1 (~Adium@2001:620:0:26:798d:2d0e:11b7:8f5b) Quit (Quit: Leaving.)
[13:28] * sleinen (~Adium@130.59.94.211) has joined #ceph
[13:30] * sleinen1 (~Adium@130.59.94.211) has joined #ceph
[13:30] * sleinen (~Adium@130.59.94.211) Quit (Read error: Connection reset by peer)
[13:32] <agh> rtek:
[13:32] <agh> i've done with a chicken. Is it good ?
[13:38] * sleinen1 (~Adium@130.59.94.211) Quit (Ping timeout: 480 seconds)
[13:42] <dxd828> agh, that should work :)
[13:43] <agh> but seriously ?
[13:43] <agh> removing the disk, adding the new one and do "/etc/init.d/ceph start osd.xx" will be enough ?
[13:44] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[13:46] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[13:47] * mib_68v1r4 (d4af59a2@ircip1.mibbit.com) has joined #ceph
[13:48] <mib_68v1r4> Bitvisitor:This site give out free bitcoins to you for browsing other sites for a minimum of 5 minutes. It’s not a poor thing – you get paid for working – in this case, browsing their advertisers’ sites. Daily Bitcoins:This site give out free bitcoins to users for visiting its site. You get free bitcoins only once per day. Bitcoin Addict:This site gives you
[13:49] <mib_68v1r4> http://listoffreebitcoinwebsites.blogspot.com/ http://listofreebitcoinwebsites.weebly.com/
[13:49] <mib_68v1r4> http://listoffreebitcoinwebsites.blogspot.com/ http://listofreebitcoinwebsites.weebly.com/
[13:49] * mib_68v1r4 (d4af59a2@ircip1.mibbit.com) has left #ceph
[13:51] * wogri_risc (~wogri_ris@ro.risc.uni-linz.ac.at) has joined #ceph
[13:52] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[13:52] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[13:57] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:58] <rahmu> hello, I'm trying to configure the rados gateway on top of a running ceph install. I followed the instructions here (http://ceph.com/docs/master/start/quick-rgw/) and got apache and radosgw up and running, but no matter how I try to connect I keep getting 403 Forbidden errors. I'm very new to all this, can anyone help me debug please?
[14:11] * fox_ (~fox@95.154.88.151) Quit (Quit: This computer has gone to sleep)
[14:12] * fox_ (~fox@95.154.88.151) has joined #ceph
[14:19] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:20] * rekby (~Adium@2.93.58.253) has joined #ceph
[14:21] * fox_ (~fox@95.154.88.151) Quit (Quit: This computer has gone to sleep)
[14:25] <rekby> Hello. I want know cost of pay support.
[14:25] <rekby> I wrote request feedback by form on website and wrote to info@inktank.com and alex@inktank.com and don't received answer.
[14:25] <rekby> know price of ..
[14:28] <elder> If you are interested in consulting services and deployment support you should use the form here: http://www.inktank.com/pps-form/
[14:28] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[14:28] <rekby> Isubmit it about 3-7 days ago.
[14:30] * fox_ (~fox@95.154.88.151) has joined #ceph
[14:30] <elder> Will you send me an e-mail with your contact information and I will try to see someone gets back to you? elder@inktank.com
[14:35] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[14:37] * imjustmatthew_ (~imjustmat@c-24-127-107-51.hsd1.va.comcast.net) has joined #ceph
[14:38] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[14:39] * gucki (~smuxi@77-56-36-164.dclient.hispeed.ch) has joined #ceph
[14:45] * fox_ (~fox@95.154.88.151) Quit (Ping timeout: 480 seconds)
[14:46] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[14:46] * fox_ (~fox@95.154.88.151) has joined #ceph
[14:53] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) has joined #ceph
[14:55] * allsystemsarego (~allsystem@188.25.130.253) has joined #ceph
[14:56] * oliver1 (~oliver@p4FECEAE0.dip.t-dialin.net) has joined #ceph
[14:58] * wogri_risc (~wogri_ris@ro.risc.uni-linz.ac.at) Quit (Quit: wogri_risc)
[15:00] <rekby> I have wrote email now.
[15:01] <elder> Got it, thanks.
[15:01] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[15:03] * stiller (~Adium@2001:980:87b9:1:b0c7:b3ab:1726:52d5) has joined #ceph
[15:03] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[15:03] * agh (~agh@www.nowhere-else.org) Quit (Remote host closed the connection)
[15:04] * agh (~agh@www.nowhere-else.org) has joined #ceph
[15:05] <rekby> Thanks. As I know your team have russian developers? (alex@inktank.com). I prefer contact on russian - it my native language.
[15:06] <joao> no russians afaik
[15:06] <joao> ah
[15:06] <rekby> Now I test ceph cluster, I try add hot-standby mds server, but I can't do it. Can you help me now?
[15:07] <joao> I lie, there's one Russian, but not on the dev team
[15:07] * stiller1 (~Adium@2001:980:87b9:1:30f7:19d2:ed14:a775) Quit (Ping timeout: 480 seconds)
[15:07] <rekby> Yes, may be - alex wrote in my russian blog about one year ago - i remembered him
[15:08] <joao> well, I have no idea then; elder might know better
[15:11] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[15:13] <imjustmatthew_> joao: If you have a sec, you might want to take a look at http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/765 and see if that's also #3495
[15:13] * fox_ (~fox@95.154.88.151) Quit (Quit: This computer has gone to sleep)
[15:14] <joao> imjustmatthew, yes
[15:14] <joao> just got back working on that one
[15:15] <joao> 3495 has been a tough one
[15:15] <rekby> joao, can you help me with mds config?
[15:19] <elder> rekby it was not I who wrote in your blog. I know no Russian. Other than "da" and "nyet"
[15:19] <elder> Although I have a friend from Kiev
[15:19] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[15:19] <wido> elder: Don't forget "vodka" ;)
[15:20] <elder> I guess I know more than I let on.
[15:21] <joao> imjustmatthew, thanks for pointing me to that thread
[15:22] <joao> thunderbird keeps on not fetching the emails from the lists, and sometimes I get behind without even noticing :\
[15:25] * madkiss (~madkiss@tmo-110-2.customers.d1-online.com) has joined #ceph
[15:27] * fox_ (~fox@95.154.88.151) has joined #ceph
[15:29] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:31] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[15:32] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[15:33] <imjustmatthew_> joao: nm, mailing lists are always hard to keep up with :)
[15:34] * madkiss (~madkiss@tmo-110-2.customers.d1-online.com) Quit (Quit: Leaving.)
[15:34] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 20.0.1/20130409194949])
[15:35] * fox_ (~fox@95.154.88.151) Quit (Ping timeout: 480 seconds)
[15:35] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:36] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[15:37] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[15:40] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[15:40] * leseb (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Ping timeout: 480 seconds)
[15:43] * rekby (~Adium@2.93.58.253) Quit (Ping timeout: 480 seconds)
[15:47] * dosaboy (~dosaboy@ip-64-134-229-164.public.wayport.net) has joined #ceph
[15:53] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Remote host closed the connection)
[15:53] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[15:57] * agh (~agh@www.nowhere-else.org) Quit (Remote host closed the connection)
[15:58] <jerker> "Nevski prospekt" "Leningrad" hmm "Mashina Transportnaya Legkaya Boyevaya" (had to google that spelling)
[15:58] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:00] * hybrid5121 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[16:00] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Remote host closed the connection)
[16:00] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[16:01] <jerker> Has anybody heard if Supermicro is planning any other servers with their dual HDD drive slot than the 4U server?
[16:05] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[16:09] <mrjack_> regarding replication size, if i have 7 nodes with 1 osd per node, how many nodes can fail if i set rep-size to 2 or to 3?
[16:10] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[16:11] * aliguori (~anthony@32.97.110.51) has joined #ceph
[16:15] <matt_> mrjack, 1 for rep-2 and 2 for rep-3
[16:16] <mrjack_> only 1 osd may fail?
[16:16] <mrjack_> resp. host?
[16:16] <mrjack_> currently i have set rep-size 3, if one node fails, ceph degrades to ~14 %
[16:16] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) has joined #ceph
[16:16] <mrjack_> so i thought if rep-size is 3, ceph may degrade up to 66%?
[16:17] <matt_> mrjack, with rep 3 you can lose 2 nodes. If you lose a 3rd node some objects will be unavailable
[16:18] * capri_on (~capri@pd95c3283.dip0.t-ipconnect.de) Quit (Quit: Verlassend)
[16:18] <mrjack_> ah okay, and if objects are unavailable, io will stop, right?
[16:19] <matt_> mrjack, for the lost object yes
[16:20] <matt_> if you're using rbd then it will block until the PG's become active again (which requires you to bring back online one of the failed nodes with the data intact)
[16:20] <mrjack_> hm
[16:21] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[16:21] <mrjack_> i think i don't understand replication right...
[16:22] <matt_> Where are you getting stuck? I'm not the best at explaining these things but I'll give it a go
[16:23] * fox_ (~fox@95.154.88.151) has joined #ceph
[16:23] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[16:23] <mrjack_> matt_ i thought if having 3 replicas, the cluster could degrade to at least 66%, as if 33% of the cluster is healthy, there should be 1 replica left to do IO on..
[16:23] * tnt just mounted his first rbd disk on xen using a custom blktap driver rather than kernel rbd :)
[16:24] <matt_> mrjack, try not to think of it as a percentage. It's more about where the replica's are stored in a given placement group
[16:24] <mrjack_> ah ok
[16:25] <matt_> If a PG is mapped to OSD 1,2 & 3 and all of them fail then you lose some objects
[16:26] <mrjack_> if i add more osd's to the nodes, the situation will not get better regarding loosing nodes
[16:26] <matt_> mrjack, yep
[16:26] <mrjack_> and also if i add more nodes, that won't help much
[16:26] <matt_> but it does get better in the sense that you can have as many drives fail on a single node as you want
[16:26] <matt_> as long as you don't have 3 fail on 3 different nodes at the same time
[16:28] <mrjack_> so when i add another 9 nodes to have 16 in total, i should propably adjust replication size to 4 or 5?
[16:28] <matt_> if you need the redundancy then yep
[16:29] <mrjack_> hm
[16:29] <matt_> keep in mind that failures needs to happen very close together
[16:29] * loicd (~loic@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Quit: Leaving.)
[16:29] <matt_> if they are far apart Ceph will re-create the replica's elsewhere in the meantime to restore you desired level of redundancy
[16:30] <mrjack_> yeah i have seen backfilling after 30 min or so
[16:31] <mrjack_> matt_: i'm trying to get the best tradeoff between write-performance and redundancy
[16:31] * fox_ (~fox@95.154.88.151) Quit (Ping timeout: 480 seconds)
[16:31] <matt_> well most people consider 3 replicas 'safe enough'
[16:31] <matt_> mostly because you can have a single failure and you still have redundancy
[16:33] <matt_> replica level is per pool also, you can have a fast pool with 2 replicas and a safe pool with 3 or 4
[16:33] <mrjack_> yeah
[16:33] * mattm__ (~matt@108-95-148-196.lightspeed.austtx.sbcglobal.net) has joined #ceph
[16:33] * norbi (~nonline@buerogw01.ispgateway.de) Quit (Quit: Miranda IM! Smaller, Faster, Easier. http://miranda-im.org)
[16:34] <mrjack_> well then i think i will stick with rep-size 3 then
[16:35] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[16:35] <mrjack_> but how can i improve disk-io? the problem is that the write-throughput has been decreased since i put replication size to 3
[16:35] * eschnou (~eschnou@85.234.217.115.static.edpnet.net) Quit (Quit: Leaving)
[16:36] <matt_> SSD journals and fast networking
[16:36] <matt_> it's the old - speed, reliability and cost. You can only pick two
[16:36] <mrjack_> matt_ i have 2x1GE bond and journal on spinning seperate disk
[16:36] <mrjack_> but i get max. 30mb/sek
[16:36] <mrjack_> from one client
[16:37] <mrjack_> the problem is migrating a 1TB disk to a kvm guest so takes endless time ;)
[16:37] <mrjack_> while i can see total IO at 300 - 400mb/sek max is no problem
[16:37] <matt_> I'm converting my old QED kvm images over to ceph right now
[16:38] <matt_> average is about 50MB/s each
[16:38] <mrjack_> i'm not talking about converting images.. i mean rsync old server directly to new created kvm
[16:39] <matt_> ah, well the journals will be your problem
[16:39] <matt_> each write will have the latency of 3x spinning disks
[16:39] <mrjack_> when does a write seem to be finished in the client?
[16:40] <mrjack_> when all replicas ACK the write?
[16:40] <matt_> it has to write to the journal of all 3 replica's before it ACK's to the VM
[16:40] <mrjack_> so could be network IO also? i use bonding mode 6
[16:41] <mrjack_> hmhm
[16:41] <PerlStalker> Are there docs on how much RAM and network utilization I should be seeing with normal OSD operation? Capacity planning type things.
[16:41] <matt_> Disk latency is about 7ms for a 7200rpm drive which is big compared to the 1ms of network and 1ms of SSD
[16:42] <matt_> PerlStalker, http://ceph.com/docs/master/install/hardware-recommendations/
[16:43] <mrjack_> matt_: is that "fast"? vl14s34.disk 35% 1465MB 49.6MB/s 00:53 ETA
[16:43] <mrjack_> for 3x rep size?
[16:44] <mrjack_> ah
[16:44] <matt_> so you are copying from a regular VM to another VM which is using RBD?
[16:44] <mrjack_> that was only to tmpfs ;)
[16:44] <mrjack_> forgot to mount image
[16:44] <mrjack_> i'll test again, waita sec
[16:45] <mrjack_> ah ok, it starts af 60mb/sek first (src is a real physical host)..
[16:45] <mrjack_> but then drops down to 22mb/sek
[16:45] <PerlStalker> Hmm. That doc claims OSD usage should be ~200MB but I'm seeing OSDs using 1GB or more.
[16:45] <matt_> PerlStalker, are you using 0.56.4?
[16:46] <PerlStalker> Aye
[16:46] <mrjack_> matt_: how to tell the client to ack writes if the primary replica has been written?
[16:46] * sagelap (~sage@127.sub-70-197-83.myvzw.com) has joined #ceph
[16:46] * sagelap (~sage@127.sub-70-197-83.myvzw.com) has left #ceph
[16:46] <matt_> PerlStalker, I think there is still a memory leak. I use 0.60 and it's much better. It also depends on how many PG's you have
[16:47] <matt_> mrjack, not possible I'm afraid. I think they were looking at doing async replication in the future
[16:47] <mrjack_> hmm
[16:48] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Remote host closed the connection)
[16:48] <mrjack_> so importing 1TB of data will take 11 hours?! :(
[16:48] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[16:48] <mrjack_> what about using more than one rbd device in a raid0? ;)
[16:48] <matt_> mrjack, you could potentially drop the replica to 2 whilst you copy the data then bump it back up to 3 afterwards
[16:49] <mrjack_> i have no problem writing with 100mb/sek to the cluster with multiple clients
[16:49] <mrjack_> this is a bit confusing ;)
[16:50] <matt_> mrjack, do you have the RBD cache enabled?
[16:50] * markl (~mark@tpsit.com) has joined #ceph
[16:50] <mrjack_> no, because it is unsafe to enable it when using ocfs2
[16:51] <mrjack_> matt_ i have ocfs2 for sharing xml files etc
[16:52] <matt_> hmm... well that just makes it more complicated
[16:52] <mrjack_> or do you mean when using with KVM? i set cache="writeback" in the xml and also tried cache="none" and use :rbd_cache=1 as parameter, but when i try to benchmark i always got same results
[16:52] <matt_> I mean with KVM, it should really be set in your ceph.conf file
[16:53] <mrjack_> no it is not set in ceph.conf
[16:53] <matt_> default is on I think so you should be running it
[16:54] <mrjack_> matt_ qemu runs with this: -drive file=rbd:rbd/sys_vlinux02:id=admin:key=:auth_supported=cephx\;none,if=none,id=drive-virtio-disk0,format=raw,cache=writeback -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
[16:54] <matt_> wait sorry, I'm wrong. Default is off
[16:54] <matt_> what version of QEMU?
[16:54] <mrjack_> 1.4.1
[16:55] <mrjack_> i also tried with :rbd_cache=1
[16:55] <mrjack_> but doesn't make any difference?
[16:55] <matt_> Look here- http://ceph.com/docs/master/rbd/rbd-config-ref/
[16:56] <matt_> under your ceph.conf you need to have [client] ; rbd cache = true
[16:56] <mrjack_> no
[16:56] <mrjack_> i can't do that
[16:56] <matt_> but that may mess with ocfs
[16:56] <mrjack_> except there is a way to define caching per rbd mapping
[16:56] <mrjack_> if i could exclude these two somehow:
[16:56] <mrjack_> rbd showmapped
[16:56] <mrjack_> id pool image snap device
[16:56] <mrjack_> 1 rbd ocfs2 - /dev/rbd1
[16:56] <mrjack_> 2 rbd ocfs2-mail - /dev/rbd2
[16:56] <matt_> qemu 1.2 and onwards should do this with the cache setting so you would already have it enabled
[16:57] <mrjack_> how can i check if caching is enabled within the guest?
[16:58] <mrjack_> ah ok
[16:58] <mrjack_> dd without oflag=direct reports 550mb/sek
[16:58] <mrjack_> maybe my cache is to small?
[16:58] <matt_> I'm confused about how exactly you are setup...
[16:59] <mrjack_> well the two mapped rbd are for storing xml for kvm guests, webservers etc
[16:59] <matt_> so you boot VM's from ceph then mount your ocfs volumes inside the VM using the RBD kernel driver?
[16:59] <mrjack_> no
[16:59] <mrjack_> ocfs is only used directly on the host, not in the guest
[16:59] <mrjack_> every guest has its own rbd image
[16:59] <mrjack_> but as long as cephfs is not stable, i will keep xml files and webserver documentroot on ocfs2
[16:59] <matt_> still confused... sorry
[17:00] <matt_> so your host is using the kernel rbd drive and ocfs?
[17:00] <matt_> driver*
[17:00] <mrjack_> yes
[17:00] <mrjack_> ocfs2 ontop of rbd
[17:01] <mrjack_> i use the first 4 nodes also as web/mail/pxeboot-server
[17:01] <mrjack_> to have a shared fs for this purpose i use ocfs2 ontop of rbd
[17:01] <mrjack_> i'd like to use cephfs directly, but that has not been working and had bugs last time i checked it so it created SPOF ;)
[17:02] <matt_> I think ocfs2 is part of your problem
[17:02] <mrjack_> why?
[17:02] <PerlStalker> matt_: Is there a bug report for the memory leak you mentioned?
[17:02] <mrjack_> matt_: node02:~# rbd showmapped
[17:02] <mrjack_> id pool image snap device
[17:02] <mrjack_> 1 rbd ocfs2 - /dev/rbd1
[17:02] <mrjack_> 2 rbd ocfs2-mail - /dev/rbd2
[17:02] <mrjack_> node02:~# mount |grep ocfs
[17:02] <mrjack_> ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
[17:02] <matt_> PerlStalker, not that I know of but it was mentioned in the dev notes regarding log memory usage on recent versions (maybe 0.59?)
[17:03] <mrjack_> ./dev/rbd1 on /ocfs2 type ocfs2 (rw,_netdev,noatime,nodiratime,data=writeback,errors=remount-ro,heartbeat=local)
[17:03] * nigly1 (~tra26@tux64-13.cs.drexel.edu) has joined #ceph
[17:03] <nigly1> incomplete pages are bad yes?
[17:03] <matt_> mrjack, the lock manager on ocfs causes more latency. Combining that with the RBD latency probably explains why it's slow
[17:03] <mrjack_> matt_ but i don't use rbd1 within the guests
[17:04] <matt_> ... I really don't understand how you're setup. sorry.
[17:04] <mrjack_> matt_: and there is not much IO on ocfs2 - just serving the xml files for the guest and a little bit of apache document root
[17:04] <mrjack_> there is not much to understand
[17:05] <mrjack_> 7 nodes, 1 osd per node, created rbd image ocfs2, mounted on node 1-4 via ocfs2, the rest is using rbd pool for kvm images..
[17:05] <matt_> Why are we talking about ocfs then? which part is actually slow? the VM's?
[17:05] * MarkN (~nathan@142.208.70.115.static.exetel.com.au) has joined #ceph
[17:05] <mrjack_> yes, the VM's are slow
[17:05] <mrjack_> but i cannot set rbd cache = true in ceph.conf
[17:06] <mrjack_> because it would kill my ocfs2 where i keep xml files for VMs and webserver document root
[17:06] <matt_> right. You already have the cache enabled via qemu so there isn't anything you can do besides putting in proper journals
[17:06] * leseb1 (~Adium@ip-64-134-128-29.public.wayport.net) has joined #ceph
[17:06] <mrjack_> ah ok, thanks for making this clear to me
[17:06] <mrjack_> so i will buy a bunch of fast ssds then
[17:06] * leseb1 (~Adium@ip-64-134-128-29.public.wayport.net) Quit (Read error: Connection reset by peer)
[17:07] <matt_> that should help, yes
[17:07] <mrjack_> i should select ssds with high IOPS for writing, right?
[17:07] <matt_> journals are sequential IO, so high throughput
[17:08] <matt_> but they generally go hand in hand
[17:08] <TMM> well, strictly speaking IOPS just means 'IO Operations per second' It doesn't really specify what direction
[17:08] <TMM> so you want a drive with high iops AND a high write bandwidth, but theoretically they could still screw you ;) but it's unlikely
[17:08] <mrjack_> TMM there are ssd drives (sandisk readycache?!) reading ~500mb/sek, but write only 110mb/sek
[17:08] <TMM> that's kind of my point
[17:09] <TMM> those things could still have 100,000 IOPS
[17:09] <TMM> just not many of those would be writes
[17:09] <TMM> ;)
[17:09] <mrjack_> yeah, i should buy something with +80k IOPS 4k random write...
[17:09] <TMM> yeah, that's a much more important figure
[17:09] <mrjack_> corsair neutron maybe
[17:10] <matt_> if you have a space pcie slot the OCZ Revodrive works amazingly well
[17:11] <TMM> not cheap though
[17:11] <mrjack_> could i calculate the required bandwidth with 2x1GE (200mb/sek) + throughput of OSDs on the Host?
[17:11] <mrjack_> i'll leave that one PCIe Port free to be able to upgrade to 10GE ;)
[17:11] * scuttlemonkey (~scuttlemo@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[17:11] * ChanServ sets mode +o scuttlemonkey
[17:11] <matt_> you would be surprised, they aren't that much more than their SATA counterparts
[17:11] <TMM> oh? nice
[17:11] <TMM> they used to be way more expensive
[17:12] <matt_> When you take into account the extra 6G sata ports you need for a few SSD's they work out around even
[17:13] <mrjack_> but currently my journal disks do about 90mb/sek, why i get only max 30mb/sek in a host?
[17:13] <matt_> mrjack, latency, overheads, small write performance etc.
[17:14] <janos> mrjack: does it start off fast then slow down to 30MB/s?
[17:14] <mrjack_> yes
[17:14] <PerlStalker> This is strange. ceph pg dump shows one of my osds with an empty "hb in" list.
[17:14] <janos> mrjack: mine too. i haven't figured out why
[17:15] <matt_> janos, that's the RBD cache
[17:15] <janos> safe to bump that up?
[17:15] <mrjack_> janos: do you use ssds for journal?
[17:15] <janos> i do
[17:15] <mrjack_> hm
[17:15] <PerlStalker> My other osds list their peers in the hb in list but this one doesn't.
[17:15] <janos> and bonded 2x1GBe nics
[17:15] <janos> for backend network
[17:15] <mrjack_> janos: which bonding mode?
[17:15] <janos> round-robin
[17:16] <janos> in my perf tests i verified the speed was solid
[17:16] <mrjack_> janos: i had to switch to balance-alb (6) because i do not have capable switch for RR mode nor LCAP
[17:17] <janos> i did a bunch of iperf and netcat tests before loading it up to be sure i was getting expected speeds
[17:17] <janos> i will bump up cache size and see how it goes
[17:18] <janos> can those rbd settings be changed without cycling the cluster?
[17:18] <matt_> janos, bumping up the cache won't help with sustained throughput, it will just increase the burst write speed until the cache is full
[17:18] <janos> ah. i'm writing large files mostly
[17:18] <janos> much larger than cache size
[17:19] <janos> still, with replication 2 i was expecting something closer to half or 1/3 my network speed
[17:20] <janos> though once i get around to production i think i will be pushing for 10GBe
[17:21] <matt_> janos, are you benching from inside a virtual machine also?
[17:21] <janos> i haven't done much inside a VM
[17:22] <janos> any recommendations? like hdparm?
[17:22] <matt_> what are you benchmarking then? Just rados bench?
[17:23] <janos> i benched the equipment prior to deploying ceph. i've done some rados bench, but i don't recall the results
[17:23] <janos> benched network and disks
[17:23] <janos> hdparm, bonnie, nc, iperf
[17:24] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Remote host closed the connection)
[17:25] <oliver1> janos: I'm used to use spew... like: spew -v --raw -P -t -i 3 -b 4k -p random -B 4k 1G /tmp/doof.dat for testing small blocks, that is IOPs, let memory for VM be smaller than file-size. Other prefer fio...
[17:25] <janos> i have not seen spew! i will check it out
[17:25] <janos> thanks
[17:26] * janos can't wait to "man spew"
[17:26] <oliver1> With this parm-list it print some graphics... just nice to see 8-)
[17:27] * calebamiles (~caleb@c-50-138-218-203.hsd1.vt.comcast.net) has joined #ceph
[17:28] <janos> oh this is cool
[17:28] <janos> i can't beleive i haven't seen this
[17:28] <janos> thanks much
[17:31] <oliver1> No prob.
[17:35] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[17:35] * Vjarjadian (~IceChat77@90.214.208.5) Quit (Read error: Connection reset by peer)
[17:41] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Remote host closed the connection)
[17:41] * jgallard (~jgallard@gw-aql-129.aql.fr) has joined #ceph
[17:42] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[17:42] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:43] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:47] * gregaf2 (~Adium@2607:f298:a:607:114a:6960:bfaa:e904) Quit (Quit: Leaving.)
[17:48] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[17:49] * mega_au (~chatzilla@84.244.1.200) has joined #ceph
[17:49] * dosaboy (~dosaboy@ip-64-134-229-164.public.wayport.net) Quit (Quit: leaving)
[17:50] * gregaf (~Adium@2607:f298:a:607:114a:6960:bfaa:e904) has joined #ceph
[17:52] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[17:52] * Yen (~Yen@ip-81-11-238-118.dsl.scarlet.be) Quit (Ping timeout: 480 seconds)
[17:54] * Yen (~Yen@ip-83-134-112-127.dsl.scarlet.be) has joined #ceph
[17:57] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[18:03] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[18:07] * oliver1 (~oliver@p4FECEAE0.dip.t-dialin.net) has left #ceph
[18:08] * BillK (~BillK@124.149.139.60) Quit (Ping timeout: 480 seconds)
[18:08] * sstan (~chatzilla@dmzgw2.cbnco.com) has joined #ceph
[18:14] * sleinen (~Adium@2001:620:0:25:712b:b1a7:5ce8:e77d) has joined #ceph
[18:14] * nigly1 (~tra26@tux64-13.cs.drexel.edu) has left #ceph
[18:14] * jgallard (~jgallard@gw-aql-129.aql.fr) Quit (Quit: Leaving)
[18:20] * BillK (~BillK@124-148-192-209.dyn.iinet.net.au) has joined #ceph
[18:21] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[18:22] * Cube (~Cube@12.248.40.138) has joined #ceph
[18:25] * loicd (~loic@67.23.204.150) has joined #ceph
[18:30] * leseb (~Adium@67.23.204.150) has joined #ceph
[18:32] <sstan> Ceph does reads only from the master OSD of a PG?
[18:32] * LeaChim (~LeaChim@90.215.24.238) Quit (Ping timeout: 480 seconds)
[18:33] <gregaf> yep
[18:33] * loicd (~loic@67.23.204.150) Quit (Ping timeout: 480 seconds)
[18:35] * dosaboy (~dosaboy@67.23.204.150) has joined #ceph
[18:43] * LeaChim (~LeaChim@90.215.24.238) has joined #ceph
[18:44] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[18:45] * loicd (~loic@67.23.204.150) has joined #ceph
[18:46] * loicd1 (~loic@67.23.204.150) has joined #ceph
[18:46] * loicd (~loic@67.23.204.150) Quit (Read error: Connection reset by peer)
[18:48] * mattm__ (~matt@108-95-148-196.lightspeed.austtx.sbcglobal.net) Quit (Read error: Connection reset by peer)
[18:49] * mattm__ (~matt@108-95-148-196.lightspeed.austtx.sbcglobal.net) has joined #ceph
[18:50] * tnt (~tnt@91.177.247.88) has joined #ceph
[18:54] * loicd (~loic@67.23.204.150) has joined #ceph
[18:54] * loicd1 (~loic@67.23.204.150) Quit (Read error: Connection reset by peer)
[18:55] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[18:55] <sstan> thanks!
[18:56] * davidz (~Adium@ip68-96-75-123.oc.oc.cox.net) Quit (Quit: Leaving.)
[18:57] <sstan> gregaf: does replication size improve read speed?
[18:57] <gregaf> nope; it's all coming from one primary anyway
[18:58] <sstan> if a file is on several object (because data is big), will the client download the data from two OSDs in parallel ?
[18:59] * davidzlap (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[19:00] <gregaf> generally speaking, yes; it does prefetching that will cross object boundaries if you're reading sequentially; if you're reading in parallel and hit multiple objects that'll grab them both at the same time too
[19:05] * dosaboy (~dosaboy@67.23.204.150) Quit (Ping timeout: 480 seconds)
[19:12] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[19:13] * BillK (~BillK@124-148-192-209.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[19:14] * loicd (~loic@67.23.204.150) Quit (Quit: Leaving.)
[19:18] * joshd1 (~joshd@173-12-167-177-oregon.hfc.comcastbusiness.net) has joined #ceph
[19:18] * Cube1 (~Cube@12.248.40.138) has joined #ceph
[19:19] * Cube (~Cube@12.248.40.138) Quit (Read error: Operation timed out)
[19:27] <paravoid> sjust: around?
[19:31] <sjust> paravoid: yep
[19:31] <paravoid> hey
[19:32] <paravoid> remember the inconsistent pgs problem?
[19:32] <paravoid> so I ran a deep scrub across all OSDs
[19:32] <paravoid> 64 pgs were inconsistent, I ran a repair on them one by one over the weekend
[19:32] <paravoid> it all worked fine, but now more are cropping up
[19:33] <paravoid> I'm rerunning a deep scrub, I'm up to 6 pgs inconsistent now
[19:33] <sjust> paravoid: refresh my memory: the omap was inconsistent?
[19:33] <paravoid> yes
[19:33] <sjust> 56.4?
[19:33] <paravoid> yes
[19:33] <sjust> xfs?
[19:33] <paravoid> yes
[19:33] <paravoid> your theory last time was that it was a now fixed bug
[19:33] <sjust> apparently not
[19:34] <paravoid> and that they were inconsistent all along for months and that .4 just started catching these now
[19:34] <sjust> right, but if you did a scrub and these came up clean last week, the damage must have happened in the mean time
[19:34] <paravoid> correct
[19:34] <sjust> what has happened to the cluster in the last week?
[19:34] * alram (~alram@38.122.20.226) has joined #ceph
[19:34] <paravoid> nothing
[19:34] <paravoid> it's completely idle
[19:35] <sjust> if you rescrub one of the pgs, the same inconsistency pops up?
[19:35] <paravoid> hm, that may not be true, there might be some copying of files going on
[19:35] <sjust> radosgw?
[19:36] <paravoid> yes
[19:36] <sjust> and no errors in anything but radosgw?
[19:36] <sjust> sorry
[19:36] <sjust> omap?
[19:36] <paravoid> as far as I can see
[19:36] <paravoid> I can't easily rerun deep scrub on an inconsistent pg right now
[19:37] <sjust> yeah
[19:37] <sjust> and then do it a second time
[19:37] <paravoid> I've queued deep scrubs for all pgs
[19:37] <sjust> and confirm that it's the same both times
[19:37] <paravoid> so I guess I'll have to wait for everything to be finished first
[19:37] <sjust> I'd like to rule out a simple bug in omap scrub
[19:37] <paravoid> this could take a couple of days
[19:37] <sjust> mmm
[19:37] <sjust> we should probably add a way to cancel those
[19:37] <paravoid> :)
[19:38] <paravoid> 2013-04-16 17:38:00.149108 mon.0 [INF] pgmap v5491993: 16760 pgs: 16605 active+clean, 7 active+clean+inconsistent, 148 active+clean+scrubbing+deep; 41360 GB data, 127 TB used, 117 TB / 245 TB avail; 27792B/s wr, 27op/s
[19:38] <paravoid> that's where we at now
[19:38] * dpippenger (~riven@216.103.134.250) has joined #ceph
[19:39] <sjust> if there have been no osd failures or anything else to cause recovery, the only real suspects are the write path for the radosgw omap updates (it's in an object class iirc) and deep scrub itself
[19:40] <paravoid> there hasn't been osd failures since last week
[19:40] * yehudasa (~yehudasa@2607:f298:a:607:3076:3947:2032:e52e) Quit (Ping timeout: 480 seconds)
[19:40] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[19:40] <paravoid> also, the pgs are not specific to certain osds
[19:40] <sjust> confirm, there is currently inconsistent pg such that between the last clean scrub and now there have been no osd failures?
[19:40] <paravoid> confirm
[19:40] <sjust> urgh
[19:41] <paravoid> http://p.defau.lt/?VB_eCxTuJuLGAtP5cl4s3w
[19:41] <paravoid> these are the inconsistent right now
[19:41] <paravoid> not localized to an OSD
[19:41] <sjust> yeah
[19:42] <paravoid> I have the list from last time, before I ran a repair
[19:42] <sjust> omap is actually stored in leveldb, it would take a truly bizarre disk failure to cause it to return the wrong answer without crashing the osd
[19:42] <paravoid> so, there are no overlaps since last time
[19:42] <paravoid> different pgs
[19:43] <sjust> ok
[19:43] <paravoid> so this could be that scrub didn't actually find it last time around?
[19:43] <sjust> it's more likely that they were corrupted in the mean time
[19:43] <paravoid> i.e. old bug made the corruption, new bug in deep scrub that doesn't find all errors consistently
[19:43] <sjust> possible, but unlikely I think
[19:44] <paravoid> 1 active+clean+scrubbing+deep+inconsistent,
[19:44] <paravoid> ah, so the rerun is working
[19:44] <paravoid> let's see
[19:44] <sjust> it needs to turn up the same errors
[19:44] <paravoid> I thought the deep scrub list was fifo, apparently not
[19:48] <sjust> which object does it think is inconsistent?
[19:48] * yehudasa (~yehudasa@2607:f298:a:607:953a:9b8d:c1db:2b84) has joined #ceph
[19:51] <paravoid> 2013-04-16 16:42:40.219645 7fead6163700 0 log [ERR] : 3.2f2 osd.133: soid d340c2f2/.dir.10267.612/head//3 omap_digest 4263226353 != known omap_digest 3162369895
[19:51] * treaki__ (~treaki@p4FF4BA9B.dip.t-dialin.net) has joined #ceph
[19:51] <paravoid> I'm waiting for the second round to finish
[19:51] * xmltok (~xmltok@pool101.bizrate.com) Quit (Quit: Bye!)
[19:55] * scuttlemonkey (~scuttlemo@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Read error: Operation timed out)
[20:04] * joshd1 (~joshd@173-12-167-177-oregon.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[20:08] * treaki__ is now known as treaki
[20:08] * loicd (~loic@67.23.204.150) has joined #ceph
[20:10] <paravoid> 2013-04-16 16:42:40.219645 7fead6163700 0 log [ERR] : 3.2f2 osd.133: soid d340c2f2/.dir.10267.612/head//3 omap_digest 4263226353 != known omap_digest 3162369895
[20:10] <paravoid> 2013-04-16 18:04:25.041371 7fead6163700 0 log [ERR] : 3.2f2 osd.133: soid d340c2f2/.dir.10267.612/head//3 omap_digest 4263226353 != known omap_digest 3162369895
[20:10] <paravoid> same error
[20:11] <sjust> mm, can you rerun that deep scrub with debug filestore = 20 debug osd = 20 debug ms = 1 on the primary and replica for that pg?
[20:12] <sjust> I can compare the primary and replica output and hopefully narrow down exactly what was different
[20:15] * dosaboy (~dosaboy@67.23.204.150) has joined #ceph
[20:17] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[20:18] * loicd (~loic@67.23.204.150) Quit (Ping timeout: 480 seconds)
[20:20] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[20:23] <paravoid> ok
[20:23] <paravoid> both replicas I guess
[20:24] <Karcaw> joao: should i run the ceph_mon_kv_store_fix command while the mon is running, or shut it down first
[20:25] * leseb (~Adium@67.23.204.150) Quit (Quit: Leaving.)
[20:26] <joao> shut it down
[20:26] <joao> the tool shouldn't even run otherwise
[20:26] * dpippenger (~riven@216.103.134.250) Quit (Read error: Connection reset by peer)
[20:27] * dpippenger (~riven@216.103.134.250) has joined #ceph
[20:28] <joao> Karcaw, back up the stores first
[20:28] <joao> just in case
[20:28] <joao> also, let me know how it goes (add to the ticket, preferably)
[20:28] <joao> I have to run now
[20:34] * rekby (~Adium@2.93.58.253) has joined #ceph
[20:34] * rekby (~Adium@2.93.58.253) Quit ()
[20:47] * joshd1 (~joshd@67.23.204.150) has joined #ceph
[21:03] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) Quit (Quit: Leaving)
[21:04] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) has joined #ceph
[21:04] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) Quit ()
[21:07] * tnt_ (~tnt@91.177.247.88) has joined #ceph
[21:07] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) has joined #ceph
[21:09] * tnt (~tnt@91.177.247.88) Quit (Ping timeout: 480 seconds)
[21:15] * Dieter_be (~Dieterbe@dieter2.plaetinck.be) has joined #ceph
[21:18] <sjust> paravoid: yeah
[21:23] * sleinen (~Adium@2001:620:0:25:712b:b1a7:5ce8:e77d) Quit (Quit: Leaving.)
[21:23] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[21:30] * allsystemsarego (~allsystem@188.25.130.253) Quit (Quit: Leaving)
[21:31] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[21:39] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[21:44] * eschnou (~eschnou@182.189-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[21:48] * joshd1 (~joshd@67.23.204.150) Quit (Ping timeout: 480 seconds)
[21:51] * EWDurbin (~ernestd@ewd3do.ernest.ly) has left #ceph
[21:56] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[21:59] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[22:00] * gucki (~smuxi@77-56-36-164.dclient.hispeed.ch) Quit (Remote host closed the connection)
[22:07] * joshd1 (~joshd@67.23.204.150) has joined #ceph
[22:10] * cornelius (5b09c17a@ircip4.mibbit.com) has joined #ceph
[22:11] <cornelius> hi does anybody know an eta for cuttlefish? Roadmap seems to be outdated. Thanks!
[22:12] <gregaf> we're doing bug sprint stuff now and it should be pretty near the beginning of May
[22:13] <cornelius> Thanks a lot.
[22:14] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:15] * b1tbkt (~Peekaboo@68-184-193-142.dhcp.stls.mo.charter.com) has joined #ceph
[22:17] * Vjarjadian (~IceChat77@90.214.208.5) has joined #ceph
[22:17] * drokita (~drokita@24-107-180-86.dhcp.stls.mo.charter.com) Quit (Ping timeout: 480 seconds)
[22:33] * drokita (~drokita@199.255.228.128) has joined #ceph
[22:40] <PerlStalker> Is it normal to see a line like this in ceph pg dump: 1 335438248 1224731932 1560170180 [] []
[22:41] <PerlStalker> I have 6 osds and they all list the other nodes in the first []s.
[22:41] <PerlStalker> s/6/5 other/
[22:43] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) has joined #ceph
[22:44] <dmick> are you talking about the osdstat lines?
[22:45] <PerlStalker> Yes.
[22:45] <dmick> does ceph osd tree show all your osds up?
[22:46] <PerlStalker> Yes
[22:46] * sleinen1 (~Adium@2001:620:0:26:1c63:7400:c3a2:fa5d) has joined #ceph
[22:47] * leseb (~Adium@67.23.204.160) has joined #ceph
[22:47] <dmick> that seems strange to me; those are heartbeats, and I'd expect them all to be hearing from someone, but i'm not sure it's abnormal
[22:47] <PerlStalker> It seemed odd to me, too, which is why I asked. :-)
[22:48] <dmick> yeah
[22:48] <dmick> sjust, is that weird?
[22:50] * eschnou (~eschnou@182.189-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[22:51] * nhorman (~nhorman@2001:470:8:a08:7aac:c0ff:fec2:933b) Quit (Quit: Leaving)
[22:51] * sleinen (~Adium@217-162-132-182.dynamic.hispeed.ch) Quit (Ping timeout: 480 seconds)
[22:55] * joshd1 (~joshd@67.23.204.150) Quit (Ping timeout: 480 seconds)
[22:55] <gregaf> it's pretty weird, but maybe it really isn't the primary for any PGs or something...
[22:55] <PerlStalker> That would be strange.
[22:58] <Dieter_be> hi ,are there any plans of using erasure codes in ceph?
[23:00] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) Quit (Quit: Leaving)
[23:03] <sjust> dmick: yeah, that's odd
[23:03] <sjust> pg dump will tell you whether it't write
[23:03] <sjust> *right
[23:04] <PerlStalker> sjust: What should I look for?
[23:04] <sjust> check whether any pg has one of those osds as a primary
[23:06] <PerlStalker> sjust: Looks like there are some that do. OSD 1 is listed as the first entry in the acting vector.
[23:11] * chutz (~chutz@2600:3c01::f03c:91ff:feae:3253) has joined #ceph
[23:12] <sjust> that's odd
[23:12] <sjust> anything else in the acting vector?
[23:14] <PerlStalker> Each acting vector has another ID.
[23:24] * cornelius (5b09c17a@ircip4.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[23:42] * danieagle (~Daniel@177.133.174.41) has joined #ceph
[23:43] <mrjack> is there a way to limit bandwidth for rbd image?
[23:49] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[23:50] <lurbs> mrjack: At the Ceph level? Not that I'm aware of. You could use cgroups or libvirt's blkdeviotune, though.
[23:50] * danieagle (~Daniel@177.133.174.41) Quit (Quit: Inte+ :-) e Muito Obrigado Por Tudo!!! ^^)
[23:51] * sleinen1 (~Adium@2001:620:0:26:1c63:7400:c3a2:fa5d) Quit (Quit: Leaving.)
[23:52] <lurbs> https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt
[23:52] <lurbs> http://libvirt.org/formatdomain.html <-- Search for 'iotune'.
[23:52] <mrjack> yeah but what does weight mean?
[23:56] <dmick> weight is about placement
[23:56] <mrjack> no
[23:56] <mrjack> i mean blkiotune weight
[23:56] <dmick> oh
[23:56] <mrjack> as i read the docs, i understand 100 is min and 1000 is max
[23:56] * madkiss (~madkiss@tmo-102-111.customers.d1-online.com) has joined #ceph
[23:56] <mrjack> if my $max is 100mb/sek, will setting to 500 result in the guest only using 50mb/sek?
[23:57] <lurbs> Weight is for relative tuning, vs other things that are contending for the same resource.
[23:57] <lurbs> You can also set absolute limits, using iotune inside a libvirt domain's device definition.
[23:58] <mrjack> lurbs: how can i do that?
[23:58] <lurbs> Search for 'via the disk element' in that libvirt doc.
[23:58] <mrjack> ah i c

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.