#ceph IRC Log

Index

IRC Log for 2013-04-05

Timestamps are in GMT/BST.

[0:02] <Elbandi_> 30-40 dirs, with ~2k file/dir
[0:03] <gregaf> hmm
[0:03] <gregaf> well, anyway, if it's working now I'm happy for the moment
[0:03] <gregaf> you probably want to go back to 1 active MDS now, though
[0:04] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[0:04] <Elbandi_> we want to use ceph, but now, i just tested
[0:05] <Elbandi_> functional, performance, etc
[0:05] <gregaf> always a good plan
[0:06] * aliguori (~anthony@32.97.110.51) Quit (Quit: Ex-Chat)
[0:06] <gregaf> just so's you know that Inktank is very careful about CephFS in production right now — we have a couple customers on it but we qualified them and their use cases pretty carefully; it needs to go through a lot more QA etc before we're comfortable advertising it for general use
[0:07] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) has joined #ceph
[0:09] * madkiss (~madkiss@tmo-098-254.customers.d1-online.com) has joined #ceph
[0:11] <PerlStalker> How does ceph decide that's it time to start scrubbing a pg?
[0:13] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[0:16] <Elbandi_> gregaf: how long it takes to make the real good fix?
[0:32] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[0:33] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) has joined #ceph
[0:35] * drico (~drico@188-230-177-62.dynamic.t-2.net) has joined #ceph
[0:35] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[0:35] * jskinner (~jskinner@50-80-32-52.client.mchsi.com) Quit (Remote host closed the connection)
[0:36] <drico> Hi everybody,
[0:36] <drico> Got a strange problem to mount my ceph cluster using ceph-fuse
[0:36] <drico> cluster is working ok, HEALTH ok
[0:37] * madkiss (~madkiss@tmo-098-254.customers.d1-online.com) Quit (Quit: Leaving.)
[0:38] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[0:39] <drico> but nothing get mounted
[0:39] <drico> i got this with debug enabled on the server i try to mount : http://pastebin.com/yU35Dtma
[0:41] <drico> does it represent some authentification failure ?
[0:42] * ivotron (~ivo@eduroam-225-108.ucsc.edu) Quit (Remote host closed the connection)
[0:42] * ivotron (~ivo@dhcp-59-178.cse.ucsc.edu) has joined #ceph
[0:46] * Vjarjadian_ (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[0:48] * ivotron (~ivo@dhcp-59-178.cse.ucsc.edu) Quit (Quit: leaving)
[0:48] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[0:50] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) Quit (Ping timeout: 480 seconds)
[0:52] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) has joined #ceph
[0:54] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) Quit (Ping timeout: 480 seconds)
[1:00] * tnt (~tnt@pd95cfcb4.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:06] * BillK (~BillK@124-169-229-107.dyn.iinet.net.au) has joined #ceph
[1:21] <dmick> drico: you mean the "client.?" part?
[1:34] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[1:41] * sjustlaptop (~sam@sccc-66-78-236-243.smartcity.com) Quit (Ping timeout: 480 seconds)
[1:47] * sjustlaptop (~sam@mce0536d0.tmodns.net) has joined #ceph
[1:48] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[1:48] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit ()
[1:48] <davidz> PerlStalker: If load is below osd_scrub_load_threshold (5) then once a day (osd_scrub_min_interval) a normal scrub will occur. Otherwise, if loads prevented this then a scrub will trigger once a week (osd_scrub_max_interval). A deep scrub will occur once a week (osd_deep_scrub_interval). Since osd_scrub_max_interval == osd_deep_scrub_interval by default, a delayed scrub will simply become a weekly deep scrub. IIRC
[1:48] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) has joined #ceph
[1:52] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Quit: Leaving.)
[1:57] * The_Bishop__ (~bishop@f052097229.adsl.alicedsl.de) has joined #ceph
[1:59] * alram (~alram@38.122.20.226) Quit (Quit: leaving)
[2:02] * The_Bishop_ (~bishop@e179010094.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[2:02] * The_Bishop_ (~bishop@f052099178.adsl.alicedsl.de) has joined #ceph
[2:04] * rustam (~rustam@5e0f5b1e.bb.sky.com) has joined #ceph
[2:07] * The_Bishop__ (~bishop@f052097229.adsl.alicedsl.de) Quit (Read error: Operation timed out)
[2:12] * The_Bishop_ (~bishop@f052099178.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[2:14] * dpippenger (~riven@216.103.134.250) Quit (Remote host closed the connection)
[2:20] * Loffler (~Loffler@182-239-166-220.ip.adam.com.au) has joined #ceph
[2:21] * Merv31000 (~Merv.rent@gw.logics.net.au) has joined #ceph
[2:22] <Merv31000> Today we had an issue with our system and it mostly recovered but all VMs did not come back.
[2:22] <Merv31000> ceph -w shows faults as it tries to communicate with all mons
[2:23] <Merv31000> I cannot start the missing VMs
[2:23] <Merv31000> I think the mons must be down? How do I restart them?
[2:25] <Merv31000> [root@alpha ~]# virsh start xena.csolve.com.au
[2:25] <Merv31000> error: Failed to start domain xena.csolve.com.au
[2:25] <Merv31000> error: Unable to read from monitor: Connection reset by peer
[2:47] * LeaChim (~LeaChim@027f1746.bb.sky.com) Quit (Ping timeout: 480 seconds)
[2:48] * sjustlaptop (~sam@mce0536d0.tmodns.net) Quit (Read error: Connection reset by peer)
[2:56] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[2:56] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[3:02] <joelio> scuttlemonkey: all working now, and how well it works too :)
[3:04] <scuttlemonkey> joelio: awesome!
[3:04] <scuttlemonkey> merv31000: asking to see if I can get you one of the experts
[3:04] <Merv31000> Great that will be most helpful
[3:04] <scuttlemonkey> can you pastebin your 'ceph -s' ?
[3:08] * chutzpah (~chutz@199.21.234.7) Quit (Quit: Leaving)
[3:12] <Merv31000> OK, I pasted as merv31000
[3:13] <dmick> Merv31000: a URL is traditionally the way you publicize your paste
[3:13] <scuttlemonkey> it should give you a url
[3:13] <scuttlemonkey> http://pastebin.com/
[3:13] <Merv31000> OK, sorry never used it before ...
[3:13] <scuttlemonkey> yeah, after you submit you can just copy/paste the url from your address bar
[3:13] <scuttlemonkey> or take the share link
[3:14] <Merv31000> Here it is; http://pastebin.com/SWaNh4mV
[3:15] <dmick> looks like your monitor(s) is(are) dead
[3:16] <Merv31000> Yes.
[3:16] <scuttlemonkey> what is in your mon log?
[3:17] <Merv31000> I did an /etc/rc.d/init.d/ceph restart mon on the first host and it has gone out of the error list ...
[3:17] <dmick> and the mon processes are started like other daemons...like that :)
[3:18] <Merv31000> I am a ceph newbie ... is that a standard log in /var/log?
[3:19] <Merv31000> I found a ceph sub-folder in logs and a ceph-mon.0.log on this first host
[3:22] <scuttlemonkey> Merv31000: sent you a phone number to grab urgent support
[3:33] <Merv31000> Got that thanks ... in shock here ...
[4:01] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[4:01] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) Quit (Remote host closed the connection)
[4:04] * portante (~user@c-24-63-226-65.hsd1.ma.comcast.net) has joined #ceph
[4:05] * jamespage (~jamespage@culvain.gromper.net) Quit (Quit: Coyote finally caught me)
[4:05] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[4:10] * slang (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[4:24] * KindTwo (~KindOne@h200.215.89.75.dynamic.ip.windstream.net) has joined #ceph
[4:26] * sjustlaptop (~sam@64.168.229.50) has joined #ceph
[4:28] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[4:28] * KindTwo is now known as KindOne
[4:36] * sjustlaptop (~sam@64.168.229.50) Quit (Ping timeout: 480 seconds)
[4:53] * Merv31000 (~Merv.rent@gw.logics.net.au) Quit (Quit: Leaving)
[5:08] * Loffler_ (~Loffler@182-239-162-210.ip.adam.com.au) has joined #ceph
[5:08] * sjustlaptop (~sam@184.169.45.218) has joined #ceph
[5:12] * sjustlaptop (~sam@184.169.45.218) Quit (Read error: Connection reset by peer)
[5:13] * Loffler (~Loffler@182-239-166-220.ip.adam.com.au) Quit (Ping timeout: 480 seconds)
[5:14] * sjustlaptop (~sam@184.169.45.218) has joined #ceph
[5:19] * Cube (~Cube@12.248.40.138) Quit (Quit: Leaving.)
[5:35] * rustam (~rustam@5e0f5b1e.bb.sky.com) Quit (Remote host closed the connection)
[5:35] * sjustlaptop (~sam@184.169.45.218) Quit (Read error: Connection reset by peer)
[5:40] * tristanz (~tristanz@c-71-204-142-149.hsd1.ca.comcast.net) has joined #ceph
[5:42] <tristanz> Is CephFS with snapshots completely hopeless for foreseeable future?
[5:45] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[5:45] * Loffler_ (~Loffler@182-239-162-210.ip.adam.com.au) Quit (Ping timeout: 480 seconds)
[5:56] * rekby (~Adium@2.92.10.160) has joined #ceph
[6:22] * Loffler (~Loffler@150.101.235.251) has joined #ceph
[6:34] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[6:38] * hijacker (~hijacker@213.91.163.5) Quit (Remote host closed the connection)
[6:45] * Merv31000 (~Merv.rent@150.101.235.251) has joined #ceph
[7:03] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) Quit (Read error: Connection reset by peer)
[7:03] * matt_ (~matt@220-245-1-152.static.tpgi.com.au) has joined #ceph
[7:04] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[7:13] * tristanz (~tristanz@c-71-204-142-149.hsd1.ca.comcast.net) Quit (Quit: tristanz)
[7:14] * madkiss (~madkiss@tmo-107-118.customers.d1-online.com) has joined #ceph
[7:16] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[7:34] * madkiss (~madkiss@tmo-107-118.customers.d1-online.com) Quit (Quit: Leaving.)
[7:40] * loicd (~loic@2a01:e35:2eba:db10:1c65:f513:6a7e:67ab) has joined #ceph
[7:44] * BillK (~BillK@124-169-229-107.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[7:58] * Loffler (~Loffler@150.101.235.251) Quit (Quit: Leaving)
[8:07] * norbi (~nonline@buerogw01.ispgateway.de) has joined #ceph
[8:08] * loicd (~loic@2a01:e35:2eba:db10:1c65:f513:6a7e:67ab) Quit (Quit: Leaving.)
[8:10] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) has joined #ceph
[8:11] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[8:13] <norbi> mornin ceph
[8:14] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Remote host closed the connection)
[8:14] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) has joined #ceph
[8:17] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[8:35] * eschnou (~eschnou@191.208-201-80.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:46] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) has joined #ceph
[8:46] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[8:49] * sleinen (~Adium@2001:620:0:26:61e1:fd9b:2656:6cf) has joined #ceph
[8:52] * Merv31000 (~Merv.rent@150.101.235.251) Quit (Quit: Leaving)
[8:56] * eschnou (~eschnou@191.208-201-80.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:00] * JohansGlock_ (~quassel@kantoor.transip.nl) has joined #ceph
[9:02] * darkfaded (~floh@88.79.251.60) has joined #ceph
[9:03] * Vjarjadian_ (~IceChat77@5ad6d005.bb.sky.com) Quit (Quit: Never put off till tomorrow, what you can do the day after tomorrow)
[9:04] * Gugge_47527 (gugge@kriminel.dk) has joined #ceph
[9:04] * darkfader (~floh@88.79.251.60) Quit (Read error: Connection reset by peer)
[9:06] * Gugge-47527 (gugge@kriminel.dk) Quit (Ping timeout: 480 seconds)
[9:06] * Gugge_47527 is now known as Gugge-47527
[9:06] * ShaunR (ShaunR@ip68-96-89-159.oc.oc.cox.net) Quit (Ping timeout: 480 seconds)
[9:07] * dennis (~dweazle@2a02:2770::21a:4aff:fee2:5724) Quit (Remote host closed the connection)
[9:07] * dennis (~dweazle@tilaa.krul.nu) has joined #ceph
[9:07] * ShaunR (~ShaunR@staff.ndchost.com) has joined #ceph
[9:07] * KindTwo (KindOne@h200.215.89.75.dynamic.ip.windstream.net) has joined #ceph
[9:07] * JohansGlock (~quassel@kantoor.transip.nl) Quit (Ping timeout: 480 seconds)
[9:08] * KindOne (~KindOne@0001a7db.user.oftc.net) Quit (Ping timeout: 480 seconds)
[9:08] * KindTwo is now known as KindOne
[9:11] * loicd (~loic@185.10.252.15) has joined #ceph
[9:15] * Morg (b2f95a11@ircip3.mibbit.com) has joined #ceph
[9:21] * leseb (~Adium@83.167.43.235) has joined #ceph
[9:23] <alexxy[home]> gregaf: patch doesnt work =\
[9:23] <alexxy[home]> mds still crashed
[9:23] * rekby (~Adium@2.92.10.160) Quit (Quit: Leaving.)
[9:23] <alexxy[home]> gregaf: http://bpaste.net/show/89127/
[9:24] <alexxy[home]> norbi: did you tryed patch from bug?
[9:29] <mega_au> gregaf: monitors still lose quorum. Spent long time looking through the logs. Apparently there's issue with paxos. I guess I need to speak to Joao. What time he's coming around?
[9:31] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:31] <norbi> hi alexxy, no i dont hae tried the patch
[9:31] <norbi> have
[9:34] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:37] * hijacker (~hijacker@bgva.sonic.taxback.ess.ie) has joined #ceph
[9:42] * l0nk (~alex@83.167.43.235) has joined #ceph
[9:44] * tnt (~tnt@dhcp92.vr.in-berlin.de) has joined #ceph
[9:44] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:45] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Remote host closed the connection)
[9:45] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[9:55] * sleinen (~Adium@2001:620:0:26:61e1:fd9b:2656:6cf) Quit (Quit: Leaving.)
[9:55] * sleinen (~Adium@130.59.94.71) has joined #ceph
[9:56] * stacker666 (~stacker66@192.pool85-58-190.dynamic.orange.es) has joined #ceph
[9:59] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Remote host closed the connection)
[10:00] * ctrl (~ctrl@83.149.9.177) has joined #ceph
[10:03] * sleinen (~Adium@130.59.94.71) Quit (Ping timeout: 480 seconds)
[10:04] * sleinen (~Adium@130.59.94.71) has joined #ceph
[10:06] * sleinen1 (~Adium@2001:620:0:25:5d65:4635:fe54:e658) has joined #ceph
[10:10] * espeer (~espeer@105-236-102-214.access.mtnbusiness.co.za) has joined #ceph
[10:10] <espeer> hello, is there anyone who can help me get to the bottom of a rados write performance problem?
[10:10] <espeer> rados -p rbd bench 60 write - gives about 45MB/s
[10:10] <espeer> but, rados -p rbd bench 60 write -b 4096 drops to 0.1 MB/s
[10:11] <espeer> 3.8.5 kernel, btrfs osds, gigabit interlink, default ceph configs
[10:12] * sleinen (~Adium@130.59.94.71) Quit (Ping timeout: 480 seconds)
[10:15] <matt_> espeer, I had the same issue. I never got the bottom of it but BTRFS seemed to be the issue
[10:15] <matt_> XFS was infintely more stable
[10:16] <espeer> yeah, i saw some posts on the mailing list with people having similar issue, but no resolution
[10:21] * ctrl (~ctrl@83.149.9.177) Quit (Ping timeout: 480 seconds)
[10:21] <espeer> so is the general consensus that production should be running on XFS at the moment?
[10:27] * KindOne (KindOne@0001a7db.user.oftc.net) Quit (Read error: Connection reset by peer)
[10:27] * ShaunR (~ShaunR@staff.ndchost.com) Quit (Read error: Connection reset by peer)
[10:27] <schlitzer|work> i guess so. btrfs is beta anyway, so not meant to be used in production
[10:27] * ShaunR (~ShaunR@staff.ndchost.com) has joined #ceph
[10:29] * LeaChim (~LeaChim@027f1746.bb.sky.com) has joined #ceph
[10:29] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[10:32] * rzerres (~ralf@bermuda.daywalker-studios.de) has joined #ceph
[10:32] <norbi> with xfs i got 7.8MB, without -b 4096 about 90.6MB
[10:32] <norbi> but can't test with btrfs
[10:33] * ctrl (~ctrl@83.149.9.41) has joined #ceph
[10:36] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) has joined #ceph
[10:50] <topro> over here 3 hosts with 3 osds each, linux 3.8.5, XFS, gigabit...
[10:51] <topro> 90MB/s without -b 4096
[10:51] <topro> 0.677 MB/s with -b 4096
[10:56] <Cube> topro: journal on disk?
[10:56] <topro> osd on 10k hdd, journal on same disk, yes
[10:57] <joao> mega_au, here now, what's up?
[10:59] <mega_au> joao: have weird issue with quorum. .Have 5 mons, but at any given time only three in quorum. And then quorum is lost all together. Mons keep running but nothing happening (except syncing between them). clients cannot connect of course.
[11:00] <joao> 0.59+?
[11:01] <mega_au> restarting mons which were in quorum last brings cluster back but only for a minute - then quorum is lost again. Looking through logs it appears that elector calls only one other mon and third one is not contacted on reelection. Therefore no ack and "so far I have" has only two mons which is not enough. 0.60
[11:02] <mega_au> Spent whole day going through logs and it appears at first glance to be paxos issue. Joining mons say that their paxos version is 0 and they trying to get remote copy. But they don't join in the end.
[11:03] <mega_au> Can send you a ton of mon logs and shell access to the cluster if you need it.
[11:03] <joao> logs would be appreciated
[11:04] <mega_au> pastebin or email?
[11:04] <joao> sftp cephdrop@ceph.com
[11:04] <mega_au> Will do in couple of hours.
[11:05] <joao> drop them in some directory and point me to them
[11:05] <joao> sure
[11:05] * BillK (~BillK@124-169-162-41.dyn.iinet.net.au) has joined #ceph
[11:05] <joao> let me know when it happens; meanwhile will be working on some other issue
[11:05] <mega_au> Had idea to downgrade cluster to single mon and then add fresh mons but a bit scared now - it is production stuff.
[11:05] <mega_au> I'll drop you a line here when I will send it off. Thanks!
[11:06] <joao> you shouldn't be running dev versions in production :\
[11:07] <joao> btw, mega_au, are you able to run a 'ceph health detail --format=json' on the cluster?
[11:07] <joao> and if so, pastebin it?
[11:18] <espeer> topro, so XFS doesn't solve the problem?
[11:19] <topro> thats what I guess, just wanted to give some XFS figures to compare
[11:22] * ivotron (~ivo@adsl-76-254-17-170.dsl.pltn13.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[11:27] <espeer> xfs + older kernel is perhaps the solution
[11:27] <espeer> i saw something about kernels after 3.7 having the issue
[11:33] * mikedawson (~chatzilla@c-98-220-189-67.hsd1.in.comcast.net) Quit (Ping timeout: 480 seconds)
[11:37] * tryggvil (~tryggvil@17-80-126-149.ftth.simafelagid.is) Quit (Quit: tryggvil)
[11:42] * drico (~drico@188-230-177-62.dynamic.t-2.net) Quit (Quit: so long, and thank you for all the bits)
[11:43] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) has joined #ceph
[11:48] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) has joined #ceph
[11:49] <topro> and I was told to use latest kernel as they have fixes for XFS as well :/
[11:50] * vipr (~root@78-23-117-241.access.telenet.be) Quit (Quit: leaving)
[11:51] * vipr (~root@78-23-117-241.access.telenet.be) has joined #ceph
[11:51] * vipr (~root@78-23-117-241.access.telenet.be) Quit ()
[11:53] * ctrl (~ctrl@83.149.9.41) Quit (Ping timeout: 480 seconds)
[11:55] * vipr (~root@78-23-117-241.access.telenet.be) has joined #ceph
[11:55] * vipr (~root@78-23-117-241.access.telenet.be) Quit ()
[11:55] * vipr (~root@78-23-117-241.access.telenet.be) has joined #ceph
[12:02] * sleinen1 (~Adium@2001:620:0:25:5d65:4635:fe54:e658) Quit (Quit: Leaving.)
[12:05] * sleinen (~Adium@2001:620:0:26:30f7:af29:a425:cf6b) has joined #ceph
[12:13] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[12:14] <rzerres> @elder: hey there
[12:15] <rzerres> i found an old track on ceph developer list concerning a bug when mapping rbd's (On 12/21/2012 05:39)
[12:16] <rzerres> i do still have this problem when i try to mount an existsing rbd on kernel 3.6.9-030609-generic.
[12:16] <rzerres> is theire any news on this issue?
[12:18] <rzerres> of corse, i do use rbd format 1 for the tests.
[12:19] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[12:21] * diegows (~diegows@190.190.2.126) has joined #ceph
[12:31] * ShaunR (~ShaunR@staff.ndchost.com) Quit (Ping timeout: 480 seconds)
[12:34] <Gugge-47527> rzerres: what problem?
[12:36] * vipr (~root@78-23-117-241.access.telenet.be) Quit (Quit: leaving)
[12:44] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[12:49] * ctrl (~ctrl@83.149.8.172) has joined #ceph
[12:54] * dosaboy (~user1@host86-161-164-218.range86-161.btcentralplus.com) Quit (Quit: Leaving.)
[12:55] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Read error: Operation timed out)
[12:57] * dosaboy (~user1@host86-161-164-218.range86-161.btcentralplus.com) has joined #ceph
[12:58] <rzerres> @gugge: if i map a rbd the kernel hangs immediately (watchdog detected hard LOCKUP)
[13:00] <Gugge-47527> do you run any of the ceph daemons on the server you map the rbd on?
[13:00] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[13:02] <rzerres> yes, i do. mds, osd, and mon
[13:03] <Gugge-47527> dont
[13:04] <Gugge-47527> http://ceph.com/docs/master/faq/#how-can-i-give-ceph-a-try
[13:05] <alexxy[home]> joao: i still have problems with bugzie registration
[13:06] <joao> alexxy[home], sorry, forgot to mention that to rturk last night
[13:06] <alexxy[home]> well its not a problem
[13:06] <joao> scuttlemonkey, poke me whenever you're around
[13:06] <alexxy[home]> also i tested patch from bugzie
[13:06] <alexxy[home]> but it doesent work
[13:06] <alexxy[home]> mds still crashes
[13:06] <rzerres> well, i could try to map on a non cluster node.
[13:07] * alexxy[home] (~alexxy@79.173.81.171) Quit (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.)
[13:07] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[13:07] <rzerres> just like to use a simple way to mount an existing rbd. What do you reccoment?
[13:08] <rzerres> is iSCSI an option? I got hold of an tgt implementation.
[13:09] <rzerres> Why do i need to do it: Expect an btrfs on the rbd and you need to fsck ....
[13:14] * backspinner (~oftc-webi@a82-161-160-72.adsl.xs4all.nl) has joined #ceph
[13:15] <Gugge-47527> rzerres: map it on a server where you dont run the ceph deamons
[13:17] <backspinner> I try to install Ceph on Debian 6.0.7
[13:17] <backspinner> But "modprobe rbd" always gives "FATAL: Module rbd not found."
[13:17] <backspinner> Where to get the rbd kernel module
[13:27] <matt_> joao, did you happen to have any luck with our mon issue?
[13:28] <joao> matt_, talked to sam last night, there may be some issues on the osd side if my workaround were to be used (then again, maybe not, but there's some risk)
[13:28] <joao> on the other hand, there is a possible workaround
[13:28] <joao> granted you still have the original store around
[13:29] <joao> I'm working on a tool to test the fix
[13:29] <joao> and am going to use your stores to test it asap
[13:30] <joao> and by asap I mean as soon as I finish working on it
[13:33] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[13:36] * ScOut3R_ (~ScOut3R@212.96.47.215) has joined #ceph
[13:40] <matt_> sounds great
[13:41] <matt_> Thanks again
[13:41] <joao> thank you, for your patience and your stores
[13:41] <joao> they'll be playing a crucial part in this whole thing :p
[13:42] <matt_> Well if I'm good at anything, it's breaking stuff
[13:43] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[13:45] * loicd (~loic@185.10.252.15) Quit (Read error: Operation timed out)
[13:52] <rzerres> @gugge: beside the fact that a production environment should not stack multiple daemons on on cluster server, in a testing environment it should work anyway. And i did it on former releases of ceph (eg v0.56). since v0.57 onwards i do recognized the watchdog crashes. And i can't imagine, it is on purpose ....
[13:57] * BManojlovic (~steki@91.195.39.5) Quit (Remote host closed the connection)
[14:02] * ctrl (~ctrl@83.149.8.172) Quit (Ping timeout: 480 seconds)
[14:02] <Gugge-47527> rzerres: the fact that mapping rbd or mouting cephfs on a server running osd's, is why you should not do it
[14:03] <Gugge-47527> if you really want to, use fuse :)
[14:03] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[14:04] <mega_au> @joao: uploaded logs and health detail to logics
[14:04] <joao> thanks mega_au, will take look after lunch
[14:05] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[14:05] <mega_au> If you would have some suggestion we will appreciate it. Otherwise we may blow the whole thing and reload images tomorrow.
[14:05] <mega_au> But something tells me that loss of quorum is not something what you'd like to see... :)
[14:09] <scuttlemonkey> joao: I'm actually off today and getting ready for an 8 hour drive
[14:09] <scuttlemonkey> can I help real quick before I jet?
[14:10] <joao> uh, just that alexxy is unable to get his tracker registration email
[14:10] <joao> but I'll poke someone else about that
[14:10] <joao> :)
[14:10] <scuttlemonkey> oh
[14:10] <scuttlemonkey> yeah, Ross is the only one I know of with permissions to poke the google apps setup
[14:16] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) has joined #ceph
[14:20] * scuttlemonkey (~scuttlemo@fl-184-1-34-163.dhcp.embarqhsd.net) Quit (Read error: Operation timed out)
[14:22] <BillK> Iam using .58 and have 1182 GB data, 3545 GB used, 1958 GB / 5509 GB avail
[14:22] <BillK> but the replication is 2, so I would expect 2 times the data, not 3 as this shows
[14:23] <BillK> what gives? - from what I can remember, .56 showed 2 times before I upgraded ...
[14:30] * Rocky (~r.nap@188.205.52.204) Quit (Quit: **Poof**)
[14:30] * trond (~trond@trh.betradar.com) has joined #ceph
[14:30] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[14:31] * Rocky (~r.nap@188.205.52.204) Quit ()
[14:31] <trond> Hi, did format 2 support for the kernel module make it to Linuxe Kernel 3.9 ?
[14:48] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[14:49] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[14:51] * ctrl (~ctrl@83.149.8.241) has joined #ceph
[14:52] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:53] * rzerres (~ralf@bermuda.daywalker-studios.de) has left #ceph
[14:54] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) has joined #ceph
[14:57] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[14:59] * portante (~user@c-24-63-226-65.hsd1.ma.comcast.net) Quit (Ping timeout: 480 seconds)
[14:59] * ctrl (~ctrl@83.149.8.241) Quit (Ping timeout: 480 seconds)
[15:03] <trond> nobody knows? ^
[15:03] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:06] * mikedawson_ (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[15:07] <Azrael> hmm. more ceph + chef cookbook bugs.
[15:08] <Azrael> it uses ceph-disk-prepare, which will take a disk (e.g. /dev/sdb) and partition it such that /dev/sdb1 (journal) and /dev/sdb2 (data) exist
[15:08] <Azrael> but the rest of the system isn't aware of that style of setup
[15:08] <Azrael> arg
[15:08] * slang1 (~slang@207-229-177-80.c3-0.drb-ubr1.chi-drb.il.cable.rcn.com) has joined #ceph
[15:11] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[15:12] * mikedawson_ is now known as mikedawson
[15:14] * madkiss (~madkiss@business-213-023-158-038.static.arcor-ip.net) Quit (Quit: Leaving.)
[15:46] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:53] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[16:00] * Morg (b2f95a11@ircip3.mibbit.com) Quit (Quit: http://www.mibbit.com ajax IRC Client)
[16:06] * vipr (~root@78-23-117-241.access.telenet.be) has joined #ceph
[16:12] * ScOut3R (~ScOut3R@212.96.47.215) has joined #ceph
[16:17] * espeer (~espeer@105-236-102-214.access.mtnbusiness.co.za) Quit (Read error: Connection reset by peer)
[16:18] * espeer (~espeer@105-236-102-214.access.mtnbusiness.co.za) has joined #ceph
[16:19] * ScOut3R_ (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[16:20] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) has joined #ceph
[16:21] <Azrael> ahaa
[16:21] <Azrael> ok
[16:21] <Azrael> thats good of ceph-disk-prepare
[16:21] <Azrael> *but*
[16:21] <Azrael> debian's udev is broken, even in wheezy
[16:21] <Azrael> and sid
[16:22] <Azrael> the path /dev/disk/by-partuuid doesn't exist. and this is what /var/lib/ceph/osd/$cluster-$id/journal will point to.
[16:22] <Azrael> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=681809
[16:24] * vata (~vata@2607:fad8:4:6:cd7e:e419:ff6d:fc94) has joined #ceph
[16:28] * espeer (~espeer@105-236-102-214.access.mtnbusiness.co.za) Quit (Quit: Konversation terminated!)
[16:35] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) Quit (Remote host closed the connection)
[16:49] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[16:49] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) has joined #ceph
[16:49] * norbi (~nonline@buerogw01.ispgateway.de) Quit (Quit: Miranda IM! Smaller, Faster, Easier. http://miranda-im.org)
[16:51] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[16:57] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[17:01] * Vjarjadian (~IceChat77@5ad6d005.bb.sky.com) has joined #ceph
[17:02] * gerard_dethier (~Thunderbi@85.234.217.115.static.edpnet.net) Quit (Quit: gerard_dethier)
[17:13] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:20] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Quit: Leaving.)
[17:24] * sagelap (~sage@2600:1012:b00d:38c:452e:1ee6:2851:5565) has joined #ceph
[17:24] <sagelap> elder: xfstest 139 hung on umount
[17:24] * ShaunR (~ShaunR@staff.ndchost.com) has joined #ceph
[17:25] <sagelap> on plana92
[17:25] <paravoid> health HEALTH_ERR 10 pgs inconsistent; 10 scrub errors
[17:25] <paravoid> all on different OSDs
[17:25] <paravoid> different osd tuples
[17:25] <paravoid> and all kind of a sudden
[17:26] * hybrid5121 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) has joined #ceph
[17:26] <sagelap> paravoid: if you look in the ceph.log file you'll see the detail about the inconsistencies..
[17:26] <sagelap> grep for ERR
[17:27] <elder> sagelap, there are two unmounts in that test. I'll have to see the output to know which hung.
[17:28] <paravoid> 2013-04-04 07:09:53.261082 osd.81 10.64.32.12:6862/7110 37 : [ERR] 3.9da deep-scrub 0 missing, 1 inconsistent objects
[17:28] <paravoid> 2013-04-04 10:51:31.672719 osd.51 10.64.32.10:6837/17752 43 : [ERR] 3.2e1 deep-scrub 0 missing, 1 inconsistent objects
[17:28] <paravoid> 2013-04-04 18:17:14.137904 osd.103 10.64.32.14:6842/2835 27 : [ERR] 3.15ea deep-scrub 0 missing, 1 inconsistent objects
[17:28] <paravoid> 2013-04-04 16:26:06.169251 osd.52 10.64.32.10:6840/17776 41 : [ERR] 3.1d23 deep-scrub 0 missing, 1 inconsistent objects
[17:29] <paravoid> 2013-04-04 20:53:45.242954 osd.143 10.64.0.178:6853/3994 17 : [ERR] 3.43b osd.82: soid 760d843b/.dir.4143.142/head//3 omap_digest 3735166186 != known omap_digest 3450559740
[17:29] <paravoid> 2013-04-04 20:55:38.584154 osd.143 10.64.0.178:6853/3994 18 : [ERR] 3.43b deep-scrub 0 missing, 1 inconsistent objects
[17:29] <sagelap> interesting. i bet they showed up now becuase the deep scrub only goes every 1-2 weeks
[17:29] <paravoid> let me pastebin that for you
[17:29] <sagelap> try manually triggering a deep scrub on one of those pgs and see what comes back
[17:29] <sagelap> i'm surprised it's not telling you which object is inconsistent
[17:29] * hybrid512 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[17:29] <paravoid> http://p.defau.lt/?c1A_j9bEA_XT8Wb1Fdih4Q
[17:30] <sagelap> (maybe that log level is WRN?)
[17:30] <sagelap> oh it is.
[17:30] <paravoid> yeah, sorry :)
[17:30] <paravoid> I had one pg two days ago and I ran a repair
[17:30] <paravoid> then more cropped up
[17:30] <paravoid> note that I upgraded to 0.56.4 three or four days ago or so
[17:31] <paravoid> restarted all OSDs in the process
[17:31] <sagelap> 0.56.4 enabled the omap deep scrub, that is probably what changed :) before it was only scrubbing attrs and data payload
[17:31] <paravoid> ah!
[17:31] <sagelap> brb
[17:31] * sagelap (~sage@2600:1012:b00d:38c:452e:1ee6:2851:5565) has left #ceph
[17:34] * hybrid5121 (~walid@LPoitiers-156-86-25-85.w193-248.abo.wanadoo.fr) Quit (Quit: Leaving.)
[17:40] <gregaf> Elbandi_: I'll get you a patch today
[17:41] <gregaf> alexxy: I don't think I had a patch for you, so huh?
[17:41] * mrjack (mrjack@office.smart-weblications.net) has joined #ceph
[17:41] <mrjack> i
[17:41] <mrjack> hi
[17:41] <mrjack> rbd export kvm00000000943 - |gzip -1 > /data/backup/kvm00000000943.gz
[17:41] <mrjack> rbd: export error: (-2147483648) Unknown error 18446744071562067968
[17:42] <mrjack> any idea?
[17:43] <mrjack> strange that error is - (2048*1024*1204)?
[17:43] <mrjack> ceph --version
[17:43] <mrjack> rbceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca)
[17:43] <mrjack> rbd --version
[17:43] <mrjack> ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca)
[17:45] <topro> mounting cephfs using fuse root can't acces all files. is there a way to "root_squash" like there is with nfs
[17:45] <topro> ^^permission denied
[17:46] <topro> I would need that for rsync based backups
[17:51] <joao> matt_, still around?
[17:53] <gregaf> topro: that's fixed in v0.60
[17:53] <gregaf> it's just a client fix
[17:53] <topro> a fuse-client fix?
[17:53] <gregaf> yeah
[17:53] <topro> and 0.61 will be an LTS release, right?
[17:53] <gregaf> if you can build it should also cherry-pick easily; commit d87035c0c4ff
[17:53] <gregaf> yep
[17:54] <topro> gregaf: i could build but I would prefer not to becaus waaayyy to many stuff to do here right now ;)
[17:55] <topro> so I will stick to 0.56 till there is 0.61 as I read about some trouble that's still there with 0.60 currently
[17:56] <topro> like crashing mds
[17:57] * ScOut3R (~ScOut3R@212.96.47.215) Quit (Ping timeout: 480 seconds)
[18:06] * sleinen (~Adium@2001:620:0:26:30f7:af29:a425:cf6b) Quit (Quit: Leaving.)
[18:06] * sleinen (~Adium@130.59.94.71) has joined #ceph
[18:09] * mattch (~mattch@pcw3047.see.ed.ac.uk) Quit (Quit: Leaving.)
[18:12] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[18:13] * loicd (~loic@3.46-14-84.ripe.coltfrance.com) Quit (Quit: Leaving.)
[18:14] * sleinen (~Adium@130.59.94.71) Quit (Ping timeout: 480 seconds)
[18:24] <sig_wall> mrjack: -(2048*1024*1204) is (signed int)0x80000000
[18:25] <sig_wall> -2147483648 = (signed)2147483648 = (signed)2^31, damn C :)
[18:26] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[18:29] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) has joined #ceph
[18:34] * alram (~alram@38.122.20.226) has joined #ceph
[18:35] * topro (~topro@host-62-245-142-50.customer.m-online.net) Quit (Quit: Konversation terminated!)
[18:39] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[18:39] * topro (~topro@host-62-245-142-50.customer.m-online.net) has joined #ceph
[18:43] * sstan (~chatzilla@dmzgw2.cbnco.com) Quit (Ping timeout: 480 seconds)
[18:44] * backspinner (~oftc-webi@a82-161-160-72.adsl.xs4all.nl) Quit (Quit: Page closed)
[18:45] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) has joined #ceph
[18:47] * sstan (~chatzilla@dmzgw2.cbnco.com) has joined #ceph
[18:48] * stacker666 (~stacker66@192.pool85-58-190.dynamic.orange.es) Quit (Ping timeout: 480 seconds)
[18:51] * Cube (~Cube@cpe-76-172-67-97.socal.res.rr.com) Quit (Quit: Leaving.)
[18:56] * rahmu (~rahmu@83.167.43.235) has joined #ceph
[18:57] * l0nk (~alex@83.167.43.235) Quit (Quit: Leaving.)
[19:04] <paravoid> sage, sagewk: around?
[19:05] * rahmu (~rahmu@83.167.43.235) Quit (Remote host closed the connection)
[19:05] * leseb (~Adium@83.167.43.235) Quit (Quit: Leaving.)
[19:05] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:05] * chutzpah (~chutz@199.21.234.7) has joined #ceph
[19:08] * danieagle (~Daniel@177.205.180.100.dynamic.adsl.gvt.net.br) has joined #ceph
[19:14] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) Quit (Remote host closed the connection)
[19:20] * Cube (~Cube@12.248.40.138) has joined #ceph
[19:27] <dmick> busy day; what's up paravoid
[19:27] * sjustlaptop (~sam@38.122.20.226) has joined #ceph
[19:29] * stacker666 (~stacker66@192.pool85-58-190.dynamic.orange.es) has joined #ceph
[19:30] * drokita (~drokita@199.255.228.128) has joined #ceph
[19:33] <drokita> I just found out that my OS disks in my ceph cluster are 15k SAS drives. My instincts tell me that is a bit overkill. Any opinions on that?
[19:34] <dmick> if you're doing it right you don't hit the OS disks much
[19:34] <drokita> That is what I was thinking, although it may be a bit reckless to assume that I am doing it right
[19:34] <drokita> All of my OSDs are on dedicated spindles.
[19:35] <dmick> unlike other matters of rectitude, one can actually measure that one :)
[19:35] <drokita> You just forced me to look up 'rectitude'
[19:35] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[19:35] * loicd (~loic@magenta.dachary.org) has joined #ceph
[19:35] * senner (~Wildcard@68-113-232-90.dhcp.stpt.wi.charter.com) Quit (Quit: Leaving.)
[19:36] <dmick> so, it's a good day all around
[19:39] <sstan> drokita : probably. One could maybe even not have to use disks for OS (pxe boot, configuration stored on the network)
[19:40] <drokita> sstan: Yeah, that is where I am planning on going with the config. That is a much better way to handle. It is going to take me a while to get there though.
[19:40] * stacker666 (~stacker66@192.pool85-58-190.dynamic.orange.es) Quit (Ping timeout: 480 seconds)
[19:40] <sstan> how many computers running osds do you plan on having ?
[19:40] <drokita> sstan: In those configurations, is it prudent to maintain a physical swap disk or does it run fine given enough RAM
[19:41] <sstan> ideally you don't want it to use swap at all
[19:41] <drokita> I have 2 clusters, 6 chassis in each cluster and 16 disks in each chassis
[19:41] <drokita> Will likely double in the next 18 mos
[19:41] <sstan> so you'll have 24 computers
[19:41] <drokita> yeah
[19:42] <sstan> cool
[19:42] <drokita> At 24 I think that the current operations plan will break.
[19:43] * dpippenger (~riven@216.103.134.250) has joined #ceph
[19:47] <drokita> Running iostat is showing that my root volume is pretty active with 6000kB being written per second
[19:47] <drokita> that is higher than most, but not all of the OSDs
[19:48] * Cube (~Cube@12.248.40.138) Quit (Quit: Leaving.)
[19:50] <dmick> hm. one wonders what that traffic would be. Logs?
[19:50] * rturk-away is now known as rturk
[19:51] <drokita> dmick: that was my guess... still seems high though
[19:51] <gregaf> logs are pretty much all it could be that Ceph is generating
[19:51] <gregaf> it's easy to get much more than that, depending on what your settings are
[19:52] * sleinen (~Adium@2001:620:0:25:e5b2:9d54:3bf4:6407) has joined #ceph
[19:52] <drokita> I never really tweaked the logging settings of Ceph, they are likely running default.
[19:54] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[19:56] <drokita> I see in the documentation that 'btrfs is not quite stable enough for production'. Do you guys still feel this way?>
[20:01] <gregaf> it's less about how stable it is (though we have seen issues with this from time to time), and more about issues with the performance dropping off
[20:02] <gregaf> it starts out very speedy but our workload is pretty bad for it so it drops off quite quickly as time goes on
[20:02] <sstan> so the next best FS is xfs ?
[20:03] * tristanz (~tristanz@64.7.84.114) has joined #ceph
[20:04] <drokita> We are using XFS right now. No complaints so far.
[20:05] <drokita> ok, one complaint... xfs.repair might take 3 days to complete on a 3TB drive
[20:08] * sstan (~chatzilla@dmzgw2.cbnco.com) Quit (Remote host closed the connection)
[20:12] * sjustlaptop (~sam@38.122.20.226) Quit (Ping timeout: 480 seconds)
[20:16] * dosaboy (~user1@host86-161-164-218.range86-161.btcentralplus.com) Quit (Quit: Leaving.)
[20:17] * jjgalvez (~jjgalvez@cpe-76-175-30-67.socal.res.rr.com) Quit (Quit: Leaving.)
[20:20] <tristanz> Is there any honest guidance on CephFS with snapshotting? Absolutely not ready for production for X months?
[20:21] <gregaf> yeah, not ready for production
[20:22] <gregaf> some people use it successfully but there are a lot of problems you can run into and I wouldn't feel comfortable with it
[20:23] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[20:23] <tristanz> Is there a rough timeline for production ready CephFS and production ready with snapshot? Or at least, definitely not before X?
[20:24] <gregaf> not at this time; sorry — it depends a lot on some decisions which we're in the process of making now
[20:24] <gregaf> snapshots work fine with RBD, if that solves your problem ;)
[20:25] * ivotron (~ivo@69-170-63-251.static-ip.telepacific.net) has joined #ceph
[20:27] <tristanz> I looked at that a little but it seemed like it would be hard to make work. I'm looking for mountable user directories, with snapshots, that are mounted r/w by many compute nodes. Classic HPC use case.
[20:27] * loicd (~loic@magenta.dachary.org) Quit (Quit: Leaving.)
[20:27] <gregaf> just making sure
[20:29] <gregaf> of course it depends on how permanent you need your data to be; if you just want a data store that you can do checkpoints on and then export final results to a different storage system every couple days that would be different
[20:29] <tristanz> Is there any stable architecture you are aware that would support that? We've drawn a blank.
[20:30] <drokita> Do you need it to scale across chassis?
[20:30] <gregaf> sorry, but as far as I know the snapshots are a big shiny that nobody else does well (and that we need more QA on)
[20:30] <maswan> tristanz: being in the hpc field, I think you're stuck with very expensive and very underperforming enterprise solutions, if you need more than just a big fileserver with zfs on
[20:31] <drokita> I was going to throw zfs out there, but it seemed so obvious that I figured you already dismissed it.
[20:33] <tristanz> ZFS is perfect, and we can limit users (projects) to a single machine, potentially, but we do need to shard users across many machines. We started down that path but worried that we were implementing to much logic ourselves.
[20:33] <drokita> So, what should one think of pgs that are showing up as 'stuck unclean' but also have no unfound objects.
[20:34] <maswan> tristanz: well, zfs+nfs
[20:34] <maswan> tristanz: the downside being that you're stuck with size and BW for what you can stick in a big server
[20:35] <drokita> Can pNFS solve any of those problems?
[20:35] <maswan> drokita: well, yes, but the server-side implementations I know of also don't support snapshots
[20:36] <drokita> ZFS could handle the snapshotting... it would be at the volume level, but better than nothing
[20:37] <maswan> drokita: of course, if you are only looking to support a moderate ammount of metadata ops (~100Hz open/close/stat) and are willing to fund a bit of development, snapshots should be doable in dCache. Has the constraint that it is roughly WORM though, once you close() a file, you can't open it for writing.
[20:37] <tristanz> right, we definitely need to shard users across servers, which introduces complexity, and we also worried about bandwidth / scaling bottlenecks. CephFS + Snapshots, seemed to solve those. But maturity is a big issue.
[20:38] <maswan> drokita: for the rest, I think pNFS are mostly an (optional) protocol for the above-mentioned enterprise storages today.
[20:39] <tristanz> Are there any projects out there that help with sharding, maybe even rebalancing, across cluster of ZFS + NFS servers? Or is this custom.
[20:40] * yasu` (~yasu`@dhcp-59-149.cse.ucsc.edu) has joined #ceph
[20:40] <drokita> tristanz: no, that is the real reason I decided to rally behind ceph
[20:40] <drokita> Honestly, my last company wasted a ton of money trying to get the zfs/pNFS thing in a workable state
[20:41] <tristanz> drokita: yes, we're trying to head that off! But are worried CephFS isn't ready, which sounds like its the case.
[20:43] <drokita> My money says that they will be ready before you can solve it by customizing these other solutions. I have plenty of experience in that can o' worms
[20:43] <maswan> tristanz: you need a common namespace for pNFS
[20:43] * apatil (~anand@li582-249.members.linode.com) has joined #ceph
[20:44] <maswan> tristanz: so it isn't just rounding up a bunch of zfs servers
[20:44] <tristanz> maswan: is pNFS ready?
[20:44] * rturk is now known as rturk-away
[20:44] <maswan> tristanz: as a protocol, yes, it seems to work on the linux client side. we're just starting to deploy some and see how it works. but that's with dCache as server.
[20:45] <Elbandi_> gregaf: ok
[20:45] <tristanz> maswan: is the motivation for you that snapshotting is absolutely required for your workloads?
[20:45] <maswan> tristanz: pNFS is basically NFS but with redirects from the metadata server to data servers for data transfers.
[20:46] <maswan> tristanz: oh, we don't need snapshotting, for us pNFS is just another way of accessing semi-online storage in dCache
[20:47] <maswan> tristanz: which is more of a distributed ftp/webdav/gridprotocols server than an online filesystem
[20:47] <drokita> Not trying to spam, but I fear this question got lost in the conversation: 'So, what should one think of pgs that are showing up as 'stuck unclean' but also have no unfound objects.'
[20:47] <maswan> so slightly out of focus here
[20:49] <tristanz> maswan: right. we're balancing the appeal of something like ZFS snapshotting vs. stability/simplicity of something like GlusterFS (until CephFS is announced as mature).
[20:55] * mcclurmc_laptop (~mcclurmc@firewall.ctxuk.citrix.com) Quit (Ping timeout: 480 seconds)
[20:59] <maswan> tristanz: also for consideration, how soon would you need to scale beyond what you can reasonably do with a single beefy zfs server?
[20:59] <maswan> which would be whatt, 100-200T/20-40Gbit/s?
[21:00] <maswan> maybe larger but slower
[21:00] <tristanz> we're in AWS, so it would be their high storage instance.
[21:02] <tristanz> it would likely work to start, but the problem is we're delivering a feature, snapshots, we can't deliver long term.
[21:05] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[21:07] <maswan> tristanz: *google* ok, so that would be a lot smaller than what I sketched at
[21:09] <tristanz> maswan: yes, definitely imposes limits on us.
[21:15] * rekby (~Adium@2.92.10.160) has joined #ceph
[21:20] <rekby> Hello.
[21:20] <rekby> I have slow performace CephFS during work with a lot of files. For example:
[21:20] <rekby> [root@localhost t1]# wget http://joomlacode.org/gf/download/frsrelease/17965/78413/Joomla_3.0.3-Stable-Full_Package.tar.gz
[21:20] <rekby> [root@localhost t1]# time tar -zxf Joomla_3.0.3-Stable-Full_Package.tar.gz 
[21:20] <rekby> real 8m58.914s
[21:20] <rekby> user 0m0.414s
[21:20] <rekby> sys 0m0.757s
[21:20] <rekby> Is it normal performace for CephFS or it can be tuned for work fine (for example set local cache)?
[21:24] <tristanz> rekby: saw that mailing list post, haven't done benchmarks myself, but would be interested in response.
[21:25] * sleinen (~Adium@2001:620:0:25:e5b2:9d54:3bf4:6407) Quit (Quit: Leaving.)
[21:25] <rekby> Yes, I sent it to mail list few days ago, but don't receive answer from anybody :(
[21:28] <tristanz> my second hand understanding is performance for tons of small files isn't great right now.
[21:29] * davidzlap (~Adium@ip68-96-75-123.oc.oc.cox.net) has joined #ceph
[21:31] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[21:31] <rekby> I want use ceph for webhosting storage and CephFS more comfortable than rbd.
[21:31] <rekby> In my usage only one host will access for files, it can switch to other host only if first server is broken.
[21:31] <rekby> I want aggressive cache max files and metadata for fast work with a lot of small files. Usually it is very seldom change and I don't need access to any file simultaneously from different computer and sync changes between it.
[21:31] * madkiss1 (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[21:37] * Cube (~Cube@12.248.40.138) has joined #ceph
[21:39] * madkiss (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[21:39] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[21:40] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[21:40] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[21:42] * jjgalvez (~jjgalvez@12.248.40.138) has joined #ceph
[21:42] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[21:45] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[21:46] * andreask (~andreas@h081217068225.dyn.cm.kabsi.at) has left #ceph
[21:55] * mcclurmc_laptop (~mcclurmc@cpc10-cmbg15-2-0-cust205.5-4.cable.virginmedia.com) has joined #ceph
[21:58] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has joined #ceph
[22:01] * jskinner_ (~jskinner@69.170.148.179) has joined #ceph
[22:08] * jskinner (~jskinner@69.170.148.179) Quit (Ping timeout: 480 seconds)
[22:14] <slang1> rekby: are you using the kernel module of cephfs or fuse?
[22:15] * mrjack_ (mrjack@office.smart-weblications.net) has joined #ceph
[22:15] <mrjack_> what could that be:
[22:15] <mrjack_> rbd export kvm00000000943 - | gzip >kvm00000000943.gz
[22:15] <mrjack_> rbd: export error: (-2147483648) Unknown error 18446744071562067968
[22:18] <rekby> I try both variants it result from kernel mode
[22:19] * madkiss (~madkiss@089144192030.atnat0001.highway.a1.net) has joined #ceph
[22:19] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) has joined #ceph
[22:20] * madkiss1 (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Read error: No route to host)
[22:21] <fghaas> mrjack_: you might want to try with --debug-rbd=10 or even 20
[22:21] * hr (~hr@p549F50AE.dip.t-dialin.net) has joined #ceph
[22:21] <fghaas> to see what precedes the error
[22:21] <slang1> rekby: both do caching at the client, the kernel module is more performant though
[22:22] * jskinner_ (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[22:22] * madkiss1 (~madkiss@chello062178057005.20.11.vie.surfer.at) has joined #ceph
[22:24] <rekby> slang1, may be it cache data, but not metadata?
[22:24] <rekby> I have good performance with one big file and bad for a lot of small file.
[22:24] <rekby> can you try it on your CephFS, what time you need for unpack Joomla on it?
[22:25] * yanzheng (~zhyan@jfdmzpr06-ext.jf.intel.com) Quit (Remote host closed the connection)
[22:27] * madkiss (~madkiss@089144192030.atnat0001.highway.a1.net) Quit (Ping timeout: 480 seconds)
[22:27] * diegows (~diegows@190.190.2.126) has joined #ceph
[22:29] * ivotron_ (~ivo@ma50536d0.tmodns.net) has joined #ceph
[22:31] * ivotron (~ivo@69-170-63-251.static-ip.telepacific.net) Quit (Ping timeout: 480 seconds)
[22:36] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[22:37] * tnt (~tnt@dhcp92.vr.in-berlin.de) Quit (Ping timeout: 480 seconds)
[22:41] * tryggvil (~tryggvil@rtr1.tolvusky.sip.is) Quit (Quit: tryggvil)
[22:41] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:42] * madkiss1 (~madkiss@chello062178057005.20.11.vie.surfer.at) Quit (Quit: Leaving.)
[22:44] <dmick> mrjack_: what version?
[22:46] * tristanz (~tristanz@64.7.84.114) Quit (Quit: tristanz)
[22:50] * dpippenger1 (~riven@216.103.134.250) has joined #ceph
[22:50] * dpippenger (~riven@216.103.134.250) Quit (Read error: Connection reset by peer)
[22:53] * jskinner (~jskinner@69.170.148.179) has joined #ceph
[22:53] * jskinner (~jskinner@69.170.148.179) Quit (Remote host closed the connection)
[22:55] * sjustlaptop (~sam@2607:f298:a:697:8dde:284a:e6d0:818b) has joined #ceph
[22:57] * tnt (~tnt@pd95cfcb7.dip0.t-ipconnect.de) has joined #ceph
[23:03] * tnt_ (~tnt@ip-34-175-205-91.static.contabo.net) has joined #ceph
[23:04] * yanzheng (~zhyan@134.134.137.75) has joined #ceph
[23:05] * tnt (~tnt@pd95cfcb7.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[23:14] * tristanz (~tristanz@c-71-204-142-149.hsd1.ca.comcast.net) has joined #ceph
[23:14] * sjustlaptop (~sam@2607:f298:a:697:8dde:284a:e6d0:818b) Quit (Ping timeout: 480 seconds)
[23:21] <Elbandi_> gregaf: then, we can say that my bug is not related to multi mds, right?
[23:22] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Remote host closed the connection)
[23:22] <slang1> rekby: what performance difference do you see?
[23:23] <rekby> in fuse it take about 15-20 minutes, I can't try it right now - i broke test cluster for manual setup and understand authorizetion and other details of settings.
[23:28] <mrjack_> fghaas ok will try that
[23:31] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[23:31] <mrjack_> fghaas: pastebin.com/7XbTUUyx
[23:32] <fghaas> mrjack_: ok that's not exactly more useful than what you already knew :)
[23:32] <mrjack_> fghaas: yep
[23:32] <fghaas> echoing dmick's earlier question, which version is this?
[23:33] <mrjack_> ceph --version
[23:33] <mrjack_> ceph version 0.56.4 (63b0f854d1cef490624de5d6cf9039735c7de5ca)
[23:33] <fghaas> and your osds are the same version?
[23:33] <mrjack_> yes
[23:34] <mrjack_> node01:/data/backup# rbd export kvm00000000943 kvm00000000943.img
[23:34] <mrjack_> Exporting image: 100% complete...done.
[23:34] <mrjack_> so i expect something to be wrong when using <stdin>
[23:35] <fghaas> stdout, yes
[23:35] <mrjack_> eh, well yes ;)
[23:36] <dmick> this isn't by chance a 32-bit platform, is it?
[23:36] <mrjack_> is it because of aio?
[23:36] <mrjack_> dmick: Linux node01 3.4.36-vs2.3.3.9 #1 SMP Sat Mar 16 18:04:37 CET 2013 x86_64 GNU/Linux
[23:36] <joshd> no, it's just a bad return code from read_iterate
[23:36] * ivotron_ (~ivo@ma50536d0.tmodns.net) Quit (Read error: Operation timed out)
[23:36] <joshd> it's stuffing a uint64 into an int64
[23:37] <joshd> I just fixed that for the new diff_iterate interface
[23:37] <mrjack_> oh, will it be in the next stable update?
[23:37] <fghaas> well there you go mrjack_, joshd is your guy :)
[23:38] <joshd> yeah, the simplest fix is to add a read_iterate2 that doesn't try to return the whole length read
[23:39] <gregaf> Elbandi_: yeah, not multi-MDS at all — testing a fix now and will update the bug when I have it :)
[23:40] <joshd> mrjack_: as it is, despite the error reported, all the data was exported, but the file might need to be resized if there's blank space at the end
[23:41] <mrjack_> i'll gunzip and check md5 of normally exported file
[23:41] * vata (~vata@2607:fad8:4:6:cd7e:e419:ff6d:fc94) Quit (Quit: Leaving.)
[23:42] * fghaas (~florian@91-119-65-118.dynamic.xdsl-line.inode.at) has left #ceph
[23:42] * drokita (~drokita@199.255.228.128) Quit (Ping timeout: 480 seconds)
[23:46] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[23:48] * sage (~sage@76.89.177.113) Quit (Read error: Operation timed out)
[23:49] <liiwi> win 12
[23:49] <liiwi> blerp
[23:49] * pioto (~pioto@pool-96-235-30-25.pitbpa.fios.verizon.net) Quit (Quit: leaving)
[23:55] * pioto (~pioto@pool-96-235-30-25.pitbpa.fios.verizon.net) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.