#ceph IRC Log

Index

IRC Log for 2013-07-10

Timestamps are in GMT/BST.

[0:00] <saaby> Machske: yes. if nodown is set, you haev to make sure they are running. - or else I/O to them just waits.
[0:00] <Machske> fridudad_: is the degraded % value decreasing anyway ?
[0:00] <Kdecherf> gregaf: Hm, where can I found a debug log of the client (never did it)?
[0:00] <fridudad_> Machske: yes
[0:00] <saaby> well thats good..
[0:00] <Machske> well personally, I would wait untill everything is rebalanced
[0:00] <Machske> and then see next
[0:01] <saaby> what does "ceph health details | grep inactive" say?
[0:01] <fridudad_> Machske: yeah but how to get there when the osd processes keep suiciding?
[0:02] <Kdecherf> gregaf: Hm, the client seems to not get the mdsmap
[0:02] <saaby> are they still suiciding? I thought you said that they where all staying up now?
[0:02] <Machske> fridudad_: you should at least see something why they are suiciding, ods logs, system logs ?
[0:03] <fridudad_> saaby: no not as long as i set nodown
[0:03] <fridudad_> saaby: ceph health details | grep inactive HEALTH_WARN 820 pgs backfill; 64 pgs backfilling; 159 pgs degraded; 75 pgs peering; 15 pgs recovery_wait; 2 pgs stuck inactive; 974 pgs stuck unclean; recovery 153843/2451817 degraded (6.275%); recovering 297 o/s, 1141MB/s; nodown,noout flag(s) set pg 4.f4d is stuck inactive for 742.258593, current state peering, last acting [11,33,6] pg 4.eed is stuck inactive for 742.344905, current state remap
[0:03] * KindOne (KindOne@0001a7db.user.oftc.net) has joined #ceph
[0:03] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[0:03] <fridudad_> Machske: they can't talk to themself so the healthcheck suicide timeout comes into play
[0:04] <Machske> damn 1141/MB/s, I'm jealous
[0:04] <saaby> fridudad_: ok, so there is peering pg's now again...
[0:04] <Machske> If you don't mind me asking, what kind of HW are you running ?
[0:05] <fridudad_> Machske: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz, 48GB RAM, 2x SSD as journal + 4x 1TB Seagate Constellation as OSD per Host
[0:05] <Machske> 10gig nics ?
[0:05] <fridudad_> Machske: yes dual bonded 10gig nics
[0:06] <Machske> bonded how ?
[0:06] <markit> fridudad_: 2x ssd? raid1?
[0:06] * aliguori (~anthony@32.97.110.51) Quit (Quit: Ex-Chat)
[0:06] * jjgalvez (~jjgalvez@ip72-193-215-88.lv.lv.cox.net) Quit (Server closed connection)
[0:06] <markit> (and with OS installed there, I suppose)
[0:06] <fridudad_> Machske: LACP or what do you mean?
[0:06] <Machske> idd
[0:06] <Machske> but I could not get the speed bonded
[0:06] <saaby> fridudad_: you have peering pg's because you unset "nodown" ?
[0:06] * jjgalvez (~jjgalvez@ip72-193-215-88.lv.lv.cox.net) has joined #ceph
[0:07] <Machske> it would be 2 times 1 Gig instead of 2Gig
[0:07] <fridudad_> markit: 2x SSD with one partitiion for OS as raid the journal itself do not run as raid
[0:07] <Machske> anyway, not your problem at hand
[0:07] <markit> fridudad_: so software raid?
[0:07] <fridudad_> Machske: you need a switch capable of doing an active active LACP HP can do it with their IRF
[0:07] <fridudad_> markit: yes md
[0:08] <Machske> and using the linux bonding driver ?
[0:08] <Machske> module
[0:08] <saaby> fridudad_: also, I don't understand why you have so much backfilling to do, if maps and weights are the same as originally...
[0:08] <fridudad_> saaby: most probably cause i had it at 0 for some hours for these OSDs
[0:08] <saaby> for some hours.. ok.
[0:09] <Machske> at 1GB/s :)
[0:09] <fridudad_> saaby: i restored the values when we started talking
[0:09] <Machske> did you change the values for more then 1 osd ?
[0:09] <markit> fridudad_: tell me when you are out of troubles, I would love to know more about your config / have some suggestions... or could you give me your email in private so I can contact you tomorrow? (me going to sleep soon)
[0:09] <fridudad_> Machske: yes
[0:09] <fridudad_> Machske: which values?
[0:09] <Machske> weights
[0:10] <fridudad_> Machske: yes all OSDs keeping crashing
[0:10] <Machske> really crashing, like process gone Oor hanging
[0:11] <Machske> if they crash, you should be able to find some clues in your logs
[0:11] <fridudad_> Machske: both right now hanging since i set nodown before crashing / killing themself via the heartbeat
[0:12] <Machske> k, new to me, didn't know those osd's have an "internal" heartbeat
[0:12] <Machske> sounds scary
[0:14] <Machske> what I find strange, your osd's crash, but it is still recovering, does your "degraded %" increase when such an osd goes down ?
[0:14] * \ask (~ask@oz.develooper.com) has joined #ceph
[0:14] <saaby> fridudad_: I dont think I am to much help anymore.. I don't understand the rootcause of the osd's burning cpu time or the crashes.
[0:14] <fridudad_> markit how to start a private chat?
[0:15] <saaby> I hope some of the ceph gents will be able to help you.
[0:15] <Kdecherf> gregaf: does it exist any known issue about missing mdsmap on clients ?
[0:15] <Kdecherf> for ceph 0.65
[0:16] <gregaf> Kdecherf: not that pops into my head; if you can pastebin some relevant logs or something I could take a peek
[0:16] * keruspe (~keruspe@slave.imagination-land.org) has joined #ceph
[0:16] <Kdecherf> hm, wait
[0:16] <fridudad_> saaby: thanks. Hopefully it will recover before i have to get up again
[0:17] <saaby> lets hope so.
[0:17] <Kdecherf> gregaf: I have that on the active mds http://bpaste.net/show/l77VVS8hvVuvxIfQcZWw/
[0:17] <Kdecherf> gregaf: and nothing relevant in the mon's log, but I'm reading it
[0:17] <gregaf> Kdecherf: client logs, where the mdsmap is apparently disappearing :)
[0:18] * markit (~marco@88-149-177-66.v4.ngi.it) Quit ()
[0:19] <Kdecherf> gregaf: how can I retrieve it?
[0:19] <gregaf> turn on a client with log=<filename> --debug-client 20 --debug-ms 1
[0:20] <Kdecherf> (i don't know how to get debug logs on the client, right.)
[0:20] <Kdecherf> thx
[0:20] <gregaf> that's for userspace
[0:20] <gregaf> if it's a kernel client it's pretty annoying
[0:21] <Kdecherf> it's a kernel client :(
[0:21] <gregaf> ah, k
[0:21] <gregaf> give me a minute
[0:21] <gregaf> anything in dmesg? that's actually about all that pops into my head right now
[0:22] <Kdecherf> I have just a note about the session succesfully established with a mon
[0:22] <Kdecherf> nothing else
[0:23] * markbby (~Adium@168.94.245.4) Quit (Quit: Leaving.)
[0:24] <gregaf> okay, so inject some debugging (debug mds = 20, debug ms = 1) into the mds and then try to connect again
[0:24] <gregaf> we can look and see what it gets from the client
[0:25] * haomaiwang (~haomaiwan@117.79.232.241) has joined #ceph
[0:26] <Kdecherf> well
[0:26] <Kdecherf> I guess it's a perm issue for the user :(
[0:26] <fridudad_> sage sagewk sjust somebody around?
[0:26] <Kdecherf> admin can mount the pool, but not a "normal" user
[0:27] <Kdecherf> I will check permissions
[0:27] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Quit: Ja odoh a vi sta 'ocete...)
[0:28] * xdeller (~xdeller@91.218.144.129) Quit (Server closed connection)
[0:28] * xdeller (~xdeller@91.218.144.129) has joined #ceph
[0:29] * terje- (~root@135.109.216.239) Quit (Ping timeout: 480 seconds)
[0:29] <Kdecherf> hm
[0:29] * PerlStalker (~PerlStalk@72.166.192.70) Quit (Quit: ...)
[0:31] * nwat (~nwatkins@eduroam-226-128.ucsc.edu) Quit (Server closed connection)
[0:31] * nwat (~nwatkins@eduroam-226-128.ucsc.edu) has joined #ceph
[0:33] <Kdecherf> gregaf: stupid question, where is the "default" keyring for client users other than admin?
[0:33] * haomaiwang (~haomaiwan@117.79.232.241) Quit (Ping timeout: 480 seconds)
[0:33] <gregaf> the same locations
[0:33] <Kdecherf> hm
[0:34] <Kdecherf> I don't remember how i added a new client
[0:34] <Psi-jack> Hmmm
[0:35] <Kdecherf> need some sleep.
[0:35] <Psi-jack> Having an odd issue with Ceph cuttlefish on one node.. Keeping the load averages at 41.
[0:37] * terje- (~root@135.109.216.239) has joined #ceph
[0:37] * nwat (~nwatkins@eduroam-226-128.ucsc.edu) has left #ceph
[0:38] <Psi-jack> And the odd thing is... I can't seem to find what's actually doing it...
[0:39] <Kdecherf> hm ceph auth list, really need to sleep
[0:40] <Kdecherf> gregaf: what are the required caps for a client to use cephfs now?
[0:41] <gregaf> rw access to the data pool, r on the monitors, and mds allow should be it
[0:43] <Kdecherf> hm, it's already set
[0:43] <Kdecherf> and it does not work
[0:44] <Kdecherf> (admin can mount)
[0:44] <gregaf> dump the caps for each that are output by ceph auth list?
[0:45] <Kdecherf> caps: [mds] allow
[0:45] <Psi-jack> Hmm, so ceph-osd.8 was causing it. Stuck in an uninteruptable sleep.
[0:45] <Kdecherf> caps: [mon] allow *
[0:45] <Kdecherf> caps: [osd] allow *
[0:45] <Kdecherf> for admin
[0:45] <Kdecherf> gregaf: http://bpaste.net/show/dkX9XSqYa84UwfsUTQgV/
[0:45] <fridudad_> saaby: FYI ceph osd tree def. shows the OSDs down even with nodown set
[0:46] <gregaf> Kdecherf: hmm, the caps syntax might have changed but I'm not sure what that extra * on the mon line does
[0:47] <gregaf> dmick: do those caps look right to you otherwise?
[0:47] <Kdecherf> hm
[0:49] <dmick> I think mon was always 'allow <perms>'; mds was the only weird one IIRC
[0:49] * Cube (~Cube@12.248.40.138) Quit (Ping timeout: 480 seconds)
[0:49] * jtang1 (~jtang@142.176.24.2) Quit (Quit: Leaving.)
[0:50] * oddomatik (~Adium@12.248.40.138) Quit (Quit: Leaving.)
[0:50] <dmick> certainly I have allow * on my client.admin
[0:51] <dmick> Kdecherf: did you elide the keys from the output??
[0:51] <Kdecherf> dmick: yep
[0:51] <dmick> (sorry, only one ? necessary)
[0:52] <dmick> oh I see
[0:52] <dmick> 'allow r *' seems wrong
[0:52] <dmick> does 'allow r' make it better?
[0:52] <Kdecherf> I check
[0:52] * mschiff (~mschiff@85.182.236.82) Quit (Remote host closed the connection)
[0:53] <Kdecherf> dmick: how can i change the cap with ceph auth?
[0:54] <Kdecherf> oh, nevermind
[0:54] <Kdecherf> found it
[0:56] <Kdecherf> yep, fixed now
[0:56] <Kdecherf> dmick: thx
[0:56] <dmick> cool
[0:56] <Kdecherf> dmick: 'allow r *' worked since our update to 0.65
[0:56] <dmick> I could believe that
[0:57] <dmick> but it was always meaningless
[0:57] <dmick> I don't know whether it would have been the same as * or just r
[0:57] <dmick> one *might* argue that it should be rejected :)
[0:58] <dmick> (and indeed maybe it is now after the rewrite, dunno)
[0:58] <Kdecherf> :)
[1:05] * dpippenger1 (~riven@tenant.pas.idealab.com) Quit (Quit: Leaving.)
[1:06] * dpippenger (~riven@tenant.pas.idealab.com) has joined #ceph
[1:15] * mtanski (~mtanski@69.193.178.202) Quit (Quit: mtanski)
[1:21] * ScOut3R (~ScOut3R@540240D7.dsl.pool.telekom.hu) has joined #ceph
[1:25] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[1:33] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Ping timeout: 480 seconds)
[1:43] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[1:47] * tnt (~tnt@109.130.77.55) Quit (Ping timeout: 480 seconds)
[1:47] * DLange is now known as Guest2695
[1:48] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[1:48] * ScOut3R (~ScOut3R@540240D7.dsl.pool.telekom.hu) Quit (Ping timeout: 480 seconds)
[1:49] * Guest2695 (~DLange@dlange.user.oftc.net) Quit (Ping timeout: 480 seconds)
[1:50] * oddomatik (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[1:51] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[2:00] * Pauline (~middelink@2001:838:3c1:1:be5f:f4ff:fe58:e04) Quit (Quit: Leaving)
[2:13] * scuttlemonkey_ (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) has joined #ceph
[2:14] * scuttlemonkey (~scuttlemo@c-69-244-181-5.hsd1.mi.comcast.net) Quit (Ping timeout: 480 seconds)
[2:28] * psiekl (psiekl@wombat.eu.org) Quit (Remote host closed the connection)
[2:28] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[2:28] * fridudad_ (~oftc-webi@p4FC2C8FD.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[2:28] * psieklFH (psiekl@wombat.eu.org) has joined #ceph
[2:31] * Tamil (~tamil@38.122.20.226) Quit (Quit: Leaving.)
[2:34] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Ping timeout: 480 seconds)
[2:36] * sagelap (~sage@2600:1012:b02a:274c:98cc:b5d6:b7b3:9525) has joined #ceph
[2:40] * portante is now known as portante|afk
[2:48] * yy (~michealyx@218.74.35.50) has joined #ceph
[3:01] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) Quit (Quit: Leaving.)
[3:05] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[3:07] * portante|afk is now known as portante
[3:09] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[3:22] * oddomatik (~Adium@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[3:32] * andreask (~andreask@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[3:32] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[3:39] * LeaChim (~LeaChim@2.220.252.14) Quit (Ping timeout: 480 seconds)
[3:39] * julian (~julianwa@125.69.104.140) has joined #ceph
[3:54] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[4:01] * xmltok_ (~xmltok@relay.els4.ticketmaster.com) has joined #ceph
[4:09] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Quit: Leaving.)
[4:14] * dpippenger (~riven@tenant.pas.idealab.com) Quit (Quit: Leaving.)
[4:35] * mikedawson (~chatzilla@206.246.156.8) Quit (Ping timeout: 480 seconds)
[4:39] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[4:40] * TiCPU (~jeromepou@209.52.17.78) has joined #ceph
[4:51] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[4:58] * TiCPU (~jeromepou@209.52.17.78) Quit (Remote host closed the connection)
[5:01] * fireD (~fireD@93-142-218-171.adsl.net.t-com.hr) has joined #ceph
[5:02] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) has joined #ceph
[5:03] * TiCPU (~jeromepou@209.52.17.78) has joined #ceph
[5:07] * fireD1 (~fireD@93-139-162-89.adsl.net.t-com.hr) Quit (Ping timeout: 480 seconds)
[5:11] * jtang1 (~jtang@blk-222-209-164.eastlink.ca) has joined #ceph
[5:17] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[5:21] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[5:36] * jlogan1 (~Thunderbi@2600:c00:3010:1:1::40) Quit (Ping timeout: 480 seconds)
[5:40] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[5:41] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Remote host closed the connection)
[5:43] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[5:43] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley_)
[5:50] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[5:51] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[5:57] <grepory> is there good documentation somewhere on how to evenly distribute read operations across a ceph cluster?
[5:57] <grepory> i mean the metadata operations, not actually block retrieval
[5:57] <grepory> RR for MONs?
[5:57] <grepory> is it just that simple?
[5:57] <grepory> seems like it would add latency. does librados handle this for you?
[5:58] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[6:02] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[6:08] * yy (~michealyx@218.74.35.50) has left #ceph
[6:17] * rektide (~rektide@192.73.236.68) Quit (Ping timeout: 480 seconds)
[6:17] * TiCPU (~jeromepou@209.52.17.78) Quit (Ping timeout: 480 seconds)
[6:25] * portante is now known as portante|afk
[6:28] * sagelap1 (~sage@76.89.177.113) has joined #ceph
[6:35] * sagelap (~sage@2600:1012:b02a:274c:98cc:b5d6:b7b3:9525) Quit (Ping timeout: 480 seconds)
[6:41] * julian_ (~julianwa@125.69.104.140) has joined #ceph
[6:48] * julian (~julianwa@125.69.104.140) Quit (Ping timeout: 480 seconds)
[6:50] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[6:50] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[6:58] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[7:06] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) Quit (Quit: Leaving.)
[7:11] * yy (~michealyx@218.74.35.50) has joined #ceph
[7:26] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[7:26] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[7:27] * xmltok_ (~xmltok@relay.els4.ticketmaster.com) Quit (Ping timeout: 480 seconds)
[7:37] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[7:52] * tnt (~tnt@109.130.77.55) has joined #ceph
[7:56] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[8:08] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[8:21] * LiRul (~lirul@91.82.105.2) has joined #ceph
[8:21] * rongze1 (~zhu@173-252-252-212.genericreverse.com) has joined #ceph
[8:26] * rongze (~zhu@173-252-252-212.genericreverse.com) Quit (Ping timeout: 480 seconds)
[8:33] * sleinen (~Adium@2001:620:0:26:c5be:c423:539c:a9c2) has joined #ceph
[8:49] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Remote host closed the connection)
[8:49] * haomaiwang (~haomaiwan@101.78.195.61) has joined #ceph
[8:59] * Kioob (~kioob@2a01:e35:2432:58a0:21e:8cff:fe07:45b6) Quit (Quit: Leaving.)
[9:07] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:08] * mschiff (~mschiff@pD951155A.dip0.t-ipconnect.de) has joined #ceph
[9:12] * hybrid512 (~walid@106-171-static.pacwan.net) has joined #ceph
[9:12] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[9:13] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit ()
[9:14] * haomaiwa_ (~haomaiwan@li565-182.members.linode.com) has joined #ceph
[9:20] * andreask (~andreask@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[9:20] * ChanServ sets mode +v andreask
[9:21] * haomaiwang (~haomaiwan@101.78.195.61) Quit (Ping timeout: 480 seconds)
[9:24] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[9:25] * tnt (~tnt@109.130.77.55) Quit (Ping timeout: 480 seconds)
[9:26] * leseb1 (~Adium@2a04:2500:0:d00:a000:f527:4a9f:8085) has joined #ceph
[9:26] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) has joined #ceph
[9:32] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[9:36] * grifferz_ (~andy@specialbrew.392abl.bitfolk.com) Quit (Ping timeout: 480 seconds)
[9:39] * andreask (~andreask@h081217068225.dyn.cm.kabsi.at) Quit (Ping timeout: 480 seconds)
[9:45] * syed_ (~chatzilla@1.38.41.236) has joined #ceph
[9:46] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[9:52] * stacker666 (~stacker66@90.163.235.0) has joined #ceph
[10:14] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:18] * LeaChim (~LeaChim@2.220.252.14) has joined #ceph
[10:24] * itamar_ (~itamar@82.166.185.149) has joined #ceph
[10:25] * infinitytrapdoor (~infinityt@134.95.27.132) has joined #ceph
[10:28] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[10:33] * john_barbee (~jbarbee@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 22.0/20130618035212])
[10:34] <ccourtaut> morning!
[10:43] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[10:46] * syed_ (~chatzilla@1.38.41.236) Quit (Read error: Connection reset by peer)
[11:07] * ScOut3R_ (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) has joined #ceph
[11:13] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[11:14] * X3NQ (~X3NQ@195.191.107.205) Quit (Read error: No route to host)
[11:14] * yy (~michealyx@218.74.35.50) has left #ceph
[11:25] * Macheske (~Bram@d5152D87C.static.telenet.be) has joined #ceph
[11:28] * Machske (~Bram@d5152D87C.static.telenet.be) Quit (Ping timeout: 480 seconds)
[11:28] * IndraPr (~indra@203.126.171.206) has joined #ceph
[11:29] <IndraPr> hi - anyone can help?
[11:29] <IndraPr> i am having problem adding CEPH RBD primary storage into Cloudstack 4.1.0
[11:45] * johnugeorge (~chatzilla@99-31-208-175.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[11:47] <johnugeorge> hi
[11:50] <johnugeorge> I have some doubts regarding the openstack on ceph architecture. How can we implement locking for block device if we expose same block device to all vms on opoenstack?
[11:53] * fridudad_ (~oftc-webi@fw-office.allied-internet.ag) has joined #ceph
[11:55] <stacker666> somebody get succefully compiled tgt with ceph support in ubuntu/debian?
[11:56] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[11:56] <stacker666> the 0.35-37 versions i found are for redhat :(
[11:59] * deadsimple (~infinityt@134.95.27.132) has joined #ceph
[12:02] * fridudad (~oftc-webi@fw-office.allied-internet.ag) Quit (Quit: Page closed)
[12:03] * n3c8-35575 (~mhattersl@pix.office.vaioni.com) Quit (Read error: No route to host)
[12:03] * n3c8-35575 (~mhattersl@pix.office.vaioni.com) has joined #ceph
[12:06] * infinitytrapdoor (~infinityt@134.95.27.132) Quit (Ping timeout: 480 seconds)
[12:12] * johnugeorge (~chatzilla@99-31-208-175.lightspeed.sntcca.sbcglobal.net) Quit (Quit: ChatZilla 0.9.90 [Firefox 22.0/20130627161625])
[12:13] * n3c8-35575 (~mhattersl@pix.office.vaioni.com) Quit (Ping timeout: 480 seconds)
[12:20] * humbolt (~elias@80.123.121.182) has joined #ceph
[12:22] * leseb1 (~Adium@2a04:2500:0:d00:a000:f527:4a9f:8085) Quit (Quit: Leaving.)
[12:31] * haomaiwang (~haomaiwan@117.79.232.209) has joined #ceph
[12:31] * haomaiwa_ (~haomaiwan@li565-182.members.linode.com) Quit (Read error: Connection reset by peer)
[12:33] * humbolt (~elias@80.123.121.182) Quit (Quit: humbolt)
[12:47] * AfC (~andrew@2001:44b8:31cb:d400:7c45:f9b8:8d74:6b09) has joined #ceph
[12:51] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[12:53] * leseb1 (~Adium@2a04:2500:0:d00:9c30:b9f:8c5f:349c) has joined #ceph
[12:58] * AfC (~andrew@2001:44b8:31cb:d400:7c45:f9b8:8d74:6b09) Quit (Ping timeout: 480 seconds)
[13:02] * leseb1 (~Adium@2a04:2500:0:d00:9c30:b9f:8c5f:349c) Quit (Ping timeout: 480 seconds)
[13:03] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[13:10] * hybrid512 (~walid@106-171-static.pacwan.net) Quit (Ping timeout: 480 seconds)
[13:10] * BillK (~BillK-OFT@124-169-221-120.dyn.iinet.net.au) Quit (Read error: Connection reset by peer)
[13:25] * tnt (~tnt@212-166-48-238.win.be) has joined #ceph
[13:28] * jtang1 (~jtang@blk-222-209-164.eastlink.ca) Quit (Quit: Leaving.)
[13:28] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Ping timeout: 480 seconds)
[13:39] * leseb (~Adium@2a04:2500:0:d00:9493:423e:fd48:e58a) has joined #ceph
[13:40] * grifferz (~andy@specialbrew.392abl.bitfolk.com) has joined #ceph
[13:41] * leseb1 (~Adium@2a04:2500:0:d00:c852:3d24:dbab:b006) has joined #ceph
[13:44] * tnt (~tnt@212-166-48-238.win.be) Quit (Ping timeout: 480 seconds)
[13:47] * leseb (~Adium@2a04:2500:0:d00:9493:423e:fd48:e58a) Quit (Ping timeout: 480 seconds)
[13:47] * tnt (~tnt@212-166-48-236.win.be) has joined #ceph
[13:56] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[13:57] * diegows (~diegows@190.190.2.126) has joined #ceph
[14:02] * hybrid512 (~walid@106-171-static.pacwan.net) has joined #ceph
[14:07] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) has joined #ceph
[14:09] * Rocky (~r.nap@188.205.52.204) Quit (Quit: **Poof**)
[14:09] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[14:10] * Rocky (~r.nap@188.205.52.204) Quit ()
[14:13] * Rocky (~r.nap@188.205.52.204) has joined #ceph
[14:14] * mxmln (~maximilia@212.79.49.65) Quit (Read error: Connection reset by peer)
[14:14] * mxmln (~maximilia@212.79.49.65) has joined #ceph
[14:14] * IndraPr (~indra@203.126.171.206) Quit (Ping timeout: 480 seconds)
[14:16] * piti (~piti@82.246.190.142) Quit (Ping timeout: 480 seconds)
[14:16] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:25] <saaby> fridudad_: here?
[14:25] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) has joined #ceph
[14:27] * julian_ (~julianwa@125.69.104.140) Quit (Quit: afk)
[14:29] * ScOut3R__ (~ScOut3R@rock.adverticum.com) has joined #ceph
[14:29] * haomaiwang (~haomaiwan@117.79.232.209) Quit (Remote host closed the connection)
[14:29] * ScOut3___ (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) has joined #ceph
[14:30] * haomaiwang (~haomaiwan@notes4.com) has joined #ceph
[14:30] <fridudad_> saaby:yes
[14:31] * markbby (~Adium@168.94.245.4) has joined #ceph
[14:31] <saaby> fridudad_: you ok?
[14:32] * ScOut3R_ (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[14:32] <saaby> did you figure out what the root cause was?
[14:32] <fridudad_> saaby: everything went fine and is online again. I was also able to fix the 100% cpu usage problem by settig max backfills to 2
[14:32] <saaby> aaaah
[14:32] <saaby> ok
[14:32] <saaby> so
[14:32] <fridudad_> saaby: thanks again for your help
[14:32] <saaby> np
[14:32] <saaby> so
[14:32] <saaby> what triggered the event?
[14:33] <fridudad_> saaby: a crashed OSD and then after the timout ceph started to backfill / recover
[14:34] <saaby> right ok. so simple.. So, the osd's burned so much cpu time because your net and disk configuration is capable of moving data quite quick..
[14:34] * ScOut3R (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[14:35] * Machske (~Bram@d5152D87C.static.telenet.be) has joined #ceph
[14:36] * Macheske (~Bram@d5152D87C.static.telenet.be) Quit (Ping timeout: 480 seconds)
[14:37] * ScOut3R__ (~ScOut3R@rock.adverticum.com) Quit (Ping timeout: 480 seconds)
[14:38] * portante|afk is now known as portante
[14:41] * jtang2 (~jtang@142.176.24.2) has joined #ceph
[14:41] * jtang1 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[14:45] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) has joined #ceph
[14:45] * jtang2 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[14:45] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[14:49] * jtang1 (~jtang@142.176.24.2) Quit ()
[14:49] * infinitytrapdoor (~infinityt@134.95.27.132) has joined #ceph
[14:55] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[14:55] * deadsimple (~infinityt@134.95.27.132) Quit (Ping timeout: 480 seconds)
[14:55] * AfC (~andrew@2001:44b8:31cb:d400:7c45:f9b8:8d74:6b09) has joined #ceph
[14:57] * aliguori_ (~anthony@cpe-70-112-157-87.austin.res.rr.com) has joined #ceph
[14:58] * AfC (~andrew@2001:44b8:31cb:d400:7c45:f9b8:8d74:6b09) Quit ()
[14:59] * piti (~piti@82.246.190.142) has joined #ceph
[15:02] * mikedawson (~chatzilla@mobile-166-147-103-162.mycingular.net) has joined #ceph
[15:05] * aliguori (~anthony@cpe-70-112-157-87.austin.res.rr.com) Quit (Ping timeout: 480 seconds)
[15:09] * jtang2 (~jtang@142.176.24.2) has joined #ceph
[15:09] * jtang1 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[15:10] * mtanski (~mtanski@cpe-74-65-252-48.nyc.res.rr.com) Quit (Quit: mtanski)
[15:14] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[15:14] * jtang2 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[15:16] * jtang2 (~jtang@142.176.24.2) has joined #ceph
[15:18] * jtang1 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[15:19] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[15:19] * mnash_ (~chatzilla@vpn.expressionanalysis.com) has joined #ceph
[15:21] * rtek_ (~sjaak@rxj.nl) has joined #ceph
[15:21] * jtang1 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[15:21] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[15:22] * mikedawson (~chatzilla@mobile-166-147-103-162.mycingular.net) Quit (Read error: No route to host)
[15:22] * ninkotech_ (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[15:22] * mikedawson (~chatzilla@23-25-19-9-static.hfc.comcastbusiness.net) has joined #ceph
[15:23] * iggy2 (~iggy@theiggy.com) has joined #ceph
[15:23] * infernix (nix@cl-1404.ams-04.nl.sixxs.net) Quit (Ping timeout: 480 seconds)
[15:23] * IndraPr (~indra@203.126.171.206) has joined #ceph
[15:24] * morse_ (~morse@supercomputing.univpm.it) has joined #ceph
[15:24] * infernix (nix@cl-1404.ams-04.nl.sixxs.net) has joined #ceph
[15:24] * [fred]_ (fred@konfuzi.us) has joined #ceph
[15:24] * jtang2 (~jtang@142.176.24.2) Quit (Ping timeout: 480 seconds)
[15:24] * __jt___ (~james@rhyolite.bx.mathcs.emory.edu) has joined #ceph
[15:24] * SWAT_ (~swat@cyberdyneinc.xs4all.nl) has joined #ceph
[15:24] * jtang1 (~jtang@142.176.24.2) Quit ()
[15:24] * [fred] (fred@konfuzi.us) Quit (Ping timeout: 480 seconds)
[15:25] * diegows (~diegows@190.190.2.126) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * mynameisbruce (~mynameisb@tjure.netzquadrat.de) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * bwesemann (~bwesemann@2001:1b30:0:6:990e:2128:78c7:8573) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * mnash (~chatzilla@vpn.expressionanalysis.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * Tribaal (uid3081@id-3081.ealing.irccloud.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * ccourtaut (~ccourtaut@2001:41d0:1:eed3::1) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * vhasi (vhasi@vha.si) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * MooingLemur (~troy@phx-pnap.pinchaser.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * morse (~morse@supercomputing.univpm.it) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * rtek (~sjaak@rxj.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * brother (foobaz@vps1.hacking.dk) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * Cybertinus (~Cybertinu@2001:828:405:30:83:96:177:42) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * iggy (~iggy@theiggy.com) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * joelio (~Joel@88.198.107.214) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * SWAT (~swat@cyberdyneinc.xs4all.nl) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * __jt__ (~james@rhyolite.bx.mathcs.emory.edu) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * MK_FG (~MK_FG@00018720.user.oftc.net) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * asadpanda (~asadpanda@2001:470:c09d:0:20c:29ff:fe4e:a66) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * raso (~raso@deb-multimedia.org) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (reticulum.oftc.net solenoid.oftc.net)
[15:25] * [fred]_ is now known as [fred]
[15:25] * mnash_ is now known as mnash
[15:26] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[15:26] * brother (foobaz@vps1.hacking.dk) has joined #ceph
[15:26] * bwesemann (~bwesemann@2001:1b30:0:6:dd4e:600d:f687:2650) has joined #ceph
[15:26] * PerlStalker (~PerlStalk@72.166.192.70) has joined #ceph
[15:27] * MooingLemur (~troy@phx-pnap.pinchaser.com) has joined #ceph
[15:27] * Karcaw (~evan@68-186-68-219.dhcp.knwc.wa.charter.com) has joined #ceph
[15:28] * diegows (~diegows@190.190.2.126) has joined #ceph
[15:28] * vhasi (vhasi@vha.si) has joined #ceph
[15:28] * joelio_ (~Joel@88.198.107.214) has joined #ceph
[15:29] * bwesemann (~bwesemann@2001:1b30:0:6:dd4e:600d:f687:2650) Quit (Killed (reticulum.oftc.net (Nick collision (new))))
[15:29] * mynameisbruce (~mynameisb@tjure.netzquadrat.de) has joined #ceph
[15:29] * bwesemann (~bwesemann@2001:1b30:0:6:990e:2128:78c7:8573) has joined #ceph
[15:29] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) has joined #ceph
[15:29] * Tribaal (uid3081@id-3081.ealing.irccloud.com) has joined #ceph
[15:29] * ccourtaut (~ccourtaut@2001:41d0:1:eed3::1) has joined #ceph
[15:29] * Cybertinus (~Cybertinu@2001:828:405:30:83:96:177:42) has joined #ceph
[15:29] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) has joined #ceph
[15:29] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[15:29] * rtek (~sjaak@rxj.nl) has joined #ceph
[15:29] * iggy (~iggy@theiggy.com) has joined #ceph
[15:29] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[15:29] * SWAT (~swat@cyberdyneinc.xs4all.nl) has joined #ceph
[15:29] * MK_FG (~MK_FG@00018720.user.oftc.net) has joined #ceph
[15:29] * joelio (~Joel@88.198.107.214) has joined #ceph
[15:29] * raso (~raso@deb-multimedia.org) has joined #ceph
[15:29] * asadpanda (~asadpanda@2001:470:c09d:0:20c:29ff:fe4e:a66) has joined #ceph
[15:29] * ninkotech (~duplo@static-84-242-87-186.net.upcbroadband.cz) Quit (Ping timeout: 480 seconds)
[15:29] * rtek (~sjaak@rxj.nl) Quit (Ping timeout: 480 seconds)
[15:29] * SWAT (~swat@cyberdyneinc.xs4all.nl) Quit (Ping timeout: 480 seconds)
[15:29] * iggy (~iggy@theiggy.com) Quit (Remote host closed the connection)
[15:29] * Cybertinus (~Cybertinu@2001:828:405:30:83:96:177:42) Quit (Ping timeout: 480 seconds)
[15:29] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[15:29] * Cybertinus (~Cybertinu@2001:828:405:30:83:96:177:42) has joined #ceph
[15:29] * bwesemann_ (~bwesemann@2001:1b30:0:6:dd4e:600d:f687:2650) has joined #ceph
[15:30] * scuttlemonkey_ is now known as scuttlemonkey
[15:30] * bwesemann (~bwesemann@2001:1b30:0:6:990e:2128:78c7:8573) Quit (Ping timeout: 484 seconds)
[15:30] * jtang1 (~jtang@142.176.24.2) Quit ()
[15:30] * joelio (~Joel@88.198.107.214) Quit (Ping timeout: 484 seconds)
[15:31] * leseb (~Adium@2a04:2500:0:d00:d142:7b2b:9c0d:51d) has joined #ceph
[15:31] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) has joined #ceph
[15:35] * leseb1 (~Adium@2a04:2500:0:d00:c852:3d24:dbab:b006) Quit (Ping timeout: 480 seconds)
[15:39] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley_)
[15:41] * maswan (maswan@kennedy.acc.umu.se) Quit (Remote host closed the connection)
[15:41] * maswan (maswan@kennedy.acc.umu.se) has joined #ceph
[15:41] * jluis (~JL@89.181.157.135) has joined #ceph
[15:42] * leseb (~Adium@2a04:2500:0:d00:d142:7b2b:9c0d:51d) Quit (Quit: Leaving.)
[15:43] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[15:47] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[15:49] * joao (~JL@89.181.157.135) Quit (Ping timeout: 480 seconds)
[15:51] * leseb (~Adium@2a04:2500:0:d00:8084:516e:2669:3424) has joined #ceph
[16:03] * mtk (~mtk@ool-44c35983.dyn.optonline.net) Quit (Remote host closed the connection)
[16:06] * mtk (~mtk@ool-44c35983.dyn.optonline.net) has joined #ceph
[16:10] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[16:13] * fridudad_ (~oftc-webi@fw-office.allied-internet.ag) Quit (Quit: Page closed)
[16:14] * vata (~vata@2607:fad8:4:6:4120:5799:7bb8:eff8) Quit (Ping timeout: 480 seconds)
[16:17] * jtang2 (~jtang@142.176.24.2) has joined #ceph
[16:18] * vata (~vata@2607:fad8:4:6:f836:7ff0:1c58:dad5) has joined #ceph
[16:18] * jtang2 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[16:19] * s2r2 (uid322@id-322.charlton.irccloud.com) Quit (Ping timeout: 480 seconds)
[16:20] * jtang2 (~jtang@142.176.24.2) has joined #ceph
[16:21] * jtang1 (~jtang@142.176.24.2) Quit (Ping timeout: 480 seconds)
[16:22] * scalability-junk (uid6422@id-6422.charlton.irccloud.com) Quit (Ping timeout: 480 seconds)
[16:25] * ismell_ (~ismell@host-24-56-171-198.beyondbb.com) has left #ceph
[16:26] * jtang2 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[16:26] * jtang3 (~jtang@142.176.24.2) has joined #ceph
[16:29] * tnt_ (~tnt@212-166-48-236.win.be) has joined #ceph
[16:29] <itamar_> Hi,
[16:30] <itamar_> where can I download 0.56.5 rpms?
[16:30] <itamar_> I need to upgrade from bobtail to cuttlefish
[16:30] * tnt (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[16:31] <itamar_> and I understand that's the one I must go through
[16:31] * LiRul (~lirul@91.82.105.2) Quit (Quit: Leaving.)
[16:31] <itamar_> anyone?
[16:31] * s2r2 (uid322@id-322.ealing.irccloud.com) has joined #ceph
[16:35] * scalability-junk (uid6422@id-6422.ealing.irccloud.com) has joined #ceph
[16:36] * mtanski (~mtanski@69.193.178.202) has joined #ceph
[16:36] * jtang1 (~jtang@142.176.24.2) has joined #ceph
[16:37] * jtang3 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[16:39] <joelio_> itamar_: 0.56 is bobtail is it not?
[16:39] * joelio_ is now known as joelio
[16:40] * jtang1 (~jtang@142.176.24.2) Quit (Read error: Connection reset by peer)
[16:40] <joelio> itamar_: ahh, I see
[16:44] * doubleg (~doubleg@69.167.130.11) has joined #ceph
[16:44] <itamar_> joelio: hey, any idea?
[16:45] <joelio> itamar_: sorry, Ubuntu users. Can't see any rpms on ceph's site for 0.56
[16:48] <itamar_> thanks..
[16:50] <stacker666> hi all
[16:50] <stacker666> somebody has successful using tgt with ceph feature?
[16:51] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) has joined #ceph
[16:51] * sleinen1 (~Adium@2001:620:0:46:fcce:f5d1:d85c:bc23) has joined #ceph
[16:52] * iggy2 is now known as iggy
[16:55] * allsystemsarego (~allsystem@188.27.165.68) has joined #ceph
[16:57] * sleinen (~Adium@2001:620:0:26:c5be:c423:539c:a9c2) Quit (Ping timeout: 480 seconds)
[16:57] * vata (~vata@2607:fad8:4:6:f836:7ff0:1c58:dad5) Quit (Ping timeout: 480 seconds)
[17:08] * tnt_ (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[17:09] * infinitytrapdoor (~infinityt@134.95.27.132) Quit (Ping timeout: 480 seconds)
[17:11] * vata (~vata@2607:fad8:4:6:f836:7ff0:1c58:dad5) has joined #ceph
[17:13] * sputnik13 (~sputnik13@wsip-68-105-248-60.sd.sd.cox.net) Quit (Quit: sputnik13)
[17:14] * leseb (~Adium@2a04:2500:0:d00:8084:516e:2669:3424) Quit (Quit: Leaving.)
[17:15] * leseb (~Adium@2a04:2500:0:d00:8be:5666:2263:4a84) has joined #ceph
[17:19] * itamar_ (~itamar@82.166.185.149) Quit (Quit: Leaving)
[17:19] * tnt (~tnt@109.130.77.55) has joined #ceph
[17:23] * jlogan (~Thunderbi@2600:c00:3010:1:1::40) has joined #ceph
[17:27] * rongze1 (~zhu@173-252-252-212.genericreverse.com) Quit (Remote host closed the connection)
[17:32] * stacker666 (~stacker66@90.163.235.0) Quit (Ping timeout: 480 seconds)
[17:34] <Gugge-47527> anyone know why rbd5 is shown as emcpowerig in iostat?
[17:38] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[17:42] * ScOut3___ (~ScOut3R@catv-89-133-17-71.catv.broadband.hu) Quit (Ping timeout: 480 seconds)
[17:53] * joshd1 (~jdurgin@2602:306:c5db:310:18fa:6b5e:f522:84cd) has joined #ceph
[17:54] * diegows (~diegows@190.190.2.126) Quit (Ping timeout: 480 seconds)
[17:57] <IndraPr> <IndraPr> hi - anyone can help?
[17:57] <IndraPr> <IndraPr> i am having problem adding CEPH RBD primary storage into Cloudstack 4.1.0
[17:58] <Gugge-47527> what did you try, and what went wrong?
[17:59] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[18:00] * oddomatik (~Adium@cpe-76-95-217-129.socal.res.rr.com) has joined #ceph
[18:09] <joelio> I really wish people would take the time to read this - http://www.catb.org/esr/faqs/smart-questions.html
[18:12] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[18:18] * grepory (~Adium@50-115-70-146.static-ip.telepacific.net) has joined #ceph
[18:19] * jluis is now known as joao
[18:21] * mikedawson (~chatzilla@23-25-19-9-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[18:23] * sleinen1 (~Adium@2001:620:0:46:fcce:f5d1:d85c:bc23) Quit (Ping timeout: 480 seconds)
[18:26] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) has joined #ceph
[18:28] <Gugge-47527> joelio :)
[18:29] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) Quit (Read error: Connection reset by peer)
[18:40] * joshd1 (~jdurgin@2602:306:c5db:310:18fa:6b5e:f522:84cd) Quit (Quit: Leaving.)
[18:40] * rturk-away is now known as rturk
[18:51] * markl (~mark@tpsit.com) Quit (Quit: leaving)
[18:52] * leseb (~Adium@2a04:2500:0:d00:8be:5666:2263:4a84) Quit (Quit: Leaving.)
[18:52] * markl (~mark@tpsit.com) has joined #ceph
[18:54] * Cube (~Cube@cpe-76-95-217-129.socal.res.rr.com) Quit (Quit: Leaving.)
[18:54] * IndraPr- (~indra@203.126.171.206) has joined #ceph
[18:56] * leseb (~Adium@2a04:2500:0:d00:6c4f:d37c:66e6:169a) has joined #ceph
[18:56] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) has joined #ceph
[18:58] * sleinen1 (~Adium@2001:620:0:26:8cd0:df6a:2984:ad22) has joined #ceph
[18:58] * IndraPr (~indra@203.126.171.206) Quit (Ping timeout: 480 seconds)
[19:04] * leseb (~Adium@2a04:2500:0:d00:6c4f:d37c:66e6:169a) Quit (Quit: Leaving.)
[19:04] * sleinen (~Adium@77-58-245-10.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[19:05] <sage> gregaf: 10 second reivew on wip-5581?
[19:06] * Midnightmyth (~quassel@93-167-84-102-static.dk.customer.tdc.net) has joined #ceph
[19:06] <gregaf> looking
[19:07] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[19:09] <gregaf> sage: in a quick grep it looks like this problem might exist elsewhere, did you audit or just see this one causing a problem?
[19:09] <sage> just saw that one. i'll look for others
[19:09] * sjustlaptop (~sam@24-205-35-233.dhcp.gldl.ca.charter.com) has joined #ceph
[19:10] <sage> in this case it's the confusion between osdmap and pgmap versions
[19:10] <gregaf> yeah
[19:10] <gregaf> there's a bunch more of them using their 'v' field or whatever though, and I don't know if they're all the right one or not
[19:11] * markl (~mark@tpsit.com) Quit (Quit: leaving)
[19:11] * markl (~mark@tpsit.com) has joined #ceph
[19:12] <sage> MMonCommand is wrong oto
[19:13] <sage> others are ok
[19:14] <gregaf> k
[19:14] <gregaf> sounds good then
[19:14] <gregaf> looks good too ;)
[19:14] <gregaf> did we just never see it because the pgmap increments so much faster than the osdmap?
[19:14] * sagelap1 (~sage@76.89.177.113) Quit (Ping timeout: 480 seconds)
[19:15] <sage> yeah i think so
[19:15] * Cube (~Cube@12.248.40.138) has joined #ceph
[19:18] <sage> the MMonCOmmand one isn't even used on the mon side. adding anothe rpatch to not populate it with a bogus value.
[19:19] * Tamil (~tamil@38.122.20.226) has joined #ceph
[19:22] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) has joined #ceph
[19:25] * yehudasa_ (~yehudasa@2602:306:330b:1410:9de3:9265:e904:18d3) Quit (Ping timeout: 480 seconds)
[19:30] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) Quit (Quit: Leaving.)
[19:31] * AndroUser (~androirc@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[19:33] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) has joined #ceph
[19:33] * AndroUser is now known as itamar_
[19:36] <sagelap> sjust sjustlaptop: the 5518 fix looks ok otherwise?
[19:36] <sagelap> bumped it up to 100, the same # of maps we pack into a single message
[19:36] <sjustlaptop> sagelap: yeah
[19:36] <sjustlaptop> sounds god
[19:36] <sjustlaptop> *good
[19:36] <sagelap> cool
[19:37] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) Quit (Quit: AndroIRC - Android IRC Client ( http://www.androirc.com ))
[19:37] <paravoid> sjustlaptop: hey, can I move this slow peering again without bugging you much? e.g. want to give me a few scenarios to run/debugging options that you'd like so that we could do this less interactivelY?
[19:38] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) Quit (Read error: Connection reset by peer)
[19:38] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[19:39] <sjustlaptop> paravoid: do you see wrongly marked down messages while your osds are slow peering?
[19:39] <paravoid> no
[19:39] <sjustlaptop> k
[19:40] * wer (~wer@206-248-239-142.unassigned.ntelos.net) Quit (Ping timeout: 480 seconds)
[19:40] <sjustlaptop> in that case, debug ms = 1, debug osd = 20, debug filestore = 20 on the osd you are going to restart and some set of osds on other machines (perhaps choose 10 other machines and 1 osd on each?) as well as debug ms = 1, debug mon = 20 on the monitors?
[19:40] <sjustlaptop> that way, we might get lucky and snag two replicas from one of the slow peering pgs?
[19:41] <paravoid> heh, that's a lot of logs
[19:41] <paravoid> okay
[19:41] <sjustlaptop> paravoid: yeah
[19:41] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) Quit ()
[19:42] <sjustlaptop> paravoid: while it's peering, it would be great if you could identify a slow one and grab 'ceph pg <pgid> query' while it's stuck
[19:42] <paravoid> so, you're not seeing this in your test clusters at all?
[19:42] <sjustlaptop> paravoid: not yet, but I'm putting together one hopefully to test it tomorrow or the day after
[19:42] <sjustlaptop> paravoid: I suspect radosgw is the x factor
[19:42] <paravoid> how come?
[19:43] <sjustlaptop> that is, it causes more traffic on leveldb than rbd
[19:43] <paravoid> okay
[19:43] <paravoid> the other unusual bit of our setup is that we have tons of very small files
[19:43] <sjustlaptop> yeah
[19:43] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[19:43] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) has joined #ceph
[19:43] <sjustlaptop> btw, how many buckets do you have? (100k-500k?)
[19:43] <paravoid> 37k
[19:43] <sjustlaptop> k
[19:44] <sjustlaptop> want to try to mimic your object/bucket distribution when I populate the cluster
[19:44] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) Quit ()
[19:44] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[19:45] <paravoid> objects are not uniformly distributed, but the largest would be 800k objects or so
[19:45] <sjustlaptop> ok
[19:45] <sjustlaptop> sizes around 4k?
[19:45] <paravoid> more most probably
[19:46] * diegows (~diegows@200.68.116.185) has joined #ceph
[19:46] <sjustlaptop> ok
[19:46] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) Quit (Remote host closed the connection)
[19:46] <sjustlaptop> 64k-256k?
[19:46] <paravoid> for half of the files, yes
[19:46] * dpippenger (~riven@cpe-76-166-208-83.socal.res.rr.com) has joined #ceph
[19:46] <sjustlaptop> rest are larger?
[19:46] <paravoid> other half would be larger files
[19:46] <paravoid> yes
[19:46] <sjustlaptop> ok, 4MB-128MB?
[19:46] * wer (~wer@206-248-239-142.unassigned.ntelos.net) has joined #ceph
[19:46] <paravoid> we keep original images, scaled thumbnails and a tiny portion of videos
[19:47] <sjustlaptop> ok
[19:47] <paravoid> original images would be 4MB-128MB for the most part, yes
[19:47] <paravoid> closer to 4MB than 128MB probably :)
[19:47] <sjustlaptop> ok
[19:47] <paravoid> videos would be gigabytes but it's a tiny data set really
[19:47] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) Quit ()
[19:48] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[19:48] <paravoid> maybe 100k videos?
[19:48] <paravoid> something on that range
[19:49] <paravoid> total obj count is 272 million right now
[19:49] <sjustlaptop> k
[19:50] <paravoid> more than 220 million is thumbnails
[19:51] <paravoid> so half in size, but the significant percentage object count wise
[19:51] * itamar_ (~androirc@IGLD-84-229-155-50.inter.net.il) Quit ()
[19:51] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[19:51] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[20:01] * janisg (~troll@85.254.50.23) Quit (Server closed connection)
[20:01] * janisg (~troll@85.254.50.23) has joined #ceph
[20:01] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) has joined #ceph
[20:03] * mikedawson (~chatzilla@23-25-46-97-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[20:09] * mikedawson (~chatzilla@23-25-19-9-static.hfc.comcastbusiness.net) has joined #ceph
[20:10] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) Quit (Quit: AndroIRC - Android IRC Client ( http://www.androirc.com ))
[20:10] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[20:13] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) Quit ()
[20:13] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[20:20] * IndraPr- (~indra@203.126.171.206) Quit (Ping timeout: 480 seconds)
[20:23] * fridudad (~oftc-webi@p4FC2D6A8.dip0.t-ipconnect.de) has joined #ceph
[20:24] * fridudad_ (~oftc-webi@p4FC2D6A8.dip0.t-ipconnect.de) has joined #ceph
[20:24] * fridudad (~oftc-webi@p4FC2D6A8.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[20:24] <fridudad_> saaby: was disconnected now online again
[20:27] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) Quit (Remote host closed the connection)
[20:30] * buck (~buck@c-24-6-91-4.hsd1.ca.comcast.net) has joined #ceph
[20:37] * nhorman (~nhorman@hmsreliant.think-freely.org) has joined #ceph
[20:37] * mschiff (~mschiff@pD951155A.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[20:38] * tdb (~tdb@willow.kent.ac.uk) Quit (Server closed connection)
[20:38] * tdb (~tdb@willow.kent.ac.uk) has joined #ceph
[20:46] <sagelap> sjustlaptop: can you ping me when you have the new set of logs on slider (or wherever)?
[20:47] <sjustlaptop> sagewk: ah, didn't realize there was another set
[20:47] <sjustlaptop> downloading and untaring now
[20:47] * sagelap (~sage@2600:1012:b00b:c734:2993:205:8718:2fda) Quit (Read error: No route to host)
[20:49] * rturk is now known as rturk-away
[20:49] * LeaChim (~LeaChim@2.220.252.14) Quit (Remote host closed the connection)
[20:53] * rturk-away is now known as rturk
[20:53] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) Quit (Quit: smiley_)
[20:53] * LeaChim (~LeaChim@2.220.252.14) has joined #ceph
[20:57] * Tamil (~tamil@38.122.20.226) Quit (Quit: Leaving.)
[20:58] * guppy (~quassel@guppy.xxx) Quit ()
[20:58] * guppy (~quassel@guppy.xxx) has joined #ceph
[21:01] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) Quit (Ping timeout: 480 seconds)
[21:06] <jks> on Fedora doing a "service ceph start osd.15" returns immediately outputting nothing and not starting any osds... what have I done wrong?
[21:06] <jks> (osd.15 is local to the machine where I issued the command)
[21:08] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) has joined #ceph
[21:15] <sjustlaptop> anyone know what radosgw caps are required to create a bucket?
[21:15] <sjustlaptop> yehuda_hm, yehudasa: ^
[21:17] * `10__ (~10@juke.fm) Quit (Server closed connection)
[21:17] * `10__ (~10@juke.fm) has joined #ceph
[21:17] <jks> nevermind... remembered that this happens when the hostname of the host doesn't match what is in ceph.conf
[21:27] <yehudasa> sjustlaptop: radosgw caps?
[21:27] <sjustlaptop> nvm, that one turned out to be that I had to create the pool
[21:28] * Tamil (~tamil@38.122.20.226) has joined #ceph
[21:36] * MrNPP (~MrNPP@216.152.240.194) Quit (Server closed connection)
[21:36] * MrNPP (~MrNPP@216.152.240.194) has joined #ceph
[21:37] * sagelap (~sage@67.159.191.98) has joined #ceph
[21:39] * nwat (~nwatkins@eduroam-226-128.ucsc.edu) has joined #ceph
[21:44] * Psi-jack (~psi-jack@psi-jack.user.oftc.net) Quit (Server closed connection)
[21:44] * Psi-Jack_ is now known as Psi-Jack
[21:44] * Psi-Jack_ (~psi-jack@psi-jack.user.oftc.net) has joined #ceph
[21:55] * Coyo (~coyo@00017955.user.oftc.net) Quit (Server closed connection)
[21:55] * Coyo (~coyo@thinks.outside.theb0x.org) has joined #ceph
[21:56] * Coyo is now known as Guest2843
[21:59] * sjust (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) Quit (Ping timeout: 480 seconds)
[22:01] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:04] * nhorman (~nhorman@hmsreliant.think-freely.org) Quit (Quit: Leaving)
[22:07] <sjustlaptop> sagewk: untarred in the same place on slider
[22:09] * jerker (jerker@Psilocybe.Update.UU.SE) Quit (Server closed connection)
[22:09] * jerker (jerker@Psilocybe.Update.UU.SE) has joined #ceph
[22:10] * jakes (~oftc-webi@dhcp-171-71-119-30.cisco.com) has joined #ceph
[22:11] * sagelap (~sage@67.159.191.98) Quit (Ping timeout: 480 seconds)
[22:11] <jakes> Hi, Can we provide locking mechanism for ceph block device?
[22:12] <gregaf> you have one? or you want one?
[22:13] <jakes> Sorry :).. I need to know if something exists. If i expose the block device to many clients, it will create issues without locking right?
[22:13] <gregaf> ah
[22:14] * sagelap (~sage@216.3.101.62) has joined #ceph
[22:14] <gregaf> there's a voluntary locking mechanism that you can use via the APIs or the CLI tool to coordinate stuff
[22:15] <gregaf> there's not any fencing as providing the guarantees you'd want is fairly expensive
[22:15] <jakes> is there any tricky way to solve this situation?.I was planning to have openstack with ceph as object store. So, If i need any shared folder ( file level access), across all VMs, is it possible?
[22:16] <dmick> jakes: think of the block device as a disk. You don't share one disk among multiple machines.
[22:16] <gregaf> right now the best solution is probably to set up an nfs server that's backed by RBD
[22:16] <gregaf> some day CephFS will do that, but today is not that day
[22:16] * mschiff (~mschiff@85.182.236.82) has joined #ceph
[22:16] <gregaf> but yeah, what dmick said
[22:18] <jakes> nfs in one vm and mount nfs share in other vms.?
[22:19] <jakes> but, This will again become single point of failure right?
[22:20] <dmick> sure, if you only set up one
[22:21] * ofu (ofu@dedi3.fuckner.net) Quit (Server closed connection)
[22:21] * ofu (ofu@dedi3.fuckner.net) has joined #ceph
[22:21] * sleinen1 (~Adium@2001:620:0:26:8cd0:df6a:2984:ad22) Quit (Quit: Leaving.)
[22:24] <jakes> but, If we are having multiple NFS servers, I feel , we are not taking advantages of ceph object store which does replication. Say for applications like hadoop over openstack, I am trying to see if we can remove hdfs layer and take advantages of ceph. The main reason is, ceph takes care of mjor functionalities of hdfs and HDFS layer is not necessary.
[22:25] <buck> jakes: to run Hadoop, you'd need the Ceph file system, which runs on top of the block layer and has more bells and whistles (file locking being one of those)
[22:26] * markbby (~Adium@168.94.245.4) Quit (Quit: Leaving.)
[22:26] <dmick> jakes: of course. You need a shared storage system. RBD images are not that (although they are stored on one, so benefit from resilience to failures of hosts, networks, disks, etc.)
[22:26] <dmick> there are many building blocks and many ways to be redundant/fault-tolerant
[22:27] * xmltok (~xmltok@pool101.bizrate.com) Quit (Quit: Leaving...)
[22:27] <jakes> but, The problem is , how is the file system CEPHFS exposed to VM's over openstack in the architecture http://ceph.com/docs/master/rbd/rbd-openstack/
[22:27] * xmltok (~xmltok@pool101.bizrate.com) has joined #ceph
[22:28] * sagelap (~sage@216.3.101.62) Quit (Ping timeout: 480 seconds)
[22:28] * sagelap (~sage@199.sub-70-197-73.myvzw.com) has joined #ceph
[22:30] <dmick> that documents how to use RBD with openstack. RBD and CephFS are not the same thing
[22:30] * itamar_ (~congfoo@IGLD-84-229-155-50.inter.net.il) Quit (Read error: Connection reset by peer)
[22:31] * smiley_ (~smiley@pool-173-73-0-53.washdc.fios.verizon.net) has joined #ceph
[22:31] <jakes> dmick,I understand. But, can we integrate CephFs and Openstack ?.. Wht can be the possible solutions?
[22:31] <jakes> dmick, I also heard from webinars that CephFS is not well tested as RBD. Thats why, i was more inclined to RBD. Can we get this functionaliy with CephFS?
[22:35] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit (Quit: Leaving.)
[22:36] * markbby (~Adium@168.94.245.2) has joined #ceph
[22:37] <dmick> Nothing stops you from creating a CephFS and accessing it from VMs, Openstack or otherwise. As far as automatically provisioning CephFS with Openstack, I don't think that functionality exists (I could be wrong). And, yes, CephFS is not as stable as RBD, but it also has different capabilities. CephFS is a shared, distributed filesystem. RBD is a block-device image stored on reliable distributed object storage.
[22:38] * mikedawson (~chatzilla@23-25-19-9-static.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[22:41] * Steki (~steki@fo-d-130.180.254.37.targo.rs) has joined #ceph
[22:42] <jakes> Dmick i am little confused. AFAIK, For my situtaion, if I need to use CephFS instead of HDFS , I have to install CEPH again for each VM's in openstack, right?. This will be equivalent to , using HDFS .
[22:46] * _robbat21irssi (nobody@www2.orbis-terrarum.net) has joined #ceph
[22:46] * BManojlovic (~steki@fo-d-130.180.254.37.targo.rs) Quit (Ping timeout: 480 seconds)
[22:46] * Ludo (~Ludo@falbala.zoxx.net) Quit (Server closed connection)
[22:46] * Ludo (~Ludo@falbala.zoxx.net) has joined #ceph
[22:46] <buck> jakes: I believe that is correct. I think it would be possible for the ceph FS and RBD to be served from the same storage cluster, but they would be using different objects in that cluster (since RBD and ceph FS behave very differently, they can't / shouldn't share objects).
[22:47] <_robbat21irssi> yo, trying to resolve some weirdness with radosgw & the S3 api; around subuser permissions to objects
[22:47] * scheuk (~scheuk@204.246.67.78) Quit (Server closed connection)
[22:47] * scheuk (~scheuk@204.246.67.78) has joined #ceph
[22:47] <buck> jakes: but I'd take others word for it over mine
[22:47] <_robbat21irssi> if I try to set as public, I get a 403 denied, but doing 'info' on the object works
[22:47] <jakes> thanks buck
[22:48] <_robbat21irssi> i also can't do info on a directory, or set any perms on a dir
[22:48] <_robbat21irssi> those give me 404 not found
[22:48] <_robbat21irssi> yet the files in the dir exist
[22:48] <dmick> jakes: certainly if you want to use CephFS, you need the CephFS client module or library, yes
[22:51] * rturk is now known as rturk-away
[22:51] * rturk-away is now known as rturk
[22:51] <_robbat21irssi> http://bpaste.net/show/113407/ <--- pastebin of 404 on directory info, 403 on file setacl public
[22:52] <jakes> is there any way to use cephFs other than installing in each of the VM's?.. If I am installing in each VM's, I will lose the purpose of removing HDFS. Will I get any performance improvement when compared to HDFS?.
[22:53] <buck> jakes: check out the thread from the last few days titled "Hadoop/Ceph and DFS IO tests" in the ceph mailing list (discussion of Ceph vs HDFS performance at present, or at least aspects of it). One qualitative benefit of using Ceph FS is that it allows for mutable data (which HDFS doesn't).
[22:54] <buck> jakes: the thread on the ceph mailing list
[22:54] <tnt> jakes: mmm, with the kernel client I think you don't need anything in the client VM except a recent kernel.
[22:54] <buck> tnt: I think the hadoop bindings leverage libcephfs
[22:54] <tnt> ah well then ...
[22:54] <dmick> Of course one problem is that the VMs need to be running an OS that supports Ceph
[22:55] <_robbat21irssi> anybody for RGW acl/dir behavior?
[22:55] <dmick> jakes, I'm not clear on what your requirements are and am not trying to design a solution for you, just pointing out what can work and what can't, and what rbd is vs. cephfs
[22:58] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) has joined #ceph
[22:58] <jakes> Dmick, I am trying to integrate hadoop in VMS over openstack. As i saw ceph had good features of hdfs and it has support for openstack, I was thinking of some solutions which can give better performance. I am curious to know if libcephfs give better perofrmance. Is there any reason for it?
[22:59] * allsystemsarego (~allsystem@188.27.165.68) Quit (Quit: Leaving)
[23:00] <jakes> Referring to what buck said, can we somehow map objects from rbd and CEphFS ?. I thought it would be same, as it is same object store. I was surprised to hear that they would be different objects in the cluster. Correct me if i am wrong
[23:00] * andreask (~andreask@h081217068225.dyn.cm.kabsi.at) has joined #ceph
[23:00] * ChanServ sets mode +v andreask
[23:01] <buck> jakes: the alternative would be to run two separate storage clusters (block storage for VMs and HDFS), correct?
[23:01] <gregaf> _robbat21irssi: not quite sure what all you're talking about since object stores don't have directories (although I'm not super familiar with the swift nomenclature I don't think it uses those)
[23:01] * leseb (~Adium@pha75-6-82-226-32-84.fbx.proxad.net) Quit ()
[23:01] <jakes> yes right..
[23:01] <gregaf> but you might try the mailing list; most of the rgw users/devs are more active there than here on irc
[23:02] * KevinPerks1 (~Adium@cpe-066-026-239-136.triad.res.rr.com) has joined #ceph
[23:03] <dmick> jakes: an object is a bag of bits.
[23:03] <buck> dmick: but RBD and Ceph FS shouldn't share bags, right?
[23:03] <dmick> rbd uses several bags of bits to represent a disk. There is structure imposed by rbd/librbd such that those bags represent one disk image.
[23:04] <dmick> cephfs uses different bags of bits to represent a filesystem. It also involves its own structure on top of those bags.
[23:04] <dmick> and S3/Swift use yet different bags, and different structure.
[23:04] <dmick> same object store, but at the object store level, they're just opaque bags of bits
[23:04] <_robbat21irssi> gregaf: ok, the S3 api docs state that Object acls are supported. what i'm trying to accomplish is giving multiple subusers write permissions to specific directories within a single bucket
[23:05] * markbby (~Adium@168.94.245.2) Quit (Quit: Leaving.)
[23:05] <dmick> so no, there is *no relationship at all* between rbd images and CephFS filesystems other than that they are both stored in the RADOS cluster
[23:05] <buck> jakes: the performance question is very valid (check out that mailing list thread I mentioned). Even thought RBD and Ceph FS use different sets of objects, you'd still be managing one storage cluster and would have 100% of your throughput for both VMs and Hadoop. Seems like it would make sizing easier, but there is the big caveat that Ceph FS isn't as tested as RBD. I'd only try this with data that was reproducible / not mission critical.
[23:05] * KevinPerks (~Adium@cpe-066-026-239-136.triad.res.rr.com) Quit (Ping timeout: 480 seconds)
[23:06] <buck> jakes: but that's just another qualitative argument and we all know that the devil is in the (performance) details.
[23:07] <jakes> Dmick, I understand. But is there any mapping possible for files stored using rbd and FS?
[23:08] <dmick> I don't understand the question.
[23:11] <jakes> Dmick, I didn't understand the concept saying rbd and cephfs lie on top of same object store. what is the advantage of this?
[23:12] <jakes> buck, you mean, block storage for vms images and cephfs installed for each VM in openstack?
[23:12] <dmick> jakes: I don't think about it in terms of advantage; that's just the way it is
[23:13] <dmick> if you start with an object store, you could just use it as an object store
[23:13] <gregaf> _robbat21irssi: yeah, it supports ACLs, but I don't think there's any such thing as "directory" ACLs unless I'm confused or there's been a spec revision I didn't hear about
[23:13] <dmick> or you could invent more-complex storage utilities on top of it, which are rbd, cephfs, radosgw
[23:13] <dmick> also a key-value store
[23:13] <gregaf> directories in Swift (or any other object store) are entirely faked, so you've generally got to do permissions on the container or the object level
[23:14] <_robbat21irssi> gregaf: on sec, i'm going to dig in the spec myself
[23:14] <_robbat21irssi> as the first reference that comes up is a windows client: http://s3browser.com/share-folders-on-amazon-s3.php
[23:14] <jakes> ok. I got it.. When I read the manuals, I got a feeling that files can be interchangeably used. I felt that files stored from one can be used in the other. Two separate entities right?
[23:15] <jakes> Buck, I didn't get he point of one storage cluster?
[23:15] <buck> jakes: ah, I think I misread your earlier message
[23:15] <dmick> (02:05:04 PM) dmick: so no, there is *no relationship at all* between rbd images and CephFS filesystems other than that they are both stored in the RADOS cluster
[23:15] <gregaf> I'm trying to google and not finding any references to this in the openstack docs, but maybe I'm just on the wrong page
[23:15] <jakes> oh. fine.
[23:16] <dmick> but, as buck points out: that means you can set up one set of clustered hosts with disks as the RADOS cluster
[23:16] <buck> jakes: I was thinking you described a situation where the VMs would boot from RBD and then process data stored in Ceph FS that was running on the same nodes as RBD. Now, upon rereading it, you were suggesting running a Ceph cluster on to of the VMs.
[23:16] <dmick> and then use the same cluster for many disjoint applications
[23:16] <dmick> still managing only one cluster of hosts
[23:17] <jks> on cuttlefish getting this log message: "cephx: verify_authorizer could not get service secret for service osd secret_id=6446" - what does it mean?
[23:17] <jks> (came out of the blue on an otherwise functioning osd)
[23:18] <buck> jakes: actually, now that I think about it, you'd need to run either HDFS or Ceph FS off of centralized storage, right? Is that your plan?
[23:19] <jakes> yes :)
[23:19] <buck> jakes: either directly or via mounted block devices
[23:19] <gregaf> _robbat21irssi: interesting, I'm seeing some references to a prefix element in their forums but not the docs
[23:19] <gregaf> I'm afraid I'm going to have to pass as this is beyond what I know about the relevant APIs
[23:19] <gregaf> you should go to the mailing list ;)
[23:20] <jakes> buck, right. I just need only one distributed storage( for both VM's and for hadoop inside vm's )
[23:20] * fridudad_ (~oftc-webi@p4FC2D6A8.dip0.t-ipconnect.de) Quit (Remote host closed the connection)
[23:20] <buck> jakes: ah, ok. So my assumption wasn't wrong. You're looking to both host your VM images and the hadoop data on central storage. So my earlier comment about managing one cluster instead of 2 (block storage and HDFS) holds, along with the warnings about Ceph FS being less tested / mature (compared to RBD)
[23:20] <_robbat21irssi> gregaf: link to the doc you found?
[23:20] <gregaf> https://forums.aws.amazon.com/thread.jspa?threadID=66764, http://docs.aws.amazon.com/AmazonS3/latest/dev/ACLOverview.html
[23:21] <buck> jakes: so while you cannot share data between object storage and file storage, you would have one thing to manage instead of two, which may have benefits (sharing all the spindles / IOPs in teh cluster rather than partitioning them, etc.)
[23:21] <gregaf> _robbat21irssi: ah, actually, those only work with "IAM Policies"
[23:21] <gregaf> which I'm not certain about but I don't think the gateway implements (I've never heard of them, anyway)
[23:22] <_robbat21irssi> ugh :-(
[23:22] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) has joined #ceph
[23:22] <_robbat21irssi> so let me lay out what i'm trying to do, then maybe y'all can suggest an alternative
[23:24] <_robbat21irssi> creating a bucket for the storage backend of a site, with an internal directory structure of /${country}/${region}/${org}
[23:24] <_robbat21irssi> org names happen to be unique, and no org should be able to write the dir for another org
[23:24] <_robbat21irssi> so I had created a subuser per org
[23:24] <gregaf> any reason they can't just each get their own bucket?
[23:24] <jakes> buck , i didn't understand your architecture. How is it a single object store?.. you would be having RBD over RADOS installed on host OS. for managing VM's. Also, CEPHFs over RADOS separately on each VM. Am I wrong?
[23:24] <_robbat21irssi> gregaf: if I could put slashes in bucket names...
[23:25] <_robbat21irssi> as it's a 1:1 mapping to the site
[23:25] <jakes> How can CephFs inside guest VM talk to RADOS on the host machine?
[23:26] <gregaf> that sounds like something you fix with redirects and rewrite rules, though admittedly I know very little about web architecting
[23:26] <_robbat21irssi> site in this case is not only the website, but also some of the organizations will be mirroring to other storage systems
[23:26] <_robbat21irssi> with the same directory layout
[23:27] <gregaf> anyway, if yehudasa is around he might know more about this, otherwise you should take it to the list as I'm out of my depth
[23:27] * jcsp (~john@82-71-55-202.dsl.in-addr.zen.co.uk) has joined #ceph
[23:27] <_robbat21irssi> eg those fools that want non-ceph storage
[23:27] <buck> jakes: I believe you could have both RBD and Ceph FS using the same storage cluster, they'd just be accessing disjoint sets of objects. So you could have the VMs mounting a Ceph FS that was using pools on the host OS (but those pools would be disjoint fro mthe pools storing the VM images)
[23:27] <_robbat21irssi> (it's their money, i'm not stopping them from having 60TB of NFS)
[23:28] <buck> so the VMs would just run the cephfs client essentially (well, libcephfs, which is a different binary but is functionally very similar) and the data would be stored in objects hosted by the host OS
[23:29] <buck> jakes: now the locality issue is entirely separate, but one thing at a time
[23:31] <_robbat21irssi> gregaf: ok, a semi-related but much simplier question for now.
[23:31] <_robbat21irssi> # s3cmd -c s3-nnels.cfg setacl -P s3://download.nnels.ca/AU/WA/ABWA/tech/odt2dtbook.m4b.zip
[23:32] <_robbat21irssi> fails with a 403 to the object
[23:32] <_robbat21irssi> -P being set public acl
[23:32] <jakes> Buck, you mean to say , object store installed in host OS, while cephfs client installed in each of the guest OS ?
[23:33] <gregaf> well, that's some kind of permission issue, there should be a little more detail in the response I believe
[23:34] <dmick> jakes: the object store is a network service running on a cluster.
[23:36] <buck> jakes: pretty much, yes. It's a library and not the kernel client, but from a high level standpoint, yes.
[23:37] <buck> jakes: does that make sense? I take 'client' to mean the ceph client (possibly the kernel client) and this is a distinctly different binary but it serves a very comparable role, just in user space.
[23:37] <_robbat21irssi> gregaf: http://bpaste.net/show/113419/ <-- info on the object works, debug info doesn't have anything useful th
[23:38] <jakes> i thought client has to connect locally to object store
[23:39] <gregaf> _robbat21irssi: probably the user in question has read/write access to the object, but not to the ACL — I forget exactly what the levels are there but they're different
[23:39] <gregaf> or at least can be
[23:40] <_robbat21irssi> that seems weird, as this is the first user (not a subuser) I created for it, but i'll check
[23:40] <buck> jakes: nope. The Ceph FS client can run on an entirely separate host from the object store. In this case, it's a logically separate host that just happens to be running on the same hardware as the object store host. Hadoop is the same way; it can run on nodes that are distinct from HDFS (but that's not typically done, due to performance concerns)
[23:43] <jakes> so you mean, mount ceph fs like in http://ceph.com/docs/master/cephfs/kernel/ for each guest
[23:44] <buck> jakes: no exactly like that. Check this out: http://ceph.com/docs/master/cephfs/hadoop/
[23:45] <buck> jakes: you'd just need the ceph.conf file that pointed at the monitors / mds for the cluster and the libcephfs library installed and then the Hadoop/ceph bindings would get the data over the network from the object store hosted on the host OSs
[23:46] <buck> jakes: that being said, I have never tried this with VMs. Only on 'real' hardware. But I don't see any reason it would be different.
[23:47] * BillK (~BillK-OFT@124-148-212-240.dyn.iinet.net.au) Quit (Ping timeout: 480 seconds)
[23:47] <buck> jakes: that should have been 'not exactly like that'
[23:50] <jakes> :). Buck, I am little confused. I didn't understand. I initially thought of mounting ceph as kernel client and then use this to communicate to object store. How is this different?
[23:54] <buck> it's essentially the same via a different mechanism (libcephfs).
[23:54] <mtanski> buck: that's basically amazon hadoop (EMR)
[23:54] <buck> jakes: sorry if I contributed to the confusion. Multi-tasking ATM but wanted to help (if I could)
[23:54] <buck> mtanski: from what I know of EMR, I'd concur
[23:55] * TiCPU (~jeromepou@209.52.17.78) has joined #ceph
[23:55] <buck> mtanski: (that's just me couching due to having looked at EMR a while ago and not having used it)
[23:56] <mtanski> I have a kclient internals question. I noticed there's a rsize option for the mount point, but it looks like the kclient ignores it
[23:56] <jakes> :).. No problems. If you have time, I would just like to know the difference in their mechanisms and their performance comparisons
[23:57] <mtanski> Instead it simply makes multiple requests with the max size of rsize
[23:57] <mtanski> At least that's my understanding of addr.c ceph_readpages
[23:58] * jakes (~oftc-webi@dhcp-171-71-119-30.cisco.com) Quit (Remote host closed the connection)
[23:58] <buck> jakes: in terms of mechanisms, I think it comes down to whether the VM images are co-located with the FS data (you'd have to read up on the Amazon side to know how they do it). Performance wise, that's a giant question that I'm not equipped to answer. This (and the associated posts) may help you get a general idea of RBD performance: http://ceph.com/performance-2/ceph-cuttlefish-vs-bobtail-part-2-4k-rbd-performance/
[23:59] <_robbat21irssi> gregaf: http://blogs.aws.amazon.com/security/post/Tx1P2T3LFXXCNB5/Writing-IAM-policies-Grant-access-to-user-specific-folders-in-an-Amazon-S3-bucke <-- found that too

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.