#ceph IRC Log

Index

IRC Log for 2016-07-21

Timestamps are in GMT/BST.

[0:05] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[0:07] * ira (~ira@c-24-34-255-34.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[0:08] * `10` (~10@69.169.91.14) has joined #ceph
[0:12] * cathode (~cathode@50.232.215.114) has joined #ceph
[0:13] * gluco (~gluco@84.88.65.65) Quit (Ping timeout: 480 seconds)
[0:14] * Zyn (~ChauffeR@61TAAAQSF.tor-irc.dnsbl.oftc.net) Quit ()
[0:15] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[0:17] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) Quit (Ping timeout: 480 seconds)
[0:18] * hellertime (~Adium@pool-71-174-156-250.bstnma.east.verizon.net) has joined #ceph
[0:18] * dgurtner (~dgurtner@5.32.72.140) Quit (Ping timeout: 480 seconds)
[0:18] * w2k (~vegas3@torrelay4.tomhek.net) has joined #ceph
[0:20] * hellertime1 (~Adium@pool-71-174-154-26.bstnma.east.verizon.net) Quit (Ping timeout: 480 seconds)
[0:21] * dneary (~dneary@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[0:22] * gluco (~gluco@milhouse.imppc.org) has joined #ceph
[0:22] * vata1 (~vata@207.96.182.162) Quit (Quit: Leaving.)
[0:24] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[0:25] * sto (~sto@121.red-2-139-229.staticip.rima-tde.net) Quit (Quit: leaving)
[0:26] * sto (~sto@121.red-2-139-229.staticip.rima-tde.net) has joined #ceph
[0:31] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[0:46] * hellertime (~Adium@pool-71-174-156-250.bstnma.east.verizon.net) Quit (Quit: Leaving.)
[0:47] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) Quit (Ping timeout: 480 seconds)
[0:48] * w2k (~vegas3@5AEAAAGLE.tor-irc.dnsbl.oftc.net) Quit ()
[0:48] * Shesh (~W|ldCraze@exit.tor.uwaterloo.ca) has joined #ceph
[0:49] <rkeene> Almost done getting Ceph 10.2.2 compiled -- highly buggy build process.
[0:50] * rendar (~I@host249-177-dynamic.37-79-r.retail.telecomitalia.it) Quit (Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!)
[0:55] <nils_> rkeene, how so?
[0:56] * bene3 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[1:10] * oms101 (~oms101@p20030057EA087A00C6D987FFFE4339A1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[1:14] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[1:18] * Shesh (~W|ldCraze@5AEAAAGL6.tor-irc.dnsbl.oftc.net) Quit ()
[1:18] * kalmisto (~Salamande@87.236.194.23) has joined #ceph
[1:19] * oms101 (~oms101@p20030057EA6C2D00C6D987FFFE4339A1.dip0.t-ipconnect.de) has joined #ceph
[1:24] * johnavp1989 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) has joined #ceph
[1:24] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[1:25] * EthanL (~lamberet@cce02cs4037-fa12-z.ams.hpecore.net) has joined #ceph
[1:31] * EthanL (~lamberet@cce02cs4037-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[1:33] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[1:37] * fsimonce (~simon@host99-64-dynamic.27-79-r.retail.telecomitalia.it) Quit (Remote host closed the connection)
[1:37] * sudocat (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[1:40] * xarses (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[1:47] * borei1 (~dan@216.13.217.230) Quit (Ping timeout: 480 seconds)
[1:48] * kalmisto (~Salamande@61TAAAQUF.tor-irc.dnsbl.oftc.net) Quit ()
[1:48] * eXeler0n (~KapiteinK@relay1.tor.openinternet.io) has joined #ceph
[2:08] * jdillaman (~jdillaman@pool-108-18-97-95.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[2:18] * eXeler0n (~KapiteinK@9YSAAARHD.tor-irc.dnsbl.oftc.net) Quit ()
[2:18] * FNugget (~Kwen@atlantic850.dedicatedpanel.com) has joined #ceph
[2:19] * truan-wang (~truanwang@220.248.17.34) has joined #ceph
[2:21] * kuku (~kuku@112.203.6.241) has joined #ceph
[2:25] * axion3joey (~oftc-webi@static-108-47-170-18.lsanca.fios.frontiernet.net) has joined #ceph
[2:26] <axion3joey> Hey Everyone,
[2:27] <axion3joey> Does any know why ceph would increase writes to the disk by 20 times
[2:27] <axion3joey> our data doesn't change very much. historically we say 5-6 gigs writes per day per drive (measured using smart utils)
[2:28] <axion3joey> now with that same workload on ceph we see 130 gigs per day per disk
[2:30] <nils_> it's to be expected that it's amplified by the number of replica as well as the journal
[2:30] * LostSoul (~LostSoul@78-11-189-153.static.ip.netia.com.pl) Quit (Read error: Connection reset by peer)
[2:30] * LostSoul (~LostSoul@78-11-189-153.static.ip.netia.com.pl) has joined #ceph
[2:31] <nils_> and then you have a bit of housekeeping etc., also depends on the kinds of writes you do
[2:31] <axion3joey> thanks for the response nils
[2:31] <axion3joey> we write 3 times. we used all ssd for the cluster and the journal is on each disk
[2:32] <nils_> I'm just trying to get familiar with the filestore code myself
[2:32] <nils_> in any case, for every 1 GiB written you'll at least have an amplification of factor 6
[2:33] <axion3joey> With this load we may have caused ourselves a problem with buying prosumer grade ssd's. They're only spec'd for 80TB of writes. At this point we'll hit that in less than 2 years
[2:33] <axion3joey> wow. that's a lot
[2:33] <axion3joey> any way to trim that down?
[2:33] <axion3joey> there's a big price difference between Sandisk SSD's and Intels
[2:33] <nils_> well you have 3 replica + everything is written to the journal and then to the disk
[2:34] <nils_> yeah we really got burned with Samsung
[2:34] <nils_> but using spinning rust for the data and SSD journals
[2:34] <axion3joey> i wonder if there's a way to disable to journal since we're writing to ssd's anyways
[2:34] <nils_> it's possible, but your data won't be sae
[2:35] <nils_> safe
[2:35] <nils_> in fact I don't think it's safe on prosumer SSD since they tend to not honor flushes/syncs
[2:35] <axion3joey> got it
[2:35] <nils_> what did you set filestore max sync interval and filestore min sync interval to?
[2:36] <axion3joey> filestore max sync interval = 30
[2:36] <nils_> I think the Samsung crapped out after 30 TBW
[2:36] <axion3joey> filestore min sync interval = 29
[2:36] <nils_> I see
[2:36] <axion3joey> filestore flusher = false
[2:36] <axion3joey> filestore queue max ops = 10000
[2:36] <nils_> using XFS?
[2:37] <axion3joey> i read somewhere that those were recommended settings
[2:37] <nils_> I also need the threads
[2:37] <nils_> there isn't really a lot of good documentation on it
[2:37] <nils_> it very much depends on the hardware so what one would need is sort of a calculator
[2:37] <axion3joey> what do you mean by threads?
[2:38] <axion3joey> and yes we're using xfs
[2:38] <nils_> filestore op threads
[2:38] <axion3joey> I don't have that in my config
[2:38] <axion3joey> googling
[2:38] <nils_> likely the default then
[2:39] <axion3joey> yeah. I'm going to grab the runtime value
[2:39] <nils_> xfs separates your disk into allocation groups and it sort of spreads the data out. In case the alignment is of this may lead to a lot of erase blocks being written and rewritten
[2:39] <axion3joey> got it.
[2:40] <axion3joey> what would you do?
[2:40] <nils_> but you can already see that this is very complicated because we have to go through the full stack, from cephfs (if you use it) to rados to osd
[2:41] <axion3joey> I also see that the default osd journal size is 0
[2:41] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[2:41] <axion3joey> and I don't have that specified in my config
[2:42] * kuku (~kuku@112.203.6.241) has joined #ceph
[2:43] <nils_> that is odd
[2:43] <nils_> how did you create your ceph setup? ceph-deploy?
[2:43] <axion3joey> ceph-deploy
[2:44] <axion3joey> I'm going to try changing my min sync interval to .1
[2:44] <nils_> yeah the default is 0.
[2:44] <axion3joey> and setting my journal size to 10gigs
[2:44] <axion3joey> that's what the docs seem to recommend
[2:44] <nils_> can you log into an OSD and check the journal
[2:44] <nils_> yeah I'm reading through the filestore code to figure it out
[2:45] <axion3joey> logged in now
[2:45] <axion3joey> how do I check the journal?
[2:45] <nils_> if it's even there
[2:45] * dnunez (~dnunez@c-73-38-0-185.hsd1.ma.comcast.net) has joined #ceph
[2:45] <nils_> there should be a bunch of devices mounted
[2:45] <nils_> /var/lib/ceph/osd/ceph-?
[2:45] <nils_> there should be a file journal there
[2:45] <axion3joey> I don't see them in mount
[2:46] <axion3joey> i see them though if I do a ceph-deploy disk list
[2:46] <nils_> can you pastebin that?
[2:47] <axion3joey> on it
[2:47] <nils_> I actually never used ceph deploy I always used the low level stuff
[2:48] <nils_> usually it would create two partitions, the first one for the journal the second one for the data
[2:48] <nils_> so I would suggest you set a journal size
[2:48] <axion3joey> here's the mount data
[2:48] <axion3joey> http://pastebin.com/MQpmzkD1
[2:48] <nils_> and re-create the osd
[2:48] * FNugget (~Kwen@26XAAAHH9.tor-irc.dnsbl.oftc.net) Quit ()
[2:48] * Linkshot (~Plesioth@212.7.192.148) has joined #ceph
[2:49] <nils_> ls -l /var/lib/ceph/osd/ceph-5/journal
[2:49] <axion3joey> lrwxrwxrwx 1 ceph ceph 58 Apr 16 22:12 /var/lib/ceph/osd/ceph-5/journal -> /dev/disk/by-partuuid/bb964a29-cf33-44bd-86ee-d70816ff6888
[2:49] <nils_> readlink -f /var/lib/ceph/osd/ceph-5/journal
[2:50] <nils_> man this IRC based shell has a lot of latency ;)
[2:50] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[2:51] <axion3joey> here's the ceph-deploy disk list output
[2:51] <axion3joey> http://pastebin.com/9bvAdWyw
[2:52] <nils_> don't you just hate it that the mdadm devices get renamed to md{127..125}
[2:52] <axion3joey> yup
[2:52] <axion3joey> almost as bad as when nics decide to change assignments after a reboot
[2:52] <axion3joey> readlink -f /var/lib/ceph/osd/ceph-5/journal
[2:52] <nils_> yeah even though they have persistent naming
[2:52] <nils_> there's your journal disk
[2:52] <nils_> looks like you actually have a journal
[2:53] <nils_> I have two systems running Ubuntu 16.04, one system names the network card ens2 the other one names it enos1p4 or something like that
[2:53] <axion3joey> yup
[2:53] <nils_> can you check with gdisk /dev/sdb
[2:53] <nils_> just do a "p" to see the journal disk size
[2:54] <axion3joey> and dell's have different name based on the nic
[2:54] <axion3joey> this was the output of readlink. /dev/sdg2
[2:54] <nils_> never really bought Dell, I can only afford Supermicro ;)
[2:55] <axion3joey> dude. used Dells are the way to go
[2:55] <axion3joey> you can get them so cheap
[2:56] <axion3joey> looks like my journals are all 5gigs
[2:56] <nils_> I'm in Germany, electricity is so expensive here that running inefficient cpus isn't cost effective. Basically you're just deferring payment
[2:56] <nils_> what are you using your ceph cluster for?
[2:56] <axion3joey> that makes sense
[2:57] <axion3joey> we're a small voip company
[2:57] <nils_> I really like the Xeon-D systems
[2:57] <axion3joey> we use ceph for vm storage for all of our customers
[2:57] <nils_> alright so you're using rbd
[2:57] <axion3joey> yup
[2:57] <axion3joey> what do you use ceph for?
[2:57] <nils_> same
[2:57] <axion3joey> seriously?
[2:57] <nils_> well it's a pretty common use case
[2:58] <axion3joey> true
[2:58] <nils_> running libvirt and KVM
[2:58] <axion3joey> got it
[2:58] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:58] <axion3joey> we use lxc
[2:58] <nils_> are you using the kernel rbd driver (rbd map)?
[2:58] * hellertime (~Adium@pool-71-174-156-250.bstnma.east.verizon.net) has joined #ceph
[2:59] <axion3joey> checking
[2:59] <axion3joey> that was setup through the hypervisor (Proxmox)
[2:59] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[2:59] <nils_> I see
[3:00] <nils_> checking the mounts on the hypervisor should tell us more
[3:00] <nils_> also it depends on how much data you have on your cluster, how many writes yu do etc.
[3:01] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[3:01] <nils_> or just running rbd map
[3:01] <axion3joey> logging in there now?
[3:02] * debian112 (~bcolbert@64.235.157.198) Quit (Quit: Leaving.)
[3:04] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) has joined #ceph
[3:05] <nils_> you have to keep in mind that ceph will have to do quite a bit of housekeeping as well.
[3:05] * BrianA (~BrianA@fw-rw.shutterfly.com) Quit (Read error: Connection reset by peer)
[3:06] <axion3joey> the docs contradict themselves
[3:06] <nils_> also I'd like to know the object size (parameter is called order)
[3:06] <nils_> yeah the docs need some serious work
[3:06] <nils_> actually one of the reasons I'm digging into the source
[3:06] <axion3joey> in the beginning of the journal settings section it says the journal size is set to 0
[3:07] <axion3joey> later on the in the page it says it's 5120
[3:07] <nils_> yeah that threw me as well
[3:07] <axion3joey> I confirmed that it's 5120 in my live cluster
[3:07] <nils_> the docs aren't versioned
[3:07] * kuku (~kuku@112.203.6.241) has joined #ceph
[3:09] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) Quit (Quit: Leaving)
[3:09] * yanzheng (~zhyan@125.70.23.222) has joined #ceph
[3:09] * hellertime (~Adium@pool-71-174-156-250.bstnma.east.verizon.net) Quit (Quit: Leaving.)
[3:15] * yanzheng1 (~zhyan@125.70.23.222) has joined #ceph
[3:16] * Jeffrey4l (~Jeffrey@110.252.43.15) has joined #ceph
[3:16] * Racpatel (~Racpatel@2601:87:0:24af::1fbc) Quit (Quit: Leaving)
[3:18] * Linkshot (~Plesioth@26XAAAHI0.tor-irc.dnsbl.oftc.net) Quit ()
[3:19] * yanzheng (~zhyan@125.70.23.222) Quit (Ping timeout: 480 seconds)
[3:23] * Ralth (~clusterfu@198.50.200.137) has joined #ceph
[3:25] * debian112 (~bcolbert@173-164-167-194-SFBA.hfc.comcastbusiness.net) has joined #ceph
[3:29] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[3:34] * LostSoul_ (~LostSoul@78-11-189-153.static.ip.netia.com.pl) has joined #ceph
[3:36] * LostSoul (~LostSoul@78-11-189-153.static.ip.netia.com.pl) Quit (Ping timeout: 480 seconds)
[3:43] * Vacuum__ (~Vacuum@88.130.201.24) has joined #ceph
[3:50] * Vacuum_ (~Vacuum@88.130.195.35) Quit (Ping timeout: 480 seconds)
[3:53] * Ralth (~clusterfu@5AEAAAGO8.tor-irc.dnsbl.oftc.net) Quit ()
[3:53] * CobraKhan007 (~DougalJac@114-227-47-212.rev.cloud.scaleway.com) has joined #ceph
[4:00] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[4:01] * cyphase (~cyphase@2601:640:c401:969a:468a:5bff:fe29:b5fd) has joined #ceph
[4:04] * kefu (~kefu@183.193.182.196) has joined #ceph
[4:13] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[4:16] * kefu_ (~kefu@114.92.96.253) has joined #ceph
[4:22] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Ping timeout: 480 seconds)
[4:23] * CobraKhan007 (~DougalJac@5AEAAAGP1.tor-irc.dnsbl.oftc.net) Quit ()
[4:23] * neobenedict (~n0x1d@5.135.65.145) has joined #ceph
[4:24] * kefu (~kefu@183.193.182.196) Quit (Ping timeout: 480 seconds)
[4:25] * EinstCra_ (~EinstCraz@58.247.119.250) has joined #ceph
[4:27] * dnunez (~dnunez@c-73-38-0-185.hsd1.ma.comcast.net) Quit (Quit: Leaving)
[4:32] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[4:32] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[4:33] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[4:35] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[4:38] * Jeffrey4l_ (~Jeffrey@110.252.56.47) has joined #ceph
[4:39] <rkeene> nils_, 1. Unchecked dependency on curl (configure script doesn't check for it, but it won't build without it); 2. Configure script fails with --without-radosgw because it looks for LDAP headers, which are only used by RADOSGW; 3. Build process fails if you don't enable radosgw (linking failure); 4. FCGI upstream no longer exists, but is a dependency; 5. Tries to use pip/virtualenv for dumb reasons, pip especially making reproducible builds difficult
[4:39] <rkeene> nils_, Off the top of my head
[4:39] * HappyLoaf (~HappyLoaf@cpc93928-bolt16-2-0-cust133.10-3.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[4:40] <nils_> rkeene, wow
[4:40] <nils_> rkeene, I had my trouble with building the stable packages
[4:40] * HappyLoaf (~HappyLoaf@cpc93928-bolt16-2-0-cust133.10-3.cable.virginm.net) has joined #ceph
[4:40] * Jeffrey4l (~Jeffrey@110.252.43.15) Quit (Read error: Connection reset by peer)
[4:42] <rkeene> This was 10.2.2
[4:42] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[4:42] * kuku (~kuku@112.203.6.241) has joined #ceph
[4:44] <nils_> rkeene, never quite felt comfortable with the whole autoconf/automake ecosystem
[4:44] * kuku (~kuku@112.203.6.241) Quit (Read error: Connection reset by peer)
[4:45] * kuku (~kuku@112.203.6.241) has joined #ceph
[4:45] <rkeene> automake is terrible, autoconf is good -- it would take a lot of work to build something better
[4:45] <nils_> yeah
[4:46] <rkeene> So far I haven't found anything better than autoconf -- cmake, autosetup, xmkmf, imake, smake, etc haven't done it for me
[4:48] <rkeene> And it would take a really smart system to automatically deal with the above problems -- most of them are the fact that optional/non-optional dependencies are determined manually... but making a system smart enough to work all that out would be extremely tedious
[4:48] <nils_> yeah it's an age old problem
[4:49] <rkeene> I now have a Ceph 10.2.2 package -- it's 250MB, but I have it.
[4:49] <rkeene> Autobuilders are running through tests and we'll see how it fares tomorrow
[4:50] <rkeene> Has to rebuild my entire Linux distribution and setup several clusters to test all kinds of things, like multiple monitors, single monitors, upgrading from older releases
[4:50] <rkeene> Tests alone take like 3 hours now, compiling the whole distribution takes another 3 on the build box
[4:51] <nils_> I guess for most developers it's tedious to make something work for a system they are not comfortable running
[4:51] <nils_> what distribution are you building?
[4:51] <rkeene> It's my own distribution, the company I work for sells Ceph appliances
[4:51] <nils_> oh I see
[4:51] <nils_> is it based on anything one would know?
[4:51] <rkeene> Runs from initramfs, atomic upgrades, etc
[4:51] <nils_> I think you already told me about it once
[4:52] <rkeene> The pkginfo files are similar to Arch, they are used to create a Makefile that has a global dependency tree...
[4:52] <rkeene> It's all cross-compiled all the time
[4:52] <rkeene> It's not really based on anything
[4:52] <nils_> are you building for multiple architectures?
[4:52] <rkeene> Just Linux/x86_64 right now
[4:53] <nils_> I would like to have a statically linked binary for ceph-mon and ceph-osd so I can pluck them into a very streamlined container
[4:53] <rkeene> But I build it on a variety of systems to ensure that it continues to work
[4:53] * neobenedict (~n0x1d@5AEAAAGQT.tor-irc.dnsbl.oftc.net) Quit ()
[4:53] * luckz (~Unforgive@Relay-J.tor-exit.network) has joined #ceph
[4:53] <rkeene> I build on Linux/x86_64 and Linux/arm and almost on Mac OS X/x86_64
[4:54] <rkeene> It would be easy to make a statically linked ceph-mon, ceph-osd, etc -- it'd just be big :-D
[4:54] <nils_> considering LTO
[4:54] <nils_> how large a binary would you think?
[4:55] <rkeene> 100MB or so, probably -- I could test it out
[4:55] <rkeene> 100MB each, that is
[4:57] <nils_> that's not too bad.
[4:57] <rkeene> Building Ceph now
[4:57] <rkeene> Added -static to ceph-mon and ceph-osd, so we'll see
[4:57] <nils_> thanks
[4:57] <rkeene> I'm going to bed, so it'll be morning before we kno
[4:57] <rkeene> know, rather
[4:58] <rkeene> Ceph takes like 45 minutes to compile on my laptop
[4:59] <nils_> EUR 0.50 for 100MiB of RAM
[5:00] <rkeene> That's size on disk, nothing to do with RAM (unless you're talking about it running from initramfs)
[5:01] <nils_> that is actually something I would consider
[5:01] <rkeene> My whole distribution runs from initramfs, which is why I don't statically link anything
[5:02] <rkeene> Especially Ceph, which is by far the largest package
[5:03] * kefu_ (~kefu@114.92.96.253) Quit (Quit: Textual IRC Client: www.textualapp.com)
[5:08] <nils_> makes sense
[5:11] * analbeard (~shw@host109-149-32-128.range109-149.btcentralplus.com) has joined #ceph
[5:12] * kefu (~kefu@183.193.182.196) has joined #ceph
[5:12] * analbeard (~shw@host109-149-32-128.range109-149.btcentralplus.com) Quit ()
[5:14] <nils_> what a great nickname that was...
[5:14] <nils_> I'm waiting for a mainboard for my xeon CPU, then I can actually do builds on 14 cores
[5:15] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) Quit (Ping timeout: 480 seconds)
[5:16] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[5:17] * Vacuum_ (~Vacuum@88.130.221.118) has joined #ceph
[5:20] * EinstCra_ (~EinstCraz@58.247.119.250) Quit (Ping timeout: 480 seconds)
[5:21] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) Quit (Quit: Leaving.)
[5:23] * luckz (~Unforgive@5AEAAAGRI.tor-irc.dnsbl.oftc.net) Quit ()
[5:23] * ZombieTree (~Fapiko@torland1-this.is.a.tor.exit.server.torland.is) has joined #ceph
[5:24] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[5:24] * Vacuum__ (~Vacuum@88.130.201.24) Quit (Ping timeout: 480 seconds)
[5:25] * kefu_ (~kefu@114.92.96.253) has joined #ceph
[5:32] * kefu (~kefu@183.193.182.196) Quit (Ping timeout: 480 seconds)
[5:39] * karnan (~karnan@106.51.137.148) has joined #ceph
[5:43] * vimal (~vikumar@114.143.165.70) has joined #ceph
[5:45] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[5:49] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[5:50] * kefu_ is now known as kefu
[5:53] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) has joined #ceph
[5:53] * ZombieTree (~Fapiko@26XAAAHMF.tor-irc.dnsbl.oftc.net) Quit ()
[5:53] * skrblr (~ghostnote@cry.ip-eend.nl) has joined #ceph
[5:53] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[5:56] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[6:01] * jamespage (~jamespage@culvain.gromper.net) Quit (Read error: No route to host)
[6:01] * jamespage (~jamespage@culvain.gromper.net) has joined #ceph
[6:01] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[6:02] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[6:08] * truan-wang (~truanwang@220.248.17.34) Quit (Ping timeout: 480 seconds)
[6:15] * rdas (~rdas@121.244.87.116) has joined #ceph
[6:22] * vimal (~vikumar@114.143.165.70) Quit (Quit: Leaving)
[6:23] * skrblr (~ghostnote@5AEAAAGSO.tor-irc.dnsbl.oftc.net) Quit ()
[6:23] * Revo84 (~Szernex@172.98.67.128) has joined #ceph
[6:27] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[6:32] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) Quit (Ping timeout: 480 seconds)
[6:35] * truan-wang (~truanwang@220.248.17.34) has joined #ceph
[6:36] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[6:37] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) has joined #ceph
[6:41] * vimal (~vikumar@121.244.87.116) has joined #ceph
[6:42] * vikhyat (~vumrao@121.244.87.116) has joined #ceph
[6:45] * Blfrg (~Blfrg@186.4.22.14) has joined #ceph
[6:45] * Blfrg (~Blfrg@186.4.22.14) Quit (Remote host closed the connection)
[6:51] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[6:53] * Revo84 (~Szernex@172.98.67.128) Quit ()
[6:53] * sese_ (~pakman__@178.32.53.131) has joined #ceph
[6:54] * truan-wang (~truanwang@220.248.17.34) Quit (Ping timeout: 480 seconds)
[6:55] * reed (~reed@184-23-0-196.dsl.static.fusionbroadband.com) Quit (Quit: Ex-Chat)
[7:02] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[7:07] * rotbeard (~redbeard@2a02:908:df13:bb00:953f:d96a:d234:916f) has joined #ceph
[7:08] * truan-wang (~truanwang@58.247.8.186) has joined #ceph
[7:08] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Remote host closed the connection)
[7:09] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[7:19] * brians (~brian@80.111.114.175) has joined #ceph
[7:23] * sese_ (~pakman__@9YSAAARON.tor-irc.dnsbl.oftc.net) Quit ()
[7:26] * onyb (~ani07nov@27.34.248.72) has joined #ceph
[7:29] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Read error: Connection reset by peer)
[7:29] * EinstCrazy (~EinstCraz@58.247.119.250) has joined #ceph
[7:43] * EinstCrazy (~EinstCraz@58.247.119.250) Quit (Remote host closed the connection)
[7:57] * Revo84 (~rapedex@Relay-J.tor-exit.network) has joined #ceph
[7:59] * swami1 (~swami@49.38.2.251) has joined #ceph
[8:05] * post-factum (~post-fact@vulcan.natalenko.name) Quit (Killed (NickServ (Too many failed password attempts.)))
[8:05] * post-factum (~post-fact@vulcan.natalenko.name) has joined #ceph
[8:14] * hroussea (~hroussea@000200d7.user.oftc.net) has joined #ceph
[8:24] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[8:27] * Revo84 (~rapedex@5AEAAAGUP.tor-irc.dnsbl.oftc.net) Quit ()
[8:27] * raindog (~ZombieL@torland1-this.is.a.tor.exit.server.torland.is) has joined #ceph
[8:29] * dugravot6 (~dugravot6@l-p-dn-in-4a.lionnois.site.univ-lorraine.fr) has joined #ceph
[8:32] * hroussea (~hroussea@000200d7.user.oftc.net) Quit (Quit: Client exited.)
[8:34] * hroussea (~hroussea@goyavier-10ge-vl69-4-2.par01.moulticast.net) has joined #ceph
[8:39] * isti (~isti@BC06E559.dsl.pool.telekom.hu) has joined #ceph
[8:45] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[8:51] * derjohn_mob (~aj@p578b6aa1.dip0.t-ipconnect.de) Quit (Ping timeout: 480 seconds)
[8:57] * raindog (~ZombieL@61TAAAQ4F.tor-irc.dnsbl.oftc.net) Quit ()
[8:58] * branto (~branto@ip-78-102-208-181.net.upcbroadband.cz) has joined #ceph
[9:04] * isti (~isti@BC06E559.dsl.pool.telekom.hu) Quit (Remote host closed the connection)
[9:13] * briner (~briner@129.194.16.54) has joined #ceph
[9:17] * EthanL (~lamberet@cce02cs4037-fa12-z.ams.hpecore.net) has joined #ceph
[9:18] * dgurtner (~dgurtner@209.132.186.254) has joined #ceph
[9:21] * art_yo (~kvirc@149.126.169.197) has joined #ceph
[9:21] <art_yo> Hi guys. Could you help me? I can not make a fs on ceph.
[9:22] <art_yo> mkfs.ext4 -m0 -E nodiscard
[9:22] <art_yo> and it stops on "Writing inoda tables - 1200/xxxxxxxx"
[9:23] <art_yo> I see io activity on ceph mon untill mkfs.ext4 stops
[9:23] <art_yo> What am I supposed to check? Is it problem with rbd or with ceph in general?
[9:25] <sep> ceph -s ; ceph health detail , do you see any io ? or andy slow requests that are stuck
[9:26] * EthanL (~lamberet@cce02cs4037-fa12-z.ams.hpecore.net) Quit (Ping timeout: 480 seconds)
[9:26] * rendar (~I@host79-131-dynamic.59-82-r.retail.telecomitalia.it) has joined #ceph
[9:26] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[9:32] * kuku (~kuku@112.203.6.241) has joined #ceph
[9:40] <art_yo> Yes, I see io
[9:41] <art_yo> But untill mkfs stops on 1297 indoe
[9:41] <art_yo> inode
[9:41] * onyb (~ani07nov@27.34.248.72) Quit (Quit: raise SystemExit())
[9:43] * kuku (~kuku@112.203.6.241) Quit (Remote host closed the connection)
[9:44] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) has joined #ceph
[9:48] <art_yo> dmesg shows a lot of messages: libceph: osd6 192.168.127.11:6808 socket closed (con state OPEN)
[9:48] * DanFoster (~Daniel@office.34sp.com) has joined #ceph
[9:53] * garphy`aw is now known as garphy
[9:54] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[9:57] <art_yo> Perhaps this problem is related to OSD fullness
[9:58] <art_yo> ceph -w shows: 2016-07-21 11:01:19.165607 osd.4 [ERR] OSD full dropping all updates 97% full
[10:14] * jarrpa (~jarrpa@2602:43:489:2100:eab1:fcff:fe47:f680) Quit (Ping timeout: 480 seconds)
[10:15] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[10:23] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[10:23] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) has joined #ceph
[10:24] * kefu (~kefu@114.92.96.253) has joined #ceph
[10:26] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[10:26] * InIMoeK (~InIMoeK@95.170.93.16) has joined #ceph
[10:27] * kefu (~kefu@114.92.96.253) has joined #ceph
[10:27] * vend3r (~pakman__@176.10.99.207) has joined #ceph
[10:29] * rraja (~rraja@121.244.87.117) has joined #ceph
[10:30] * zhen (~Thunderbi@192.94.118.200) has joined #ceph
[10:31] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[10:31] * zhen (~Thunderbi@192.94.118.200) Quit ()
[10:32] * kefu (~kefu@114.92.96.253) has joined #ceph
[10:33] * owasserm (~owasserm@nat-pool-ams-t.redhat.com) has joined #ceph
[10:33] * fsimonce (~simon@host99-64-dynamic.27-79-r.retail.telecomitalia.it) has joined #ceph
[10:41] * TMM (~hp@185.5.121.201) has joined #ceph
[10:42] * truan-wang (~truanwang@58.247.8.186) Quit (Ping timeout: 480 seconds)
[10:44] <art_yo> I decided to create RBD on another server
[10:48] * penguinRaider (~KiKo@103.6.219.219) Quit (Ping timeout: 480 seconds)
[10:57] * vend3r (~pakman__@61TAAAQ64.tor-irc.dnsbl.oftc.net) Quit ()
[10:57] * DJComet (~Arcturus@tsn109-201-154-201.dyn.nltelcom.net) has joined #ceph
[11:02] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Remote host closed the connection)
[11:04] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[11:04] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[11:11] * r0lland (~r0lland@121.244.155.12) has joined #ceph
[11:13] * stiopa (~stiopa@cpc73832-dals21-2-0-cust453.20-2.cable.virginm.net) has joined #ceph
[11:16] <mistur> Hello
[11:16] <mistur> on an infernalis cluster I want to give access to a user to multiple pool with a namespace
[11:17] <mistur> should I do something like :
[11:17] <mistur> ceph auth caps client.user1 mon 'allow r' osd 'allow rw pool=pool1 namespace=space rw pool=pool2 namespace=space pool=pool3 namespace=space'
[11:17] <mistur> or just
[11:18] * willi (~willi@2a00:1050:4:0:4c9d:69eb:45dd:e22b) has joined #ceph
[11:18] <willi> hey guys
[11:18] <mistur> ceph auth caps client.user1 mon 'allow r' osd 'allow rw pool=pool1 rw pool=pool2 rw pool=pool3 namespace=space'
[11:18] <willi> Is anyone of you out there who has an Intel P3700 for Journal an can give me back test results with:
[11:18] <willi> rados bench -p rbd 60 write -b 4M -t 1
[11:20] * krogon_ (~krogon@134.134.137.75) has joined #ceph
[11:20] <boolman> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
[11:20] <boolman> willi:
[11:20] <willi> i know
[11:20] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[11:20] <willi> but he uses only a fio test
[11:20] <willi> i want a rados
[11:22] * krogon (~krogon@irdmzpr02-ext.ir.intel.com) Quit (Remote host closed the connection)
[11:22] * bara (~bara@213.175.37.12) has joined #ceph
[11:27] * DJComet (~Arcturus@tsn109-201-154-201.dyn.nltelcom.net) Quit ()
[11:27] * ggarg (~ggarg@nat.nue.novell.com) Quit (Read error: Connection reset by peer)
[11:30] <mistur> what is the syntax to give access to multiple pool for 1 user ?
[11:34] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[11:38] * penguinRaider (~KiKo@103.6.219.219) Quit (Remote host closed the connection)
[11:42] <sep> art_yo, 97% full ? that's quite a lot more then the full warning you get at 85%....
[11:43] <art_yo> what do you mean?
[11:44] <sep> do you not get a warning in ceph -s oabout osd.4 beeing near-full when it's above 85% ?
[11:45] <sep> so you know you need to start adding more disk. the free space on disks is basically yout failure domain. since that's where data go when a disk dies.
[11:47] <art_yo> I just didn't look at it
[11:51] <art_yo> I am ready to remove all data (it's just a test server). How am I supposed to do it?
[11:52] * penguinRaider (~KiKo@103.6.219.219) has joined #ceph
[11:53] * sebastian-w (~quassel@212.218.8.138) Quit (Remote host closed the connection)
[11:53] * sebastian-w (~quassel@212.218.8.138) has joined #ceph
[11:56] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) Quit (Remote host closed the connection)
[11:57] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) has joined #ceph
[11:57] * ZombieTree (~tuhnis@politkovskaja.torservers.net) has joined #ceph
[11:58] * F|1nt (~F|1nt@host37-212.lan-isdn.imaginet.fr) Quit ()
[12:03] * kefu (~kefu@114.92.96.253) Quit (Max SendQ exceeded)
[12:04] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[12:15] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[12:16] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[12:17] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[12:18] <chrome0> hey, i've got a pretty busy ceph cluster and need to replace a node. the osd's are all down. if i rm the osds from crush there will be a lot of rebalancing going on which i'd like to avoid, how do people do that usually?
[12:25] * untoreh (~fra@151.45.246.178) has joined #ceph
[12:26] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[12:26] * ngoswami_ (~ngoswami@121.244.87.116) has joined #ceph
[12:27] * ZombieTree (~tuhnis@9YSAAARUK.tor-irc.dnsbl.oftc.net) Quit ()
[12:27] * tuhnis (~Kyso@61TAAAQ9L.tor-irc.dnsbl.oftc.net) has joined #ceph
[12:29] <untoreh> this is not nice http://i.imgur.com/sDEuz6f.png , rude.
[12:33] * rraja (~rraja@121.244.87.117) has joined #ceph
[12:36] * bara (~bara@213.175.37.12) Quit (Ping timeout: 480 seconds)
[12:40] * Kurt (~Adium@2001:628:1:5:2439:619d:6184:b145) Quit (Quit: Leaving.)
[12:43] * whatevsz (~quassel@185.22.140.59) Quit (Ping timeout: 480 seconds)
[12:45] * yanzheng (~zhyan@118.116.114.34) has joined #ceph
[12:46] * bara (~bara@nat-pool-brq-t.redhat.com) has joined #ceph
[12:47] * yanzheng1 (~zhyan@125.70.23.222) Quit (Ping timeout: 480 seconds)
[12:56] * chopmann (~sirmonkey@2a02:8108:8b40:5600:afc9:f9c5:160c:f885) has joined #ceph
[12:57] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[12:57] * tuhnis (~Kyso@61TAAAQ9L.tor-irc.dnsbl.oftc.net) Quit ()
[13:05] * willi (~willi@2a00:1050:4:0:4c9d:69eb:45dd:e22b) Quit ()
[13:05] * b0e (~aledermue@213.95.25.82) has joined #ceph
[13:06] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[13:07] <MrBy> chrome0: norecover/nobackfill ?
[13:10] <MrBy> chrome0: ceph osd set --help
[13:11] <chrome0> MrBy : hm, i see those mentioned but not much else :-) any context what those do?
[13:13] * ngoswami__ (~ngoswami@121.244.87.116) has joined #ceph
[13:13] * ngoswami_ (~ngoswami@121.244.87.116) Quit (Read error: Connection reset by peer)
[13:13] <MrBy> chrome0: afaik the cluster does not any recover nor backfill, so you can replace the failed osd, once you integrate it only the new osd shall be backfilled (assumption: you removed the norecover flag)
[13:13] * ngoswami__ (~ngoswami@121.244.87.116) Quit ()
[13:14] <chrome0> ok
[13:14] <chrome0> cheers MrBy
[13:14] * ngoswami_ (~ngoswami@121.244.87.116) has joined #ceph
[13:14] <MrBy> chrome0: do you have a test cluster where you can try it in advance
[13:15] <chrome0> MrBy : yes, luckily
[13:15] <MrBy> chrome0: ceph osd set noout would also be an option. You can google those options to get a closer idea what they do
[13:16] <chrome0> right, using those
[13:18] <MrBy> chrome0: https://access.redhat.com/documentation/en/red-hat-ceph-storage/version-1.2.3/red-hat-ceph-storage-123-red-hat-ceph-administration-guide/#setting_unsetting_overrides here is a little explanation. could not find them on the ceph website, but there are almost self explaining imho
[13:18] <chrome0> kk
[13:21] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[13:25] * ngoswami__ (~ngoswami@121.244.87.116) has joined #ceph
[13:25] * ngoswami_ (~ngoswami@121.244.87.116) Quit (Read error: Connection reset by peer)
[13:26] * T1 (~the_one@5.186.54.143) Quit (Ping timeout: 480 seconds)
[13:26] * bniver (~bniver@pool-173-48-58-27.bstnma.fios.verizon.net) Quit (Remote host closed the connection)
[13:27] * yanzheng (~zhyan@118.116.114.34) Quit (Ping timeout: 480 seconds)
[13:27] * ade (~abradshaw@tmo-080-125.customers.d1-online.com) has joined #ceph
[13:28] * ade (~abradshaw@tmo-080-125.customers.d1-online.com) Quit ()
[13:28] * ade (~abradshaw@tmo-080-125.customers.d1-online.com) has joined #ceph
[13:31] * art_yo|3 (~kvirc@149.126.169.140) has joined #ceph
[13:32] * b0e (~aledermue@213.95.25.82) Quit (Ping timeout: 480 seconds)
[13:33] * yanzheng (~zhyan@118.116.114.34) has joined #ceph
[13:38] * art_yo (~kvirc@149.126.169.197) Quit (Ping timeout: 480 seconds)
[13:40] * jarrpa (~jarrpa@2602:3f:e183:a600:eab1:fcff:fe47:f680) has joined #ceph
[13:40] * Hemanth (~hkumar_@121.244.87.117) has joined #ceph
[13:42] * art_yo (~kvirc@149.126.169.197) has joined #ceph
[13:42] * art_yo|3 (~kvirc@149.126.169.140) Quit (Ping timeout: 480 seconds)
[13:45] * b0e (~aledermue@213.95.25.82) has joined #ceph
[13:49] * ade (~abradshaw@tmo-080-125.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[13:49] * T1 (~the_one@5.186.54.143) has joined #ceph
[13:51] * bniver (~bniver@71-9-144-29.static.oxfr.ma.charter.com) has joined #ceph
[13:57] * cryptk (~shishi@chulak.enn.lu) has joined #ceph
[14:00] * Racpatel (~Racpatel@2601:87:0:24af::1fbc) has joined #ceph
[14:03] * willi (~willi@p200300774E328EFCADBA1E08CECDF3CB.dip0.t-ipconnect.de) has joined #ceph
[14:03] * willi (~willi@p200300774E328EFCADBA1E08CECDF3CB.dip0.t-ipconnect.de) Quit ()
[14:07] * dgurtner (~dgurtner@209.132.186.254) Quit (Ping timeout: 480 seconds)
[14:17] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) has joined #ceph
[14:18] * i_m (~ivan.miro@deibp9eh1--blueice1n0.emea.ibm.com) has joined #ceph
[14:21] * badone (~badone@66.187.239.16) Quit (Remote host closed the connection)
[14:22] * jarrpa (~jarrpa@2602:3f:e183:a600:eab1:fcff:fe47:f680) Quit (Ping timeout: 480 seconds)
[14:27] * cryptk (~shishi@26XAAAHV0.tor-irc.dnsbl.oftc.net) Quit ()
[14:31] * jarrpa (~jarrpa@63.225.131.166) has joined #ceph
[14:39] * vimal (~vikumar@121.244.87.116) Quit (Quit: Leaving)
[14:41] * ira (~ira@nat-pool-bos-t.redhat.com) has joined #ceph
[14:46] * mattbenjamin (~mbenjamin@12.118.3.106) has joined #ceph
[14:46] * penguinRaider (~KiKo@103.6.219.219) Quit (Read error: Connection reset by peer)
[14:46] * ngoswami__ (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[14:46] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Coyote finally caught me)
[14:47] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[14:47] * johnavp1989 (~jpetrini@pool-100-14-10-2.phlapa.fios.verizon.net) Quit (Ping timeout: 480 seconds)
[14:48] * dneary (~dneary@nat-pool-bos-u.redhat.com) has joined #ceph
[14:57] * CobraKhan007 (~Jamana@atlantic850.dedicatedpanel.com) has joined #ceph
[14:58] * ira (~ira@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[14:58] * nils_ (~nils_@doomstreet.collins.kg) Quit (Quit: This computer has gone to sleep)
[14:59] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Coyote finally caught me)
[14:59] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[15:00] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[15:02] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[15:03] * penguinRaider (~KiKo@23.94.218.244) has joined #ceph
[15:06] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:08] * neurodrone_ (~neurodron@pool-100-35-225-168.nwrknj.fios.verizon.net) Quit (Quit: neurodrone_)
[15:12] * derjohn_mob (~aj@fw.gkh-setu.de) has joined #ceph
[15:15] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) Quit (Quit: Ex-Chat)
[15:15] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[15:17] * art_yo (~kvirc@149.126.169.197) Quit (Ping timeout: 480 seconds)
[15:20] * rmart04 (~rmart04@support.memset.com) has joined #ceph
[15:27] * CobraKhan007 (~Jamana@61TAAARDM.tor-irc.dnsbl.oftc.net) Quit ()
[15:27] * Coestar (~Dragonsha@85.93.218.204) has joined #ceph
[15:28] * johnavp1989 (~jpetrini@8.39.115.8) has joined #ceph
[15:28] <- *johnavp1989* To prove that you are human, please enter the result of 8+3
[15:35] * wjw-freebsd2 (~wjw@smtp.digiware.nl) has joined #ceph
[15:35] * squizzi (~squizzi@107.13.31.195) has joined #ceph
[15:37] * yanzheng (~zhyan@118.116.114.34) Quit (Quit: This computer has gone to sleep)
[15:37] * dgurtner (~dgurtner@209.132.186.254) has joined #ceph
[15:39] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[15:41] * swami1 (~swami@49.38.2.251) Quit (Quit: Leaving.)
[15:44] * owasserm (~owasserm@nat-pool-ams-t.redhat.com) Quit (Ping timeout: 480 seconds)
[15:48] * dosaboy (~dosaboy@33.93.189.91.lcy-02.canonistack.canonical.com) Quit (Quit: leaving)
[15:52] * dosaboy (~dosaboy@33.93.189.91.lcy-02.canonistack.canonical.com) has joined #ceph
[15:55] * dnunez (~dnunez@nat-pool-bos-u.redhat.com) has joined #ceph
[15:57] * sudocat1 (~dibarra@192.185.1.19) has joined #ceph
[15:57] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[15:57] * Coestar (~Dragonsha@9YSAAAR0H.tor-irc.dnsbl.oftc.net) Quit ()
[15:58] * sudocat2 (~dibarra@192.185.1.20) has joined #ceph
[15:58] * EinstCrazy (~EinstCraz@58.39.17.213) has joined #ceph
[16:02] * sudocat1 (~dibarra@192.185.1.19) Quit (Read error: Connection reset by peer)
[16:02] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[16:02] * sudocat2 (~dibarra@192.185.1.20) Quit ()
[16:03] * dnunez (~dnunez@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[16:04] * derjohn_mob (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[16:04] * bene2 (~bene@nat-pool-bos-t.redhat.com) has joined #ceph
[16:04] * vbellur (~vijay@71.234.224.255) Quit (Ping timeout: 480 seconds)
[16:05] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[16:05] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) has joined #ceph
[16:11] * vikhyat (~vumrao@121.244.87.116) Quit (Quit: Leaving)
[16:14] * dnunez (~dnunez@nat-pool-bos-t.redhat.com) has joined #ceph
[16:23] * lpabon (~quassel@nat-pool-bos-t.redhat.com) has joined #ceph
[16:25] * squizzi (~squizzi@107.13.31.195) Quit (Quit: bye)
[16:27] * ira (~ira@nat-pool-bos-u.redhat.com) has joined #ceph
[16:28] * vbellur (~vijay@nat-pool-bos-t.redhat.com) has joined #ceph
[16:31] * EinstCrazy (~EinstCraz@58.39.17.213) Quit (Remote host closed the connection)
[16:32] * EinstCrazy (~EinstCraz@58.39.17.213) has joined #ceph
[16:32] * lpabon (~quassel@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[16:33] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[16:43] * xarses (~xarses@64.124.158.100) has joined #ceph
[16:44] * derjohn_mob (~aj@88.128.80.27) has joined #ceph
[16:51] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) Quit (Quit: Leaving.)
[16:51] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) has joined #ceph
[16:51] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) Quit ()
[16:52] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) has joined #ceph
[16:52] * davidzlap (~Adium@rrcs-74-87-213-28.west.biz.rr.com) Quit ()
[16:57] * Shadow386 (~Hejt@185.100.86.69) has joined #ceph
[16:58] * vbellur (~vijay@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[17:00] * owasserm (~owasserm@2001:984:d3f7:1:5ec5:d4ff:fee0:f6dc) has joined #ceph
[17:00] * arcimboldo (~antonio@84-75-174-248.dclient.hispeed.ch) has joined #ceph
[17:01] <arcimboldo> hi all, I'm using ceph for openstack cinder/ephemeral images. I wonder if there is an easy way to interrogate ceph on the IO performed by single clients
[17:01] <arcimboldo> something like "ceph -s" output but split by IPs
[17:02] <[arx]> i don't think there is a way
[17:04] * haplo37 (~haplo37@199.91.185.156) Quit (Ping timeout: 480 seconds)
[17:05] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[17:05] * sudocat1 (~dibarra@192.185.1.20) Quit (Ping timeout: 480 seconds)
[17:05] * lpabon (~quassel@nat-pool-bos-t.redhat.com) has joined #ceph
[17:07] * joshd1 (~jdurgin@2602:30a:c089:2b0:886b:8cf4:786c:c202) has joined #ceph
[17:07] * vata1 (~vata@207.96.182.162) has joined #ceph
[17:10] * vbellur (~vijay@nat-pool-bos-u.redhat.com) has joined #ceph
[17:12] * haplo37 (~haplo37@199.91.185.156) has joined #ceph
[17:13] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[17:13] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) has joined #ceph
[17:19] * karnan (~karnan@106.51.137.148) Quit (Quit: Leaving)
[17:20] * EinstCrazy (~EinstCraz@58.39.17.213) Quit (Remote host closed the connection)
[17:20] * kefu (~kefu@183.193.182.196) has joined #ceph
[17:25] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) has joined #ceph
[17:27] * Shadow386 (~Hejt@26XAAAH1I.tor-irc.dnsbl.oftc.net) Quit ()
[17:27] * Hidendra (~Throlkim@37.59.72.135) has joined #ceph
[17:30] * ircolle (~Adium@adsl-76-195-223-64.dsl.chmpil.sbcglobal.net) has joined #ceph
[17:34] * scg (~zscg@146-115-134-246.c3-0.nwt-ubr1.sbo-nwt.ma.cable.rcn.com) Quit (Ping timeout: 480 seconds)
[17:34] * malevolent (~quassel@124.red-88-11-251.dynamicip.rima-tde.net) Quit (Quit: http://quassel-irc.org - Chatee c?modamente donde sea.)
[17:35] * malevolent (~quassel@124.red-88-11-251.dynamicip.rima-tde.net) has joined #ceph
[17:36] * Hemanth (~hkumar_@121.244.87.117) Quit (Ping timeout: 480 seconds)
[17:37] * sudocat1 (~dibarra@192.185.1.20) Quit (Quit: Leaving.)
[17:37] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[17:39] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[17:39] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[17:43] * ChanServ sets mode +o scuttlemonkey
[17:45] * ircolle1 (~Adium@adsl-76-195-223-64.dsl.chmpil.sbcglobal.net) has joined #ceph
[17:45] * squizzi (~squizzi@2001:420:2240:1268:a0b7:f4b7:490:2105) has joined #ceph
[17:46] * TMM (~hp@185.5.121.201) Quit (Quit: Ex-Chat)
[17:47] * davidzlap (~Adium@cpe-172-91-154-245.socal.res.rr.com) has joined #ceph
[17:47] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[17:48] * noahw (~noahw@eduroam-169-233-225-12.ucsc.edu) has joined #ceph
[17:48] * danieagle (~Daniel@179.98.86.132) has joined #ceph
[17:49] * ircolle (~Adium@adsl-76-195-223-64.dsl.chmpil.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[17:49] * rmart04 (~rmart04@support.memset.com) Quit (Quit: rmart04)
[17:50] * dgurtner (~dgurtner@209.132.186.254) Quit (Ping timeout: 480 seconds)
[17:51] * ronrib (~boswortr@45.32.242.135) Quit (Remote host closed the connection)
[17:52] * bara (~bara@nat-pool-brq-t.redhat.com) Quit (Quit: Bye guys! (??????????????????? ?????????)
[17:54] * rotbeard (~redbeard@2a02:908:df13:bb00:953f:d96a:d234:916f) Quit (Quit: Leaving)
[17:56] * b0e (~aledermue@213.95.25.82) has joined #ceph
[17:56] * ircolle (~Adium@adsl-76-195-220-139.dsl.chmpil.sbcglobal.net) has joined #ceph
[17:56] * ircolle1 (~Adium@adsl-76-195-223-64.dsl.chmpil.sbcglobal.net) Quit (Read error: Connection reset by peer)
[17:57] * rraja (~rraja@121.244.87.117) Quit (Quit: Leaving)
[17:57] * Hidendra (~Throlkim@9YSAAAR43.tor-irc.dnsbl.oftc.net) Quit ()
[17:59] * rakeshgm (~rakesh@121.244.87.117) has joined #ceph
[17:59] * branto (~branto@ip-78-102-208-181.net.upcbroadband.cz) Quit (Quit: Leaving.)
[18:00] * jcsp (~jspray@82-71-16-249.dsl.in-addr.zen.co.uk) has joined #ceph
[18:02] * derjohn_mob (~aj@88.128.80.27) Quit (Ping timeout: 480 seconds)
[18:05] * swami1 (~swami@27.7.165.198) has joined #ceph
[18:08] * kefu is now known as kefu|afk
[18:10] * InIMoeK (~InIMoeK@95.170.93.16) Quit (Ping timeout: 480 seconds)
[18:10] * wjw-freebsd2 (~wjw@smtp.digiware.nl) Quit (Ping timeout: 480 seconds)
[18:11] * RayTracer (~RayTracer@153.19.13.27) has joined #ceph
[18:26] * b0e (~aledermue@213.95.25.82) Quit (Quit: Leaving.)
[18:27] * kefu|afk (~kefu@183.193.182.196) Quit (Max SendQ exceeded)
[18:27] * Shadow386 (~Plesioth@61TAAARK0.tor-irc.dnsbl.oftc.net) has joined #ceph
[18:31] * kefu (~kefu@183.193.182.196) has joined #ceph
[18:31] * dyasny (~dyasny@cable-192.222.152.136.electronicbox.net) Quit (Ping timeout: 480 seconds)
[18:32] * debian112 (~bcolbert@173-164-167-194-SFBA.hfc.comcastbusiness.net) Quit (Quit: Leaving.)
[18:34] * SuzukiB12s (~eduardo@138.186.43.20) has joined #ceph
[18:37] * ircolle (~Adium@adsl-76-195-220-139.dsl.chmpil.sbcglobal.net) Quit (Quit: Leaving.)
[18:39] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[18:41] * swami1 (~swami@27.7.165.198) Quit (Quit: Leaving.)
[18:46] * swami1 (~swami@27.7.165.198) has joined #ceph
[18:49] * vasu (~vasu@c-73-231-60-138.hsd1.ca.comcast.net) has joined #ceph
[18:50] * rakeshgm (~rakesh@121.244.87.117) Quit (Quit: Leaving)
[18:53] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[18:53] <salwasser> I have an osd that has been in the booting state for a day or two now. The osd data directory is no longer growing in size. any ideas how I can troubleshoot what is going on with this osd?
[18:54] * RayTracer (~RayTracer@153.19.13.27) Quit (Remote host closed the connection)
[18:55] * RayTracer (~RayTracer@153.19.13.27) has joined #ceph
[18:55] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:57] * Shadow386 (~Plesioth@61TAAARK0.tor-irc.dnsbl.oftc.net) Quit ()
[18:57] * pakman__ (~Atomizer@61TAAARLU.tor-irc.dnsbl.oftc.net) has joined #ceph
[19:00] <Anticimex> doing upgrade from infernalis to jewel. works fine
[19:01] <Anticimex> until i rebooted 1 osd node to make sure it comes up fine. after reboot getting: "cluster [WRN] failed to encode map e33776 with expected crc" from all osds on it
[19:01] <Anticimex> "1 osd node" = "1 server with bunch of osds on it"
[19:01] <Anticimex> according to http://www.spinics.net/lists/ceph-devel/msg30450.html the message means that the OSDs are pulling the map directly from the mon's
[19:01] <Anticimex> but that's also a bit old
[19:03] * RayTracer (~RayTracer@153.19.13.27) Quit (Ping timeout: 480 seconds)
[19:03] * DanFoster (~Daniel@office.34sp.com) Quit (Quit: Leaving)
[19:03] <Anticimex> i upgraded mons one at a time, and all are upgraded. all osd daemons and mon daemons (no mds exist) are now running jewel
[19:03] * chopmann (~sirmonkey@2a02:8108:8b40:5600:afc9:f9c5:160c:f885) Quit (Quit: chopmann)
[19:05] * vata (~vata@cable-173.246.3-246.ebox.ca) Quit (Ping timeout: 480 seconds)
[19:06] * i_m (~ivan.miro@deibp9eh1--blueice1n0.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[19:07] * debian112 (~bcolbert@64.235.157.198) has joined #ceph
[19:10] * EthanL (~lamberet@cce02cs4043-fa12-z.ams.hpecore.net) has joined #ceph
[19:12] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) has joined #ceph
[19:14] * vata1 (~vata@207.96.182.162) Quit (Remote host closed the connection)
[19:15] * swami1 (~swami@27.7.165.198) Quit (Quit: Leaving.)
[19:16] * vata (~vata@cable-173.246.3-246.ebox.ca) has joined #ceph
[19:17] * EthanL (~lamberet@cce02cs4043-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[19:17] <salwasser> ~salwasser
[19:21] * vata1 (~vata@207.96.182.162) has joined #ceph
[19:23] * Jeffrey4l_ (~Jeffrey@110.252.56.47) Quit (Ping timeout: 480 seconds)
[19:23] * garphy is now known as garphy`aw
[19:24] * SuzukiB12s (~eduardo@138.186.43.20) has left #ceph
[19:24] * vata (~vata@cable-173.246.3-246.ebox.ca) Quit (Ping timeout: 480 seconds)
[19:27] * pakman__ (~Atomizer@61TAAARLU.tor-irc.dnsbl.oftc.net) Quit ()
[19:27] * Kaervan (~Kyso_@freeciv.nmte.ch) has joined #ceph
[19:28] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:37] * vata (~vata@cable-173.246.3-246.ebox.ca) has joined #ceph
[19:41] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[19:55] * linuxkidd_ (~linuxkidd@107.72.98.254) has joined #ceph
[19:56] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) Quit (Ping timeout: 480 seconds)
[19:57] * RayTracer (~RayTracer@153.19.13.27) has joined #ceph
[19:57] * kefu (~kefu@183.193.182.196) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[19:57] * Kaervan (~Kyso_@61TAAARMR.tor-irc.dnsbl.oftc.net) Quit ()
[19:57] * richardus1 (~W|ldCraze@atlantic480.us.unmetered.com) has joined #ceph
[20:00] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) Quit (Remote host closed the connection)
[20:00] * squizzi (~squizzi@2001:420:2240:1268:a0b7:f4b7:490:2105) Quit (Quit: bye)
[20:01] * squizzi (~squizzi@2001:420:2240:1268:a0b7:f4b7:490:2105) has joined #ceph
[20:03] * BrianA1 (~BrianA@c-24-130-77-13.hsd1.ca.comcast.net) has joined #ceph
[20:03] * BrianA1 (~BrianA@c-24-130-77-13.hsd1.ca.comcast.net) has left #ceph
[20:03] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[20:07] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) has joined #ceph
[20:09] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Ping timeout: 480 seconds)
[20:11] * jarrpa (~jarrpa@63.225.131.166) Quit (Ping timeout: 480 seconds)
[20:13] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[20:13] * EthanL (~lamberet@cce02cs4043-fa12-z.ams.hpecore.net) has joined #ceph
[20:18] * arcimboldo (~antonio@84-75-174-248.dclient.hispeed.ch) Quit (Quit: Ex-Chat)
[20:19] * RayTracer (~RayTracer@153.19.13.27) Quit (Quit: Leaving...)
[20:19] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[20:20] * jarrpa (~jarrpa@67-4-129-232.mpls.qwest.net) has joined #ceph
[20:20] * EthanL (~lamberet@cce02cs4043-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[20:25] * mykola (~Mikolaj@91.245.73.11) has joined #ceph
[20:27] * richardus1 (~W|ldCraze@26XAAAH6W.tor-irc.dnsbl.oftc.net) Quit ()
[20:27] * Oddtwang (~dusti@cry.ip-eend.nl) has joined #ceph
[20:30] * shaunm (~shaunm@74.83.215.100) Quit (Ping timeout: 480 seconds)
[20:35] * dougf_ (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[20:37] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Ping timeout: 480 seconds)
[20:38] * BrianA1 (~BrianA@c-24-130-77-13.hsd1.ca.comcast.net) has joined #ceph
[20:40] * visbits (~textual@8.29.138.28) Quit (Quit: Textual IRC Client: www.textualapp.com)
[20:45] * Hemanth (~hkumar_@103.228.221.160) has joined #ceph
[20:47] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[20:48] * sudocat1 (~dibarra@192.185.1.20) Quit (Read error: Connection reset by peer)
[20:48] * sudocat1 (~dibarra@192.185.1.20) has joined #ceph
[20:48] * sudocat (~dibarra@192.185.1.20) Quit (Read error: Connection reset by peer)
[20:51] * sudocat1 (~dibarra@192.185.1.20) Quit ()
[20:51] * sudocat (~dibarra@192.185.1.20) has joined #ceph
[20:51] * BrianA1 (~BrianA@c-24-130-77-13.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[20:55] * xarses_ (~xarses@64.124.158.100) has joined #ceph
[20:56] * xarses (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[20:57] * Oddtwang (~dusti@61TAAAROA.tor-irc.dnsbl.oftc.net) Quit ()
[21:01] * Hemanth (~hkumar_@103.228.221.160) Quit (Ping timeout: 480 seconds)
[21:02] * asdf1 (~jabbe@32.97.110.56) has joined #ceph
[21:03] * cathode (~cathode@50.232.215.114) has joined #ceph
[21:07] * blizzow (~jburns@50.243.148.102) Quit (Ping timeout: 480 seconds)
[21:15] * dougf_ (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) Quit (Ping timeout: 480 seconds)
[21:16] * dougf (~dougf@96-38-99-179.dhcp.jcsn.tn.charter.com) has joined #ceph
[21:20] * wes_dillingham (~wes_dilli@140.247.242.44) has joined #ceph
[21:21] <wes_dillingham> I have and rbd device which appears to have an exclusive lock that i cannot remove, I believe it is from a failed snapshot attempt earlier in the day. my goal is to delete this image. it has the followign features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
[21:22] <wes_dillingham> when i attempted to snapshot the device earlier it failed with: 2016-07-19 02:52:24.368818 7f17be4f2700 -1 ObjectPlayer: : partial record at offset 13593922
[21:22] <wes_dillingham> 2016-07-19 02:52:24.368823 7f17be4f2700 -1 ObjectPlayer: : corruption range [13593922, 33554432)
[21:22] <wes_dillingham> when i lock ls the device i get:
[21:23] <wes_dillingham> client.7638752 auto 139774610388880 10.31.25.135:0/2895393653
[21:23] * jarrpa (~jarrpa@67-4-129-232.mpls.qwest.net) Quit (Ping timeout: 480 seconds)
[21:23] <wes_dillingham> i attempt to remove the lock using: rbd lock rm <image-spec> 139774610388880 client.7638752
[21:23] <wes_dillingham> and get the error: rbd: releasing lock failed: (2) No such file or directory
[21:24] <wes_dillingham> the ip of the lock is a monitor???. the monitor from which I am attempting to perform these operations
[21:24] <wes_dillingham> running 10.2.0
[21:26] * cathode (~cathode@50.232.215.114) Quit (Quit: Leaving)
[21:26] <asdf1> I'm having a weird issue...whenever I do "rbd ls", I get some weird errors...
[21:26] <asdf1> 2016-07-21 12:25:58.974513 7efcbc67d700 0 -- :/1382572333 >> 10.0.0.22:6789/0 pipe(0x55a0689ea550 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x55a0689e4000).fault
[21:26] <asdf1> 2016-07-21 12:26:01.975332 7efcaf8aa700 0 -- :/1382572333 >> 10.0.0.22:6789/0 pipe(0x7efca4000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7efca4004ef0).fault
[21:26] <asdf1> 2016-07-21 12:26:04.975072 7efcbc67d700 0 -- :/1382572333 >> 10.0.0.22:6789/0 pipe(0x7efca40080e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7efca400c380).fault
[21:26] <asdf1> 2016-07-21 12:26:07.975596 7efcaf8aa700 0 -- :/1382572333 >> 10.0.0.22:6789/0 pipe(0x7efca4000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7efca4006480).fault
[21:26] <asdf1> any idea what this might be from?
[21:26] <asdf1> I can't really google it
[21:27] <wes_dillingham> where are you doing the rbd ls from, can you reach the monitors listed in the conf file ?
[21:27] <[arx]> are you able to ... that
[21:27] <asdf1> I'm running them from a server that's connected to the monitors
[21:27] * nastidon (~Borf@109.236.90.209) has joined #ceph
[21:28] <asdf1> but the same thing happens on each monitor
[21:28] <[arx]> is the ip address always 10.0.0.22?
[21:29] <asdf1> there are three...21, 22 and 23
[21:29] <asdf1> hmm, when I run the command from 22, it cycles through 21, 22, 23
[21:29] <asdf1> over and over
[21:31] <[arx]> yea, i normally only see that error when i can't connect to a monitor server.
[21:31] * jarrpa (~jarrpa@2602:3f:e183:a600:eab1:fcff:fe47:f680) has joined #ceph
[21:32] * rendar (~I@host79-131-dynamic.59-82-r.retail.telecomitalia.it) Quit (Ping timeout: 480 seconds)
[21:32] * reed (~reed@216.38.134.18) has joined #ceph
[21:32] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[21:33] <wes_dillingham> can you probe the other monitors ports?
[21:34] <asdf1> ok, I'm brand new to ceph (disclaimer0
[21:34] <wes_dillingham> telnet $monitorIP 6789
[21:34] <asdf1> oh ok...well I can ssh to them
[21:34] <wes_dillingham> monitors run by default on port 6789, see if you can reach that port
[21:35] <asdf1> ok trying
[21:35] <asdf1> # telnet 10.0.0.22 6789
[21:35] <asdf1> Trying 10.0.0.22...
[21:35] <asdf1> telnet: Unable to connect to remote host: Connection refused
[21:36] <asdf1> is that bad?
[21:39] <wes_dillingham> yes
[21:39] <wes_dillingham> that means you cant communicate with the monitor
[21:39] <wes_dillingham> is the monitor daemon running on that host?
[21:40] <asdf1> what is the command to check that? (forgive my ignorance)
[21:40] <wes_dillingham> service ceph-mon@* status
[21:40] <wes_dillingham> on the mon..
[21:41] <asdf1> ok
[21:41] <wes_dillingham> check the firewall on your mon(s)
[21:42] <asdf1> ok I rebooted one of the nodes and I can do the telnet connection to that host now
[21:43] <asdf1> rebooting the others to see if that fixes it!
[21:43] <asdf1> thank you so much wes_dillingham for your help so far!
[21:43] <wes_dillingham> np
[21:43] <dillaman> wes_dillingham: the journal appears to be corrupt so the image is refusing to open
[21:44] <wes_dillingham> dillaman: i was able to remove the lock, but i cant disable journaling or remove the image
[21:44] <wes_dillingham> rbd feature disable 1ss_root_disk/one-52 journaling
[21:44] <wes_dillingham> 2016-07-21 15:38:54.392738 7f520cb46700 -1 ObjectPlayer: : partial record at offset 13593922
[21:44] <wes_dillingham> 2016-07-21 15:38:54.392749 7f520cb46700 -1 ObjectPlayer: : corruption range [13593922, 33554432)
[21:44] <wes_dillingham> 2016-07-21 15:38:55.327119 7f520cb46700 -1 ObjectPlayer: : partial record at offset 4570308
[21:44] <wes_dillingham> 2016-07-21 15:38:55.327129 7f520cb46700 -1 ObjectPlayer: : corruption range [4570308, 33554432)
[21:44] <dillaman> wes_dillingham: 10.2.0 had a bug that resulted in large writes (> than the journal object size) overflowing the journal
[21:45] <dillaman> wes_dillingham: can you run "rbd info" against the image and record the journal id
[21:45] <wes_dillingham> are you going to have me drop to rados and blast the objects?
[21:45] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[21:45] <wes_dillingham> journal: 319e64238e1f29
[21:46] <dillaman> wes_dillingham: than you can run "rados ls | grep -E '^journal_data\.<YOUR JOURNAL ID HERE>\.' | xargs -n 100 | rados rm" to delete the corrupt journal data
[21:46] <dillaman> wes_dillingham: ha .. indeed
[21:46] <wes_dillingham> curious, would i have to play this game if i upgraded to 10.2.2
[21:46] <wes_dillingham> could i simply do a rbd rm <img> in this state ?
[21:47] * MentalRay (~MentalRay@office-mtl1-nat-146-218-70-69.gtcomm.net) has joined #ceph
[21:47] <wes_dillingham> or would i still have to blast the objects via rados
[21:47] <dillaman> wes_dillingham: the journal would still be corrupt after upgrade
[21:47] <wes_dillingham> ok
[21:48] <dillaman> wes_dillingham: the goal would be for "rbd rm" or "rbd feature disable <xyz> journaling" to get you out of the state, but I can't say we've tested all corner cases
[21:48] <dillaman> wes_dillingham: if it doesn't work, we will fix it
[21:48] * penguinRaider (~KiKo@23.94.218.244) Quit (Ping timeout: 480 seconds)
[21:48] <wes_dillingham> ok, im going to give it a go
[21:49] <dillaman> wes_dillingham: upgrade or direct delete?
[21:50] <wes_dillingham> delete.
[21:50] <wes_dillingham> if upgrading doesnt help me in current state???.
[21:50] <wes_dillingham> but i am planning on upgrading anyways.
[21:50] <wes_dillingham> not now though
[21:50] <dillaman> ok
[21:52] * derjohn_mob (~aj@x4db27b7f.dyn.telefonica.de) has joined #ceph
[21:52] <wes_dillingham> dillaman: that earlier command you gave should be rados ls <pool> right ?
[21:53] <wes_dillingham> i get nervous removing stuff at rados level...
[21:53] <dillaman> wes_dillingham: dry-run it first: rados -p <pool> ls | grep -E '^journal_data\.319e64238e1f29\.'
[21:54] <wes_dillingham> returned nothing
[21:54] <dillaman> ... then append on " | xargs -n 100 rados -p <pool> rm" when ready -- or just rm them one at a time
[21:56] <wes_dillingham> nvm, i dont need your help with bash, sorry
[21:56] * lpabon (~quassel@nat-pool-bos-t.redhat.com) Quit (Remote host closed the connection)
[21:57] <wes_dillingham> do these look like the objects i need to blast dillaman ? https://paste.fedoraproject.org/393668/14691310/
[21:57] * nastidon (~Borf@61TAAARQA.tor-irc.dnsbl.oftc.net) Quit ()
[21:57] * Phase (~dicko@atlantic480.us.unmetered.com) has joined #ceph
[21:58] <dillaman> wes_dillingham: yup
[21:58] * rendar (~I@host79-131-dynamic.59-82-r.retail.telecomitalia.it) has joined #ceph
[21:58] * Phase is now known as Guest3895
[21:59] <wes_dillingham> why the -n 100 to prevent a malformed command from deleting more than 100 objects?
[22:00] * georgem (~Adium@85.204.4.209) has joined #ceph
[22:01] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:ed7c:2028:533d:c626) Quit (Ping timeout: 480 seconds)
[22:02] * penguinRaider (~KiKo@23.94.218.244) has joined #ceph
[22:05] <dillaman> wes_dillingham: just to batch multiple removals into a single invocation but not large enough to hit most argument length limits
[22:05] <wes_dillingham> thansk for the help dillaman i was able to remove th rbd device.
[22:06] <wes_dillingham> after removing the corrput jounral
[22:07] <dillaman> wes_dillingham: np
[22:17] * theTrav (~theTrav@CPE-124-188-218-238.sfcz1.cht.bigpond.net.au) has joined #ceph
[22:22] * wes_dillingham (~wes_dilli@140.247.242.44) Quit (Ping timeout: 480 seconds)
[22:23] * SamYaple (~SamYaple@162.209.126.134) Quit (Quit: leaving)
[22:23] * SamYaple (~SamYaple@162.209.126.134) has joined #ceph
[22:25] * mykola (~Mikolaj@91.245.73.11) Quit (Quit: away)
[22:27] * Guest3895 (~dicko@5AEAAAHEY.tor-irc.dnsbl.oftc.net) Quit ()
[22:27] * W|ldCraze (~fauxhawk@192.87.28.28) has joined #ceph
[22:28] * debian112 (~bcolbert@64.235.157.198) Quit (Ping timeout: 480 seconds)
[22:33] * JeremyP (~phamth7@68.140.239.132) Quit (Quit: Leaving)
[22:36] * shaunm (~shaunm@74.83.215.100) has joined #ceph
[22:36] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[22:37] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[22:40] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[22:45] * mattbenjamin (~mbenjamin@12.118.3.106) Quit (Ping timeout: 480 seconds)
[22:51] * debian112 (~bcolbert@64.235.157.198) has joined #ceph
[22:53] * allaok (~allaok@ARennes-658-1-158-189.w92-139.abo.wanadoo.fr) has joined #ceph
[22:56] * allaok (~allaok@ARennes-658-1-158-189.w92-139.abo.wanadoo.fr) has left #ceph
[22:56] * georgem (~Adium@85.204.4.209) Quit (Quit: Leaving.)
[22:57] * W|ldCraze (~fauxhawk@9YSAAASEU.tor-irc.dnsbl.oftc.net) Quit ()
[22:57] * w0lfeh (~rikai@tor-exit-ovh-ca1.toupeira.org) has joined #ceph
[22:58] * shaunm (~shaunm@74.83.215.100) Quit (Ping timeout: 480 seconds)
[22:59] * analbeard (~shw@support.memset.com) has joined #ceph
[23:00] * haplo37 (~haplo37@199.91.185.156) Quit (Remote host closed the connection)
[23:00] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Read error: Connection reset by peer)
[23:01] * analbeard (~shw@support.memset.com) has left #ceph
[23:03] * vbellur (~vijay@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[23:04] * cathode (~cathode@50.232.215.114) has joined #ceph
[23:08] * rwheeler (~rwheeler@nat-pool-bos-t.redhat.com) Quit (Quit: Leaving)
[23:10] * DaveOD (~DaveOD@nomadesk.com) Quit (Ping timeout: 480 seconds)
[23:15] * newbie (~kvirc@host217-114-156-249.pppoe.mark-itt.net) Quit (Ping timeout: 480 seconds)
[23:16] * ira (~ira@nat-pool-bos-u.redhat.com) Quit (Ping timeout: 480 seconds)
[23:17] * fdmanana (~fdmanana@2001:8a0:6e0c:6601:454c:daa6:f349:3b58) has joined #ceph
[23:18] * xarses_ (~xarses@64.124.158.100) Quit (Ping timeout: 480 seconds)
[23:18] * linuxkidd (~linuxkidd@ip70-189-207-54.lv.lv.cox.net) has joined #ceph
[23:25] * linuxkidd_ (~linuxkidd@107.72.98.254) Quit (Ping timeout: 480 seconds)
[23:26] * reed (~reed@216.38.134.18) Quit (Ping timeout: 480 seconds)
[23:27] * w0lfeh (~rikai@26XAAAIBX.tor-irc.dnsbl.oftc.net) Quit ()
[23:27] * Pettis (~Gecko1986@178.32.53.131) has joined #ceph
[23:35] * squizzi (~squizzi@2001:420:2240:1268:a0b7:f4b7:490:2105) Quit (Quit: bye)
[23:35] * JPGainsborough (~jpg@71.122.174.182) Quit (Ping timeout: 480 seconds)
[23:37] * cronburg (~cronburg@nat-pool-bos-t.redhat.com) Quit (Ping timeout: 480 seconds)
[23:38] * cyphase (~cyphase@000134f2.user.oftc.net) Quit (Ping timeout: 480 seconds)
[23:39] * cyphase (~cyphase@000134f2.user.oftc.net) has joined #ceph
[23:40] * bene2 (~bene@nat-pool-bos-t.redhat.com) Quit (Quit: Konversation terminated!)
[23:40] * EthanL (~lamberet@cce02cs4044-fa12-z.ams.hpecore.net) has joined #ceph
[23:47] * EthanL (~lamberet@cce02cs4044-fa12-z.ams.hpecore.net) Quit (Read error: Connection reset by peer)
[23:55] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph
[23:57] * Pettis (~Gecko1986@5AEAAAHGM.tor-irc.dnsbl.oftc.net) Quit ()
[23:57] * wjw-freebsd (~wjw@smtp.digiware.nl) Quit (Read error: Connection reset by peer)
[23:58] * wjw-freebsd (~wjw@smtp.digiware.nl) has joined #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.