#ceph IRC Log

Index

IRC Log for 2015-06-29

Timestamps are in GMT/BST.

[0:00] * oro (~oro@84-72-180-62.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[0:11] * sage (~quassel@2607:f298:6050:709d:c4ef:e7a1:7751:6071) Quit (Ping timeout: 480 seconds)
[0:12] * linjan (~linjan@80.179.241.26) Quit (Ping timeout: 480 seconds)
[0:16] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[0:32] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[0:33] * Nacer (~Nacer@2001:41d0:fe82:7200:58f1:b488:7091:af82) Quit (Ping timeout: 480 seconds)
[0:38] * badone (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[0:40] * badone__ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[0:40] * xarses (~xarses@173-164-194-206-SFBA.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[0:45] * badone_ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[0:50] * Nacer (~Nacer@2001:41d0:fe82:7200:bd53:b9fd:328c:54c8) has joined #ceph
[0:57] * bgleb (~bgleb@94.19.146.224) has joined #ceph
[0:58] * daviddcc (~dcasier@ADijon-653-1-192-159.w90-39.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[1:03] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) Quit (Ping timeout: 480 seconds)
[1:05] * TheSov2 (~TheSov@c-50-158-169-178.hsd1.il.comcast.net) Quit (Read error: Connection reset by peer)
[1:05] * Rickus (~Rickus@office.protected.ca) Quit (Read error: Connection reset by peer)
[1:06] * TheSov2 (~TheSov@c-50-158-169-178.hsd1.il.comcast.net) has joined #ceph
[1:06] * Rickus (~Rickus@office.protected.ca) has joined #ceph
[1:07] * TMM (~hp@178-84-46-106.dynamic.upc.nl) has joined #ceph
[1:09] * bgleb (~bgleb@94.19.146.224) Quit (Remote host closed the connection)
[1:20] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) Quit (Quit: doppelgrau)
[1:26] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[1:26] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[1:40] * xarses (~xarses@173-164-194-206-SFBA.hfc.comcastbusiness.net) has joined #ceph
[1:44] * LeaChim (~LeaChim@host86-132-233-125.range86-132.btcentralplus.com) Quit (Ping timeout: 480 seconds)
[1:54] * b0e (~aledermue@cpe-75-85-69-61.socal.res.rr.com) has joined #ceph
[1:58] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[2:10] * bgleb (~bgleb@94.19.146.224) has joined #ceph
[2:13] * xarses (~xarses@173-164-194-206-SFBA.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[2:14] * trey (~trey@trey.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:19] * xarses (~xarses@173-164-194-206-SFBA.hfc.comcastbusiness.net) has joined #ceph
[2:21] * b0e (~aledermue@cpe-75-85-69-61.socal.res.rr.com) Quit (Quit: Leaving.)
[2:26] * bgleb (~bgleb@94.19.146.224) Quit (Ping timeout: 480 seconds)
[2:31] * trey (~trey@trey.user.oftc.net) has joined #ceph
[2:36] * ghartz (~ghartz@AStrasbourg-651-1-176-226.w90-40.abo.wanadoo.fr) has joined #ceph
[2:37] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[2:39] * fam_away is now known as fam
[2:39] * trey (~trey@trey.user.oftc.net) Quit (Ping timeout: 480 seconds)
[2:43] * lucas (~Thunderbi@218.76.52.64) has joined #ceph
[2:45] * xarses (~xarses@173-164-194-206-SFBA.hfc.comcastbusiness.net) Quit (Ping timeout: 480 seconds)
[2:54] * burley (~khemicals@cpe-98-28-239-78.cinci.res.rr.com) has joined #ceph
[3:03] * shyu (~Shanzhi@119.254.120.66) has joined #ceph
[3:10] * Debesis (~0x@213.119.116.89.mobile.mezon.lt) Quit (Ping timeout: 480 seconds)
[3:13] * yanzheng (~zhyan@182.139.205.179) has joined #ceph
[3:33] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) has joined #ceph
[3:34] * fam is now known as fam_away
[3:36] * fam_away is now known as fam
[3:47] * aj__ (~aj@x590d8fe0.dyn.telefonica.de) has joined #ceph
[3:50] * yanzheng (~zhyan@182.139.205.179) Quit (Ping timeout: 480 seconds)
[3:55] * derjohn_mobi (~aj@x4db27f39.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[3:55] * yanzheng (~zhyan@182.139.205.179) has joined #ceph
[3:55] * shang (~ShangWu@175.41.48.77) has joined #ceph
[4:08] * bgleb (~bgleb@94.19.146.224) has joined #ceph
[4:16] * fam is now known as fam_away
[4:18] * flisky (~Thunderbi@106.39.60.34) has joined #ceph
[4:22] * MRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) has joined #ceph
[4:22] * shang (~ShangWu@175.41.48.77) Quit (Ping timeout: 480 seconds)
[4:23] * bgleb (~bgleb@94.19.146.224) Quit (Ping timeout: 480 seconds)
[4:23] * MRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit ()
[4:26] * fam_away is now known as fam
[4:29] * MentalRay (~MRay@MTRLPQ42-1176054809.sdsl.bell.ca) Quit (Ping timeout: 480 seconds)
[4:38] * zhaochao (~zhaochao@125.39.8.229) has joined #ceph
[4:40] * vbellur (~vijay@c-71-198-3-24.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[5:06] * kefu (~kefu@114.92.113.55) has joined #ceph
[5:13] * zhaochao_ (~zhaochao@111.161.77.233) has joined #ceph
[5:17] * zhaochao (~zhaochao@125.39.8.229) Quit (Ping timeout: 480 seconds)
[5:17] * zhaochao_ is now known as zhaochao
[5:20] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[5:22] * squ (~Thunderbi@46.109.36.167) has joined #ceph
[5:27] * wq32 (ba11b0ae@107.161.19.109) has joined #ceph
[5:28] <wq32> Hi, I was reading about Ceph and a guide
[5:28] <wq32> And the guy configured the cluster as file server
[5:28] <wq32> I had the impression you could use Ceph as Object, Block and File system
[5:29] <wq32> Do you need to create different clusters for this? Or can one Ceph clusters handle all of them?
[5:29] <wq32> Example, is offering Object storage like S3, and also for VM (Block Storage) from the same cluster
[5:30] <lurbs> One cluster handles all of them, it puts the data into different pools.
[5:30] * vbellur (~vijay@c-71-198-3-24.hsd1.ca.comcast.net) has joined #ceph
[5:31] * Vacuum_ (~Vacuum@i59F79413.versanet.de) has joined #ceph
[5:34] <wq32> Thanks
[5:34] <wq32> I'm referring to this guide
[5:34] <wq32> http://www.virtualtothecore.com/en/adventures-ceph-storage-part-1-introduction/
[5:35] <wq32> I could not find now where part, but in one he explains he configured it as a repository so the OSDs where formatted for it
[5:35] <wq32> This is what confused me.
[5:36] <wq32> As if he was formatting it specially for one type of use.
[5:38] * Vacuum__ (~Vacuum@88.130.216.214) Quit (Ping timeout: 480 seconds)
[5:38] * shang (~ShangWu@175.41.48.77) has joined #ceph
[5:40] * fam is now known as fam_away
[5:41] <wq32> Is ZFS with Ceph a good idea?
[5:42] <wq32> It seems the benefits of ZFS are actually provided partially already by using Ceph, in my case data integrity
[5:42] <wq32> My concern is losing data.
[5:43] <wq32> And Ceph sounds like a great idea if it can store data in 3 places. 2 separate machines + 1 another one in another datacenter
[5:43] <lurbs> XFS is the current recommended underlying filesystem. Ceph itself handles the redundancy and periodic consistency checking.
[5:44] <wq32> Is XFS suggested even over BTRFS?
[5:45] <wq32> I guess since CentOS 7 changed to XFS it makes more sense
[5:47] <wq32> Are there any commercial services based on Ceph which I can use, test or trial?
[5:47] <wq32> I'm wondering if this is just paper project or its actually running on production
[5:47] <lurbs> http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/#filesystems
[5:47] <wq32> The concept is just go great to be true
[5:48] <Nats_> yahoo is known to run their flickr photo service ontop of ceph
[5:48] <lurbs> CERN also use it heavily.
[5:48] <wq32> I was watching the videos on Youtube, and they said Xen was coming
[5:48] <wq32> Is Xen already supported?
[5:48] <Nats_> and no shortage of openstack users using ceph for block storage
[5:48] <lurbs> wq32: http://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store-object-storage-at
[5:49] <lurbs> Yeah, our use case is for OpenStack.
[5:49] <wq32> Ok, that sounds great. I will check that.
[5:49] <Nats_> it works better with kvm than xen
[5:50] <wq32> Yes, but the presentations from Ross Turk
[5:50] <wq32> Are 2 years old
[5:50] <wq32> In 2 different ones, he said Xen support is coming
[5:50] <wq32> Has this changed?
[5:50] <wq32> I mean, is it supported already now in 2015?
[5:50] <wq32> I run heavily on XenServer
[5:50] <lurbs> No idea, we use KVM.
[5:51] <wq32> KVM is nice, but I made the choice years back and KVM was not mature enough
[5:51] <Nats_> you need to use 'native' qemu
[5:51] * bobrik (~bobrik@83.243.64.45) Quit (Read error: Connection reset by peer)
[5:51] <wq32> Otherwise today I would probably have gone with KVM
[5:51] * bobrik (~bobrik@83.243.64.45) has joined #ceph
[5:51] <Nats_> as opposed to xen's traditional qemu fork.
[5:52] <lurbs> You could always use the kernel RBD client, but there are a few caveats there. Deadlocks if the machine accessing the RBD is also part of the cluster, and you need a reasonably recent kernel in order to get access to some RBD features.
[5:52] <Nats_> last time i looked into it. we run a mixture of kvm and xen in production
[5:53] <Nats_> similar story as you, started with xen years ago when kvm was still immature
[5:54] <lurbs> CVE-2015-3456 says it still is immature. ;)
[5:54] <wq32> I read somewhere reads/writes to small files is horrible
[5:54] <wq32> Storing VM disks would work
[5:54] <wq32> But if this is true how does Yahoo do it?
[5:55] <wq32> Or websites running from Ceph would be a disaster is this is true
[5:55] <wq32> Not sure if I read it or saw in some video, that small files where actually bad
[5:55] <wq32> Less than 4 MB.
[5:56] <wq32> I thought CVE-2015-3456 was patched
[5:56] <wq32> I remember installing the patches
[5:56] <lurbs> It is.
[5:56] <wq32> At least Xenserver released the patches
[5:56] <Nats_> CVE-2015-3456 affected HVM xen and kvm equally
[5:57] * fam_away is now known as fam
[5:57] <wq32> Yes, very nasty. I was expecting this for years.
[5:57] <wq32> As more environments are virtualized. They are going to be bigger attack vectors.
[6:02] <wq32> So what would be the best approach on hosting small files, for example a web app or website on a Ceph cluster?
[6:02] <wq32> On a VM and the virtual disk on Ceph?
[6:02] <wq32> Because if its true about the small files...
[6:03] <wq32> What about storing Dockers
[6:04] <Nats_> 'horrible' is probably just referring to the individual request latency
[6:04] <wq32> My main interest is object storage, like images, backups, others.
[6:04] <wq32> But if the cluster is up, why not use it as well for other uses.
[6:04] <Nats_> being milliseconds instead of microseconds
[6:04] * bgleb (~bgleb@94.19.146.224) has joined #ceph
[6:04] <wq32> Yes, but that is expected with any cluster solution
[6:05] <wq32> Including database clusters, or storage, what ever
[6:05] <wq32> Latency and performance will never match direct drive access on the same host
[6:05] <wq32> I expect that
[6:05] <wq32> But my question is if its acceptable
[6:05] <wq32> Because I tried to run a Wordpress site once (as testing) on NFS
[6:05] <wq32> And while it worked
[6:05] <wq32> Its slow
[6:06] <wq32> Was not happy. File operations where very slow.
[6:06] <wq32> A file management systems in PHP gave tons of errors because of how much latency there was
[6:06] <Nats_> in terms of VM block-storage, i can confirm its acceptable
[6:06] <wq32> And that was one test app alone.
[6:07] <wq32> Did you tied using the Ceph own file system instead?
[6:07] <wq32> Yes, objects would be awful I think
[6:07] <wq32> It would be block or file system
[6:07] <wq32> Not sure which one is a better fit for smaller files
[6:08] <wq32> In particular reads
[6:08] <wq32> If you get 100,000 hits per minute, that is tons of small reads, unless its cached in memory
[6:09] * MentalRay (~MRay@107.171.161.165) has joined #ceph
[6:10] <lurbs> That's basically it. Something that heavily hit really should be in disk cache.
[6:10] * sleinen1 (~Adium@2001:620:0:82::101) has joined #ceph
[6:11] <wq32> Using SSD's or can Ceph use Ram as cache for frequent access?
[6:12] <m0zes> ceph will used the servers file cache in memory.
[6:12] <m0zes> s/file/page/
[6:12] <wq32> 1 GB of RAM required per 1 TB correct?
[6:12] <wq32> Per the reading I did.
[6:12] <m0zes> it is recommended you have that at least. in general, more is better ;)
[6:13] <wq32> Sure, but if I build a small enviroment
[6:13] <wq32> Lets say 10 TB
[6:13] <wq32> I assume with 16 GB it should work
[6:13] <wq32> 16 GB ram I mean with 1 x 1 TB
[6:13] <wq32> I can't go big for testing
[6:14] <Nats_> if you aren't in production then you can run less
[6:14] <wq32> As there is no point. Rather test small and scale up if I like what I see
[6:14] <wq32> Sure, but my testings need to test some real production data
[6:14] <wq32> There is no point in testing something that will not work once its live
[6:15] <wq32> I would need to test it with small enviroments that replicate small real world usage
[6:15] <wq32> Like several users copying data (backup)
[6:15] <wq32> Or reading gold templates for a few VM's
[6:16] <wq32> Or just hosting a few Wordpress or Drupal websites and faking hits and visitors to them to test if it crashes
[6:16] <Nats_> then the recommendation of 1GB/TB is fine
[6:16] <wq32> This is why I asked my first question. Objects would be great for backup, I don't care about performance here, it can be as slow as it can
[6:16] <wq32> For VM's hosting virtual machines. I will not try that yet. For that I would like an expensive setup.
[6:17] <wq32> But for webapps
[6:17] <wq32> I'm not sure
[6:17] <wq32> Biggest performance hit would be hosting real VM on the clusteer
[6:17] <wq32> But that is very intensive, regardless of which storage is used.
[6:18] <wq32> I use Raid 10 for this VM with cache and expensive raid cards
[6:18] <wq32> As well nice drives
[6:18] <wq32> And never had I/O problems
[6:18] <wq32> Well, maybe one or two times it crashes, but it was probably a Xen issue, or some bug in the hardware raid card, because it was not rebooted for months
[6:19] * amote (~amote@121.244.87.116) has joined #ceph
[6:19] <wq32> I'm afraid of proposing something like this and when I move some VM's, they are slow as hell or even crash
[6:19] <wq32> So I would rather prefer to test it out with object storage. Documents, images, backup files, compressed file....
[6:20] <wq32> If that works, test hosting some web based. Not database of course, just the web front end. Mostly PHP
[6:20] <m0zes> rbd and object storage are both very stable. the cephfs (posix network mounted filesytem) stuff is *probably* safe enough, but it isn't as mature.
[6:20] <wq32> So the web front end running from the Ceph cluster, if that works I will jump right away.
[6:21] <wq32> If that works, everythingt else should also work without issues.
[6:21] * m0zes is deploying cephfs as we speak... or at least migrating data into it.
[6:21] <wq32> What is posix supposed to replace in the future?
[6:21] * bgleb (~bgleb@94.19.146.224) Quit (Ping timeout: 480 seconds)
[6:21] <wq32> Yes, I read it was under development, but what is the idea behind its own file system?
[6:22] <m0zes> nfs, glusterfs, perhaps hpc workloads (lustre).
[6:22] <wq32> To replace NFS?
[6:22] <wq32> You mean mounting the Ceph file system directly on a machine as shared mount?
[6:22] <m0zes> yep.
[6:23] <wq32> How in the hell would that even work?
[6:23] <wq32> Most operating systems will not recognize it.
[6:23] <m0zes> there are clients for windows, osx and linux. linux has both fuse and kernel space implementations
[6:23] <wq32> If it has to be mounted directly in the OS without the gateway
[6:24] <wq32> But is it implemented in the linux kernel already?
[6:24] <m0zes> yep.
[6:24] <wq32> Well I hate NFS
[6:24] <wq32> That would be lovely if I can replace it.
[6:24] <wq32> I hate the locking and permissions issues
[6:24] <wq32> So block works fine right now. I guess I will try that.
[6:25] <wq32> The downside with block is latency sensitive.
[6:25] <wq32> With NFS you can have a bit of latency or network peaks and it will still work
[6:25] <wq32> Block storage is awful over longer distances
[6:26] <wq32> As long as they are very close in the same rack or datacenter, block works great
[6:26] <m0zes> I'm pushing 9000 iops out of 1 vm right now. not stressing anything.
[6:26] <wq32> This is something I can't figure out yet now it works with Ceph
[6:26] <wq32> If its distributing blocks in different drives in different servers.
[6:26] <wq32> I cannot imagine until I see it
[6:26] <wq32> The performance must be just very slow.
[6:27] <wq32> This is why I need to play with it and see and believe
[6:28] <wq32> Well, from what I read, and this is one of the things I love the most. If that there is no bottle necks in the systems.
[6:28] <wq32> If you have performance problems, you could add one more node, and another one and another one
[6:28] <wq32> From what I understand clients connect directly
[6:28] <wq32> They are not on a central gateway or system which could be the bottle neck
[6:29] <wq32> If this is true, I don't even care if the performance is not that great, because it should be solved by distributing workload among more nodes.
[6:29] <wq32> Is this correct?
[6:30] <wq32> Lets say your VM 9000 iops can't do anymore
[6:30] <wq32> You need another 5000 iops for another app or data device
[6:30] <m0zes> yep. there is no gateway. there are "monitors" that verify disks/hosts are behaving, and they can kick nodes out if they die.
[6:30] <wq32> You could add in this case another node or VM in you case to increase the iops?
[6:30] <wq32> Yes that part is clear, the nodes talk to each and see when one is out
[6:31] <m0zes> yes, you could easily add another node and distribute the load more.
[6:31] <wq32> But I'm talking about data access/reads/writes from end devices to their own storage
[6:31] <wq32> Does ceph do this, or you have to move data around to distribute the load?
[6:32] <wq32> The big problem, even with SAN is bandwidth. Even if you have 10Ge ports, and several ones, eventually you will scale out enough to kill them.
[6:32] <wq32> If I read correctly, Ceph does not has this problem
[6:32] <wq32> You just keep adding nodes
[6:32] <lurbs> I believe that there's work being done on data localisation, but currently all reads come from the master replica.
[6:32] <m0zes> when you add the new host, you add it to a crushmap, that tells ceph where in the hierachy the new host fits. then ceph will move data to new "proper" locations, and the load will get spread more.
[6:32] <wq32> And each one has its own performance (ram, storage, network card, etc)
[6:33] <wq32> Well that sounds great.
[6:33] <wq32> So this is actually true?
[6:33] <wq32> This sounds similar to S3
[6:33] <wq32> In someway
[6:34] <wq32> Amazon stores data in at least 2 servers in the same datacenter and 1 server in another facility
[6:34] <wq32> This sounds so similar to what Ceph can do
[6:34] <wq32> For data backup this is like magick, you could burn the whole DC and data would still be safe in another part.
[6:35] * aj__ (~aj@x590d8fe0.dyn.telefonica.de) Quit (Ping timeout: 480 seconds)
[6:35] <wq32> If they add compression and deduplication they will take over the planet
[6:35] <m0zes> yep, you can organize your data like that in a crushmap. you can spread it between machines, racks, rows, dcs, disk controllers, whatever you can think of. you should be able to map.
[6:36] <lurbs> We haven't needed either of those things, we're bound by IOPS, not capacity.
[6:36] <wq32> I researched many solutions, but most of them are similar to a big raid array. All of them had a weak link somewhere.
[6:36] <m0zes> btrfs has the compression (and maybe dedupe?) but isn't recommended for production/mission critical workloads yet.
[6:37] <wq32> Can this be done with data already saved in the cluster? Or this is one time change and cannot be modified once its live?
[6:37] <wq32> I mean the configuration on how you want data spread.
[6:37] <m0zes> yes. you can change it live. ceph does this all the time, as disks fail. as machines lose connectivity. whatever.
[6:38] <wq32> How many nodes would I need to test this?
[6:38] <wq32> 3?
[6:38] <m0zes> ideally you would start with 3. I've seen people do it in 1, but it gets hard to test some things.
[6:38] <lurbs> Three's really the minimum, yeah.
[6:39] <wq32> But if 2 fail, data is gone correct?
[6:39] <lurbs> Depends on your configuration.
[6:39] <m0zes> if I have 4 hosts, and have ceph set to keep 3 copies of the data, if a machine dies it should re-replicate the "missing" copy to guarantee you have 3 total copies of your data on machines that are "up"
[6:39] <lurbs> Default is 'size' of 3, so three copies, and only one copy per machine.
[6:40] <lurbs> Configurable per pool, which is just a logical construct/namespace.
[6:40] <m0zes> data wouldn't be "gone" writes would normally (by default) stop
[6:40] <wq32> But data would be triple correct?
[6:40] <wq32> If you save a 500 GB backup, it would actually be 500 GB x 3?
[6:40] <lurbs> Yes.
[6:40] <wq32> Ouch :(
[6:41] <wq32> Well it makes sense
[6:41] <m0zes> if you don't like that you can look at erasure coding. but you'll need a replicated pool in front of it to act as a caching tier.
[6:41] <wq32> So with 10 nodes, and 3 copies, it would still be just 1500 GB in total
[6:42] <wq32> As long as not all 3 of those nodes fail I'm save.
[6:42] <lurbs> Although you can do erasure coded pools too. k data: m coding chunks, similar to RAID 5 etc.
[6:42] <m0zes> erasure coded pools *are* slower, but you can get data integrity guarantees with fewer complete "copies"
[6:43] <wq32> 1 failing happens, 2 at the same time, is very bad luck, but 3 is extremely unlikely. Data protection policies always say 3 places besides the original
[6:43] <wq32> No, I dislike Raid 5 and never used it
[6:43] <wq32> Only use Raid 10
[6:43] <wq32> Many horror stories with raid5
[6:44] * m0zes has a k=8 m=4 erasure coded pool, behind a replica 3 cache tier. when data gets flushed to the backend I use 1.5x the space to guarantee 4 simultaneous failures won't take me out.
[6:44] <wq32> You don't lose as much data as with raid 10 but has its problems.
[6:44] <lurbs> Yeah, although Ceph allows you to work around a lot of those limitations, as you can choose k and m.
[6:45] <Nats_> m0zes, whats your experience with erasure code pool been?
[6:45] * MentalRay (~MRay@107.171.161.165) Quit (Quit: This computer has gone to sleep)
[6:45] <m0zes> that and ceph actually *knows* about the data that has been written. you aren't re-calculating hashes for entire disks.
[6:45] <wq32> The chances of 3 simultaneous failures are almost impossible. The only thing that could erase data that way is a software glitch. This mean ceph deleting it by mistake or failure.
[6:46] <m0zes> Nats_: not too bad, so far. it is *slow*, but it was one of two pools that didn't fry when some *extremely* bad luck hit a few weeks ago.
[6:46] <wq32> So big question. What is the network activity on ceph when nodes are on iddle? If nodes are talking to each other all the time there must be some network data.
[6:47] <wq32> I assume the nodes are not sync data all the time.
[6:47] <wq32> But just when they need too (writes)
[6:47] <Nats_> m0zes, yeah perf is my big concern given ceph isnt overly fast to begin with
[6:47] <m0zes> they have heartbeats to the monitors.
[6:47] <m0zes> Nats_: I just put a fairly large cache tier in front of it. currently 80TB.
[6:48] <Nats_> m0zes, no interesting failure cases for you though?
[6:48] <wq32> 80 TB cache? How did you managed to build that?
[6:49] <m0zes> Nats_: nope, data has been consistent. I had ceph fail to activate a couple pgs once when a single stripe was missing. I've not chased that one down yet, though.
[6:49] <m0zes> wq32: 24x 86TB nodes.
[6:50] <wq32> SSD?
[6:50] <Nats_> m0zes, thanks for that. i've been eying it off for a while but haven't made the leap given the relative newness vs replicated
[6:50] <wq32> Will SSD actually make any differecen with Ceph at all over spinning drives?
[6:51] <wq32> Because data over the line will still be so much slower anyway
[6:51] <m0zes> wq32: they will *if* you have 1) good ssds 2) fast cpus 3) extremely low latency network.
[6:51] <wq32> I read somewhere it was a waste.
[6:52] <wq32> But SSD are just to fast to keep up with data flowing in/out over the network
[6:52] <m0zes> currently the recommendation is to use SSDs for the ceph journals if iops are a concern and you can't afford the above three.
[6:52] <Nats_> you have to keep in mind you are optimizing for 1,000 (or whatever) concurrent users, not 1
[6:53] <m0zes> wq32: 2 ssds per host for journals/OS, 12x 6TB spinners and 4x 4TB spinners per host.
[6:53] <m0zes> my mfg hadn't certified the 6TB drives for internal use when we purchased earlier this yeat.
[6:54] <lurbs> You haven't had any issues with the OS and journals on the same drives?
[6:54] <wq32> 1 gigabit cards will still only push 125 Megabytes per second.
[6:54] <wq32> Even if you have 4 pushing at the same time, the SSD is still so much faster.
[6:54] <wq32> It does sound like a waste
[6:55] <m0zes> not because of OS and journals. mostly from the ssds themselves. apparently my boss decided to get "read optimized" ssds.
[6:55] <wq32> Ok, but they are for storage then
[6:56] <m0zes> wq32: 1gb has a much higher latency than 10 or 40gb, you generally aren't worried about the actual speed of the network, so much as the latency.
[6:56] <wq32> But for journals
[6:56] <m0zes> again ssds are for iops, not *necessarily* mbps.
[6:56] <Nats_> and beyond that you care more about iops than throughput
[6:56] <wq32> Assuming you are even pushing that without overhead.
[6:57] <wq32> But SSD for storage is so expensive
[6:57] <wq32> That is out of my budget
[6:57] <wq32> They are like 400$ per drive
[6:58] <wq32> http://www.amazon.com/Samsung-MZ-7GE960EW-845DC-2-5-INCH-CENTER/dp/B00LX2QUPI/ref=sr_1_2?ie=UTF8&qid=1435553881&sr=8-2&keywords=samsung+ssd+data+center
[6:58] <m0zes> you don't need particularly large ones for journaling. 100-200 GB ones are more than enough for most setups.
[6:59] <m0zes> iirc, there were some concerns bout the samsung ssds for ceph. most around here recommend the intel s3700 series.
[6:59] <wq32> Yes, I just put the link here
[6:59] <wq32> I actually use Intel Datacenters ones
[7:00] <wq32> What concerns? The samsungs, at least for consumers are great, Pro and Evo beat Intel
[7:00] <wq32> Not sure why their datacenter ones would be different, just with more endurance so more expensive.
[7:01] <wq32> Believe it or not, I still like spinning drive.
[7:01] <wq32> Some go on for years.
[7:01] <m0zes> something about inconsistent iops, or just latency per transaction when doing O_DSYNC writes.
[7:02] <m0zes> anyway, it is past my bed time. gotta work in the morning.
[7:02] <wq32> Thanks
[7:02] <lurbs> m0zes: Yeah, we're using the S3700.
[7:02] <wq32> I'm gone as well, I have to go to sleep as well
[7:02] <wq32> 1 AM already
[7:02] * m0zes would say avoid liteon like the plague.
[7:02] <wq32> See you guys
[7:02] <wq32> Thanks for the answers
[7:03] <wq32> I still have tons of reading to do...
[7:03] <wq32> As it's a bit over my head. Still I think I will manage to build something.
[7:04] <wq32> Bye :)
[7:04] * wq32 (ba11b0ae@107.161.19.109) has left #ceph
[7:05] <lurbs> I don't know what that m0zes guy was talking about, it's only 5pm.
[7:07] * rdas (~rdas@121.244.87.116) has joined #ceph
[7:27] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Quit: Leaving.)
[7:28] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[7:36] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[7:43] * aj__ (~aj@88.128.80.79) has joined #ceph
[7:47] * shohn (~shohn@dslb-188-102-036-251.188.102.pools.vodafone-ip.de) has joined #ceph
[7:47] <shohn> ping vimal
[7:48] * MrAbaddon (~MrAbaddon@a89-155-97-112.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[7:48] * kefu (~kefu@114.92.113.55) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[7:50] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[7:55] * yghannam (~yghannam@0001f8aa.user.oftc.net) has joined #ceph
[7:56] * sleinen (~Adium@2001:620:0:82::104) has joined #ceph
[8:02] * mykola (~Mikolaj@91.225.202.121) has joined #ceph
[8:03] * bgleb (~bgleb@94.19.146.224) has joined #ceph
[8:03] * rotbeard (~redbeard@2a02:908:df10:d300:76f0:6dff:fe3b:994d) Quit (Quit: Verlassend)
[8:05] * kefu (~kefu@114.92.113.55) has joined #ceph
[8:06] * doppelgrau (~doppelgra@pd956d116.dip0.t-ipconnect.de) has joined #ceph
[8:10] * shylesh (~shylesh@121.244.87.124) has joined #ceph
[8:10] * dlan (~dennis@116.228.88.131) Quit (Ping timeout: 480 seconds)
[8:12] * bgleb (~bgleb@94.19.146.224) Quit (Ping timeout: 480 seconds)
[8:14] * sleinen (~Adium@2001:620:0:82::104) Quit (Ping timeout: 480 seconds)
[8:18] * karnan (~karnan@121.244.87.117) has joined #ceph
[8:21] * sleinen (~Adium@2001:620:0:82::106) has joined #ceph
[8:24] * oro (~oro@84-72-180-62.dclient.hispeed.ch) has joined #ceph
[8:26] * TMM (~hp@178-84-46-106.dynamic.upc.nl) Quit (Quit: Ex-Chat)
[8:26] * overclk (~overclk@121.244.87.117) has joined #ceph
[8:27] * overclk (~overclk@121.244.87.117) Quit ()
[8:28] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[8:31] * dopesong_ (~dopesong@lb1.mailer.data.lt) has joined #ceph
[8:32] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) Quit (Quit: Leaving.)
[8:32] * Hemanth (~Hemanth@121.244.87.117) has joined #ceph
[8:35] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[8:35] * Sysadmin88 (~IceChat77@2.124.164.69) Quit (Quit: Not that there is anything wrong with that)
[8:36] * sleinen (~Adium@2001:620:0:82::106) Quit (Read error: Connection reset by peer)
[8:39] * Nacer (~Nacer@2001:41d0:fe82:7200:bd53:b9fd:328c:54c8) Quit (Remote host closed the connection)
[8:39] * aj__ (~aj@88.128.80.79) Quit (Ping timeout: 480 seconds)
[8:46] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[8:52] * sleinen1 (~Adium@2001:620:0:82::101) has joined #ceph
[8:55] * rdas (~rdas@121.244.87.116) has joined #ceph
[8:59] * oro (~oro@84-72-180-62.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[9:04] * sleinen1 (~Adium@2001:620:0:82::101) Quit (Ping timeout: 480 seconds)
[9:05] * sleinen1 (~Adium@2001:620:0:69::103) has joined #ceph
[9:06] * chasmo77 (~chas77@158.183-62-69.ftth.swbr.surewest.net) has joined #ceph
[9:10] * nardial (~ls@dslb-088-076-092-027.088.076.pools.vodafone-ip.de) has joined #ceph
[9:10] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) has joined #ceph
[9:12] * aj__ (~aj@fw.gkh-setu.de) has joined #ceph
[9:13] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[9:13] * tmrz (~quassel@198-84-192-38.cpe.teksavvy.com) Quit (Read error: Connection reset by peer)
[9:13] * tmrz (~quassel@198-84-192-38.cpe.teksavvy.com) has joined #ceph
[9:16] * tmrz_ (~quassel@198-84-192-38.cpe.teksavvy.com) has joined #ceph
[9:17] * rotbeard (~redbeard@185.32.80.238) has joined #ceph
[9:17] * dgurtner (~dgurtner@178.197.231.10) has joined #ceph
[9:18] * codice (~toodles@97-94-175-73.static.mtpk.ca.charter.com) Quit (Quit: leaving)
[9:22] * tmrz (~quassel@198-84-192-38.cpe.teksavvy.com) Quit (Ping timeout: 480 seconds)
[9:23] * DV (~veillard@2001:41d0:a:f29f::1) Quit (Remote host closed the connection)
[9:30] * linjan (~linjan@80.179.241.26) has joined #ceph
[9:30] * Kupo1 (~Kupo@ip72-200-70-110.tc.ph.cox.net) has joined #ceph
[9:30] <Kupo1> https://git.ceph.com/ down?
[9:30] * cok (~chk@2a02:2350:18:1010:e0ad:6349:a2d5:fd61) has joined #ceph
[9:31] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) has joined #ceph
[9:33] * codice (~toodles@97-94-175-73.static.mtpk.ca.charter.com) has joined #ceph
[9:38] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[9:40] * bitserker (~toni@63.pool85-52-240.static.orange.es) has joined #ceph
[9:41] * DV (~veillard@2001:41d0:1:d478::1) has joined #ceph
[9:43] * fsimonce (~simon@95.238.70.69) has joined #ceph
[9:45] * ade (~abradshaw@tmo-111-63.customers.d1-online.com) has joined #ceph
[9:59] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[10:00] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[10:04] * mattch (~mattch@pcw3047.see.ed.ac.uk) has joined #ceph
[10:05] * linjan (~linjan@80.179.241.26) Quit (Ping timeout: 480 seconds)
[10:05] * frednass (~fred@dn-infra-12.lionnois.univ-lorraine.fr) has left #ceph
[10:06] * dis (~dis@109.110.66.238) has joined #ceph
[10:07] * thomnico_ (~thomnico@2a01:e35:8b41:120:1018:5794:10ec:30df) has joined #ceph
[10:10] * thomnico (~thomnico@cro38-2-88-180-16-18.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[10:11] * oro (~oro@2001:620:20:16:cd3b:3fa3:6b41:4190) has joined #ceph
[10:12] * ksperis (~ksperis@46.218.42.103) has joined #ceph
[10:14] * Concubidated (~Adium@71.21.5.251) Quit (Quit: Leaving.)
[10:20] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) has joined #ceph
[10:24] * kapil (~ksharma@2620:113:80c0:5::2222) has joined #ceph
[10:26] * frednass (~fred@dn-infra-12.lionnois.univ-lorraine.fr) has joined #ceph
[10:27] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) has joined #ceph
[10:27] <Be-El> hi
[10:31] * TMM (~hp@sams-office-nat.tomtomgroup.com) has joined #ceph
[10:34] * daviddcc (~dcasier@ADijon-653-1-192-159.w90-39.abo.wanadoo.fr) has joined #ceph
[10:35] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) Quit (Quit: Leaving.)
[10:37] * ingslovak (~peto@178-143-108-176.dynamic.orange.sk) has joined #ceph
[10:48] * fdmanana__ (~fdmanana@bl14-143-223.dsl.telepac.pt) has joined #ceph
[10:55] * thomnico_ (~thomnico@2a01:e35:8b41:120:1018:5794:10ec:30df) Quit (Ping timeout: 480 seconds)
[10:57] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) has joined #ceph
[10:58] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) Quit (Remote host closed the connection)
[11:00] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) has joined #ceph
[11:05] * MrAbaddon (~MrAbaddon@a89-154-187-169.cpe.netcabo.pt) has joined #ceph
[11:13] * branto (~branto@178.253.154.131) has joined #ceph
[11:20] <bgleb> Hi! It seems that we faced with http://tracker.ceph.com/issues/11604 and now a lot of objects are corrupted in our buckets (total of 700k objects, 20TiB of data). Is there any way to detect such objects?
[11:28] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[11:30] * linjan (~linjan@195.110.41.9) has joined #ceph
[11:31] * Kureysli (~abbasi@88.244.114.163) has joined #ceph
[11:31] * Kureysli (~abbasi@88.244.114.163) Quit (autokilled: Do not spam other people. Mail support@oftc.net if you feel this is in error. (2015-06-29 09:31:29))
[11:35] * linjan (~linjan@195.110.41.9) Quit (Remote host closed the connection)
[11:36] * ramonskie (ab15507e@107.161.19.109) has joined #ceph
[11:37] * linjan (~linjan@195.110.41.9) has joined #ceph
[11:40] <ramonskie> anyone here :( i messed up my mon again when trying to add a monitor again
[11:41] <ramonskie> i did a backup of my mon direcotry before i did anything
[11:41] <ramonskie> and now i restored my backup but it is still probing the failed mon
[11:42] <ramonskie> and it thinks that the failed mon is leading somehow
[11:42] * jcsp (~Adium@0001bf3a.user.oftc.net) has joined #ceph
[11:50] * owasserm (~owasserm@52D9864F.cm-11-1c.dynamic.ziggo.nl) has joined #ceph
[11:51] <ramonskie> okay don't know but after serveral times of removing and restoring the data. it does not probe and elect the failed mon
[11:55] * dud225 (~hwerner@92.103.57.251) has joined #ceph
[11:56] * sleinen1 (~Adium@2001:620:0:69::103) Quit (Ping timeout: 480 seconds)
[12:10] * Debesis (~0x@213.119.116.89.mobile.mezon.lt) has joined #ceph
[12:15] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[12:27] * Lattyware (~Skyrider@79.142.70.10) has joined #ceph
[12:29] * cok (~chk@2a02:2350:18:1010:e0ad:6349:a2d5:fd61) Quit (Quit: Leaving.)
[12:29] * ramonskie (ab15507e@107.161.19.109) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[12:33] * shyu (~Shanzhi@119.254.120.66) Quit (Remote host closed the connection)
[12:39] * sep (~sep@2a04:2740:1:0:52e5:49ff:feeb:32) has joined #ceph
[12:40] * ngoswami (~ngoswami@121.244.87.116) has joined #ceph
[12:41] <bgleb> we're running investigation on corrupted data in storage and found that one of uploaded file has such rules in manifest: parts 0-1169 with part size 10m, part 1170 with part size 8m and override_prefix, parts 1171-1226 with part size 10m, part 1127 with size 8m and override_prefix and rest of parts has 10m size (except last one, which is 4m part).
[12:42] <bgleb> We've uploaded this object using aws s3 java sdk with multipart upload and retry policy. Default chunk size was 10m. Is it okay that we have parts with prefix override AND smaller size?
[12:43] <bgleb> I'm dumping all object keys related to this object from rados (rados ls --pool=.rgw.buckets | grep OBJID) and going to check which objects are in store.
[12:44] * daviddcc (~dcasier@ADijon-653-1-192-159.w90-39.abo.wanadoo.fr) Quit (Ping timeout: 480 seconds)
[12:46] <bgleb> Done. We have several classes of objects: first of all __shadow object (seems that it's caused by bug in rgw which prevents GC). 2376 of them ends with _1 and 2373 ends with _2.
[12:46] * rdas (~rdas@121.244.87.116) Quit (Quit: Leaving)
[12:48] * nardial (~ls@dslb-088-076-092-027.088.076.pools.vodafone-ip.de) Quit (Remote host closed the connection)
[12:48] <bgleb> next goes __multipart: 2374 total, 2372 of them contains .2~SOMEID after object key and two of them (parts 1170, 1227 which I listed above in prefix override) doesn't contains 2~, but ends with different SOMEID_PARTNO.
[12:51] <bgleb> I found several related issues in tracker: #11604, #11622, #11749
[12:52] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[12:55] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[12:56] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) Quit (Remote host closed the connection)
[12:56] * Lattyware (~Skyrider@9S0AABOCT.tor-irc.dnsbl.oftc.net) Quit ()
[12:58] * sleinen1 (~Adium@2001:620:0:69::100) has joined #ceph
[12:59] * sankarshan (~sankarsha@183.87.39.242) Quit (Ping timeout: 480 seconds)
[13:00] * shang (~ShangWu@175.41.48.77) Quit (Ping timeout: 480 seconds)
[13:05] * kefu (~kefu@114.92.113.55) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[13:05] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[13:10] * ira (~ira@c-71-233-225-22.hsd1.ma.comcast.net) has joined #ceph
[13:15] * kefu (~kefu@114.92.113.55) has joined #ceph
[13:17] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[13:18] * badone__ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) Quit (Ping timeout: 480 seconds)
[13:18] * kefu (~kefu@114.92.113.55) has joined #ceph
[13:19] * fam is now known as fam_away
[13:24] * Debesis (~0x@213.119.116.89.mobile.mezon.lt) Quit (Quit: Leaving)
[13:26] <joel> any simple libvirt based ceph deployment.. don't need a full openstack/opennebula etc.. just some simple scripts to help managed the storage pool and add ceph stuff
[13:26] * cok (~chk@nat-cph5-sys.net.one.com) has joined #ceph
[13:31] * Debesis (~0x@19.166.116.89.mobile.mezon.lt) has joined #ceph
[13:32] * zhaochao (~zhaochao@111.161.77.233) Quit (Quit: ChatZilla 0.9.91.1 [Iceweasel 38.0.1/20150526223604])
[13:37] * karnan (~karnan@121.244.87.117) Quit (Remote host closed the connection)
[13:38] <boolman> http://paste.ubuntu.com/11792636/ if my replication size is set 3 using ruleset 1, do I understand it correctly. It will coose 2 racks, and 3 hosts total ?
[13:39] <boolman> or does chooseleaf firstn 0 type host, means that it will select 3 hosts in both racks ?
[13:39] * cok (~chk@nat-cph5-sys.net.one.com) has left #ceph
[13:44] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Remote host closed the connection)
[13:53] * ganders (~root@190.2.42.21) has joined #ceph
[13:53] * i_m (~ivan.miro@mail.iicmos.ru) has joined #ceph
[13:56] * a1-away (~jelle@62.27.85.48) has joined #ceph
[13:56] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) Quit (Remote host closed the connection)
[14:01] * i_m (~ivan.miro@mail.iicmos.ru) Quit (Ping timeout: 480 seconds)
[14:03] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) has joined #ceph
[14:10] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[14:16] <Be-El> does the rados gateway keystone user authentication integration support keystone domains?
[14:17] * i_m (~ivan.miro@deibp9eh1--blueice4n2.emea.ibm.com) has joined #ceph
[14:21] * bene (~ben@c-24-60-237-191.hsd1.nh.comcast.net) has joined #ceph
[14:28] * amote (~amote@121.244.87.116) Quit (Quit: Leaving)
[14:28] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[14:28] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) has joined #ceph
[14:29] * kefu (~kefu@114.92.113.55) has joined #ceph
[14:30] * kawa2014 (~kawa@89.184.114.246) Quit (Ping timeout: 480 seconds)
[14:30] * kawa2014 (~kawa@212.110.41.244) has joined #ceph
[14:31] * Doodlepieguy (~ChauffeR@tor-exit-node.7by7.de) has joined #ceph
[14:38] * An_T_oine (~Antoine@192.93.37.4) has joined #ceph
[14:40] * ramonskie (ab15507e@107.161.19.53) has joined #ceph
[14:44] * KevinPerks (~Adium@2606:a000:80ad:1300:554a:b46:6165:5c33) has joined #ceph
[14:45] * kefu (~kefu@114.92.113.55) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[14:49] * sankarshan (~sankarsha@183.87.39.242) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[14:57] * squ (~Thunderbi@46.109.36.167) Quit (Quit: squ)
[14:57] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) has joined #ceph
[15:01] * Doodlepieguy (~ChauffeR@7R2AAB37M.tor-irc.dnsbl.oftc.net) Quit ()
[15:01] * wschulze (~wschulze@cpe-69-206-240-164.nyc.res.rr.com) has joined #ceph
[15:02] * sankarshan (~sankarsha@183.87.39.242) has joined #ceph
[15:04] * georgem (~Adium@fwnat.oicr.on.ca) has joined #ceph
[15:04] <zenpac> Is there a Ceph API call that summarizes external mounts ?
[15:05] <ramonskie> i think i know what go's wrong everytime i tried to add a monitor with ceph-deploy, my first monitor was not deployed with ceph-deploy. i also noticed that mon.keyring file that was in the ceph-deploy directory was not the same as the ceph.mon keyring on my current monitor
[15:06] <ramonskie> can this be the issue that when i try to add another monitor that it wants to assimilate the current monitor to the new monitor state?
[15:08] <ramonskie> zenpac: i only know of the "ceph-disk list" command
[15:08] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) Quit (Remote host closed the connection)
[15:09] * kefu (~kefu@114.92.113.55) has joined #ceph
[15:09] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) has joined #ceph
[15:09] * tupper (~tcole@2001:420:2280:1272:8900:f9b8:3b49:567e) has joined #ceph
[15:10] * shaunm (~shaunm@74.215.76.114) has joined #ceph
[15:10] * linjan (~linjan@195.110.41.9) Quit (Read error: Connection reset by peer)
[15:10] * linjan (~linjan@195.110.41.9) has joined #ceph
[15:10] * linjan (~linjan@195.110.41.9) Quit (Read error: Connection reset by peer)
[15:11] * fam_away is now known as fam
[15:11] <ramonskie> and you can use ceph commands with ceph-rest-api package
[15:14] * thomnico (~thomnico@2a01:e35:8b41:120:c970:36cd:2552:7236) has joined #ceph
[15:16] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[15:16] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) has joined #ceph
[15:17] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) Quit (Ping timeout: 480 seconds)
[15:17] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[15:17] * brad_mssw (~brad@66.129.88.50) has joined #ceph
[15:18] * kefu (~kefu@114.92.113.55) has joined #ceph
[15:18] <zenpac> My pools on the Storage cluster look like : http://paste.debian.net/265589/ .. Am I allowed to create more pools?
[15:19] * dyasny (~dyasny@173.231.115.59) has joined #ceph
[15:19] <ramonskie> you can create as many pools as you want but be carefull because you need to consider the pg groups
[15:20] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) has joined #ceph
[15:20] <ramonskie> see http://ceph.com/pgcalc/
[15:21] * harold (~hamiller@71-94-227-123.dhcp.mdfd.or.charter.com) Quit (Quit: Leaving)
[15:22] <zenpac> All I have setup so far is defaults..
[15:25] <zenpac> So I've not tried to access any PG's.. My OSD's are formatted with XFS as the fs.
[15:25] * flisky (~Thunderbi@106.39.60.34) Quit (Quit: flisky)
[15:26] * yanzheng (~zhyan@182.139.205.179) Quit (Quit: This computer has gone to sleep)
[15:26] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) has joined #ceph
[15:27] * alram (~alram@50.95.199.50) has joined #ceph
[15:28] * yanzheng (~zhyan@182.139.205.179) has joined #ceph
[15:29] * kawa2014 (~kawa@212.110.41.244) Quit (Ping timeout: 480 seconds)
[15:34] * zenpac (~zenpac3@66.55.33.66) Quit (Ping timeout: 480 seconds)
[15:37] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[15:38] * squizzi (~squizzi@nat-pool-rdu-t.redhat.com) Quit (Remote host closed the connection)
[15:38] * kawa2014 (~kawa@89.184.114.246) has joined #ceph
[15:39] * kefu (~kefu@114.92.113.55) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[15:39] * cok (~chk@2a02:2350:18:1010:6d70:861b:5256:a1f6) has joined #ceph
[15:40] <georgem> Does anybody know how to change the "rgw object stripe size" from the default 4 MB?
[15:41] * kefu (~kefu@136.0.17.243) has joined #ceph
[15:41] * vata (~vata@cable-21.246.173-197.electronicbox.net) Quit (Quit: Leaving.)
[15:42] * oro (~oro@2001:620:20:16:cd3b:3fa3:6b41:4190) Quit (Remote host closed the connection)
[15:43] * rwheeler (~rwheeler@172.56.40.194) has joined #ceph
[15:44] * wicope (~wicope@0001fd8a.user.oftc.net) has joined #ceph
[15:44] * oro (~oro@2001:620:20:16:3891:440:2218:1366) has joined #ceph
[15:45] * zenpac (~zenpac3@66.55.33.66) has joined #ceph
[15:46] * kefu_ (~kefu@114.92.113.55) has joined #ceph
[15:46] <georgem> I added "rgw object stripe size = 8" in ceph.conf on the radosgw nodes (also Mon nodes), restarted the Mon and radosgw services but I don't think it's using this new value
[15:49] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) has joined #ceph
[15:49] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Ping timeout: 480 seconds)
[15:51] * kefu (~kefu@136.0.17.243) Quit (Ping timeout: 480 seconds)
[15:51] * kefu_ (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[15:52] * kefu (~kefu@114.92.113.55) has joined #ceph
[15:55] <georgem> any advice?
[15:56] * shylesh (~shylesh@121.244.87.124) Quit (Remote host closed the connection)
[15:57] * Hemanth (~Hemanth@121.244.87.117) Quit (Quit: Leaving)
[15:58] * cok (~chk@2a02:2350:18:1010:6d70:861b:5256:a1f6) has left #ceph
[15:58] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[16:05] * dyasny (~dyasny@173.231.115.59) Quit (Remote host closed the connection)
[16:05] * rektide_ (~rektide@eldergods.com) has joined #ceph
[16:05] * rektide (~rektide@eldergods.com) Quit (Read error: Connection reset by peer)
[16:07] * dyasny (~dyasny@173.231.115.59) has joined #ceph
[16:07] * thomnico (~thomnico@2a01:e35:8b41:120:c970:36cd:2552:7236) Quit (Ping timeout: 480 seconds)
[16:10] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[16:11] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[16:11] * kefu (~kefu@114.92.113.55) has joined #ceph
[16:12] * dopeson__ (~dopesong@lb1.mailer.data.lt) has joined #ceph
[16:13] * kefu (~kefu@114.92.113.55) Quit ()
[16:13] <ramonskie> georgem: maby use the tell command
[16:16] * dopesong_ (~dopesong@lb1.mailer.data.lt) Quit (Ping timeout: 480 seconds)
[16:16] * alram (~alram@50.95.199.50) Quit (Quit: leaving)
[16:17] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) Quit (Quit: Leaving.)
[16:17] * kefu (~kefu@114.92.113.55) has joined #ceph
[16:18] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[16:28] * ramonskie (ab15507e@107.161.19.53) Quit (Quit: http://www.kiwiirc.com/ - A hand crafted IRC client)
[16:29] * wushudoin (~wushudoin@38.140.108.2) has joined #ceph
[16:30] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[16:31] * danieagle (~Daniel@191.254.218.6) Quit (Quit: Obrigado por Tudo! :-) inte+ :-))
[16:32] * rlrevell (~leer@vbo1.inmotionhosting.com) Quit (Remote host closed the connection)
[16:32] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) has joined #ceph
[16:33] * yanzheng (~zhyan@182.139.205.179) Quit (Quit: This computer has gone to sleep)
[16:33] * zenpac (~zenpac3@66.55.33.66) Quit (Ping timeout: 480 seconds)
[16:37] * capri (~capri@212.218.127.222) Quit (Ping timeout: 480 seconds)
[16:39] * zenpac (~zenpac3@66.55.33.66) has joined #ceph
[16:40] * nardial (~ls@dslb-088-076-092-027.088.076.pools.vodafone-ip.de) has joined #ceph
[16:40] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) Quit (Quit: Leaving)
[16:42] * sleinen1 (~Adium@2001:620:0:69::100) Quit (Ping timeout: 480 seconds)
[16:42] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[16:43] * dopeson__ (~dopesong@lb1.mailer.data.lt) Quit (Read error: Connection reset by peer)
[16:44] * dopesong_ (~dopesong@lb1.mailer.data.lt) has joined #ceph
[16:45] * alram (~alram@192.41.52.12) has joined #ceph
[16:46] * wicope (~wicope@0001fd8a.user.oftc.net) Quit (Remote host closed the connection)
[16:47] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[16:47] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) has joined #ceph
[16:48] * kefu (~kefu@114.92.113.55) has joined #ceph
[16:50] * squizzi (~squizzi@nat-pool-rdu-t.redhat.com) has joined #ceph
[16:50] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[16:51] * rwheeler (~rwheeler@172.56.40.194) Quit (Quit: Leaving)
[16:53] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) has joined #ceph
[16:54] * debian112 (~bcolbert@24.126.201.64) has joined #ceph
[16:55] <zenpac> In Ceph-OpenStack integration, is there any use of the CEPH api's ? I don't see OpenStack using any Ceph API's in the setup.
[16:55] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[16:56] * kefu (~kefu@114.92.113.55) has joined #ceph
[16:58] * bgleb (~bgleb@2a02:6b8:0:2309:e8ec:e947:e1e:5d55) Quit (Remote host closed the connection)
[17:03] * thomnico (~thomnico@2a01:e35:8b41:120:c970:36cd:2552:7236) has joined #ceph
[17:06] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[17:15] * mtl (~Adium@c-67-174-109-212.hsd1.co.comcast.net) has joined #ceph
[17:19] <mtl> Hi. I think we found a pretty serious bug with rbd, snapshots, and clones. I'm curious to see if anyone else has seen it.
[17:19] <rkeene> Maybe... what is the problem ?
[17:19] <zenpac> when I do "ceph osd pool create xyz 128, does that allocate a fixe amount of space, or just assign PG's to it?
[17:19] * scuttle|afk is now known as scuttlemonkey
[17:20] * TMM (~hp@sams-office-nat.tomtomgroup.com) Quit (Quit: Ex-Chat)
[17:20] * snakamoto1 (~Adium@192.16.26.2) has joined #ceph
[17:20] <mtl> If we take a snapshot, let it sit for a few days, and then protect and make a clone from that snapshot, changes made to the parent image after the snapshot was created show up on the cloned image.
[17:20] <mtl> The hard part is it's only happening about 10% of the time.
[17:21] <rkeene> I don't have a similar usage pattern, so I haven't seen it
[17:21] <mtl> So far, we've only noticed it on clones that are made from another clone originally.
[17:22] <cholcombe> how does the placement group calculation change for erasure coded pools?
[17:24] <cholcombe> nvm i found it :)
[17:24] * sleinen (~Adium@194.230.159.171) has joined #ceph
[17:24] * gregmark (~Adium@68.87.42.115) has joined #ceph
[17:24] * nardial (~ls@dslb-088-076-092-027.088.076.pools.vodafone-ip.de) Quit (Quit: Leaving)
[17:25] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) Quit (Remote host closed the connection)
[17:26] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) has joined #ceph
[17:28] * bene is now known as bene_afk
[17:28] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) Quit (Remote host closed the connection)
[17:32] * sleinen (~Adium@194.230.159.171) Quit (Ping timeout: 480 seconds)
[17:32] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) has joined #ceph
[17:33] * mtl (~Adium@c-67-174-109-212.hsd1.co.comcast.net) has left #ceph
[17:34] * moore (~moore@64.202.160.88) has joined #ceph
[17:36] * oro (~oro@2001:620:20:16:3891:440:2218:1366) Quit (Remote host closed the connection)
[17:37] * dud225 (~hwerner@92.103.57.251) has left #ceph
[17:38] * sleinen (~Adium@2001:620:0:82::102) has joined #ceph
[17:39] <zenpac> I only have 104 PG's.. How many should I use to create a pool?
[17:40] * georgem1 (~Adium@fwnat.oicr.on.ca) has joined #ceph
[17:40] * georgem (~Adium@fwnat.oicr.on.ca) Quit (Read error: Connection reset by peer)
[17:41] * sjm (~sjm@CPE001b24bd8cd3-CM001bd7096236.cpe.net.cable.rogers.com) has joined #ceph
[17:44] * fam is now known as fam_away
[17:44] <zenpac> How do I set a value in ceph.conf across all nodes?
[17:47] * MentalRay (~MRay@107.171.161.165) has joined #ceph
[17:48] <zenpac> do i modify the local copy in ~/cluster1/ceph.conf and "ceph-deploy config push node1 node1 node3" ?
[17:48] * fred__ (~fred_@192.93.37.4) has joined #ceph
[17:52] * smerz (~ircircirc@37.74.194.90) Quit (Remote host closed the connection)
[17:55] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[17:57] * reed (~reed@75-101-54-131.dsl.static.fusionbroadband.com) has joined #ceph
[17:57] * lcurtis (~lcurtis@47.19.105.250) Quit ()
[17:57] * lcurtis (~lcurtis@47.19.105.250) has joined #ceph
[17:58] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) has joined #ceph
[18:01] * ade (~abradshaw@tmo-111-63.customers.d1-online.com) Quit (Ping timeout: 480 seconds)
[18:01] * dopeson__ (~dopesong@lb1.mailer.data.lt) has joined #ceph
[18:01] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[18:02] * kefu (~kefu@li413-226.members.linode.com) has joined #ceph
[18:03] * dopesong_ (~dopesong@lb1.mailer.data.lt) Quit (Ping timeout: 480 seconds)
[18:03] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[18:04] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has left #ceph
[18:07] * dopesong (~dopesong@88-119-94-55.static.zebra.lt) Quit (Ping timeout: 480 seconds)
[18:07] * Concubidated (~Adium@71.21.5.251) has joined #ceph
[18:08] <zenpac> How do I update or change my ceph.conf file using ceph-deploy?
[18:09] * MentalRay (~MRay@107.171.161.165) Quit (Quit: This computer has gone to sleep)
[18:09] * kefu (~kefu@li413-226.members.linode.com) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[18:11] <zenpac> Do I first update the file on mon and then the osd?
[18:11] * kefu (~kefu@114.92.113.55) has joined #ceph
[18:12] * sleinen (~Adium@2001:620:0:82::102) Quit (Read error: Connection reset by peer)
[18:15] * rwheeler (~rwheeler@66-194-8-225.static.twtelecom.net) has joined #ceph
[18:19] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[18:22] * jordanP (~jordan@scality-jouf-2-194.fib.nerim.net) Quit (Quit: Leaving)
[18:22] * linjan (~linjan@80.179.241.26) has joined #ceph
[18:23] * vata (~vata@208.88.110.46) has joined #ceph
[18:23] * Be-El (~quassel@fb08-bcf-pc01.computational.bio.uni-giessen.de) Quit (Remote host closed the connection)
[18:25] * yguang11 (~yguang11@2001:4998:effd:600:b4b5:2f86:981:3153) has joined #ceph
[18:26] * fred__ (~fred_@192.93.37.4) Quit (Quit: KVIrc 4.2.0 Equilibrium http://www.kvirc.net/)
[18:30] * dugravot6 (~dugravot6@dn-infra-04.lionnois.univ-lorraine.fr) Quit (Quit: Leaving.)
[18:34] * rwheeler_ (~rwheeler@66-194-8-225.static.twtelecom.net) has joined #ceph
[18:35] * ingslovak (~peto@178-143-108-176.dynamic.orange.sk) Quit (Quit: Leaving.)
[18:35] * vbellur (~vijay@c-71-198-3-24.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[18:37] * xarses (~xarses@c-73-202-191-48.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[18:38] * kawa2014 (~kawa@89.184.114.246) Quit (Quit: Leaving)
[18:38] * dgurtner (~dgurtner@178.197.231.10) Quit (Ping timeout: 480 seconds)
[18:38] * rwheeler (~rwheeler@66-194-8-225.static.twtelecom.net) Quit (Ping timeout: 480 seconds)
[18:40] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[18:43] * MentalRay (~MRay@107.171.161.165) has joined #ceph
[18:49] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[18:49] * sjm1 (~sjm@CPE001b24bd8cd3-CM001bd7096236.cpe.net.cable.rogers.com) has joined #ceph
[18:49] * amote (~amote@1.39.96.197) has joined #ceph
[18:50] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[18:50] * sjm (~sjm@CPE001b24bd8cd3-CM001bd7096236.cpe.net.cable.rogers.com) Quit (Ping timeout: 480 seconds)
[18:51] * TheSov (~TheSov@cip-248.trustwave.com) has joined #ceph
[18:51] * ircolle (~ircolle@66-194-8-225.static.twtelecom.net) has joined #ceph
[18:51] <TheSov> is there any particular reason to avoid putting osd's on raids?
[18:51] * xarses (~xarses@12.164.168.117) has joined #ceph
[18:55] <[arx]> not wasting disks
[18:56] * branto (~branto@178.253.154.131) Quit (Ping timeout: 480 seconds)
[18:56] <TheSov> i mean raid 0's not redundancy per se
[18:56] <TheSov> like 4 disk raid 0's
[18:57] * Nacer (~Nacer@252-87-190-213.intermediasud.com) has joined #ceph
[18:57] <m0zes> then you'll be decreasing the time between failures *and* increasing the recovery complexity.
[18:57] <TheSov> oh?
[18:57] * moore (~moore@64.202.160.88) Quit (Read error: Connection reset by peer)
[18:57] <TheSov> from what i've been reading 4tb+ disks encounter URE's on a near daily basis
[18:58] * moore (~moore@64.202.160.88) has joined #ceph
[18:58] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) Quit (Ping timeout: 480 seconds)
[18:58] <TheSov> im interested in increasing my iops per osd
[18:58] * dmick (~dmick@206.169.83.146) has left #ceph
[18:59] <TheSov> keeping the number osds to a minimum, the battery on a raid card will do write caching and such
[18:59] <TheSov> i know that ceph doesnt like write caching on
[18:59] <TheSov> but since its battery backed it should be ok
[19:00] * madkiss (~madkiss@089144198222.atnat0007.highway.a1.net) Quit (Quit: Leaving.)
[19:02] * amote (~amote@1.39.96.197) Quit (Quit: Leaving)
[19:03] <georgem1> TheSov: if you loose any drive in 4 disks RAID0 you not only loose one OSD, but a very large OSD so there will be four times the data that needs to be replicated to make up for this lost OSD
[19:06] * Nacer (~Nacer@252-87-190-213.intermediasud.com) Quit (Ping timeout: 480 seconds)
[19:07] * Grum (~Rehevkor@tor-exit.squirrel.theremailer.net) has joined #ceph
[19:08] * dugravot6 (~dugravot6@2a01:e35:8bbf:4060:55aa:429f:3b56:8cd0) has joined #ceph
[19:09] * cholcombe (~chris@c-73-180-29-35.hsd1.or.comcast.net) has joined #ceph
[19:09] * sankarshan (~sankarsha@183.87.39.242) Quit (Quit: Are you sure you want to quit this channel (Cancel/Ok) ?)
[19:09] * Kupo1 (~Kupo@ip72-200-70-110.tc.ph.cox.net) Quit (Read error: Connection reset by peer)
[19:12] * Kupo1 (~Kupo@ip72-200-70-110.tc.ph.cox.net) has joined #ceph
[19:18] * rlrevell (~leer@vbo1.inmotionhosting.com) Quit (Ping timeout: 480 seconds)
[19:19] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[19:19] * HT_Sergio (~chatzilla@66.207.210.170) has joined #ceph
[19:25] * kefu (~kefu@114.92.113.55) Quit (Max SendQ exceeded)
[19:26] * oro (~oro@84-72-180-62.dclient.hispeed.ch) has joined #ceph
[19:27] * kefu (~kefu@114.92.113.55) has joined #ceph
[19:28] <TheSov> georgem1, but is that the only reason?
[19:29] <TheSov> heres the deal, my boss wants to use 4 disk raid 5's im arguing against that and for the recommended 1 osd per disk
[19:30] <TheSov> my boss wants the interns able to replace drives in ceph, they cant do it if, ceph requires them to delete a down osd and ceph requires them to add a new osd.
[19:30] * stiopa (~stiopa@cpc73828-dals21-2-0-cust630.20-2.cable.virginm.net) has joined #ceph
[19:31] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) Quit (Quit: Leaving)
[19:32] <georgem1> TheSov: I assume it's the main one; when people use RAID they usually look for redundancy, not speed, so using RAID0 is possible with Ceph but not recommended for the above reason mainly, in addition you have to create RAID containers in advance of the OS taking control of the raw disks, etc
[19:32] <georgem1> TheSov: tell your boss the fix to the problem causes more problems, and you should find a better way, and there is a better way
[19:33] <TheSov> so whats the best way to get someone who will have no access to the ceph cluster, the ability to replace down disks?
[19:34] <Kvisle> write some code that detects empty disks, and automatically initiates them into the cluster.
[19:37] <TheSov> Kvisle, that works for adding osd, but deleting them?
[19:37] * Grum (~Rehevkor@9S0AABOUS.tor-irc.dnsbl.oftc.net) Quit ()
[19:37] <TheSov> afaik you shouldnt automatic osd removal, or should you?
[19:37] * i_m (~ivan.miro@deibp9eh1--blueice4n2.emea.ibm.com) Quit (Ping timeout: 480 seconds)
[19:40] * MrAbaddon (~MrAbaddon@a89-154-187-169.cpe.netcabo.pt) Quit (Ping timeout: 480 seconds)
[19:40] <HT_Sergio> Hi all! I was looking around for a python package (in pypi) that includes rbd.py and rados.py from Ceph's repo (src/pybind/), but I couldn't find one. Is this something I could package and put up on pypi myself (following License rules of course)?
[19:41] <HT_Sergio> I don't need all the functionality/dependencies of python-cephclient, I just need rbd.py and rados.py which don't have any deps at all.
[19:42] * MentalRay (~MRay@107.171.161.165) Quit (Quit: This computer has gone to sleep)
[19:42] <HT_Sergio> I was going to call it ceph-minimal
[19:42] <HT_Sergio> *python-ceph-minimal
[19:42] <kefu> HT_Sergio, i am not sure if i follow you, but rados.py does not work without librados.so.
[19:43] * dopeson__ (~dopesong@lb1.mailer.data.lt) Quit (Ping timeout: 480 seconds)
[19:43] <HT_Sergio> sorry, I meant doesn't have any python deps. You're right :)
[19:43] <kefu> and rbd.py relies on librbd.so.
[19:43] * vbellur (~vijay@38.140.108.3) has joined #ceph
[19:46] * bitserker (~toni@63.pool85-52-240.static.orange.es) Quit (Quit: Leaving.)
[19:46] * shylesh (~shylesh@123.136.237.105) has joined #ceph
[19:46] <kefu> HT_Sergio, well, so if you pull them together into a python package, say python-ceph-minimal, it will in turn depend on python-rados and python-rbd. right?
[19:47] <HT_Sergio> there's no such thing as python-rbd or python-rados in pypi. That's what I want to solve :)
[19:48] <TheSov> im starting to think i may need to build a quick "osd portal" website that lets you delete and add osd's with crude login
[19:48] * rlrevell (~leer@vbo1.inmotionhosting.com) Quit (Ping timeout: 480 seconds)
[19:48] <TheSov> when you delete a disk from the crushmap. can you add it back to the crushmap via cli?
[19:50] <kefu> HT_Sergio, ahh, i think i understand your intention now.
[19:51] <kefu> now sure how one will package it, as its dependencies are librados and librbd.
[19:52] <kefu> if it's libxml, that would be straightforward.
[19:53] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) has joined #ceph
[19:53] <HT_Sergio> kefu: Right. So many packaging it and putting it up on pypi doesn't make sense
[19:53] <HT_Sergio> *maye
[19:53] <HT_Sergio> **maybe
[19:55] <kefu> TheSov: you can add an OSD using cli.
[19:55] <HT_Sergio> kefu: Thank you for helping me. Cheers :)
[19:56] <kefu> TheSov: "osd crush add"
[19:58] <kefu> HT_Sergio: yw =) but that's an interesting idea anyway,
[19:59] <kefu> ceph just can not fit in the model of it. i guess
[19:59] * rlrevell (~leer@vbo1.inmotionhosting.com) has joined #ceph
[19:59] * rotbeard (~redbeard@185.32.80.238) Quit (Quit: Leaving)
[20:00] <HT_Sergio> kefu: I think I will be wrapping these 2 files up somehow, since I really don't want all the stuff of python-cephclient.
[20:00] <HT_Sergio> kefu: I'm not completely sure if the code I write to do the packaging will also have to be GPLv2. Do you know?
[20:01] <kefu> does ceph offer a package named "python-cephclient"?
[20:01] <HT_Sergio> yes
[20:02] <HT_Sergio> oh it's not offical from Ceph I think
[20:02] * kefu searching debian/control
[20:02] <HT_Sergio> https://github.com/dmsimard/python-cephclient
[20:02] <dmsimard> Oh hai
[20:02] <dmsimard> :)
[20:02] * shawniverson (~shawniver@165.138.163.203) has joined #ceph
[20:02] <dmsimard> It's by no means official, indeed
[20:03] <kefu> python-cephclient is completely a different story.
[20:04] * shawniverson (~shawniver@165.138.163.203) has left #ceph
[20:04] <HT_Sergio> kefu: there is also "python-ceph" package for Ubuntu
[20:04] <HT_Sergio> same issue for me - lots of extra deps I don't need
[20:05] <kefu> python-ceph = python-rados + python-rbd
[20:05] <kefu> it's but a meta package.
[20:05] <kefu> and python-cephfs
[20:05] <HT_Sergio> I don't have those packages in 14.04. Maybe newer versions ?
[20:08] <kefu> HT_Sergio: http://eu.ceph.com/debian-hammer/pool/main/c/ceph/
[20:09] <kefu> http://packages.ubuntu.com/search?keywords=python-rados&searchon=names&suite=all&section=all
[20:09] <kefu> yes.
[20:09] * zaitcev (~zaitcev@2001:558:6001:10:61d7:f51f:def8:4b0f) has joined #ceph
[20:09] <kefu> HT_Sergio, anyway, happy hacking
[20:09] <TheSov> since osds are formatted xfs by default, can i mount an osd and look at it?
[20:09] * kefu about to call it a night.
[20:09] <TheSov> what would i see?
[20:10] <HT_Sergio> kefu: Ah, ok. I was looking only in official Ubuntu repo.
[20:10] <HT_Sergio> kefu: Thanks again!
[20:10] <TheSov> kefu, before you go, i dunno if you are involved with ceph dev but is there a reason all the armhf repos are practically empty?
[20:11] <kefu> HT_Sergio: np
[20:12] <snakamoto1> TheSov: the OSDs are already mounted by default
[20:12] <snakamoto1> You will see the files and directories they use to record the objects. Are you looking for an explanation of each file?
[20:12] <zenpac> Is there a standard reference network setup for OpenStack that might differ from a typical Ceph install?
[20:13] <TheSov> no i was just wondering
[20:13] <kefu> TheSov: i have no idea.
[20:14] * ngoswami (~ngoswami@121.244.87.116) Quit (Quit: Leaving)
[20:14] <TheSov> thanks
[20:14] <kefu> TheSov: seems the repo only has arch=all packages for armhf.
[20:14] <kefu> a wild guess is that the ceph team does not have any arm builders.
[20:15] <kefu> or have any crossbuild for armhf.
[20:17] * tobiash_ (~quassel@mail.bmw-carit.de) has joined #ceph
[20:17] <kefu> TheSov: http://tracker.ceph.com/issues/9942
[20:17] <kefu> seems it has not changed since then.
[20:17] <TheSov> holy moly
[20:18] <kefu> sorry for not being helpful.
[20:18] <TheSov> status: wont fix...sigh
[20:18] <TheSov> no problems kefu thanks for info
[20:18] <kefu> =)
[20:18] <kefu> have a good day
[20:18] * dugravot6 (~dugravot6@2a01:e35:8bbf:4060:55aa:429f:3b56:8cd0) Quit (Quit: Leaving.)
[20:18] <TheSov> you too!
[20:19] * jclm (~jclm@64.134.221.203) has joined #ceph
[20:20] * tobiash (~quassel@mail.bmw-carit.de) Quit (Ping timeout: 480 seconds)
[20:21] <TheSov> i need to find a crushmaps for dummies book
[20:26] * ChrisNBlum (~ChrisNBlu@dhcp-ip-217.dorf.rwth-aachen.de) Quit (Quit: ZNC - http://znc.in)
[20:27] * Nacer (~Nacer@203-206-190-109.dsl.ovh.fr) has joined #ceph
[20:28] * jclm (~jclm@64.134.221.203) Quit (Quit: Leaving.)
[20:30] * kefu (~kefu@114.92.113.55) Quit (Quit: My Mac has gone to sleep. ZZZzzz???)
[20:33] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) Quit (Ping timeout: 480 seconds)
[20:36] * linuxkidd (~linuxkidd@99.sub-70-193-66.myvzw.com) has joined #ceph
[20:37] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) has joined #ceph
[20:44] * shylesh (~shylesh@123.136.237.105) Quit (Remote host closed the connection)
[20:44] * Spikey (~Scaevolus@80.82.70.25) has joined #ceph
[20:46] * joshd (~jdurgin@206.169.83.146) has joined #ceph
[20:47] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[20:53] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) has joined #ceph
[20:59] * i_m (~ivan.miro@nat-5-carp.hcn-strela.ru) Quit (Ping timeout: 480 seconds)
[20:59] * dupont-y (~dupont-y@familledupont.org) has joined #ceph
[20:59] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) Quit (Remote host closed the connection)
[21:02] * MrAbaddon (~MrAbaddon@a89-155-97-112.cpe.netcabo.pt) has joined #ceph
[21:03] * alphan8 (18fb5914@107.161.19.53) has joined #ceph
[21:04] <alphan8> hey all. is flow control and jumbo frames useful in a ceph deployment?
[21:07] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) has joined #ceph
[21:10] * ChrisNBlum (~ChrisNBlu@dhcp-ip-217.dorf.rwth-aachen.de) has joined #ceph
[21:11] * jrankin (~jrankin@d53-64-170-236.nap.wideopenwest.com) Quit (Quit: Leaving)
[21:14] * Spikey (~Scaevolus@80.82.70.25) Quit ()
[21:22] * dgurtner (~dgurtner@178.197.233.252) has joined #ceph
[21:22] <georgem1> alphan8:Ceph doesn't like flow control and it's not recommended to use it but jumbo frames is recommended, especially on the replication network
[21:22] * linjan (~linjan@80.179.241.26) Quit (Remote host closed the connection)
[21:24] * HT_Sergio (~chatzilla@66.207.210.170) Quit (Ping timeout: 480 seconds)
[21:25] <alphan8> thanks geogem1
[21:35] * dgurtner (~dgurtner@178.197.233.252) Quit (Ping timeout: 480 seconds)
[21:38] * tupper (~tcole@2001:420:2280:1272:8900:f9b8:3b49:567e) Quit (Ping timeout: 480 seconds)
[21:39] * TheSov3 (~TheSov@cip-248.trustwave.com) has joined #ceph
[21:39] * nsoffer (~nsoffer@nat-pool-tlv-t.redhat.com) Quit (Ping timeout: 480 seconds)
[21:39] * evilrob00 (~evilrob00@cpe-24-28-12-195.austin.res.rr.com) has joined #ceph
[21:40] * thomnico (~thomnico@2a01:e35:8b41:120:c970:36cd:2552:7236) Quit (Ping timeout: 480 seconds)
[21:43] * ganders (~root@190.2.42.21) Quit (Quit: WeeChat 0.4.2)
[21:46] * TheSov (~TheSov@cip-248.trustwave.com) Quit (Ping timeout: 480 seconds)
[21:47] * bene_afk (~ben@c-24-60-237-191.hsd1.nh.comcast.net) Quit (Quit: Konversation terminated!)
[21:52] * aj__ (~aj@fw.gkh-setu.de) Quit (Ping timeout: 480 seconds)
[21:52] * badone__ (~brad@CPE-121-215-241-179.static.qld.bigpond.net.au) has joined #ceph
[21:58] * bitserker (~toni@77.231.154.146) has joined #ceph
[21:59] * TheSov3 is now known as Thesov
[22:00] <Thesov> funny nearly every storage archictecture i know of, wants flow control
[22:03] * tupper (~tcole@rrcs-70-62-115-203.midsouth.biz.rr.com) has joined #ceph
[22:04] <championofcyrodi> Thesov: light flow? medium flow? or heavy flow?
[22:04] <Thesov> all 3 championofcyrodi
[22:04] <championofcyrodi> these are for extra heavy overnight: http://www.amazon.com/Always-Extra-Heavy-Overnight-Flexi-Wings/dp/B002KAL6NI
[22:05] <georgem1> Thesov: Ceph is quite sensible to missed heartbeats so you if you start dropping packets because you do flow control you'll cause more problems???with the money you save by not paying netapp and emc you should afford to throw more network capacity instead of doing flow control, my 2 cents
[22:05] <Thesov> georgem1, i wasnt doubting you, im just happy i dont have to configure end to end flow control anymore
[22:06] <Thesov> its like a staple of storage annoyance
[22:08] * linjan (~linjan@80.179.241.26) has joined #ceph
[22:09] * Sysadmin88 (~IceChat77@2.124.164.69) has joined #ceph
[22:10] * pmatulis (~peter@ec2-23-23-42-7.compute-1.amazonaws.com) Quit (Read error: Connection reset by peer)
[22:10] * DV (~veillard@2001:41d0:1:d478::1) Quit (Remote host closed the connection)
[22:10] * DV (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[22:17] * DV__ (~veillard@2001:41d0:1:d478::1) Quit (Ping timeout: 480 seconds)
[22:18] * tupper (~tcole@rrcs-70-62-115-203.midsouth.biz.rr.com) Quit (Ping timeout: 480 seconds)
[22:21] * pmatulis (~peter@ec2-23-23-42-7.compute-1.amazonaws.com) has joined #ceph
[22:22] * Shnaw (~phyphor@politkovskaja.torservers.net) has joined #ceph
[22:24] <alphan8> i would have figured tcp does flow control
[22:24] * darkfaded is now known as darkfader
[22:24] <alphan8> so why bother doing it at the switch level
[22:24] <alphan8> rather. at layer 2
[22:24] <alphan8> however.. iscsi is tcp based as well and they encourage flow control
[22:24] * ingslovak (~peto@cloud.vps.websupport.sk) has joined #ceph
[22:24] <darkfader> alphan8: the *cough* reason might be that things don't just work because they should
[22:24] <alphan8> so as such the documentation didnt seem to talk about it.
[22:25] <darkfader> in a case of a switch for example, tcp can negiotiate a lot - which is good. but it can't know about congestion on the switches' uplink port
[22:25] <alphan8> yeah thats a certainty
[22:26] <alphan8> things don't work because they should.. that is.
[22:26] <alphan8> true true.
[22:26] <alphan8> thanks for the thoughts darkfader
[22:27] * DV__ (~veillard@2001:41d0:a:f29f::1) has joined #ceph
[22:27] <darkfader> np, i'm sorry i don't have a solution on my table, but i hope it helps you look at the problem relaxed and get rid of it soon :)
[22:27] * tupper (~tcole@173.38.117.86) has joined #ceph
[22:28] * linjan (~linjan@80.179.241.26) Quit (Remote host closed the connection)
[22:28] <darkfader> what's it worth, my life has been very easy once i started dedicating mon (and mds) systems only to that
[22:29] <alphan8> nice
[22:29] <alphan8> ill keep that in mind
[22:29] * linjan (~linjan@213.8.240.146) has joined #ceph
[22:38] * dyasny (~dyasny@173.231.115.59) Quit (Ping timeout: 480 seconds)
[22:38] * MentalRay (~MRay@142.169.78.141) has joined #ceph
[22:41] * dupont-y (~dupont-y@familledupont.org) Quit (Quit: Ex-Chat)
[22:42] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) has joined #ceph
[22:45] * georgem1 (~Adium@fwnat.oicr.on.ca) Quit (Quit: Leaving.)
[22:49] * wushudoin_ (~wushudoin@transit-86-181-132-209.redhat.com) has joined #ceph
[22:50] * branto (~branto@ip-213-220-232-132.net.upcbroadband.cz) has joined #ceph
[22:50] * branto (~branto@ip-213-220-232-132.net.upcbroadband.cz) has left #ceph
[22:50] * linjan (~linjan@213.8.240.146) Quit (Remote host closed the connection)
[22:51] * evilrob00 (~evilrob00@cpe-24-28-12-195.austin.res.rr.com) Quit (Remote host closed the connection)
[22:52] * Shnaw (~phyphor@7R2AAB4R3.tor-irc.dnsbl.oftc.net) Quit ()
[22:53] * linjan (~linjan@80.179.241.26) has joined #ceph
[22:54] <Thesov> when i build a crushmap how exactly do i tell ceph that i want 1 copy on a specific set of osds?
[22:54] <Thesov> for instance i know that you break it down to like host anc chassis and rack
[22:55] <Thesov> but how do i make sure there is one full replica on rack 1 and 2 and 3 4 5 6 are straw?
[22:55] <Kvisle> http://dachary.org/?p=2536
[22:55] * nsoffer (~nsoffer@109.65.255.114) has joined #ceph
[22:55] <Thesov> Kvisle, u want beer? cuz thats how you get beer.
[22:56] <Kvisle> we have three datacenters, and have one replica in each -- local radosgw's on all sites
[22:56] * MentalRay (~MRay@142.169.78.141) Quit (Quit: This computer has gone to sleep)
[22:56] <Kvisle> that's how I know
[22:56] <Kvisle> :P
[22:56] * wushudoin (~wushudoin@38.140.108.2) Quit (Ping timeout: 480 seconds)
[22:57] * wushudoin_ (~wushudoin@transit-86-181-132-209.redhat.com) Quit (Ping timeout: 480 seconds)
[22:58] * mykola (~Mikolaj@91.225.202.121) Quit (Quit: away)
[22:59] <Thesov> so if i follow this, and create 1 osd with the whole space of the other copy, i can keep 1 osd with a full copy of whats on all the other osd's correct?
[23:00] <Thesov> this replica is actually a ceph osd placed on a very large zfs volume
[23:00] <Thesov> as long as my weights are correct i assume
[23:01] <Kvisle> I think you can, but I've come across some challenges when there are only three leafs to choose from .. then having a weight that's approximatly equal in all three leafs starts to have an impact ... or pgs can get stuck in a remapped state
[23:01] <Kvisle> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042148.html <-- relevant.
[23:02] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[23:06] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) Quit (Read error: Connection reset by peer)
[23:07] * wushudoin_ (~wushudoin@38.140.108.2) has joined #ceph
[23:07] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[23:09] * rlrevell (~leer@vbo1.inmotionhosting.com) Quit (Ping timeout: 480 seconds)
[23:10] <cholcombe> ceph: is [osd] allow rwx enough to allow a client to create a pool?
[23:10] <Thesov> Kvisle, you are getting beer!
[23:11] <Thesov> you need mon rwx for client to create a pool
[23:11] <Thesov> thats a monitor deally
[23:11] <Thesov> thats a little scary to give the client though
[23:12] <cholcombe> coo
[23:12] * MentalRay (~MRay@142.169.78.222) has joined #ceph
[23:12] <cholcombe> my mon caps are just rw
[23:12] <cholcombe> that explains my perms denied
[23:12] * gregmark (~Adium@68.87.42.115) Quit (Quit: Leaving.)
[23:12] <Thesov> ceph auth get-or-create client.cephclientname osd 'allow rwx' mon 'allow rwx' -o /etc/ceph/ceph.client.cephclientname.keyring
[23:12] <Thesov> that will let u do it
[23:12] <cholcombe> nice :)
[23:12] <Thesov> just fyi that means a client has basically admin rights to ceph
[23:13] <cholcombe> yeah i know
[23:13] <cholcombe> probably not ideal
[23:13] <Thesov> as long as you know :)
[23:14] <Thesov> knowing is half the battle. G I Joooooooooe!
[23:16] * marrusl (~mark@cpe-67-247-9-253.nyc.res.rr.com) has joined #ceph
[23:16] * evilrob00 (~evilrob00@cpe-72-179-3-209.austin.res.rr.com) has joined #ceph
[23:24] * bitserker (~toni@77.231.154.146) Quit (Quit: Leaving.)
[23:24] * bitserker (~toni@77.231.154.146) has joined #ceph
[23:27] <Thesov> can someone explain to me why ceph running on rust alone has a ioping of .9 ms and my local DAS has a ioping of 10ms?
[23:28] * oro (~oro@84-72-180-62.dclient.hispeed.ch) Quit (Ping timeout: 480 seconds)
[23:28] <Thesov> something not quite kosher here
[23:29] * t0rn (~ssullivan@2607:fad0:32:a02:56ee:75ff:fe48:3bd3) Quit (Quit: Leaving.)
[23:33] * snakamoto1 (~Adium@192.16.26.2) Quit (Quit: Leaving.)
[23:37] * tupper (~tcole@173.38.117.86) Quit (Ping timeout: 480 seconds)
[23:42] * linjan (~linjan@80.179.241.26) Quit (Ping timeout: 480 seconds)
[23:44] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) Quit (Quit: jdillaman)
[23:44] * snakamoto (~Adium@192.16.26.2) has joined #ceph
[23:45] * jdillaman (~jdillaman@pool-108-18-97-82.washdc.fios.verizon.net) has joined #ceph
[23:46] * linjan (~linjan@213.8.240.146) has joined #ceph
[23:49] * scott_ (~scott@69.65.17.194) has joined #ceph
[23:49] <scott_> Can anyone help me get Ceph calamari working with Centos 7?
[23:51] * brad_mssw (~brad@66.129.88.50) Quit (Quit: Leaving)
[23:52] <scott_> Getting the following error during calamari-ctl initalize; http://pastebin.com/6RckMVnJ
[23:53] * fridim_ (~fridim@56-198-190-109.dsl.ovh.fr) Quit (Ping timeout: 480 seconds)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.