Startup diaries

Aaaaaand we're back.

Feels like it's been a long time since my last update.

Well, have no fear, I've just pushed an update to Flippity with some major changes:

• Category suggestions and subcategories
• The radius bar now goes up to 1,500 miles so you can search all of the United States visually
• You can now browse all items for a search (not just the first 100!)
• You can now filter by fixed-price / buy-it-now listings, and new / used condition items
• Cute little bunnies

I have several more important changes in the pipeline, but I'm open to feedback.

Congratulations!

Your application has passed eBay’s Compatible Application Check process. We have increased your Shopping and Finding API call limit from 5,000 per day to 1,500,000 per day.

...

You may use the eBay Compatible Application logo on your website and in printed materials.

In celebration, I've fixed a few bugs and added a few features to Flippity that I personally like. Hopefully you'll notice them on your own, but if not here's a hint. In the item details window (after you click an item), try using the J, K, and ESC keyboard shortcuts.

And yeah, I realize there's a ton of other stuff that needs fixin', but right now I'm all about baby steps.

Shhh, secret eBay release

I've decided to let the people reading this blog try out the eBay release before anyone else. It's kind of exciting because it works nationwide, unlike our original Craigslist version. And I'm not announcing it anywhere else just yet, because of the following:

• We're limited in our interaction with eBay. We can only support maybe a hundred users per day at this point. We're in the process of being certified, which can take a week or so. So I only want a limited audience at this point.
• It's not finished, there's bugs (report them to us), though I know about many of them. But it's still kinda neat !
• I want to hear the feedback of the first users, please comment here or email me (romy AT ucla D0T edu)

eBay version nearing completion

Analytics show approximately 100 people a day still accessing the now-defunct original version of Flippity. The reason I say defunct, of course, is the fact that the Craigslist data is now over a week old, and so most listings have expired by now. Sorry guys, I feel bad about this, but we're working as fast as our ADD allows to release a functioning eBay version.

Truth be told, it's already built -- I'm staring at it right now, but it's not exactly ready yet. I estimate a few more days. The biggest remaining piece involves taking eBay's massive category list (6MB of XML) and making it useful. I also have to fix a few paging and caching issues, as 98% of the backend infrastructure we've built for Craigslist is now completely unnecessary. So now you know the reasons behind the whole 'iterate quickly' mantra.

Throwing away months of work is certainly not pleasant, but it's only fatal if you allow it to be. I look at it as a learning curve -- you can't always get it right off the bat, but you can get enough momentum to keep going. And if you choose to treat each obstacle like a bullet, you are giving yourself reasons to die.

The API myth

Let me begin by calling bullshit on all the people claiming that an API would spell a company's doom. Is this anything beyond groupthink and hot air ? Specifically, do any of you know of a business that was on the up and up until it opened up its data via an API and went kaboom ? If so, speak up.

And no, this isn't another bash-the-classified-guerilla post. In the interest of debate, I'm willing to put personal biases aside for the sake of answering this one simple question:

If you were in Jim Buckmaster's shoes, would you open up the massive Craigslist database to the world ?

Let's weigh some pros and cons. Pros: a ton of buzz, increased user growth and adoption, many new traffic sources, developer loyalty, and tens of thousands of mashup projects. In the end, this means more traffic, more revenue, and much more exposure. Best of all, the world is practically guaranteed to benefit from all the new ways this information would undoubtedly be parsed, and you'd be a hero.

Cons. Developer time. Support nightmares. Significant extra server load and provisioning. New employees. A laundry list of new expenses and headaches, such as the need to police misuses and abuses. And let's, for the sake of argument, assume that this would somehow (?) help your competition. All said and done, it can seem like opening up a can of worms.

It's possible to make strong arguments for both sides. Which side you would pick ultimately depends on the answer to a simple question. Why ? What are the reasons for doing one or the other ?

Given the evidence, I believe many of the benefits of an API aren't really benefits for Buckmaster and company. Specifically things like traffic, growth, and revenue, which are normally top priorities, are not something the company is too concerned about. And hell, if your company was organically growing by an average of over a million new users per month, how much faster would you really want to grow ?

Now, I may not agree with the gestapo tactics, but the decision against an API seems, if nothing else, rational.

Jim updates Craigslist blog

For those who haven't had enough of the Craigslist & Pipes fiasco, the CEO has updated the company's blog.

I also found an interesting read from a couple years ago about Ryan Sit, who started a browse-by-image service for Craigslist and was promptly slapped in the mouth.

Moving on

So, my previous post was picked up by some major news sites including slashdot and techcrunch, and has gotten a fair amount of publicity. On the bright side, this created some debate and dialogue. On the not-so-bright side, I don't know if anything positive will ever come out of it, and I'm already tired of the negative vibes around this whole debacle. It's time to move on.

That said, the one thing Jason Kincaid got slightly wrong in the initial piece on techcrunch is that flippity has been shut down. While it's true that Craigslist was our #1 choice, we've been porting the initial idea over to several other providers who are much more open to having their data accessible, before this incident even happened. It's a little different, but it just might work. And when it's ready, I'll post a link here.

To close, I wanna say that I don't blame Craig. I'm not even sure he knew this was happening. Though if it was my company, I'd probably want to know.

Don't be evil, Craigslist

Yesterday (Nov 30th 2009), at approximately 2pm PST, Craigslist effectively killed all Yahoo Pipes projects that use Craigslist as a data source. For those unfamiliar with Yahoo Pipes, it's a nifty tool that allows you to take data from anywhere on the net, mash it up, parse it, and produce interesting results. One of its benefits is that it caches the data, so the original source does not get pummeled on every request.

Our project, known as Flippity, used pipes as the primary data source and was in its alpha stages of development. We were one of 2,111 pipes that used Craigslist on Yahoo Pipes, all of which to my knowledge were non-commercial, created solely as fun projects to provide a new perspective on data already available.

What's most interesting is perhaps the timing of all this. You see, about 4 days ago I posted a thread on hacker news asking for feedback on what my friend Dan & I have come up with so far -- a mashup that simply plots Craigslist listings on a map, and allows you to do a radius search around any location. It was, in essence, HousingMaps.com generalized across all listings.

Having exchanged emails with the Craigslist founder several weeks ago (see previous blog post) in which he expressed interest in seeing more, I decided to follow up and give him the closed alpha link.

Fri, November 27, 2009 3:14:06 PM

Hi Craig,

You asked to hear more about this, so here you go!

http://www.flippity.com/alpha

This is currently closed to the public, but I've received significant positive feedback from friends.

Would like to hear what you think,

Romy

As you can see, this was sent 4 days ago. I received no response, which is unusual as Craig responded within minutes to my original several emails. Oh well, I thought to myself, we'll just keep working on it and see what happens.

I couldn't believe my eyes when I saw what Craigslist had done. They literally added a check for "pipes.yahoo.com" in the referrer header of any HTTP request, which was then redirected to the home page. In essence, they blocked them. Really, Craig ? This is your response ? Allow me to quote the first email you ever sent me:

... thanks! and as a rule of thumb, okay to use RSS feeds for noncommercial purposes.

Well, we are using RSS feeds for noncommercial purposes. So were 2,110 other people. And you just shafted all of them, not to mention Yahoo itself. May I ask why ?

I mean really, these pipes aren't all that popular so if you told me your servers were getting hammered, I would say that's unlikely. So what is it then ? What did these people do that was so wrong that it merited such a response ?

Anyway, it's a sad day for me. I'm not too upset about my own project, as Flippity was already removing Craigslist as a data source. With the likes of eBay and Oodle not only providing open APIs but encouraging and rewarding developers, spending my time wrestling with Craigslist is just plain stupid and exhausting. I'm sure I'm not the only person to have come to that conclusion, and I wish it were different.

By the way, it's not too hard to defeat any technical measure Craigslist can put up. We could, for instance, build a peer to peer network that obtains bits and pieces of data from Craigslist via hundreds of IPs with randomized time intervals. Or build a Java applet that we distribute to our users, having them exchange data with us a la BitTorrent. There's very little Craigslist would be able to do to counter. However, it's just not worth my time. If Craigslist wants to keep its doors shut to the world, so be it.

[ If you're interested in startups, or just the future of flippity, I will be documenting our journey every step of the way. Subscribing to the blog will show me someone cares. ]

A friendly exchange with Craig

A snippet from my brief exchange with Craigslist's founder. 

Romy, thanks! and as a rule of thumb, okay to use RSS feeds for 
noncommercial purposes.

I'd like to hear more about this.

thanks!

Craig

I followed up shortly after with

... of course it'd be 10x more useful if people would actually put an address, or even cross-streets instead of "SALE! CHEAP!!", but hey we can't have it all. Although I have a feeling that if the site asked for cross-streets instead of location, maybe in two boxes, the amount of junk would go down significantly. What do you think ?

Oh, and about being noncommercial. A bunch of the RSS mashup sites I've seen, like Craiglook, throw up advertising presumably to cover server costs or make a few bucks. If I didn't want to go that route, would asking for donations be acceptable ?

A few minutes later

Romy, I can ask about that, shall I? thanks!

Craig

Thoroughly confused, I replied with

Yes please? :)

Which was met with

done!

Well shiver me timbers, who knew Craig was so nice and responsive. The only problem is, days passed and I've received no response. A follow up email was also met with a shrug from Craig. More importantly, when the alpha version was launched and over 1500 hacker news readers came to check us out, I sent Craig another email with a link, to which I've received no response. Incidentally Craiglook, the site I mentioned in my email, took down its ads shortly after. Cute coincidence, I suppose..

Hello World

Hi, 

I saw your post on HN about your craigslist mashup.  I would love to know how you went about making this?  I dont know if you blog but a blog entry explaining how you went about this project would be great.  Also, you've done an awesome job at this.  

Thanks

Oh why not, a blog can't hurt.

The mashup in question is Flippity, and it's hardly finished. Today it's just an alpha prototype, and was released primarily to get user and community feedback. You see, Dan and I have been working on the project more or less full time for about 2 months, and we've gotten kind of bored without any users. Yeah, we have ADD. More importantly, I think 2 months for a consumer internet site without any users is a long, long time. I expect future projects to have prototypes ready in under a month, but this is Dan and I's first real project together, so we also had a process to establish. Oh, and neither one of us was a web developer.

The idea was pretty simple -- plot Craigslist listings on a map. Add features like radius search, email notifications, and so forth. But the core idea was to let you see what's being sold around your home. Why ? Well, with this you can do nifty things. You can execute a proximity search so you don't have to drive too far (Craigslist only provides regions). You can isolate nearby items and visit those sellers in one trip. And best of all, without searching for anything in particular, you can see what all your 'neighbors' are selling. It's like a garage sale, but with all of Craigslist as a data set.

So version .001 scraped Craigslist -- it issued GET requests via http, parsing the resulting data set. This worked, but it was ugly and faced the immediate threat of being IP-blocked by Craigslist at any point since it was centralized. Not long after, I realized Craigslist has RSS feeds. Did you know this ? If so, congratulations, you're one of a total of seven people who do. The icon is neatly tucked in the bottom right corner of any given results page.

What good are RSS feeds ? Well, they take the parsing out of the equation, since RSS is based on XML, which aside from being bloated doesn't require much parsing. Much more importantly, RSS feeds are read and cached by RSS readers. Some of those readers, like Google and Yahoo, have an API. What does this mean in plain English ? It means that we can now get our Craigslist data through a 3rd party, without ever touching Craigslist directly. Not only that, but if anyone else using the readers pulled the same data in the last 15-30 minutes, then it comes directly from the 3rd party, and Craigslist is only hit once. Even better, there's a nifty tool called Yahoo pipes that will let you obtain data from anywhere on the internet (Craigslist included), parse it in a bunch of ways, and spit it out via JSON/XML/RSS.

This lead us to our next design. We'll have a cron job that runs every 15 minutes (Craigslist refresh rate), pulling data from our private Yahoo pipe which reads a bunch of Craigslist RSS feeds (there's 36 categories in "for sale") and spits them out as gzipped JSON. We only need to read the first page for most categories, as the Craigslist posting rate is usually less than 100 items per 15 minutes (cars and trucks is one exception). Our python code then parses this data and inserts it into a Postgres database. Another process runs continuously and uses Google's Geocoding API to geocode the locations of these new entries, if possible, into latitudes and longitudes. Since the location field in Craigslist is optional, it often contains junk, but it turns out there's enough utility nonetheless. Finally, if the geolocation fails, we'll extract the general region from the URL or posting and approximate the location, attempting to salvage it.

We wouldn't have to jump through all these hoops if Craigslist had an API. But even without an API, if they managed to get higher quality locations, Flippity would be much more useful. Exact addresses are ideal, but people may be unwilling to post those, so why not cross streets ? I emailed Craigslist's founder, Craig Newmark to get his opinion. I'll post some of that exchange in the next blog entry.